what is the best performance method for Extracting huge data from SAP ECC system?

[Former Member]

Hi All,

what is the best performance method for Extracting huge data from SAP ECC system?

As per my knowledge using ABAP data flow will increase the performance of extraction as it has the capability of reading ABAP(ECC) tables.and also accuracy of reading the data will be good with out any missing of the records.

Normal data flow will have less performance as it is less capable of reading ABAP tables and the accuracy of the data will be the challenge here.

please suggest me in above points, whether I am right or not

Scenario:

I have  8 millions of records in ECC (legacy) system table and I want to extact data using BODS with best performance method.

please suggest me the best way to extract the data using bods and which will decrease the maximum extraction time.

Thanks,

Deepthi

[werner_daehn]

This does not consider the RFC_Streaming setting for ABAP dataflows. In my opinion this will be the fasted for all scenarios.

RFC_READ_TABLE is just more convenient in case it can be used, even.

Reasons are

1. Parallel processing. While reading the ERP data, it is streamed over the network and processed in the dataflow.
2. Least amount of overhead. No files are written just to be read by ftp, etc.

[former_member431649]

Hi This is a fairly old thread

From a simple speed perspective the RFC read table will be the fastest. However you need to make sure the extract can finish within the timeout limit set by the basis administrators. This could be done by using multiple dataflows all extracting from the same table using different parameters. EG for BSAD by Fiscal Year and Period.

If the RFC read table can not be completed within the time limit, then the only answer is to use the ABAP dataflow method. Hear as mentioned above the issue is the total time is broken into separate parts (Extract from SAP System to File System, Access File on File System, and Read file on file system and write to database). Using a shared directory eliminates the time required to transfer the file from the remote SAP File System to the Local BODS file system. However as mentioned in previous comments security implications may prevent using a shared file system.

Depending on your target database, try using multiple writers or bulk load methods

[Former Member]

Hi.

In your scenario, the best way to extract data from SAP ECC would be to use ABAP data flows using SAP Data Services.

Thanks.

Regards,

Thatoyame

[former_member208246]

Hi

Since you use ECC system as the source, the best method is to use the ABAP data flow since the data is numerous in size. Normal data flow can be used for the small set of records, whereas the ABAP data flow along with the data transfer can be used if the SAP system is used as the Source

In the Normal data flow the extraction can be done but it takes ample of time to the target, whereas in the ABAP data flow, its faster and the shared directory mechanism is followed for the flat files

Thanks and regards

Sanjay

[former_member315492]

Hi,

For a normal DB's as a source. It is advisable to use as Dataflow, Where as  SAP ECC as a source ,

Better to use ABAP Data Flow. ABAP Data flow supports High volume of data compared Normal data flow.

RFC  and Shared directory are  the method to connect database intenally.

RFC provides much performance.

Regards,

Narasimha.

[Former Member]

I am not sure of your BODS Version

Before 4.1, ABAP Dataflow with Execute Preloaded settings for Datastore Shared Directory/Custom Transfer/FTP , I personally feel Shared Directory is faster. However other methods are used for enhanced security only.

4.1 and above, I bet ABAP Dataflow with Execute Preloaded - RFC Transfer is supposedly the fastest! I was able to prove it on a small table.

Let us know which approach you chose!

[Former Member]

Hi Ganesh,

Thanks for your response.

we are using 4.1 version. hopefully i can use ABAP data flow with execute Preloaded.

can you please elaborate execute preload and the transfer methods and which way it increase the performance.

lets take an example of 8 millon records in KNA1 table, the extraction time is 4 hours with normal data flow.

how much time we can reduce with using ABAP data flow with execute Preloaded option.

Thanks in advance...

Regards,

Deepthi

[Former Member]

Best approach is the ABAP Dataflow with RFC Transfer.

ABAP with Shared Directory is of course comparably faster too. However, in this approach the DS Job has to wait until the ABAP spits the .DAT file. So if a 8 million record is considered, it has to be all completely written in the file.

However, in the ABAP with RFC Transfer the DS job don't have to wait until all the 8 million records are spitted out. DS can just start collecting the chunks as and when ABAP program has started to throw! So obviously this brings down that unwanted wait time.

Cheers

Ganesh Sampath

[former_member187605]

In DS 4.2, there are basically two options for a straightforward 1-on-1 table extract:

1. An ABAP datflow with RFC transfer
2. A regular dataflow (do not use this in older DS versions, except for very small tables!), which is slightly faster

The disadvantage of the latter option is that the process times out after 30 minutes. That means if you don't achieve to pull your table in less than half an hour, only option 1 remains valid.

Note that this applies to 1-to-1 copies only. In more complex situations (sorts and group by, joins with CDHDR and CDPOS for incremental loads...), the ABAP approach will often outperform the SQL-based one, because not all relevant SQL code is pushed down to the underlying database yet.

[Former Member]

Note: In our environment, the timeout is just 10 minutes. !!!

Cheers

Ganesh Sampath

[former_member214776]

Hi Deepthi,

Q-can you please elaborate execute preload

A-

Since you are using DS 4.1,

1. Were ABAP programs loaded using earlier versions of BODS(i.e. before 4.1)

2. If yes, then was DS was upgraded using transport files and ABAP Programs regenerated.

3. If yes again, then Now If you select "Execute Preloaded" for the ABAP execution option mode in

your SAP Applications datastore, when Data Services executes a job that contains an ABAP data flow, it runs the newly generated and uploaded ABAP program. Hence, the job can take advantage of the performance improvements in the 4.1 version.

And If ABAP Program was not regenerated while upgradation process of DS 4.1 , then you need to upgrade the generated ABAP program on the SAP server to match the Data Services version. and then use " Execute peloaded option".

Thanks,

Richa Goel

[Former Member]

Point 2 is only partially true. You will have to re-upload the DS Functions in /BODS/BODS package provided by Data Services in %LINK_DIR%/admin\R3_Functions\transport\ folder

Point 2 is going to be perfectly true only if you will have to use the RFC Transfer approach

Cheers

Ganesh Sampath

[former_member214776]

Thanks for Correcting Ganesh.

[former_member189933]

Hi guys,

Good day!

I have a few questions on RFC transfer method and ABAP DF's.. Please shed some light on these...

1.	When using ABAP Dataflow with RFC data transfer, do we still need to use Data Transport and provide .DAT file name. I believe No .dat files are written in RFC Data transfer mode.
2.	When we select Data Transfer Method to “RFC”, I believe “Working directory on SAP server” is not required. We can leave it blank. Right?
3	If we use Normal Data Flow for extracting data from SAP on “RFC Data Transfer Method”, Abap programs are still created and data is transferred over RFC streaming or none of these applies to Normal DF?

Cheers,

Praga..