cancel
Showing results for 
Search instead for 
Did you mean: 

Issue regarding handling of high volume of data

Former Member
0 Kudos

Hi Experts,

I am faceing Issue regarding handling of high volume of data in PI 7.0

we are planing to start working on JDBC-XI-RFC scenario using ABAP proxy (Frequency- Monthly). but as per analisys of this XI Interface, there is issue regarding High data volume. So I require help and guide lines from your side to handle High volume of data.

Interface detail:

At legacy system (DB2 Database system) getting nearly 5 lacs no of lines in a day at database table.

Monthly = (5 * 30) = 150 lacs no of line/month

In this XI Interface, we are uploading monthly summrized data.

As per analisys few points are:

1. After applying mapping rules and summarization of data, there will be nearly ~5 lacs no of line has to upload to SAP system at the end of month.

2. size of 1 line data ~ 1000 bytes (max) then max message size will be 5lac * 1 KB = 500000 KB (~500 MB)

Issue:

1. As per my experiance ~150MB data, XI can handle easily, But I am warry about 500MB data volume.

2. As per growing of business, this data volume may be ~1024MB (1GB) in future.

Question:

1. Will XI system handle this much high volume of data?

2. Shall I need to change some parameter in XI serevr and SAP system side like Time Out parameter?

Experts, please reply soon, It is urgent.

Regards,

Umesh Laxkar

Accepted Solutions (0)

Answers (4)

Answers (4)

Former Member
0 Kudos

HI,

You can handle volume of payload either by:

1.Splitting your payload using bpm or udfs.

2.you can zip unzip your payload,that might reduce the size to some extent.

refer:

http://help.sap.com/saphelp_nw70/helpdata/en/45/da9358a1772e97e10000000a155369/frameset.htm

In addition to above steps ....you may have to fine tune certain parameters in your XI and ECC box as the system may throw timeout errors. Going forward, following link can be helpful for you :

former_member190389
Active Contributor
0 Kudos

You can use BPM for this.

while reading you can decide on a batch size and read data in number of pages and collect it in a container.

similarly while writing also you can do the same thing and write the data in loop.

Former Member
0 Kudos

Hey Umesh,

I have done something like this for very huge data. You can try splitting the Message (in case you can split it somehow) on some basis. This is called Message Bundling.

I am sharing my wiki-blog with you. Hope it helps.

https://www.sdn.sap.com/irj/sdn/wiki?path=/display/xi/splitMappingusing+UDF

If you go by this you have to use 1:n mapping.

Reward points for useful answers

Former Member
0 Kudos

Hi Umesh,

Is it possible to transfer the data in several messages? Then, you don't have to care about message size. Probably this is possible using a fitting SQL query.

Kind regards,

Dennis