Need to transfer the message more than two millions via PI 7.4

[Former Member]

Dear All,

The customer need to transfer the master data more than two millions from SAP ECC to third-party system. The PI version for customer is 7.4. Moreover, the data will be  verified when arriving to third-party system. I think the quantities is too large and need to verify. I am afraid that the queue of PI is blocked.   Anyone has the better advice to handle it? I try to find the solution via google, BUT failed.

Thank you for your support in advance.

Many thanks & best regards,

Hubery

[manoj_khavatkopp · Accepted solution]

Hubery,

How is the Sender system sending the file ?

If its a flat file you can use FCC and make use of record set so that you can read only a set of records at a time and then you can append this in target system.

Alternatively you can ask the sender system to zip the file and send and PI system can pick and drop this file.

If sender is IDOC you can use IDOC packing concept .

Additionally you can check these links :

It would be helpful if u tell what sender/receiver adapter you are using .

Br,

Manoj

[Former Member · Accepted solution]

Manoj,

Thank you for your information. The mass data is collected by report program and send to 3rd party system in batches. I will plan  the quantities of 10000 as one package.  But I am afraid that the queue of PI is blocked. Not sure whether other solution could the issues. I will check your solution.

Many thanks & best regards,

Hubery

[Former Member · Accepted solution]

Hello Hubery,
of course you have to choose the batching and processing parameters according to your usage, but I suppose 10.000 messages in batch to be quite a big amount. According to SAP PI performance guide there is a "commonly accepted" message threshold of 5 Mb, so try to adjust the batching to keep this value. Also, you can refer to that guide and revise the EO_MSG_SIZE_LIMIT and EO_MSG_SIZE_LIMIT_PARALLEL parameters on PI system prior to transferring process. But generally, 2 million of messages is not a huge number for PI, its size matters more

Best regards,
Vadym

[Former Member · Accepted solution]

Hello Vadym,

Thank you so much for your reply. Forget to say,the installation based on PI 7.4 single stack, so the parameters is not available like EO_MSG_SIZE_LIMIT and EO_MSG_SIZE_LIMIT_PARALLEL. I am not sure whether the java single-stack has some method like t-code SXMB_ADM to optimize and control the message .

Many thanks & best regards,

Hubery

[Former Member · Accepted solution]

Oh, sorry, for Java-only system these settings are definitely not applicable. Despite, you can tune icm/HTTP/max_request_size_KB parameter and refer to SAP Note #1727870 - Handling of large messages in the Messaging System which gives the general overview of possible bottlenecks during high volume message processing in adapter engine.

Best regards

[bhavesh_kantilal]

What is the End to End Flow? Options to do this depend on the flow that is implemented. Can you share the details of this?

We have loaded over 3 millions materials from SAP to a 3rd Party system via PI  in a batched mode but we had to do certain design tweaks for the same and hence the end to end flow questions..

Regards

Bhavesh

[Former Member]

Dear Bhavesh,

Could you please tell me what your batched mode is? 

Many thanks & best regards,

Hubery

[bhavesh_kantilal]

Hello Hubery,

Sorry for the delayed response. In my case we were replicating Material Master from SAP to a 3rd Party System. These were days when PI did not have a IDoc Packaging on the Sender IDoc adapter and hence we used IDoc Packaging on ECC with a File XML Port. The IDoc XML would then contain multiple Idoc's and then this was picked by PI using a File Adapter. We arrived at the ideal packet size through a trial and error method in QA to get the optimum 5 MB Payload XML and then from there on, the scenario worked like a charm.

As already pointed by Vadym, look at the end to end flow, see what is your ideal batch size and make sure it is a point to point flow with no additional lookup's etc and this should just be fine.

Also - do check if this is a one time load or a daily / regular load. Make sure if this is a daily / regular load, your server is sized for the volume of messages expected.

Regards,

Bhavesh