Large Volume of data using Proxy

[Former Member]

Dear Members,

We are facing an issue when we are trying to extract huge volume of data from SAP backend and send the same to PI using proxy.

The scenario : - Extract data from SAP and call ABAP proxy class with the data load. Now, if the volume of the data is huge, for ex: 40,000 records, which comes to be around 60Mb XML when passing to proxy, the proxy call fails.

We can see the error in sxi_monitor in ECC/CRM.

The errror in smq2 - "XI Error CLIENT_RECEIVE_FAILED.INTERNAL: Queue stopped"

Error in moni IN ECC:

  <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<!--  Call Integration Server

  -->

<SAP:Error SOAP:mustUnderstand="" xmlns:SAP="http://sap.com/xi/XI/Message/30" xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">

  <SAP:Category>XIServer</SAP:Category>

  <SAP:Code area="INTERNAL">CLIENT_RECEIVE_FAILED</SAP:Code>

  <SAP:P1>110</SAP:P1>

  <SAP:P2>HTTPIO_PLG_CANCELED</SAP:P2>

  <SAP:P3 />

  <SAP:P4 />

  <SAP:AdditionalText />

  <SAP:ApplicationFaultMessage namespace="" />

  <SAP:Stack>Error while receiving by HTTP (error code: 110 , error text: HTTPIO_PLG_CANCELED)</SAP:Stack>

  <SAP:Retry>A</SAP:Retry>

  </SAP:Error>

Is there any standard solution where we can send huge volumes of data to PI without the above error.

[Former Member · Accepted solution]

Hi Souvik,

Please check this blog posted by Sabyasachi Masanta, you might find this helpful:

http://scn.sap.com/people/smasanta/blog/2012/12/21/how-to-handle-large-volume-of-data-for-proxy-to-f...

Regards,

Richa

[Former Member · Accepted solution]

Thanks Richa. This solves our issue.

However, we also have another option where we append the records in the file by sending the file name as parameter in the source structure. After all the records are appended, we write a trigger file so that the next process (of the file being picked up) is triggered.

All this has to be done using EOIO obviously.

Thanks all for your inputs.

Souvik

[nabendu_sen]

Hi Souvik,

Please check the below thread:

http://scn.sap.com/thread/1843220

[baskar_gopalakrishnan2]

What pi version do you use? PI 7.3 version can handle large volume of message size. But the optimum message size handling range for PI 7.1 is upto 5 mb. you might want to split the data in the proxy logic and send the smaller messages for the optimum performance.

[Former Member]

Hi Baskar,

We are using PI 7.3.

We tried to split the data into multiple proxy calls but PI needs to write a single file. The merging of the files is a problem since the files has two parts:

1. First line is header

2. File name should have timestamp.

Thanks,

Souvik

[rajasekhar_reddy14]

pricessing 60MB file using middleware when transformation involved means always risky, even if you use latest version 7.3 doesnt mean that it works.You need right hardware to handle this kind of huge files.

But you need right design to handle this, like spliting the file in Proxy code and target side use append mode with filenaming scheme time stamp.

Regards,

Raj

[iaki_vila]

Hi Souvik,

If in your scenario you don't need message mapping you can try another alternative. By your explanation i guess that your scenario is PROXY-FILE. You could write a file in your abap endpoint, instead of proxy. Later with PI you will move the file from one file system to another, and in this moment you could use file chunk mode to move a huge file, check this Former Member's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2011/12/26/fileftp-adapter--large-file-trans...

Regards.