XI and Large Message Processing - should we use XI of not?

[Former Member]

We have a scenario where the legacy SAP system has allot of data extraction programs that download files and ftp them somewhere. These files range from anywhere between 10MB and 350MB in size.

We are upgrading to ECC6 and XI. We need to send the data from the abaps to XI then out using the file adapter.

Our testing has shown that once we start processing messages above approx. 25MB that XI grinds to a halt (java out of memory, cpicerr, etc). These messages actually convert to around 100MB when wrapped in XML. And this happens on the transfer of the message from ECC6 to XI - we haven't even got to the file adapter yet. We are using ABAP proxy to transfer the data which seems to be able to handle the large sizes much better than RFC.

I've gone thru all the previous forum posts about increasing the heap size etc, but my question is more architectural - ie. should we use XI for these types of data extracts?

There is a weblog (/people/william.li/blog/2006/09/08/how-to-send-any-data-even-binary-through-xi-without-using-the-integration-repository) that mentions you can send any files through XI without mapping and without repository objects by simply using a receiver determination and the file adapter. We have also tested this and notice that even though processing is much quicker - the adapter engine is still slowed down and if we have several of these processing at once we're unsure of the consequences! We only tried with a 60MB message as well.

Michal Krawczyk mentioned in a comment that it would be better to use a java proxy with some java code to copy or ftp the files instead - BUT... is it even worth using XI in this case - what does it add?

Jason.

[bhavesh_kantilal · Accepted solution]

Jason,

<i>BUT... >s it even worth using XI in this case - what does it add?</i>

I agree with you. If XI is going to add as a mere FTP then what is the point of using XI in such interfaces? One can always install a FTP server and use one instead!

Most of the times, it becomes a case that the client has an XI box and does not want to install a FTP server and so wishes XI to pick the file and push it across to some other location. it doesnt make much sense I agree , but such things actually do prevail in real time projects as well , unfortunately!

Regards

Bhavesh

[Former Member · Accepted solution]

Hi Bhavesh,

_*When there is an error , corresponding error page should be displayed instead of whole stack trace.

*_I want to achieve this with CE 7.1 EHP1.

To cary out this I am not finding HTTP property ServiceUnavailableResponse and Webcontainer property ApplicationStoppedFile in nwa system configuration.

Please can you help to know how to get this properties in nwa .

It is urgent.

Regards

Mahesh

[Former Member]

Jason,

use of XI migth help you have features like centralized monitoring , de-coupling of systems which are highly desirable in complex system landscapes.

Agreed though you cant have something in place which gives you poor performance in case of high data volumes but you can always think of alternatives as suggested by Michal...say in this case there would be a single technology layer between your file and the Java proxy and it would be using standard java io functionalities and the performance has got to be better...FTP using file adapter will have multiple technologies / protocols involved i.e. FTP protocol + File Access + Adapter Engine in Java + Integration engine layer in ABAP and so on....you can see that the parties involved in the over all data transfer increase in number...

[deidre_logan]

We are having the same issue... We have a 400MB file that they want to move through XI.

We have updated the HTTPProvider service on the Java server dispatcher to have the parameter MaxRequestContentLength = 800M

The ABAP parameters icm/HTTP/max_request_size_km set to 2097152, icm/server_port_0 TIMEOUT set to 600. rdisp/max_wprun_time = 3600, ztta/max_memreq_MB =2048.

In SXMB_ADM we add the parameter TUNING EO_MSG_SIZE_LIMIT = 40M

What are the actual limitations of XI for filesize? Any other parameters you all have updated to proces large files?

What is your limits?

We get the OutOfMemory error on the java server and it just bounces up and down until we deactivate the communication channel trying to pick up the file.

[former_member193518]

In our scenario we are processing files of max 75MB.

We have Xi 3.0 and tuned the Java stack with 2GB or heap size and large permanent size.

After 75MB the system restarts the Java stack with Out of Memory errors.

If you want to process large files (100MB+) SAP suggest using a seperate server instance on additional hardware (an apps server).

Maybe even 64bit hardware and OS could help??

Just my experience....

[deidre_logan]

We are running on HP_UX and 64bit kernel. We can process a 300MB file... I might just work with increasing the parameters mentioned above to see if we can get to 400MB.

thanks,