High volume of data!

[Former Member]

Hi folks,

I’m very worried with an issue regarding high volume of data. Our customer has a backend system that generates a huge file, and this huge file must be sent daily trough XI to SAP ERP. This file can have a size between 300mb and 400mb in a scenario File to IDoc. I’m worried because I don’t know if XI is ready to handle huge files and keep up the system performance... I spoke with the customer, and if it helps, this file can be split it in few smaller files, but they only can split it in a slices with fixed size - one file with 300mb becomes in 30 files of 10mb. (Note: this file or these files will be placed into XI file system from a FTP tool).

Well, I suppose the hardware requirements must be increased (CPU, memory, broadband, etc). But, the reason of my post is to know if anyone has faced an issue like this one and which is the best approach to handle this issue... Should customer split the huge file? And if he split it, how can I handle those files in order to maintain the links between them? Remember, we are talking about sliced files with fixed size…

Thanks in advance,

Ricardo.

[Former Member · Accepted solution]

Hi Ricardo,

for my experience you should not have troubles handling files of 10mb, so I suggest you to have the files splitted because a file with size 400mb will normally not be processed by XI (always remember that the message size inside XI will increase because of the XML tags and so on).

For mantaining the order of files you can use qos Exactly Once in Order, specifing the queue name (see: )<a href="http://help.sap.com/saphelp_nw04/helpdata/en/e3/94007075cae04f930cc4c034e411e1/frameset.htm">Configuring the Sender File/FTP Adapter</a>

Regards,

Sergio

[agasthuri_doss]

Hi,

Pls go thru these links, they give some idea

topic #3.42

https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/70ada5ef-0201-0010-1f8b-c93...

/people/sap.user72/blog/2004/11/28/how-robust-is-sap-exchange-infrastructure-xi

http://help.sap.com/bp_bpmv130/Documentation/Planning/XISizingGuide.pdf#search=%22Message%20size%20i...

A discussion that would be useful,

Regards

Agasthuri

[bhavesh_kantilal]

I would agree with the comments here.

As you have the option of Spliiting the files outside of XI, i would suggest that you go ahead and do this.

Regards

Bhavesh

[Former Member]

Hi all,

I guess I’m unable to use compression, because I need to manipulate the file content with multimapping to 3 IDoc's. I will split the huge file and I will need to read one by one to generate 3 different types of IDoc’s (WPUBON, WPUBAT and WPUFIB). If I understood Alessandro’s weblog, it is only possible use this compression feature without manipulate the file content anyhow (no content conversion, no mapping, no xml... nothing!).

It seams great, but in my case it doesn’t apply.

Cheers,

Ricardo.

[Former Member]

Yes Ricardo,

your right, file compression is only useful for simple file to file scenario with no mapping (file transfer).

You have to split the files as much as possible and use QoS EOIO to be able to respect the sequence.

Kind Regards,

Sergio

[Former Member]

Hi Ricardo,

as the previous colleague answered, the size of the message will increase a lot after the mapping, so the size could easily increase double, triple or even more.

Technically speaking, XI could handle a file up to 2Gb - but I havent seen so far such a big file! Even with files of 50mb, 100mb a lot of tweaking has to be done.

I suggest you two things:

1- Tweak performance: have a look at note:894509 and weblog

/people/prasad.illapani/blog/2007/03/08/performance-tuning-checks-in-sap-exchange-infrastructure

2- Divide the file into several files

3- Use compression

Check weblog /people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover

Hope this helps you

Regards,

Jaime