Huge size file processing in PI

[Former Member]

Hi Experts,

1. I have seen blogs which explains processing huge files. for file and sftp

Here also we have constrain that we can not do any mapping. it has to be EOIO Qos.

would it be possible, to process  1 GB size file and do mapping? which hardware factor will decide that sytem is capable of processing large size with mapping?

is it number of CPUs,Applications server(JAVA and ABAP),no of server nodes,java,heap size?

if my system if able to process 10 MB file with mapping there should be something which is determining the capability.

this kind of huge size file processing will fit into some scenarios.  for example,Proxy to soap scenario with 1GB size message exchange does not make sense. no idea if there is any web service will handle such huge file.

2. consider pi is able to process 50 MB size message with mapping. in order to increase the performance what are the options we have in PI

i have come across these two point many times during design phase of my project. looking for your suggestion

Thanks.

[iaki_vila · Accepted solution]

Hi Ram,

There are not a straight answer for your question. I can talk about my experience with messages over 400MB (JDBC or File to ABAP Proxy).

The first problem is to configure the PI to accept this messages, for example if you are in a dual stack you need to increase the SMICM parameter about the maximum message size allowed.

My main problem has been the PI memory, the message need to be in java memory during the mapping. The best solution is to have a de-central adapter engine to work with this messages, even increasing the nodes in the central adapter can be problematic. (There are some notes about the memory problem, i can search again if you need them).

About your second question you can work with the PI queues, establishing priorities to avoid a larger time to wait for the little messages. Also, to increase the memory, but you should think that the receiver system can be a bottleneck when the PI tries to send the message.

Hope this helps.

Regards.

[Former Member]

Thank you Inaki and pavan for you valuable inputs.

[former_member183908]

Hi Ram,

You have not mentioned what sort of Integration it is.You just mentioned as FILE.I presume it is FILE To FILE scenario.In this case in PI 711 i am able to process 100MB(more than 1Million records ) file size with mapping(File is in the delta extract in SAP ECC AL11).In the sender file adapter i have chosen recordset per message and processed the messages in bit and pieces.Please note this is not the actual standard chunk mode.The initial run of the sender adapter will load the 100MB file size into the memory and after that messages will be sent to IE based on recordset per message.If it is more than 100MB PI Java starts bouncing because of memory issues.Later we have redesigned the interface from proxy to file asyn and proxy will send the messages to PI in chunks.In a single run it will sent 5000 messages.

For PI 711 i believe we have the memory limtation of the cluster node.Each cluster node can't be more than 5GB again processing depends on the number of Java app servers and i think this is no more the limitation from PI 730 version and we can use 16GB memory as the cluser node.

this kind of huge size file processing will fit into some scenarios.  for example,Proxy to soap scenario with 1GB size message exchange does not make sense. no idea if there is any web service will handle such huge file.

If i understand this i think if it is asyn communication then definitely 1GB data can sent to webservice however messages from Proxy should sent to PI in batches.May be the same idea can work for Sync communication as well however timeouts in receiver channel will be the next issue.Increasing time outs globally is not best practice however if you are on 730 or later version you can increase timeouts specific to your scenario.

To handle 50 MB file size make sure you have the additional java app servers.I don't remember exactly how many app server we have in my case to handle 100 MB file size.

Thanks

[former_member2539]

Hello Pavan,

     I am the Systems Engineer supporting the PI 7.3EHP1 SP13 System.  I have been asked to confirm and get some development advice on large messages sizes.

We have some output from ECC jobs that are growing from large 80Mb to 100Mb and they are failing to process out of ECC into PI.

Yes, it will give the PI JVM an OOM error and restarts the PI VM.

We have been talking about the need to break these up and send it through but in the end we need PI to encrypt and SFTP this file to  the partner as one file.  The last part is not an issue for us.  We are leveraging Advantco's interface plug-in for the last part.  Can please share some details of how one takes an AL11 large 100MB text file that is on ECC and gets it to process from ECC - to -  PI in smaller record sets  or chunks .  Then we hope it will allow us then to process the last part  which will encrypt and SFTP it out.

Again the partner is looking for 1 file.

In this simple case we are not leveraging the PI mapping etc.  But in other cases we are.

Thanks,

Mich Wilhelmi