Java Stack goes down when processing large files

[Former Member]

Hi,

I have a case in where I have to process 150.000 records.

The 150.000 records are in a unstructured text format (18MB)

I map these to a IDoc (BOMMAT) which is going to generate approximately 110.000 IDocs.

Now, while this process is going on, the PI Java Stack goes down. I am aware that this is/might be a big chunk for PI BUT how am I going to do this?

I tried in FCC to use Recordsets per Message (5.000) but no help.

How can I achieve this???

Please help.

Thanks.

[RaghuVamseedhar]

Sorry.. I posted in this thread by mistake.

Edited by: Raghu Vamsee on Jan 9, 2011 6:56 PM

[Former Member]

Chris,

The best way is to split the large files into chunks and then running those in a separate mapping, you can create 2 mapping one for splitting the files and the other for processing them. for example breaking an order with 5000 line items to 500 line items each.

Regards,

Pavan

[abhay_rajhans2]

Hi Chris,

You can try following resolutions.

1) Convert the files into small files of around 3-4 MB. This can be done using some script on OS level. This will reduce load on the PI server and processing will be fast.

2) Check the message mapping which you are using. Earlier in our project we were processing 15 MB files. It took normally 10-15 hrs. The issue was with the message mapping and logic. Once issue resolved processing time reduced.

3)After that We processed 50 MB files at that time it took a lot time so we raised with SAP then they us a patch to update and told us to replace if functions with IFS.

4) Once you able to process the file in PI then tell the Basis team to check the parameter of lock table in SM12 transaction. It should be able to process the number of IDocs.

[baskar_gopalakrishnan2]

PI recommends file size acceptable range is not more than 5 mb. Why dont use multimapping?

I see your one file has 150000 records. you can create your root node idoc to say 1 to unbounded instead one to one.

or in other words, you can create many idocs on the sap ecc6 side, rather creating one big idoc with all records.

You can edit the idoc and import the idoc using external definition for editing the occurences.

Hope this helps for sure.

[Former Member]

Hi,

Any ideas how I can speed up the process?

I set the Recordset per Message to 5000 but,

as I understand, the file adapter takes the whole file and tries to map it to the IDocs. Each mapping operation takes then 5000 records.

How can I accelerate the process?

[Former Member]

Hi Chris,

is it one file with 150.000 records or 150.000 files?

What is the inbound adapter?

We have a similar problem with inbound SOAP and http adapter.

A 5 MB file takes 18 hours to be processed.

I suspect XML parsing to be the malefactor.

With inbound RFC Adapter 5 MB is no problem.

Another effect i realized is processing time is increasing exponential with file size (not linear).

For your scenario:

split inbound data into smaller pieces and poll them with timeoffset e.g. fileadapter if possible.

Try different inbound adapters because there are differences in processing speed.

good luck

Michael

[Former Member]

Hi,

It is one file with 150.000 records. The scenario is file to IDoc. In mapping there will be 110.000 IDocs created. It is still working right now. (2 hours and continuing)....