Large volume input data file: 300MB

[Former Member]

I have a text file whose size is 300MB, I need to transform it to a XML file using file adapters. It didn't work at the beginning and I got error message as:

Java.io.IOException: Unable to eat more bytes

Then I set the "Recordsets per message" to 10, I got error message as:

Java.lang.OutOfMemoryError: Java heap space (failed to allocated 427725576 bytes)

I am just wondering is there any solution for large volume data file?

Much appreciate it if any inputs

[baskar_gopalakrishnan2 · Accepted solution]

Talk to BASIS to increase java stack heap space memory. But still 300 mb is still larger size file. Can't you split the data file into smaller chunks like 100 mb each so that you don't overload PI for processing it. If you decrease the file size, increasing java heap space & using recordsets per message would help sure for some extent.

[Former Member · Accepted solution]

Thank you Baskar.

Yes, I emailed our Basis people about increasing Java heap space.

Also spliting the file into few smaller files is my last option. I am trying to make it work in the current PI enviroment before requesting for spliting the file.

I was just wondering if anyone else here encountered the same problem and have it resolved without splitting the file.

[former_member184681 · Accepted solution]

Dear Blue L,

To my opinion, increasing the memory resources for PI should be enough to make your scenario work. Splitting the file shouldn't be necessary, since this is exactly what the Recordsets per message feature does. So if only PI is able to read the whole file (which you have already proved to be true) and has enough operating memory to process so many messages in parallel, it should work fine. The only question that remains is the performance of your scenario, but this might be a sort of a tradeoff you could agree for.

Regards,

Greg

[Former Member · Accepted solution]

Thank you very much Greg.

I totally agree with what you said. The matter now is convince our Basis people to increase the memory size for us. And the performance shouldn't be a big problem, since this task is more of one time thing, and it won't be running daily.