Out of Memory Exception when processing SAP messages in xMII
I am using xmii 12. I have configured a message listener to listen for OPERA3 messages from SAP system and have configured a transaction to get executed when such a message comes.
The transaction will write the received XML into a file in local system and will execute a custom action which will parse the file and insert its contents to DB.
But twice when an initial download for OPERA3 has been triggered, the j2ee server has restarted with an exit code of 666(out of memory exception).
The file size of the OPERA3 xml message is 57MB and it has 60000 records in it.
I have created a custom action since standard XML Loader action will slow down handling files with larger size
1) Is the problem due to large amount of records and file size?
2) Should i have to increase the jvm parameters?
3) How should i approach messages with large number of records?
Matthias Wald replied
mass data and XML is a known issue. To process such data the complete DOM is built in memory by MII. If you want to fight this issue with
resources the only chance you have is that you use x64 Windows Server für MII with several GB for the JVM. Typically you could try to give
the JVM a heap size of 3GB/4GB (keep increasing till it works :-). Though this is architectural not a nice solution, so you should send, for
example, the message zipped and base64 encoded. This message could then be received by MII and saved to a file. Decoding and dezipping
this file should be done on file system such that a decoded and dezipped file is created. Then you can process this file with a custom action.
Remember that you will face the same resource problems in your custom action. The solution is to use an event driven parser like SAX to parse
the file and do the inserts in JDBC batch mode. With this procedure you can guarantee that a fixed amount of memory is used when parsing
a big XML file.