Error after parsing huge file in XI

[Former Member]

Hi experts,

I have a simple interface

FILE -> XI -> FILE

The outbound file is 1 big file of size more than 25MB (Contains 28K records)

The file receiver creates 1 file for each record ==> 1 input file creates more than 28K files.

But I encounter the following error after I try to send this big file into XI:

<i>503 Service Unavailable.

Dispatcher running, but no server connected!

</i>

To rectify this error, I had to restart the J2EE instances.

However, when I use a smaller file size (E.g. 10MB), it works fine.

It looks to me like there are some memory settings I can toggle with to prevent XI from crashing.

But what can I tune?

Please help!

Thanks.

Ron

[Former Member]

Hi Ron,

Is there a need to split the message, 1 record per file?

Is it possible to process this file as 1 message? 25M should still be fine without causing any memory problems.

Regards,

Bill

[prabhu_s2]

not sure if this has something to do with memory, i beleive that u will not be able to extend ur memory settings or h/w settings. check with quicksizing tool. i remember reading one of the thread before stating on partitioning

[Former Member]

>>>503 Service Unavailable.

>>>Dispatcher running, but no server connected!

Yes, you probably hit a out-of-memory error that brought down the java server and induced a restart.

Anytime you deal with files of significant size such as this one, proper system tuning is critical, especially the jvm tuning and the java heap size. Begin with Note 723909 - Java VM settings for J2EE 6.40/7.0.

If you're using File Content Conversion, then using the "Recordsets per message" parameter will help as it will split your message into separate messages according to this setting.

Regards,

Jin

[Former Member]

> If you're using File Content Conversion, then using

> the "Recordsets per message" parameter will help as

> it will split your message into separate messages

> according to this setting.

Can you please elaborate on the "Recordsets per messge" parameter, and how I can use it?

My scenario basically requires me to split 1 huge message into multiple smaller messages.

E.g.

Simplified Source format:

<OpChargesList>

<OpCharge> </OpCharge>

<OpCharge> </OpCharge>

<OpCharge> </OpCharge>

</OpChargesList>

Simplified Target format:

<OpCharge> </OpCharge>

I do not use a BPM for this.

I've set the QoS to EO.

[Former Member]

Hi Ron,

Another option is to process the file as one message, and if there is a need to split them to the receiver, you can use multi-mapping in Interface Determination to send it out as muliple files.

The advantage of this is that small message size going thru XI contains overhead.

Regards,

Bill

[Former Member]

As I mentioned, the parameter ""Recordsets per message" is only relevant if using File Content Conversion (e.g. using file adapter to convert text to xml). Based on your example, your source seems to be an XML file, so content conversion (and the parameter) is not applicable.

Check your logs (e.g. usr/sap/<SID>/DVEBMGSXX/work directory; perhaps default trace files) to verify that an outofmemory occurred. Then check the Note 723909 to start verifying your jvm settings. Depending on your access (e.g. to VA or config tool), you may need to work with your XI/Basis team.

Regards,

Jin

[Former Member]

> Hi Ron,

> Another option is to process the file as one message,

> and if there is a need to split them to the receiver,

> you can use multi-mapping in Interface Determination

> to send it out as muliple files.

>

> The advantage of this is that small message size

> going thru XI contains overhead.

>

> Regards,

> Bill

Hi Bill,

Thanks for your advise.

My actual scenario is

FILE -> XI -> SOAP.

Reason why I was using File Receiver Channel is because the web service wasn't ready.

Basically each web service accepts messages at a more granular level, hence there's a need to split the messages.

I'm NOT using BPM, and I'm doing the split by using message-mapping - SplitByValue, with the target structure occurence set to 0..unbounded.

Hope this explains my case more clearly.

Sorry for the confusion

Ron

[Former Member]

> As I mentioned, the parameter ""Recordsets per

> message" is only relevant if using File Content

> Conversion (e.g. using file adapter to convert text

> to xml). Based on your example, your source seems to

> be an XML file, so content conversion (and the

> parameter) is not applicable.

>

> Check your logs (e.g. usr/sap/<SID>/DVEBMGSXX/work

> directory; perhaps default trace files) to verify

> that an outofmemory occurred. Then check the Note

> 723909 to start verifying your jvm settings.

> Depending on your access (e.g. to VA or config

> tool), you may need to work with your XI/Basis

> team.

>

> Regards,

> Jin

Where can I configure my heap size manually?

If it's a simple java program, I typically do this by executing java -Xms512 -Xmx2048 ...

My question is where can I pass in these JVM Args to the J2EE instance?

Thanks.

Ron

[Former Member]

The heap settings apply across the java stack and can be maintained using the config tool (more info in Note 710146). You cannot maintain or pass specific heap or other jvm settings when running a particular scenario.

Regards,

Jin

[Former Member]

Hi Ron,

I believe you can still accomplish what you need by sending the complete file as 1 msg. Then, use multi-mapping to split the file into records and send each one to the SOAP adapter.

Multi-mapping does NOT need BPM. You can use it in Interface Determination. Check this help: http://help.sap.com/saphelp_nw04/helpdata/en/42/ed364cf8593eebe10000000a1553f7/frameset.htm

I believe this will alleviate much of the resource/performance problem you are having.

Regards,

Bill

[Former Member]

> The heap settings apply across the java stack and can

> be maintained using the config tool (more info in

> Note 710146). You cannot maintain or pass specific

> heap or other jvm settings when running a particular

> scenario.

>

> Regards,

> Jin

Hi,

I followed the SAPNote, and configured the maximum heap size to 2048MB.

Initially the maximum heap size was set to 1024MB.

Minimum heap size remains at 1024MB. (-Xms)

The physical memory I have on the box is 4GB.

But when I tried to restart the J2EE cluster with this configuration, the JVM crashed and the server could not start up.

May I know what is a suitable size I can set to the maximum heap size to?

Because ultimately, I want to resolve the huge file parsing problem.

Please help.

Thanks.

Ron

[Former Member]

Hi Ron -

2 GBs may be excessive. Whatever the set value, one thing to keep in mind is to set min and max (-Xms -Xmx) heap size equal. This is SAP's recommendation (also mentioned in the Note).

Have you verified that an OOM is occurring from the logs (e.g. std_server0.out in the work directory)? This file can get overwritten upon restart. To keep copies of it, you need to maintain an instance profile setting: jstartup/keep_old_logfiles=3

For a 25 MB file with no content conversion, you really shouldn't need much more than 1024MB, if any. Besides the heap, please verify the other recommended JVM settings provided in the Note.

There are many things to consider that has not been provided. What is your OS/DB? 32-bit/64-bit? What JVM version are you using? Which XI release - 3.0 or 7.0? What SPS Level? Are there other applications running on your system?

Please post your 'instance.properties' file and 'instance.properties.vmprop' files if you can - located at 'usr/sap/<SID>/DVEBMGS<XX>/j2ee/cluster'.

Please also look at the latest <a href="http://service.sap.com/~sapidb/011000358700007402452005E/XI70-Tuning-01.pdf">XI Tuning Guide</a> available on SAP SMP.

Regards,

Jin