Avoiding SAP JAVA hung situation in entire SAP landscape due to heap dump generation

[Former Member]

Hi All,

Problem: java systems getting hanged due to heap dump generation  in the path /usr/sap/ resulting in service disruption. Even with the file system alerts, the generation( which happens very fast) and size of the dump file is so big(5GB-10 GB), before  BASIS team could act and move/delete the heap dump files the system does reaches the hung state(might sometime would require a restart as well).

Solution :

Create a separate mount point like  /usr/sap/dumps and point the dump generation in

that location rather than /usr/sap/<SID>. The size of the new file system could be 5-10 times the heap size(approx. 15-20 GB).

Question:

1.What parameters I need to tune?

2. How to get heap dump generated at will to see if the new heap dump files are getting generated in the new file system?

Regards,

Ashutosh

[hemanth2 · Accepted solution]

Hi Ashutosh,
 

Hope you are doing good.

In situations like this,  we can proactively monitor the issue by triggering manual heap dump.

Please see note 1004255 - How to create a full HPROF heap dump , especially this part:

On request

• Add -XX:+HeapDumpOnCtrlBreak  to the java options of the server (See note 710146 "How to change J2EE Engine JVM Settings")
• One has to trigger a SIGQUIT (UNIX) or CTRL_BREAK event (Win). The procedure to trigger the event is the same as for getting a thread dump (see note 710154)
• Output file name format is: java_pid<pid>.hprof.<timestamp> in the /usr/sap/<SID>/<instance>/j2ee/cluster/server<N> directory

Hope this helps.

_ _ _ _ _ _ _ _ _

Kind Regards,

Hemanth
SAP AGS
_ _ _ _ _ _ _ _ _
 

[Former Member]

Hi Ashish and Hemant, One more doubt: I went through the notes and found -XX:HeapDumpPath=OOM.hprof. My concern is as per the AP note 1004255 , it says change it to: -XX:HeapDumpPath= So the question is: 1. say I created /usr/sap/dump then the path would be -XX:HeapDumpPath=/usr/sap/dump or -XX:HeapDumpPath=//usr/sap/dump/OOM.hprof Please clarify my doubt. I want the log name for out of memory same. Only the patch should be /usr/sap/dump. Regards, Vicky

[Former Member]

Additionally: I tested the full scenario:

1. Added -XX:+HeapDumpOnCtrlBreak  and then generated the dump

1. Start the JCMon program with the profile of the instance.

2. Enter "20" to go to the "Local Administration Menue".

3. Note the process ID (PID) from the Java node.

4. Issue a "kill -3 <PID>" command from the shell.

And in the path /usr/sap/<SID>/JCXX/j2ee/cluster/server0 I saw

javacore.20150609.000905.19429.0005.txt

javacore.20150609.000848.19429.0004.txt

The I removed

1. Removed -XX:+HeapDumpOnCtrlBreak  and then generated the dump

1. Start the JCMon program with the profile of the instance.

2. Enter "20" to go to the "Local Administration Menue".

3. Note the process ID (PID) from the Java node.

4. Issue a "kill -3 <PID>" command from the shell.

Till I saw  in the path /usr/sap/<SID>/JCXX/j2ee/cluster/server0 I saw

javacore.20150609.000905.19429.0005.txt

javacore.20150609.000848.19429.0004.txt

That means I get these files even when I do not use -XX:+HeapDumpOnCtrlBreak

Please suggest.

1. Adding the parameter -XX:+HeapDumpOnCtrlBreak to generate the dump does not take effect. I restared the system everytime I changed the parameter.Later cross checked as well in the configtool.

2. What would be the path

Say I created /usr/sap/dump then the path would be -XX:HeapDumpPath=/usr/sap/dump

or -XX:HeapDumpPath=//usr/sap/dump/OOM.hprof Please clarify my doubt.

I want the log name for out of memory same. Only the patch should be /usr/sap/dump.

Regards,

VIcky

[Former Member]

Hello Vicky,

Path should be :  -XX:HeapDumpPath=/usr/sap/dump

Make sure you have enough space available there.

Try generating heap dump from jsmon, follow below wiki-

How to create a heap dump of the Application Server Java? - Technology Troubleshooting Guide - SCN W...

Regards,

Ashish

[divyanshu_srivastava3]

Hi,

Have you tried using jvmmon ?

SAP JVM – Analysis of permanent generation memory usage using jvmmon - SAP Netweaver Application Ser...

Using JVMMON to Trigger a Heap Dump - SAP Manufacturing Execution (ME) - SCN Wiki

And then use MAT for this purpose.

Regards,

[divyanshu_srivastava3]

Do a quick analysis of heap dump using MAT.

Also, then tune your memory parameters.

Regards,

[Former Member]

Hello Vicky,

You can change/add below parameter to change the path of heap dump-

-XX:HeapDumpPath=<directory where to save the heap dumps>

Refer SAP notes 1004255 and 710154 to know how to generate heap dump.

Regards,

Ashish