Heap Dump file generation problem

[Former Member]

Hi,

I've configured configtool to have these 2 parameters:

-XX:+HeapDumpOnOutOfMemoryError

-XX:+HeapDumpOnCtrlBreak

In my understanding, with these 2 parameters, the heap dump files will only be generated under 2 situations, ie, when out of memory occurred, or user manually click CLTR + BREAK in MMC.

1) Unfortunately, there are many heap dump files (9 in total) generated when none of the above situation occured. I couldnt find "OutOfMemoryError" in the default trace, nor the shallow heap size of those heap dump files are anywhere near the memory limit. The consequences are our server run out of disk space.

My question is, what are the other possibilities that heap dump file will be generated?

2) In the Memory Consumption graph (NWA (http://host:port/nwa) -> System Management -> Monitoring -> Java Systems Reports), out of memory error occurred when the memory usage is reaching about 80% of the allocated memory. What are the remaining 20% or so reserved for ?

Any help would be much appreciated.

Thanks.

[krumts · Accepted solution]

Hi,

Having the -XX:+HeapDumpOnCtrlBreak option makes the VM trigger a heap dump, whenever a CTRL_BREAK event appears. The same event is used also to trigger a thread dump, an action you can do manually from the SAP Management Console, I think it is called "Dump stacks". So if there was someone triggering thread dumps for analysis of other types of problems, this has the side effect of writing also a heap dump.

Additionally, the server itself may trigger a thread dump (and by this also a heap dump if the option is present). It does this for example when a timeout appears during the start or stop of the server. A thread dump from such a moment allows us to see for example which service is unable to start.

Therefore, I would recommend that you leave only the -XX:+HeapDumpOnOutOfMemoryError, as long as you don't plan to trigger any heap dumps on your own. The latter will cause the VM to write a heap dump only once - on the first appearance of an OutOfMemoryError.

In case you need to trigger the heap dumps manually, leave the -XX:+HeapDumpOnCtrlBreak option for the moment of troubleshooting, but consider if you want to keep it afterwards.

If heap dumps were written because of an OutOfMemoryError you should be able to see this in the dev_server file in /usr/sap/<SID>/<inst>/work/ . Also there you should be able to see if indeed thread dumps were triggered (just search for "Full Thread ").

I hope this helps.

Regards,

Krum

[Former Member · Accepted solution]

Hi krum,

Thanks for the quick response. I will remove the -XX:+HeapDumpOnCtrlBreak parameter.

Anyway, do you have any idea of the explanation for question 2) ?

Thanks again.

cheers,

julius

[krumts · Accepted solution]

Hi,

Well, a sure answer I don't have. But I have a some guesses:

- the OutOfMemoryError is not thrown because of heap exhaustion. There could be other reasons, e.g.:

.....* OutOfMemoryError because of exhausted permanent generation (I think the message is "OutOfMemoryError: PermGen"),

.....* native space exhaustion - error with some complaint about swap space, or unable to create native thread, etc...

- the second explanation I can think of, is just a matter of visualisation. In HotSpot VMs the heap is separated into several areas (young generation (also split in two parts) and old generation). The max could mean all of them. And it may happen that at the moment of the OOM you have a full old generation, and some free space in the young gen. Not sure if this will make up to 20% however ...

- it may be also that the application tries to allocate one huge object (can happen with resizing StringBuffers, ArrayLists, etc...). If the single array is bigger in size than the 20% free memory, you will get an OutOfMemoryError...

I think what could help answering the question is a look at the verbose GC output at the moment of such a problem.

Regards,

Krum