cancel
Showing results for 
Search instead for 
Did you mean: 

Javacore dump in XI

Former Member
0 Kudos

Hello,

In our XI environment, we have two server nodes, quite offen i see lots of javacore?????????.txt is getting generated which causing /usr/sap/XIX is filled, due to this java goes down.

As a temp fix, i am removing those files and restarting java and server backs to normal.

I cheked std_serverx logs and default trace file. not able to get any concrete information about the cause of the crash. I could see this in std_serverx 'Caught OutOfMemoryError! '

we are using IBM JVM.

Below are the logs

tpd0101:xsxadm 195> grep Error std_server0.out

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)

java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:356)JVMDUMP006I Processing Dump Event "uncaught", detail "java/lang/OutOfMemoryError" - Please Wait.

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)

java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.security.AccessController.doPrivileged(AccessController.java:207)

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:170)

java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

tpd0101:xixadm 196> ls

tpd0101:xixadm 109> grep dump javacore.20080102.085845.4885.txt

0SECTION TITLE subcomponent dump routine

0SECTION GPINFO subcomponent dump routine

1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.

0SECTION ENVINFO subcomponent dump routine

2CIUSERARG -Xdump:heap:none

2CIUSERARG -Xdump

0SECTION MEMINFO subcomponent dump routine

0SECTION LOCKS subcomponent dump routine

0SECTION THREADS subcomponent dump routine

2XMFULLTHDDUMP Full thread dump J9SE VM (J2RE 1.4.2 IBM J9 2.2 Linux amd64-64 build 20061121_1035_LHdSMr, native threads):

0SECTION CLASSES subcomponent dump routine

3CLTEXTCLASS com/seeburger/xi/module/dump/PayloadDump(0x0000002B32F0A0D8)

0SECTION Javadump End section

tpd0101:xixadm 110> vi

0SECTION MEMINFO subcomponent dump routine

NULL =================================

1STHEAPFREE Bytes of Heap Space Free: 0SECTION MEMINFO subcomponent dump routine

NULL =================================

1STHEAPFREE Bytes of Heap Space Free: 1f57158

1STHEAPALLOC Bytes of Heap Space Allocated: 80000000

NULL

1STHEAPALLOC Bytes of Heap Space Allocated: 80000000

NULL

Can you please advice me on this.

Thanks,

Venkatesh

Accepted Solutions (0)

Answers (2)

Answers (2)

ravi_raman2
Active Contributor
0 Kudos

What os are you running on..Aix 5.3(or above).........if yes..then check the heap size..its different for aix.........let us know

Regards

Ravi Raman

Former Member
0 Kudos

Hello,

Functional team, who do the configuration for messages, confirms the messages are small. I am not sure who to check that, can you any help to check the message size?

Max heap size for both the servers nodes are 2048MB in config tool.

OS - Linux x86_64

Please let me knw for any other information.

Regards,

Venkatesh

Former Member
0 Kudos

Heap size looks ok.

you can check the message size in RWB --> Performance monitoring..then select the time duration when dump occured and check the total size processed during that time.

This might give you some clue.

-Pinkle

Former Member
0 Kudos

Hi,

I have checked the message size, messages are small.

After i restated the java, i found jlaunch occuping 99% of CPU. causing java to reboot on its own. and generation javacore file under serverx directory.

I got this informatioin from javacore file..

3LKWAITNOTIFY "JCO.ServerThread-88" (0x0000002B36639C00)

2LKREGMON Thread public flags mutex lock (0x0000000001EB5020): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000001EB5228): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000001EB5498): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000001EB5638): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000001EB56A0): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000001EB5708): <unowned>

2LKREGMON QueueSlotMonitor-3 lock (0x0000000002CDB390): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDB598): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDB8D8): <unowned>

3LKNOTIFYQ Waiting to be notified:

3LKWAITNOTIFY "JCO.ServerThread-80" (0x0000000008377F00)

2LKREGMON Thread public flags mutex lock (0x0000000002CDBA78): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDBDB8): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDBE20): <unowned>

3LKNOTIFYQ Waiting to be notified:

3LKWAITNOTIFY "XI JMS2XI[JMS_SENDER_AMTRIX/AMTRIX/GL_CARS]_6" (0x0000002B24600100)

2LKREGMON Thread public flags mutex lock (0x0000000002CDBEF0): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDBFC0): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDC298): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000002CDC3D0): <unowned>

2LKREGMON Thread public flags mutex lock (0x0000000001B6B798): <unowned>

3LKNOTIFYQ Waiting to be notified:

3LKWAITNOTIFY "JCO.ServerThread-25" (0x0000002B28941100)

i think some prob with JMS_SENDER_AMTRIX or might me the locks in java threads. not sure about the the cause for this?

Did any seen this issue......can you pls advice me on this?

Thanks.

Venkatesh

Former Member
0 Kudos

Hi Venkatesh,

I've seen high CPU usage when java is performing garbage collection.

Check whether all java parameters are tuned correctly.

-Pinkle

Former Member
0 Kudos

Check when this is happening, it might be due to large message (this you can check using RWB --> Performance monitoring).

Second thing to check the heap size.

-Pinkle