on 01-07-2008 12:17 AM
Hello,
In our XI environment, we have two server nodes, quite offen i see lots of javacore?????????.txt is getting generated which causing /usr/sap/XIX is filled, due to this java goes down.
As a temp fix, i am removing those files and restarting java and server backs to normal.
I cheked std_serverx logs and default trace file. not able to get any concrete information about the cause of the crash. I could see this in std_serverx 'Caught OutOfMemoryError! '
we are using IBM JVM.
Below are the logs
tpd0101:xsxadm 195> grep Error std_server0.out
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:356)JVMDUMP006I Processing Dump Event "uncaught", detail "java/lang/OutOfMemoryError" - Please Wait.
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.security.AccessController.doPrivileged(AccessController.java:207)
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:170)
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
tpd0101:xixadm 196> ls
tpd0101:xixadm 109> grep dump javacore.20080102.085845.4885.txt
0SECTION TITLE subcomponent dump routine
0SECTION GPINFO subcomponent dump routine
1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
0SECTION ENVINFO subcomponent dump routine
2CIUSERARG -Xdump:heap:none
2CIUSERARG -Xdump
0SECTION MEMINFO subcomponent dump routine
0SECTION LOCKS subcomponent dump routine
0SECTION THREADS subcomponent dump routine
2XMFULLTHDDUMP Full thread dump J9SE VM (J2RE 1.4.2 IBM J9 2.2 Linux amd64-64 build 20061121_1035_LHdSMr, native threads):
0SECTION CLASSES subcomponent dump routine
3CLTEXTCLASS com/seeburger/xi/module/dump/PayloadDump(0x0000002B32F0A0D8)
0SECTION Javadump End section
tpd0101:xixadm 110> vi
0SECTION MEMINFO subcomponent dump routine
NULL =================================
1STHEAPFREE Bytes of Heap Space Free: 0SECTION MEMINFO subcomponent dump routine
NULL =================================
1STHEAPFREE Bytes of Heap Space Free: 1f57158
1STHEAPALLOC Bytes of Heap Space Allocated: 80000000
NULL
1STHEAPALLOC Bytes of Heap Space Allocated: 80000000
NULL
Can you please advice me on this.
Thanks,
Venkatesh
What os are you running on..Aix 5.3(or above).........if yes..then check the heap size..its different for aix.........let us know
Regards
Ravi Raman
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello,
Functional team, who do the configuration for messages, confirms the messages are small. I am not sure who to check that, can you any help to check the message size?
Max heap size for both the servers nodes are 2048MB in config tool.
OS - Linux x86_64
Please let me knw for any other information.
Regards,
Venkatesh
Hi,
I have checked the message size, messages are small.
After i restated the java, i found jlaunch occuping 99% of CPU. causing java to reboot on its own. and generation javacore file under serverx directory.
I got this informatioin from javacore file..
3LKWAITNOTIFY "JCO.ServerThread-88" (0x0000002B36639C00)
2LKREGMON Thread public flags mutex lock (0x0000000001EB5020): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000001EB5228): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000001EB5498): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000001EB5638): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000001EB56A0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000001EB5708): <unowned>
2LKREGMON QueueSlotMonitor-3 lock (0x0000000002CDB390): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDB598): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDB8D8): <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "JCO.ServerThread-80" (0x0000000008377F00)
2LKREGMON Thread public flags mutex lock (0x0000000002CDBA78): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDBDB8): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDBE20): <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "XI JMS2XI[JMS_SENDER_AMTRIX/AMTRIX/GL_CARS]_6" (0x0000002B24600100)
2LKREGMON Thread public flags mutex lock (0x0000000002CDBEF0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDBFC0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDC298): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000002CDC3D0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0000000001B6B798): <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "JCO.ServerThread-25" (0x0000002B28941100)
i think some prob with JMS_SENDER_AMTRIX or might me the locks in java threads. not sure about the the cause for this?
Did any seen this issue......can you pls advice me on this?
Thanks.
Venkatesh
Check when this is happening, it might be due to large message (this you can check using RWB --> Performance monitoring).
Second thing to check the heap size.
-Pinkle
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
88 | |
23 | |
11 | |
9 | |
8 | |
5 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.