Java Dispatcher not coming up,Server node and SDM is running.

[Former Member]

Hi Everyone,

While starting my Java instance, Dispatcher is stopped with return code "666",Server node and SDM is running.This is all of sudden not sure what has changed.

I have changed the heap size value of dispatcher ,Did cleanipc ,But still no luck.

My heap size parameters are according to SAP note:

Note 0001024539 - Recommended Settings for NW04 >= SP14, NW 7.0 >= SP6 on AIX-BPBO.

Java version :

java version "1.4.2"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2)

Classic VM (build 1.4.2, J2RE 1.4.2 IBM AIX 5L for PowerPC (64 bit JVM) build caix64142-20080923 (SR12) (JIT enabled: jitc))

Please find below dispatcher error log.

*************************************************************************************************************************************************

Mon Jan 10 15:36:04 2011

JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.

JVMDUMP007I JVM Requesting System Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/core.20110110.153604.2523266.0001.dmp'

Note: "Enable full CORE dump" in smit is set to FALSE and as a result there will be limited threading information in core file.

Mon Jan 10 15:36:14 2011

JVMDUMP010I System Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/core.20110110.153604.2523266.0001.dmp

JVMDUMP007I JVM Requesting Snap Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/Snap.20110110.153604.2523266.0002.trc'

JVMDUMP010I Snap Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/Snap.20110110.153604.2523266.0002.trc

JVMDUMP007I JVM Requesting Heap Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/heapdump.20110110.153604.2523266.0003.phd'

Mon Jan 10 15:36:15 2011

JVMDUMP010I Heap Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/heapdump.20110110.153604.2523266.0003.phd

JVMDUMP007I JVM Requesting Java Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/javacore.20110110.153604.2523266.0004.txt'

JVMDUMP010I Java Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/javacore.20110110.153604.2523266.0004.txt

JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".

</verbosegc>

[Thr 6428] JLaunchIExitJava: exit hook is called (rc = 666)

[Thr 6428] **********************************************************************

• ERROR => The Java VM terminated with a non-zero exit code.

• Please see SAP Note 943602 , section 'J2EE Engine exit codes'

• for additional information and trouble shooting.

**********************************************************************

[Thr 6428] SigISetIgnoreAction : SIG_IGN for signal 20

[Thr 6428] JLaunchCloseProgram: good bye (exitcode = 666)

Could somebody help...

Thanks

[Former Member · Accepted solution]

Hey,

The following note may directly relate to the issue experienced

Note 977070 - OutOfMemoryError on the dispatcher node during startup

Regards,

John

[Former Member · Accepted solution]

John -Thanks a lot,Issue resolved.

[Former Member · Accepted solution]

Hi Mohanned,

Would like to know what was the resolution. We run into same issue on our new java systems based on 7.00 version.

We build the system thru migration, but the dispatcher  and server are failing to start.

[Former Member]

Hello,

as you can see, exitcode 666 represents Out Of Memory issue. As you already checked note 1024539 I'd recommend you to check the following note and forum thread:

723909 - Java VM settings for J2EE 6.40/7.0

Bear in mind that you should adjust the parameters for dispatcher, server(s) and SDM. I can see an heapdump files was generated when this happened. After you tune your system with all the recommended parameters on the documentation above, if the issue persists then we should check what information the heapdump can show us.

Regards,

Jonas Cella

[Former Member]

Hi

Thanks...I already tried the parameters in SAP Note: 723909 on dispatcher and server node but no success.Did lot trial and error by changing the heap values but still the dispatcher is restarting by itself two three times and then stopping.

Any other suggestions??

Thanks

[Former Member]

Hello,

The dispatcher does not run any applications so it is very unusual that it has an OOM issue. Can you provide more details on:

the number of server nodes it connects to?

How frquently these OOM errors occur?

It might also be useful to view the configuration data from the dispatcher. The parameters should be listed under "read node properties " within the dev_dispatcher file itself,

KR,

John

[Former Member]

Hi John ,Thanks

The dispatcher connects to only one server node.dev_dispatcher content details below.

**********************************************************************

JStartupIReadSection: read node properties [ID18122300]

-> node name : dispatcher

-> node type : dispatcher

-> node execute : yes

-> jlaunch parameters :

-> java path : /usr/java14_64

-> java parameters : -Xmn70M -Xgcpolicy:gencon -verbose:gc -Xj9 -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk -Djava.security.policy=.\java.policy -Djava.security.egd=file:/dev/urandom -Djco.jarm=1

-> java vm version : J2RE 1.4.2 IBM J9 2.3 AIX ppc64-64 j9ap64142-20080923 (JIT enabled)

-> java vm vendor : IBM J9 VM (IBM Corporation)

-> java vm type : server

-> java vm cpu : ppc64

-> heap size : 200M

-> init heap size : 200M

-> root path : /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher

-> class path : ./bin/boot/boot.jar:./bin/system/bytecode.jar:.

-> OS libs path : /usr/sap/D10/DVEBMGS01/j2ee/os_libs

-> main class : com.sap.engine.boot.Start

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : /usr/sap/D10/DVEBMGS01/exe/jstartup.jar:/usr/sap/D10/DVEBMGS01/exe/jvmx.jar

-> shutdown class : com.sap.engine.boot.Start

-> parameters :

-> debuggable : no

-> debug mode : no

-> debug port : 50000

-> shutdown timeout : 120000

**********************************************************************

[Thr 1] JLaunchISetDebugMode: set debug mode [no]

[Thr 772] JLaunchIStartFunc: Thread 772 started as Java VM thread.

**********************************************************************

JHVM_LoadJavaVM: VM Arguments of node [dispatcher]

-> stack : 1048576 Bytes

-> arg[ 0]: exit

-> arg[ 1]: abort

-> arg[ 2]: vfprintf

-> arg[ 3]: -Xmn70M

-> arg[ 4]: -Xgcpolicy:gencon

-> arg[ 5]: -verbose:gc

-> arg[ 6]: -Xj9

-> arg[ 7]: -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk

-> arg[ 8]: -Djava.security.policy=.\java.policy

-> arg[ 9]: -Djava.security.egd=file:/dev/urandom

-> arg[ 10]: -Djco.jarm=1

-> arg[ 11]: -Dsys.global.dir=/usr/sap/D10/SYS/global

-> arg[ 12]: -Dapplication.home=/usr/sap/D10/DVEBMGS01/exe

-> arg[ 13]: -Djava.class.path=/usr/sap/D10/DVEBMGS01/exe/jstartup.jar:/usr/sap/D10/DVEBMGS01/exe/jvmx.jar:./bin/boot/boot.jar:./bin/system/bytecode.jar:.

-> arg[ 14]: -Djava.library.path=/usr/java14_64/jre/bin:/usr/java14_64/jre/bin:/usr/java14_64/jre/bin/classic:/usr/java14_64/jre/bin:/usr/sap/D10/DVEBMGS01/exe:

/usr/sap/D10/DVEBMGS01/exe:/usr/sap/D10/DVEBMGS01/exe:/usr/TDAccess2.4.1:/usr/sap/D10/SYS/exe/run:/oracle/client/10x_64/instantclient:/

usr/java14_64/jre/bin/j9vm:/usr/sap/D10/DVEBMGS01/exe::/usr/lib:/usr/sap/D10/DVEBMGS01/j2ee/os_libs:/usr/sap/D10/DVEBMGS01/exe:/usr/sap/D10

/DVEBMGS01/exe:/usr/sap/D10/DVEBMGS01/exe:/usr/TDAccess2.4.1:/usr/sap/D10/SYS/exe/run:/oracle/client/10x_64/instantclient

-> arg[ 15]: -Dmemory.manager=200M

-> arg[ 16]: -Xmx200M

-> arg[ 17]: -Xms200M

-> arg[ 18]: -DLoadBalanceRestricted=no

-> arg[ 19]: -Djstartup.mode=JCONTROL

-> arg[ 20]: -Djstartup.ownProcessId=2748598

-> arg[ 21]: -Djstartup.ownHardwareId=U0234126406

-> arg[ 22]: -Djstartup.whoami=dispatcher

-> arg[ 23]: -Djstartup.debuggable=no

-> arg[ 24]: -DSAPINFO=D10_01_dispatcher

-> arg[ 25]: -DSAPSTART=1

-> arg[ 26]: -DCONNECT_PORT=64999

-> arg[ 27]: -DSAPSYSTEM=01

-> arg[ 28]: -DSAPSYSTEMNAME=D10

-> arg[ 29]: -DSAPMYNAME=eccdcs1s_D10_01

-> arg[ 30]: -DSAPPROFILE=/usr/sap/D10/SYS/profile/D10_DVEBMGS01_xxxxxxx

-> arg[ 31]: -DFRFC_FALLBACK=ON

-> arg[ 32]: -DFRFC_FALLBACK_HOST=localhost

-> arg[ 33]: -DSAPSTARTUP=1

-> arg[ 34]: -DSAPSYSTEM=01

-> arg[ 35]: -DSAPSYSTEMNAME=D10

-> arg[ 36]: -DSAPMYNAME=xxxxxxxx_D10_01

-> arg[ 37]: -DSAPDBHOST=xxxxxxx

-> arg[ 38]: -Dj2ee.dbhost=xxxxxx

**********************************************************************

Edited by: Khaiser Khan Mohammed on Jan 18, 2011 11:19 AM

[franz_hiltscher]

Hi,

you should update your JDK to the lastest recommended Version (note 1306760 - JDK SR 13 FP 6 depending on your SP Level)

Check as sidadm with shell command env if you edit the profiles correct (note 1306760)

If possible post std_dispatcher.out

Regards Franz

[Former Member]

Thanks Franz...yeah,the Java upgrade was the last option I thought.Today I will try it and let you know the results.

log of std_dispatcher.out:::

>>>>>>>>>JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.

JVMDUMP007I JVM Requesting System Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/core.20110124.152156.2805860.0001.dmp'

Note: "Enable full CORE dump" in smit is set to FALSE and as a result there will be limited threading information in core file.

JVMDUMP010I System Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/core.20110124.152156.2805860.0001.dmp

JVMDUMP007I JVM Requesting Snap Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/Snap.20110124.152156.2805860.0002.trc'

JVMDUMP010I Snap Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/Snap.20110124.152156.2805860.0002.trc

JVMDUMP007I JVM Requesting Heap Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/heapdump.20110124.152156.2805860.0003.phd'

JVMDUMP010I Heap Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/heapdump.20110124.152156.2805860.0003.phd

JVMDUMP007I JVM Requesting Java Dump using '/usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/javacore.20110124.152156.2805860.0004.txt'

JVMDUMP010I Java Dump written to /usr/sap/D10/DVEBMGS01/j2ee/cluster/dispatcher/javacore.20110124.152156.2805860.0004.txt

JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError

at java.lang.reflect.Array.newArrayImpl(Native Method)

at java.lang.reflect.Array.newInstance(Array.java:283)

at com.ibm.jvm.ExtendedSystem.resizeArray(ExtendedSystem.java:55)

at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:86)

at com.sapmarkets.bam.jmxadapter.util.LogDirectory.parseLogDirectory(LogDirectory.java:975)

at com.sapmarkets.bam.jmxadapter.util.LogDirectory.loadWritableDirectory(LogDirectory.java:186)

at com.sapmarkets.bam.jmxadapter.util.LogDirectory.<init>(LogDirectory.java:130)

at com.sapmarkets.bam.jmxadapter.util.LogDirectoryConfigurator.configure(LogDirectoryConfigurator.java:53)

at com.sapmarkets.bam.j2ee.services.logviewer.MBeanRegistrar.registerLogTypeConfigurators(MBeanRegistrar.java:75)

at com.sapmarkets.bam.j2ee.services.logviewer.MBeanRegistrar.registerLogTypes(MBeanRegistrar.java:179)

at com.sapmarkets.bam.jmx.connector.AbstractLVServer.init(AbstractLVServer.java:60)

at com.sapmarkets.bam.j2ee.services.logviewer.LogViewer.start(LogViewer.java:195)

at com.sap.engine.core.service630.container.ServiceRunner.startCommunicationServiceFrame(ServiceRunner.java:224)

at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:156)

at com.sap.engine.frame.core.thread.Task.run(Task.java:64)

at com.sap.engine.core.thread.impl6.SingleThread.execute(SingleThread.java:78)

at com.sap.engine.core.thread.impl6.SingleThread.run(SingleThread.java:148)>>>>>>>>>>>>>>>>

[Former Member]

Hi,

I ran into this issue before and this is what helped me:

1. Heap size settings in config tool are not maintained properly. The heap size for dispatcher must be either too low or too high. Ensure it is compliant with the instance.properties file. Some main memory parameters are -XMX, -XMS, Heap size..

2. Update JDK to the recommended level.

Also, I hope the page file size on the host is enough.

Rgds,

Soujanya

[Former Member]

Hello Khaiser,

Fixing your AS JAVA Stability issue depend upon the actual reason of the out of memory

To find out that , collect javacore , heap , thread dump while your system is effected.

After collecting the dumps , analyse java core with SAP Thread Viewer & IBM Support Assistant tool.

And with SAP Memory Analyser need to analyse your J9extracted memory dump.

There are also possiblity of any particular application causing the memory issue & coming to the root cause of the instability will help you to fix the issue permanently.

IBM J9 SR13 is stable version that you can use. Also cross check the application component that is using the memory too.

Regards.

Sumanta Chatterjee