Central Instance Java Server0/Server1 not starting

[Former Member]

We recently installed a new dialog instance for our CUA system (abap and java stacks). Now, whenever the Java stack fully initializes itself on the Dialong Instance first, the Server0 and Server1 on the Central Instance Java stack will not complete their startups. Both Server0 and Server1 get stuck on the "Starting Apps" status. Anybody see this before? Any suggestions on how to fix/reconfigure the java so that this does not happen?

Currently, our work around is to always let the Central Instance finish its startup and then start the Dialong Instance. However, if the CI Java crashes and restarts on its own, it will not be able to complete its startup until we shut down the DI instance.

Thanks.

[Former Member]

What directories are you sharing between the systems? This sounds like it might a simple collision of system resources (file locks etc.). Note that the issue should also show up in one of the logs in the work directory.

Hope that helps.

J. Haynes

[Former Member]

J Haynes,

The only directories being shared are on the secure user store. We have somewhat of a unique setup in that we have an SMB space defined over in our USS which resides on a mainframe.

If file locking were the issue, would not the Dialog Instance get locked out if the Central Instance were to finish its startup first? This has never happened. Only the Central Instance Java stack is adversely effected when the DI comes up first. Never vica versa.

Thanks though,

Bruce

[Former Member]

When starting the CI I'd say the database is started, too, and then the DI can connect.

Its not possible to start first the DI and then the CI, as the database would not be

up and running.

Could you pls attach the java log files when the CI crashes and can not startup.

thanks

kr

andreas

[Former Member]

Andreas, Here is a copy of the log after a crash and restart

[Thr 5648] Tue Feb 17 07:59:09 2009

[Thr 5648] JLaunchIExitJava: exit hook is called (rc = 666)

[Thr 5648]

• ERROR => The Java VM terminated with a non-zero exit code.

• Please see SAP Note 943602 , section 'J2EE Engine exit codes'

• for additional information and trouble shooting.

[Thr 5648] JLaunchCloseProgram: good bye (exitcode = 666)

-

---

trc file: "D:\usr\sap\DGD\DVEBMGS40\work\dev_server0", trc level: 1, release: "700"

-

---

node name : ID407398750

pid : 1864

system name : DGD

system nr. : 40

started at : Tue Feb 17 07:59:13 2009

arguments :

arg[00] : D:\usr\sap\DGD\DVEBMGS40\exe\jlaunch.exe

arg[01] : pf=
AXXTSMB\sapmnt\DGD\SYS\profile\DGD_DVEBMGS40_DOWNEAVM249

arg[02] : -DSAPINFO=DGD_40_server

arg[03] : pf=
AXXTSMB\sapmnt\DGD\SYS\profile\DGD_DVEBMGS40_DOWNEAVM249

arg[04] : -DSAPSTART=1

arg[05] : -DCONNECT_PORT=2952

arg[06] : -DSAPSYSTEM=40

arg[07] : -DSAPSYSTEMNAME=DGD

arg[08] : -DSAPMYNAME=DOWNEAVM249_DGD_40

arg[09] : -DSAPPROFILE=
AXXTSMB\sapmnt\DGD\SYS\profile\DGD_DVEBMGS40_DOWNEAVM249

arg[10] : -DFRFC_FALLBACK=ON

arg[11] : -DFRFC_FALLBACK_HOST=localhost

JStartupReadInstanceProperties: read instance properties [D:\usr\sap\DGD\DVEBMGS40\j2ee\cluster\instance.properties]

(parms removed to reduce size of file)

Used property files

-> files [00] : D:\usr\sap\DGD\DVEBMGS40\j2ee\cluster\instance.properties

Instance properties

(parms removed to reduce size of file)

Bootstrap nodes

(parms removed to reduce size of file)

Worker nodes

(parms removed to reduce size of file)

[Thr 1216] JLaunchRequestQueueInit: create named pipe for ipc

[Thr 1216] JLaunchRequestQueueInit: create pipe listener thread

[Thr 5552] WaitSyncSemThread: Thread 5552 started as semaphore monitor thread.

[Thr 5096] JLaunchRequestFunc: Thread 5096 started as listener thread for np messages.

[Thr 1216] NiInit3: NI already initialized; param 'maxHandles' ignored (1;202)

[Thr 1216] CPIC (version=700.2006.09.13)

[Thr 1216] [Node: server0] java home is set by profile parameter

Java Home: C:/Program Files/j2sdk1.4.2_13-x64

[Thr 1216] Tue Feb 17 07:59:20 2009

[Thr 1216] JStartupICheckFrameworkPackage: can't find framework package D:\usr\sap\DGD\DVEBMGS40\exe\jvmx.jar

JStartupIReadSection: read node properties [ID407398750]

(parms removed to reduce size of file)

[Thr 1216] JLaunchISetDebugMode: set debug mode [no]

[Thr 1452] JLaunchIStartFunc: Thread 1452 started as Java VM thread.

[Thr 1452] [JHVM_PrepareVMOptions] use java parameters set by profile parameter

Java Parameters: -Xss2m

JHVM_LoadJavaVM: VM Arguments of node [server0]

(parms removed to reduce size of file)

[Thr 1452] JHVM_LoadJavaVM: Java VM created OK.

JHVM_BuildArgumentList: main method arguments of node [server0]

[Thr 1788] Tue Feb 17 07:59:25 2009

[Thr 1788] JHVM_RegisterNatives: registering methods in com.sap.bc.krn.perf.PerfTimes

[Thr 1788] Tue Feb 17 07:59:27 2009

[Thr 1788] JHVM_RegisterNatives: registering methods in com.sap.bc.proj.jstartup.JStartupFramework

[Thr 1788] JLaunchISetClusterId: set cluster id 407398750

[Thr 1788] Tue Feb 17 07:59:40 2009

[Thr 1788] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]

[Thr 1788] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)]

[Thr 3680] Tue Feb 17 08:02:46 2009

[Thr 3680] JHVM_RegisterNatives: registering methods in com.sap.mw.rfc.driver.CpicDriver

[Thr 3680] Tue Feb 17 08:02:47 2009

[Thr 3680] JHVM_RegisterNatives: registering methods in com.sap.i18n.cp.ConverterJNI

[Thr 3680] JHVM_RegisterNatives: registering methods in com.sap.mw.rfc.engine.Compress

[Thr 2180] Tue Feb 17 08:04:57 2009

[Thr 2180] JHVM_RegisterNatives: registering methods in com.sap.security.core.server.vsi.service.jni.VirusScanInterface

Tue Feb 17 08:05:43 2009

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.4.2_13-b06 mixed mode):

"Thread-32" daemon prio=10 tid=0x000000001d373230 nid=0x1034 in Object.wait() [0x000000001c88f000..0x000000001c88fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff0118> (a java.util.TaskQueue)

at java.util.TimerThread.mainLoop(Timer.java:429)

- locked <0x000000008eff0118> (a java.util.TaskQueue)

at java.util.TimerThread.run(Timer.java:382)

"SAPEngine_CCMSCommandConsumer_0" prio=10 tid=0x000000001d3734c0 nid=0x204 in Object.wait() [0x000000001fc8f000..0x000000001fc8fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff2070> (a java.lang.Object)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.lib.util.concurrent.LinkedQueue.take(LinkedQueue.java:104)

- locked <0x000000008eff2070> (a java.lang.Object)

at com.sapmarkets.bam.jmxadapter.CCMSCommandConsumer.run(CCMSCommandConsumer.java:38)

at java.lang.Thread.run(Thread.java:534)

"Thread-29" prio=10 tid=0x000000001d372fa0 nid=0xa64 in Object.wait() [0x000000001fa8f000..0x000000001fa8fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff01f0> (a com.sap.jmx.provider.ProviderConnectionTimer)

at com.sap.jmx.provider.ProviderConnectionTimer.run(ProviderConnectionTimer.java:31)

- locked <0x000000008eff01f0> (a com.sap.jmx.provider.ProviderConnectionTimer)

at java.lang.Thread.run(Thread.java:534)

(extraneous waiting/lock msgs deleted)

"VM Thread" prio=5 tid=0x0000000001386790 nid=0xe70 runnable

"VM Periodic Task Thread" prio=10 tid=0x0000000001386990 nid=0xbe0 waiting on condition

"Suspend Checker Thread" prio=10 tid=0x0000000001386890 nid=0xb90 runnable

[Thr 1788] JLaunchISetState: change state from [Starting (2)] to [Starting applications (10)]

[Thr 200] Tue Feb 17 08:08:57 2009

[Thr 200] JHVM_RegisterNatives: registering methods in com.sap.bc.proj.jstartup.sadm.ShmCache

[Former Member]

Hi Bruce,

thanks for the logfile. It tells us however not the real cause for the problem. I guess

it is a memory problem, but to make sure I would need to see the file

std_server0.out

Please attach to this thread.

Thanks and kind regards

andreas

[Former Member]

andreas,

Here is a partial std_server0.out file. It seems that the dumping/memory/locking problem starts immediately after the Framework has started:

ServiceManager started for 354203 ms.

Framework started for 422687 ms.

SAP J2EE Engine Version 7.00 PatchLevel 109886.44 is running!

PatchLevel 109886.44 November 17, 2007 10:27 GMT

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.4.2_13-b06 mixed mode):

"DSR_Write_Rate_Monitor: DOWNEAVM249.dow_DGD_407398750" prio=5 tid=0x000000001a6b9860 nid=0x4864 runnable [0x0000000000000000..0x0000000000000000]

"Thread-33" prio=5 tid=0x000000001a6b95d0 nid=0x4230 runnable [0x0000000000000000..0x0000000000000000]

"Thread-32" daemon prio=10 tid=0x000000001a6b90b0 nid=0x5044 in Object.wait() [0x000000001de1f000..0x000000001de1fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff2080> (a java.util.TaskQueue)

at java.util.TimerThread.mainLoop(Timer.java:429)

- locked <0x000000008eff2080> (a java.util.TaskQueue)

at java.util.TimerThread.run(Timer.java:382)

"SAPEngine_CCMSCommandConsumer_0" prio=10 tid=0x000000001ac73d80 nid=0x1b1c in Object.wait() [0x000000001dc1f000..0x000000001dc1fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff0108> (a java.lang.Object)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.lib.util.concurrent.LinkedQueue.take(LinkedQueue.java:104)

- locked <0x000000008eff0108> (a java.lang.Object)

at com.sapmarkets.bam.jmxadapter.CCMSCommandConsumer.run(CCMSCommandConsumer.java:38)

at java.lang.Thread.run(Thread.java:534)

"Thread-29" prio=10 tid=0x000000001ac73860 nid=0x2a64 in Object.wait() [0x000000001da1f000..0x000000001da1fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff01d0> (a com.sap.jmx.provider.ProviderConnectionTimer)

at com.sap.jmx.provider.ProviderConnectionTimer.run(ProviderConnectionTimer.java:31)

- locked <0x000000008eff01d0> (a com.sap.jmx.provider.ProviderConnectionTimer)

at java.lang.Thread.run(Thread.java:534)

"Thread-28" prio=10 tid=0x0000000008d2dad0 nid=0x4bf4 in Object.wait() [0x000000001d81f000..0x000000001d81fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008eff0268> (a com.sap.jmx.provider.LazyMBeanLoader)

at java.lang.Object.wait(Object.java:429)

at com.sap.jmx.provider.LazyMBeanLoader.run(LazyMBeanLoader.java:133)

- locked <0x000000008eff0268> (a com.sap.jmx.provider.LazyMBeanLoader)

at java.lang.Thread.run(Thread.java:534)

"SAPEngine_Application_Thread[impl:3]_1" prio=5 tid=0x00000000012e4920 nid=0x38b0 in Object.wait() [0x0000000007daf000..0x0000000007dafb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008fb8f078> (a com.sap.engine.lib.util.WaitQueue)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.lib.util.WaitQueue.dequeue(WaitQueue.java:238)

- locked <0x000000008fb8f078> (a com.sap.engine.lib.util.WaitQueue)

at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:149)

"SAPEngine_Application_Thread[impl:3]_0" prio=5 tid=0x00000000012e4690 nid=0xc20 in Object.wait() [0x00000000079af000..0x00000000079afb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008fb8f078> (a com.sap.engine.lib.util.WaitQueue)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.lib.util.WaitQueue.dequeue(WaitQueue.java:238)

- locked <0x000000008fb8f078> (a com.sap.engine.lib.util.WaitQueue)

at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:149)

"Thread-1" prio=5 tid=0x00000000012e4170 nid=0x3a68 in Object.wait() [0x0000000007baf000..0x0000000007bafb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008fbbf530> (a com.sap.engine.core.Framework)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.core.Framework.start(Framework.java:207)

- locked <0x000000008fbbf530> (a com.sap.engine.core.Framework)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:324)

at com.sap.engine.boot.FrameThread.run(FrameThread.java:61)

at java.lang.Thread.run(Thread.java:534)

"Signal Dispatcher" daemon prio=10 tid=0x00000000012e34a0 nid=0x2334 waiting on condition [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=9 tid=0x00000000012e3210 nid=0x23c8 in Object.wait() [0x0000000006baf000..0x0000000006bafb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x00000000909192e0> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)

- locked <0x00000000909192e0> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00000000012e2f80 nid=0x1438 in Object.wait() [0x00000000069af000..0x00000000069afb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008f9a61d0> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:429)

at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)

- locked <0x000000008f9a61d0> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x00000000012e2cf0 nid=0x3cb8 in Object.wait() [0x000000000222f000..0x000000000222fc68]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000008fbbf3e0> (a com.sap.engine.boot.StartFrame)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.boot.StartFrame.work(StartFrame.java:121)

- locked <0x000000008fbbf3e0> (a com.sap.engine.boot.StartFrame)

at com.sap.engine.boot.Start.main(Start.java:34)

"VM Thread" prio=5 tid=0x000000000137a4a0 nid=0x3cac runnable

"VM Periodic Task Thread" prio=10 tid=0x000000000137a6a0 nid=0xccc waiting on condition

"Suspend Checker Thread" prio=10 tid=0x000000000137a5a0 nid=0x67c runnable

Service tc.monitoring.logviewer started. (89484 ms).

>

Login :### Excluding compile: com.sap.engine.services.webservices.jaxrpc.encoding.TypeMappingImpl::initializeRelations

1. Excluding compile: com.sap.engine.services.webservices.jaxrpc.encoding.GeneratedComplexType::_loadInto

Thanks, Bruce

[Former Member]

After working with SAP, we determined that the GRC system (java based) which was also installed on the same server as the CUA system does not support an environment that includes dialog servers.