SDM could not start the J2EE cluster on the host!

[Former Member]

Hello,

I have a problem in Solution Manager Support Package installation.

I'm installing SP10 with SUM. In the java stack, the step "DEPLOY JAVA CORE COMPONENTS" takes a long time.

In the log, I have this errors:

Feb 6, 2014 7:36:50 AM   Info: Starting: Update: Selected development component 'tc/SL/CMS/Util'/'sap.com'/'SAP AG'/'7.0213.20120926150722.0000'/'7' updates currently deployed development component 'tc/SL/CMS/Util'/'sap.com'/'SAP AG'/'7.0212.20120427162320.0000'/'0'. Feb 6, 2014 7:36:50 AM   Info: SDA to be deployed: /usr/sap/SSM/DVEBMGS00/SDM/root/origin/sap.com/tc/SL/CMS/Util/SAP AG/7/7.0213.20120926150722.0000/CMSutil.sda Feb 6, 2014 7:36:50 AM   Info: Software type of SDA: library Feb 6, 2014 7:36:50 AM   Info: ***** Begin of SAP J2EE Engine Deployment (Library component of SAP J2EE Engine) ***** Feb 6, 2014 7:36:50 AM   Info: Starting cluster instance processes. Feb 6, 2014 7:36:50 AM   Info: Starting the instance JC_srvssmprd_SSM_00 running on host srvssmprd Feb 6, 2014 7:36:56 AM   Info: Starting the instance JC_srvssmprd_SSM_00 processes. The instance is running on host srvssmprd                                           Feb 6, 2014 7:36:56 AM   Info: Starting the process dispatcher Feb 6, 2014 8:26:50 AM   Info: ***** End of SAP J2EE Engine Deployment (Library component of SAP J2EE Engine) ***** Feb 6, 2014 8:26:50 AM   Error: Aborted: development component 'tc/SL/CMS/Util'/'sap.com'/'SAP AG'/'7.0213.20120926150722.0000'/'7', grouped by : SDM could not start the J2EE cluster on the host srvssmprd! The online deployment is terminated. A timeout occured during the cluster running verification. (message ID: com.sap.sdm.serverext.servertype.inqmy.extern.EngineCompOnlineDeployerImpl.performAction(DeploymentActionTypes).STARTUP_CLUSTER) Feb 6, 2014 8:26:50 AM   Info: Starting to save the repository Feb 6, 2014 8:26:50 AM   Info: Finished saving the repository for 138 ms. Feb 6, 2014 8:26:50 AM   Info: Skipping: Update: Selected development component 'tc/SL/CMS/IDEClient'/'sap.com'/'SAP AG'/'7.0213.20120926150722.0000'/'7' updates currently deployed development component 'tc/SL/CMS/IDEClient'/'sap.com'/'SAP AG'/'7.0212.20120427162320.0000'/'0'. Feb 6, 2014 8:26:50 AM   Info: Starting to save the repository Feb 6, 2014 8:26:51 AM   Info: Finished saving the repository for 157 ms.

Feb 6, 2014 7:36:50 AM   Info: Starting: Update: Selected development component 'tc/SL/CMS/Util'/'sap.com'/'SAP AG'/'7.0213.20120926150722.0000'/'7' updates currently deployed development component 'tc/SL/CMS/Util'/'sap.com'/'SAP AG'/'7.0212.20120427162320.0000'/'0'. Feb 6, 2014 7:36:50 AM   Info: SDA to be deployed: /usr/sap/SSM/DVEBMGS00/SDM/root/origin/sap.com/tc/SL/CMS/Util/SAP AG/7/7.0213.20120926150722.0000/CMSutil.sda Feb 6, 2014 7:36:50 AM   Info: Software type of SDA: library Feb 6, 2014 7:36:50 AM   Info: ***** Begin of SAP J2EE Engine Deployment (Library component of SAP J2EE Engine) ***** Feb 6, 2014 7:36:50 AM   Info: Starting cluster instance processes. Feb 6, 2014 7:36:50 AM   Info: Starting the instance JC_srvssmprd_SSM_00 running on host srvssmprd Feb 6, 2014 7:36:56 AM   Info: Starting the instance JC_srvssmprd_SSM_00 processes. The instance is running on host srvssmprd                                           Feb 6, 2014 7:36:56 AM   Info: Starting the process dispatcher Feb 6, 2014 8:26:50 AM   Info: ***** End of SAP J2EE Engine Deployment (Library component of SAP J2EE Engine) ***** Feb 6, 2014 8:26:50 AM   Error: Aborted: development component 'tc/SL/CMS/Util'/'sap.com'/'SAP AG'/'7.0213.20120926150722.0000'/'7', grouped by : SDM could not start the J2EE cluster on the host srvssmprd! The online deployment is terminated. A timeout occured during the cluster running verification. (message ID: com.sap.sdm.serverext.servertype.inqmy.extern.EngineCompOnlineDeployerImpl.performAction(DeploymentActionTypes).STARTUP_CLUSTER) Feb 6, 2014 8:26:50 AM   Info: Starting to save the repository Feb 6, 2014 8:26:50 AM   Info: Finished saving the repository for 138 ms. Feb 6, 2014 8:26:50 AM   Info: Skipping: Update: Selected development component 'tc/SL/CMS/IDEClient'/'sap.com'/'SAP AG'/'7.0213.20120926150722.0000'/'7' updates currently deployed development component 'tc/SL/CMS/IDEClient'/'sap.com'/'SAP AG'/'7.0212.20120427162320.0000'/'0'. Feb 6, 2014 8:26:50 AM   Info: Starting to save the repository Feb 6, 2014 8:26:51 AM   Info: Finished saving the repository for 157 ms.

The error is repeted with each development component

Can anyone help me?

Thank you

Xavier Garcia

[Former Member]

Here is the std_server0.out log.

std_server0.out.zip

[Former Member]

Please upload latest defaultTrace.X.log from dispatcher/log directory.

[Former Member]

Hello Roman,

Here your are.

Thanks.

Xavier Garcia

defaultTrace.0.trc.zip

[Former Member]

Hello all,

I have new logs from restarted process.

I'll attach the std_server0.out in a new post.

Thanks,

Xavier Garcia

dev_dispatcher.zip

dev_server0.zip

std_dispatcher.out.zip

[former_member45419]

Hi Dear,

Pls refer above links,

Regards

Nayeem

[Former Member]

Hello Khaja,

I'm using SAP JVM.

Is not the same problema.

Thank you.

Xavier Garcia

[Former Member]

Hello,

I can attach std_dispatcher.out because is small enough. But I can't attach the others because are too large.

I have created a file with the content of dev_dispatcher and std_server0.out with error content:

I haven't found any relevant in dev_server0.

Thanks.

Xavier Garcia

std_dispatcher.out.zip

bigfilesmerged.txt.zip

[Former Member]

thanks for the logs. as per the logs dev_dispatcher see below:

[Thr 28] JHVM_RegisterNatives: com.sap.bc.proj.jstartup.JStartupFramework
[Thr 28] JLaunchISetClusterId: set cluster id 5230500
[Thr 28] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]
[Thr 28] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)]

[Thr 15] Wed Feb  5 18:17:00 2014
[Thr 15] JLaunchIExitJava: exit hook is called (rc = 0)
[Thr 15] SigISetIgnoreAction : SIG_IGN for signal SIGCHLD
[Thr 15] JLaunchCloseProgram: good bye (exitcode = 0)

Then the logs should say as below, but not:

[Thr 5752] JLaunchISetState: change state from [Starting (2)] to [Running (3)]

also see below:

Timed out services:

Service com.adobe~XMLFormService > service com.adobe~XMLFormService start method invoked.

some how you must attach the complete logs to analyze this issue.

[Former Member]

The strange thing is that dev_jcontrol and dev_dispatcher files contains entries with timestamps of Feb 5 18:17 2014. At the same time you uploaded std_server0 file where thread dump was generated at Feb 6 10:02 2014. Can you try to restart J2EE cluster and attach last log files as asked before?

[Former Member]

Hello Roman,

Yes, I had already realized of this one.

I think that this error doesn't have relation with this.

I have already stop the process and restarted it for have new and clean error logs.

Thanks.

Xavier Garcia

[Former Member]

Hello Roman,

I view and error in dev_jcontrol:

arg[00] = /usr/sap/SSM/DVEBMGS00/exe/jlaunch -> arg[01] = pf=/usr/sap/SSM/SYS/profile/SSM_DVEBMGS00_srvssmprd -> arg[02] = -DSAPINFO=SSM_00_bootstrap -> arg[03] = -nodeId=-1 -> arg[04] = -file=/usr/sap/SSM/DVEBMGS00/j2ee/cluster/instance.properties -> arg[05] = -syncSem=83886163 -> arg[06] = -nodeName=bootstrap_ID5230500 -> arg[07] = -jvmOutFile=/usr/sap/SSM/DVEBMGS00/work/jvm_bootstrap_ID5230500.out -> arg[08] = -stdOutFile=/usr/sap/SSM/DVEBMGS00/work/std_bootstrap_ID5230500.out -> arg[09] = -locOutFile=/usr/sap/SSM/DVEBMGS00/work/dev_bootstrap_ID5230500 -> arg[10] = -mode=BOOTSTRAP -> arg[11] = pf=/usr/sap/SSM/SYS/profile/SSM_DVEBMGS00_srvssmprd -> arg[12] = -DSAPSTART=1 -> lib path = LD_LIBRARY_PATH=/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/jre/lib/sparcv9/server:/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/jre/lib/sparcv9:/usr/sap/SSM/DVEBMGS00/exe/ sapjvm_4/jre/../lib/sparcv9:/usr/sap/SSM/DVEBMGS00/exe:/usr/sap/SSM/SYS/exe/run:/oracle/client/11x_64/instantclient:/usr/jdk/packages/lib/sparcv9:/lib/64:/usr/lib/64:/u sr/sap/SSM/DVEBMGS00/j2ee/os_libs:/usr/sap/SSM/DVEBMGS00/j2ee/os_libs:/usr/sap/SSM/DVEBMGS00/exe:/usr/sap/SSM/SYS/exe/run:/oracle/client/11x_64/instantclient -> exe path = PATH=/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/bin:/usr/sap/SSM/DVEBMGS00/j2ee/os_libs:/oracle/SSM/112_64/bin:/usr/sap/SSM/SYS/exe/jvm/sun_64/sapjvm_4.1.026/sap jvm_4/bin:.:/export/home/ssmadm:/usr/sap/SSM/SYS/exe/run:/bin:/usr/bin:/sbin:/usr/sbin:/etc:/usr/ccs/bin:/usr/ucb ********************************************************************** [Thr 01] JStartupICreateProcess: fork process (pid 29630) ********************************************************************** JControlStartJLaunch: program = /usr/sap/SSM/DVEBMGS00/exe/jlaunch -> arg[00] = /usr/sap/SSM/DVEBMGS00/exe/jlaunch -> arg[01] = pf=/usr/sap/SSM/SYS/profile/SSM_DVEBMGS00_srvssmprd -> arg[02] = -DSAPINFO=SSM_00_dispatcher -> arg[03] = -nodeId=0 -> arg[04] = -file=/usr/sap/SSM/DVEBMGS00/j2ee/cluster/instance.properties -> arg[05] = -syncSem=83886163 -> arg[06] = -nodeName=ID5230500 -> arg[07] = -jvmOutFile=/usr/sap/SSM/DVEBMGS00/work/jvm_dispatcher.out -> arg[08] = -jvmOutMode=append -> arg[09] = -stdOutFile=/usr/sap/SSM/DVEBMGS00/work/std_dispatcher.out -> arg[10] = -stdOutMode=append -> arg[11] = -traceMode=append -> arg[12] = -locOutFile=/usr/sap/SSM/DVEBMGS00/work/dev_dispatcher -> arg[13] = -mode=JCONTROL -> arg[14] = pf=/usr/sap/SSM/SYS/profile/SSM_DVEBMGS00_srvssmprd -> arg[15] = -DSAPSTART=1 -> lib path = LD_LIBRARY_PATH=/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/jre/lib/sparcv9/server:/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/jre/lib/sparcv9:/usr/sap/SSM/DVEBMGS00/exe/ sapjvm_4/jre/../lib/sparcv9:/usr/sap/SSM/DVEBMGS00/exe:/usr/sap/SSM/SYS/exe/run:/oracle/client/11x_64/instantclient:/usr/jdk/packages/lib/sparcv9:/lib/64:/usr/lib/64:/u sr/sap/SSM/DVEBMGS00/j2ee/os_libs:/usr/sap/SSM/DVEBMGS00/j2ee/os_libs:/usr/sap/SSM/DVEBMGS00/exe:/usr/sap/SSM/SYS/exe/run:/oracle/client/11x_64/instantclient -> exe path = PATH=/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/bin:/usr/sap/SSM/DVEBMGS00/j2ee/os_libs:/oracle/SSM/112_64/bin:/usr/sap/SSM/SYS/exe/jvm/sun_64/sapjvm_4.1.026/sap jvm_4/bin:.:/export/home/ssmadm:/usr/sap/SSM/SYS/exe/run:/bin:/usr/bin:/sbin:/usr/sbin:/etc:/usr/ccs/bin:/usr/ucb ********************************************************************** [Thr 01] Wed Feb  5 18:16:51 2014 [Thr 01] JStartupICreateProcess: fork process (pid 383) [Thr 01] JControlICheckProcessList: process dispatcher started (PID:383) [Thr 01] Wed Feb  5 18:17:01 2014 [Thr 01] JControlICheckProcessList: process dispatcher (pid:383) died (RUN-FLAG) [Thr 01] JControlIResetProcess: reset process dispatcher [Thr 01] JControlIResetProcess: [dispatcher] not running -> increase error count (7) [Thr 01]> exe path = PATH=/usr/sap/SSM/DVEBMGS00/exe/sapjvm_4/bin:/usr/sap/SSM/DVEBMGS00/j2ee/os_libs:/oracle/SSM/112_64/bin:/usr/sap/SSM/SYS/exe/jvm/sun_64/sapjvm_4.1.026/sap jvm_4/bin:.:/export/home/ssmadm:/usr/sap/SSM/SYS/exe/run:/bin:/usr/bin:/sbin:/usr/sbin:/etc:/usr/ccs/bin:/usr/ucb **********************************************************************

Thanks,

Xavier Garcia

[Former Member]

Please check dev(std)_dispatcher and dev(std)_serverX log files for errors or upload it here (if possible attach them as files).

[Former Member]

>>Wed Feb  5 18:17:01 2014 [Thr 01] JControlICheckProcessList: process dispatcher (pid:383) died

Please attach std_serverX.out, std_dispatcher.out from work directory.

[Former Member]

Please check log files in work directory (dev_*, std_*) to find out why J2EE cluster could not be started. If you have troubles upload it here.