JAVA stack wont come up after complete restart.....

[Former Member]

Dear Friends,

We have a very peculiar case here with our PI system.

Whenever we go complete restart(ABAP+JAVA).....our JAVA stack fails everytime. No matter what, it wont come.

All the server nodes fail immediately during 'starting framework' phase itself.

Interesting thing is, as a workaround, we go for server reboot everytime and then JAVA comes up without any problem.

Please note that this is not the case when we go for only JAVA restart i.e. there is no problem while restarting JAVA stack alone.

Problem only occurs whenever we go for complete restart.

I have went through possible each and every thread on this in SCN, but no help.

We are in dire need of any help.

Please find below the default trace of server node.

Could not register com.sap.engine.session.mbeans.DatashareMBean. Exception is: java.lang.InstantiationException: com.sap.engine.session.mbeans.MonitorSessionsOpenMBe

an#

#2.#2014 04 11 20:01:18:832#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

com.sap.ASJ.krn_srv.000002#BC-JAS-COR#kernel.sda#C0003547E44E00020000000100000AF9#602208850000000023##com.sap.engine.core.service630.container.DescriptorHelperImpl##

###436fd3bac1a311e38fb6005056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [tc~je~cfs] component [descriptor] container#

#2.#2014 04 11 20:01:18:832#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

#BC-JAS-COR#kernel.sda#C0003547E44E00020000000200000AF9#602208850000000023##com.sap.engine.core.service630.container.DescriptorHelperImpl#####436fd3bac1a311e38fb6005

056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [tc~je~cfs] component [descriptor] container

[EXCEPTION]

com.sap.engine.lib.config.api.exceptions.ClusterConfigurationException: The entry with the specified name containers-info.xml cannot be retrieved in tc~je~cfs/descri

ptors.

      at com.sap.engine.lib.config.impl.component.PersistentEntryContainerImpl.getPersistentEntryAsStream(PersistentEntryContainerImpl.java:62)

      at com.sap.engine.core.service630.container.DescriptorHelperImpl.getPersistentEntryStream(DescriptorHelperImpl.java:57)

      at com.sap.engine.services.deploy.server.cache.containers.Containers.addContainers(Containers.java:90)

      at com.sap.engine.services.deploy.server.cache.containers.Containers.initContainers(Containers.java:197)

      at com.sap.engine.services.deploy.server.DeployServiceFrame.init(DeployServiceFrame.java:175)

      at com.sap.engine.services.deploy.server.DeployServiceFrame.start(DeployServiceFrame.java:56)

      at com.sap.engine.core.service630.container.ServiceRunner.startFrame(ServiceRunner.java:165)

      at com.sap.engine.core.service630.container.ServiceRunner.startService(ServiceRunner.java:118)

      at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:65)

      at com.sap.engine.frame.core.thread.Task.run(Task.java:73)

      at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:178)

      at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:316)

Caused by: com.sap.engine.frame.core.configuration.ConfigurationException: Permission denied

      at com.sap.engine.core.configuration.impl.ConfigurationImpl.getFileFromCopy(ConfigurationImpl.java:346)

      at com.sap.engine.lib.config.impl.ConfigurationAccessor.getEntryStream(ConfigurationAccessor.java:2267)

      at com.sap.engine.lib.config.impl.component.PersistentEntryContainerImpl.getPersistentEntryAsStream(PersistentEntryContainerImpl.java:58)

      ... 11 more

Caused by: java.io.IOException: Permission denied

      at java.io.UnixFileSystem.createFileExclusively(Native Method)

      at java.io.File.checkAndCreate(File.java:1715)

      at java.io.File.createTempFile0(File.java:1736)

      at java.io.File.createTempFile(File.java:1813)

      at com.sap.engine.core.configuration.impl.ConfigurationImpl.getFileFromCopy(ConfigurationImpl.java:292)

      ... 13 more

#

#2.#2014 04 11 20:01:18:839#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

com.sap.ASJ.krn_srv.000002#BC-JAS-COR#kernel.sda#C0003547E44E00020000000400000AF9#602208850000000023##com.sap.engine.core.service630.container.DescriptorHelperImpl##

###436fd3bac1a311e38fb6005056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [tc~je~sca~integration] component [descriptor] container#

#2.#2014 04 11 20:01:18:839#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

#BC-JAS-COR#kernel.sda#C0003547E44E00020000000500000AF9#602208850000000023##com.sap.engine.core.service630.container.DescriptorHelperImpl#####436fd3bac1a311e38fb6005

056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [tc~je~sca~integration] component [descriptor] container

[EXCEPTION]

com.sap.engine.lib.config.api.exceptions.ClusterConfigurationException: The entry with the specified name containers-info.xml cannot be retrieved in tc~je~sca~integr

ation/descriptors.

      at com.sap.engine.lib.config.impl.component.PersistentEntryContainerImpl.getPersistentEntryAsStream(PersistentEntryContainerImpl.java:62)

      at com.sap.engine.core.service630.container.DescriptorHelperImpl.getPersistentEntryStream(DescriptorHelperImpl.java:57)

      at com.sap.engine.services.deploy.server.cache.containers.Containers.addContainers(Containers.java:90)

      at com.sap.engine.services.deploy.server.cache.containers.Containers.initContainers(Containers.java:197)

      at com.sap.engine.services.deploy.server.DeployServiceFrame.init(DeployServiceFrame.java:175)

      at com.sap.engine.services.deploy.server.DeployServiceFrame.start(DeployServiceFrame.java:56)

      at com.sap.engine.core.service630.container.ServiceRunner.startFrame(ServiceRunner.java:165)

      at com.sap.engine.core.service630.container.ServiceRunner.startService(ServiceRunner.java:118)

      at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:65)

      at com.sap.engine.frame.core.thread.Task.run(Task.java:73)

      at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:178)

      at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:316)

Caused by: com.sap.engine.frame.core.configuration.ConfigurationException: Permission denied

      at com.sap.engine.core.configuration.impl.ConfigurationImpl.getFileFromCopy(ConfigurationImpl.java:346)

      at com.sap.engine.lib.config.impl.ConfigurationAccessor.getEntryStream(ConfigurationAccessor.java:2267)

      at com.sap.engine.lib.config.impl.component.PersistentEntryContainerImpl.getPersistentEntryAsStream(PersistentEntryContainerImpl.java:58)

      ... 11 more

Caused by: java.io.IOException: Permission denied

      at java.io.UnixFileSystem.createFileExclusively(Native Method)

      at java.io.File.checkAndCreate(File.java:1715)

      at java.io.File.createTempFile0(File.java:1736)

      at java.io.File.createTempFile(File.java:1813)

      at com.sap.engine.core.configuration.impl.ConfigurationImpl.getFileFromCopy(ConfigurationImpl.java:292)

      ... 13 more

#

#2.#2014 04 11 20:01:18:845#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

com.sap.ASJ.krn_srv.000002#BC-JAS-COR#kernel.sda#C0003547E44E00020000000700000AF9#602208850000000023##com.sap.engine.core.service630.container.DescriptorHelperImpl##

###436fd3bac1a311e38fb6005056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [jmsconnector] component [descriptor] container#

#2.#2014 04 11 20:01:18:845#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

#BC-JAS-COR#kernel.sda#C0003547E44E00020000000800000AF9#602208850000000023##com.sap.engine.core.service630.container.DescriptorHelperImpl#####436fd3bac1a311e38fb6005

056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [jmsconnector] component [descriptor] container

[EXCEPTION]

com.sap.engine.lib.config.api.exceptions.ClusterConfigurationException: The entry with the specified name containers-info.xml cannot be retrieved in jmsconnector/des

criptors.

And finally, this message pops in trace

#2.#2014 04 11 20:01:24:187#+0200#Fatal#com.sap.engine.core.Framework#

com.sap.ASJ.krn_frm.000019#BC-JAS-COR#kernel.sda#C0003547E44E00020000005900000AF9#602208850000000830##com.sap.engine.core.Framework#####436fd3bac1a311e38fb6005056ba5

0f8##0#Service Runner [dbpool]#Plain##

Critical shutdown with exit code [2150] was requested due to [AS Java cannot be started; core service [dbpool] failed to start]#

#2.#2014 04 11 20:01:34:262#+0200#Warning#com.sap.engine.core.thread.impl3.ErrorQueueHandler#

com.sap.ASJ.krn_thd.000025#BC-JAS-COR#kernel.sda#C0003547E44E00000000000000001064###com.sap.engine.core.thread.impl3.ErrorQueueHandler#####51832c54c1a311e38428005056

ba50f8###Thread[ErrorQueueWatchDog,5,main]#Plain##

The following Error is thrown by the VM [java.lang.InternalError]. ErrorQueueHandler will just trace it. The caller component should take care to process it properly

.

java.lang.InternalError

#

#2.#2014 04 11 20:01:37:170#+0200#Warning#com.sap.engine.core.session.Manager#

com.sap.ASJ.ses.cf0023#BC-JAS-COR-SES#kernel.sda#C0003547E44E00010000000000001064###com.sap.engine.core.session.Manager########Thread[main,5,main]#Plain##

Could not register com.sap.engine.session.mbeans.SessDomRecMBean. Exception is: java.lang.InstantiationException: com.sap.engine.session.mbeans.MonitorSessionsOpenMB

ean#

#2.#2014 04 11 20:01:41:477#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

com.sap.ASJ.krn_srv.000002#BC-JAS-COR#kernel.sda#C0003547E44E00020000000100001064#602208850000000021##com.sap.engine.core.service630.container.DescriptorHelperImpl##

###544c3391c1a311e3a22b005056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [component.info.service] component [descriptor] container#

#2.#2014 04 11 20:01:41:477#+0200#Error#com.sap.engine.core.service630.container.DescriptorHelperImpl#

#BC-JAS-COR#kernel.sda#C0003547E44E00020000000200001064#602208850000000021##com.sap.engine.core.service630.container.DescriptorHelperImpl#####544c3391c1a311e3a22b005

056ba50f8##0#Service Runner [deploy]#Plain##

Error reading [containers-info.xml] file entry from [component.info.service] component [descriptor] container

[EXCEPTION]

com.sap.engine.lib.config.api.exceptions.ClusterConfigurationException: The entry with the specified name containers-info.xml cannot be retrieved in component.info.s

ervice/descriptors.

      at com.sap.engine.lib.config.impl.component.PersistentEntryContainerImpl.getPersistentEntryAsStream(PersistentEntryContainerImpl.java:62)

      at com.sap.engine.core.service630.container.DescriptorHelperImpl.getPersistentEntryStream(DescriptorHelperImpl.java:57)

      at com.sap.engine.services.deploy.server.cache.containers.Containers.addContainers(Containers.java:90)

      at com.sap.engine.services.deploy.server.cache.containers.Containers.initContainers(Containers.java:197)

      at com.sap.engine.services.deploy.server.DeployServiceFrame.init(DeployServiceFrame.java:175)

      at com.sap.engine.services.deploy.server.DeployServiceFrame.start(DeployServiceFrame.java:56)

      at com.sap.engine.core.service630.container.ServiceRunner.startFrame(ServiceRunner.java:165)

      at com.sap.engine.core.service630.container.ServiceRunner.startService(ServiceRunner.java:118)

      at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:65)

      at com.sap.engine.frame.core.thread.Task.run(Task.java:73)

      at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:178)

      at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:316)

Caused by: com.sap.engine.frame.core.configuration.ConfigurationException: Permission denied

      at com.sap.engine.core.configuration.impl.ConfigurationImpl.getFileFromCopy(ConfigurationImpl.java:346)

      at com.sap.engine.lib.config.impl.ConfigurationAccessor.getEntryStream(ConfigurationAccessor.java:2267)

Kindly help.

Thanks

Message was edited by: Moderator

[jensfieger · Accepted solution]

Vikas,

which release and SP are you running on? I'd also be interested which PL of J2EE ENGINE SERVERCORE you're using.

Rgds

Jens

[Former Member · Accepted solution]

Hello Jens,

We are on Rel 731 and SP 9.

Servercore is on PL 4.

Any clues?

Thanks

[jensfieger · Accepted solution]

Vikas,

there are some messages 'Permission denied'. Did you see that permissions of /usr/sap/<SID>/<Instance>/j2ee/cluster/server<n> change? Did you change any permissions of j2ee related folders/subfolders manually?

Rgds

Jens

[Former Member · Accepted solution]

Hello Jens,

All the server<n> folders have 755 with <sid>adm:sapsys as their permissions.

I don't remember changing anything manually..

Even I suspected this as first, but the fact that, after server reboot it just goes smoothly without any intervention, didnt make sense to me.

Thanks.

[jensfieger · Accepted solution]

Vikas,

did you check also the temp directories?

/usr/sap/<SID>/<INSTANCE>/j2ee/cluster/

--> ls -lrt --> check temp

/usr/sap/<SID>/<INSTANCE>/j2ee/cluster/server<nr>

--> ls -lrt --> check temp

Check after the crash.

Rgds

Jens

[Former Member · Accepted solution]

hello Jens,

Yes all temp folders have 755 permissions as rest.

But important thing, temp folders dont get updated when we go for complete restart, it only gets overwritten during restart after server reboot.

So, I guess its a permission issue, but the point is why can it access in normal restart when permissions are just fine?

thanks

[jensfieger · Accepted solution]

Vikas,

which umask did you set for sidadm? Did you set the profile parameter rdisp/umask?

Rgds

Jens

[Former Member · Accepted solution]

Hello Jens,

umask is set as 111.

Regards,

[jensfieger · Accepted solution]

Probably that might be the problem. As far as I know Linux system usually have umask 022 (maybe 027 in case of diagnostics agent, Note 1163751).

Umask 111 seems to be 666 --> rw- rw- rw-

• http://www.linuxnix.com/2011/12/umask-define-linuxunix.html
• Note 68718 - Problems with file authorizations and umask

Is there a reason for setting umask to 111?

Rgds

Jens

[Former Member · Accepted solution]

hello Jens,

Yes its diff then usual i.e. 022

but issue here is "permission denied"

and umask being 111 is more authoritative than 022.

So, i dont understand how it could be a issue.

But, i will try to change it to default and see.

thanks

[Former Member]

Dear All,

Its been a long time.

But finally we had the down time and we resolved our issue.

Issue was with UMASK.

Set it to standard 022 and it worked fine.

Thanks to you all.

Regards,

Vikas

[Former Member]

Thanks Tim, Sriram, RB, Jens

You all have given me several options to start over with.

I will work on these and update you with the outcome.

There might be a delay, as change management and downtime is bit of a problem .

mean while if you can think of any other thing, let me know.

Really thankful to you guys.

Cheers

[Former Member]

Hello RB,

Here, I am attaching complete log files.

Also, you mentioned core service as UME, but I dont see UME related error anywhere?

Kindly see if you could find anything else in logs.

Regards,

Vikas

std_server0.ou0.txt.zip

dev_server0.0.txt.zip

[Reagan]

Start of UME Service Failed - SAP NetWeaver™ Problem Analysis Guide (PAG) - SAP Library

A required service failed
If one or more required services failed to start, such as connector, dbpool, timeout, or configuration adapter, the UME service cannot start either. Look for a message in the defaultTrace.trc, such as: Error: Service dbpool failed to start

Can you check whether the above SAP note is valid in your case ?

Are there any contents under these directories ?

ls -la /usr/sap/PX2/DVEBMGS60/j2ee/cluster/server0/temp

ls -la /usr/sap/PX2/DVEBMGS60/j2ee/cluster/server0/temp/dbpool

Also check the solution suggested in this thread.

SAP XI server0 stopping after trying to start | SCN

Regards

RB

[Former Member]

Hi RB,

under /usr/sap/PX2/DVEBMGS60/j2ee/cluster/server0/temp, yes content is there.

but nothing under /usr/sap/PX2/DVEBMGS60/j2ee/cluster/server0/temp/dbpool.

Also, the link you suggested, neither "dispatcher" not "cpt" folders are there in suggested paths.

Thanks

[Sriram2009]

Hi Vikas

Kindly refer the SAP Notes

  1945047 - AS Java stops with exitcode 2150

1562141 - Application Server Java cannot start during upgrade to 7.30

1891477 - PI 7.1X upgrade deployment process fails due to missing ExecutionDetails class

Regards

Sriram

[Former Member]

Vikas,

These error logs show an issue in starting the DBPOOL service - this would make me think initially of the following:

1) Check the secure store properties on the OS - is it possible that the file is corrupt / bad permissions on the file?

2) Check with the offline config tool to see if there's an issue there - with SAPSR3DB schema user or SAPJSF / J2EE_ADMIN user - which ever user you have to connect back to the ABAP stack.

3) Check the j2ee schema owner to ensure it's not locked or the password has expired.

Obviously if the java stack cannot spawn dbpool connections, it can't read into the components deployed on the database.

I still find it interesting that a restart is possible from the ICM on the ABAP stack but not from the OS. The real difference there is in the mechanism used. OS startups will use the SAP Control framework (host agent) while the ICM option will restart it directly - within the existing host agent context.

The hostagent uses a combination of SAPADM (not SIDADM) and root - you might want to check the security folder as sapadm - see if you can navigate to the /sec folder and read the file as sapadm - if sapadm can't read the secure store file on the OS level, this might explain the permission denied error.

There are some great suggestions here - I'm just offering another line of thought.

Cheers,

Tim

[Reagan]

Unfortunately you only provided a part of the trace files. I would consider checking the complete log and also the latest default trace log. Based on what you have provided I see a core service failed to start.

Service [dbpool] start ================= ERROR =================
  CSN Component [BC-JAS-TRH], DC Name [sap.com/dbpool]
java.lang.NullPointerException
        at java.util.Hashtable.put(Hashtable.java:394)
        at com.sap.engine.services.dbpool.PoolFrame.getComponentsPersistentFiles(PoolFrame.java:405)
        at com.sap.engine.services.dbpool.PoolFrame.start(PoolFrame.java:140)
        at com.sap.engine.core.service630.container.ServiceRunner.startFrame(ServiceRunner.java:165)
        at com.sap.engine.core.service630.container.ServiceRunner.startService(ServiceRunner.java:118)
        at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:65)
        at com.sap.engine.frame.core.thread.Task.run(Task.java:73)
        at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:178)
        at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:316)
FATAL: Critical shutdown with exit code [2150] was requested due to [AS Java cannot be started; core service [dbpool] failed to start]

I would check the default trace and see what it contains. The default trace provided by you contains no relevant information to the error in the dev_server and std_server logs.

Start with this SAP note and see if that helps.

1592625 - J2EE engine startup fails due to java.io.IOException:no such file or directory error

Regards

RB

[Former Member]

Hello Tim,

I assume, by host control agent, you are pointing to instance specific "sapstartsrv" as this is responsible for startup and sop.

But, yes, I never thought in this direction. Really thankful to you.

And yes there is some ambiguity in EXEs of diff instances.

For CI:

/usr/sap/PX2/DVEBMGS60/exe/sapstartsrv, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:57:04

/usr/sap/PX2/DVEBMGS60/exe/disp+work, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:45:24

/usr/sap/PX2/DVEBMGS60/exe/gwrd, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:46:22

/usr/sap/PX2/DVEBMGS60/exe/msg_server, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:55:18

/usr/sap/PX2/DVEBMGS60/exe/dboraslib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 22 23:38:48

/usr/sap/PX2/DVEBMGS60/exe/dbmssslib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 23 00:02:17

/usr/sap/PX2/DVEBMGS60/exe/dbdb2slib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 22 23:36:58

/usr/sap/PX2/DVEBMGS60/exe/dbdb6slib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 22 23:38:27

/usr/sap/PX2/DVEBMGS60/exe/dbsybslib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 23 00:58:30

/usr/sap/PX2/DVEBMGS60/exe/enserver, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:56:08

/usr/sap/PX2/DVEBMGS60/exe/icman, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:48

/usr/sap/PX2/DVEBMGS60/exe/sapwebdisp, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:49

/usr/sap/PX2/DVEBMGS60/exe/jcontrol, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 11:01:07

/usr/sap/PX2/DVEBMGS60/exe/jlaunch, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 11:01:09

/usr/sap/PX2/DVEBMGS60/exe/jstart, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 11:01:04

For ASCS:

/usr/sap/PX2/ASCS61/exe/sapstartsrv, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:57:04

/usr/sap/PX2/ASCS61/exe/disp+work, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 22 20:17:23

/usr/sap/PX2/ASCS61/exe/gwrd, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:46:22

/usr/sap/PX2/ASCS61/exe/msg_server, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:55:18

/usr/sap/PX2/ASCS61/exe/dboraslib.so, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 21 19:39:19

/usr/sap/PX2/ASCS61/exe/dbmssslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 00:08:44

/usr/sap/PX2/ASCS61/exe/dbdb2slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:07

/usr/sap/PX2/ASCS61/exe/dbdb6slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:17

/usr/sap/PX2/ASCS61/exe/dbsybslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 02:24:54

/usr/sap/PX2/ASCS61/exe/enserver, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:56:08

/usr/sap/PX2/ASCS61/exe/icman, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:27:35

/usr/sap/PX2/ASCS61/exe/sapwebdisp, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:49

/usr/sap/PX2/ASCS61/exe/jcontrol, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:17

/usr/sap/PX2/ASCS61/exe/jlaunch, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:21

/usr/sap/PX2/ASCS61/exe/jstart, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:12

For SCS:

/usr/sap/PX2/SCS62/exe/sapstartsrv, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:57:04

/usr/sap/PX2/SCS62/exe/disp+work, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 22 20:17:23

/usr/sap/PX2/SCS62/exe/gwrd, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:46:22

/usr/sap/PX2/SCS62/exe/msg_server, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:55:18

/usr/sap/PX2/SCS62/exe/dboraslib.so, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 21 19:39:19

/usr/sap/PX2/SCS62/exe/dbmssslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 00:08:44

/usr/sap/PX2/SCS62/exe/dbdb2slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:07

/usr/sap/PX2/SCS62/exe/dbdb6slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:17

/usr/sap/PX2/SCS62/exe/dbsybslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 02:24:54

/usr/sap/PX2/SCS62/exe/enserver, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:56:08

/usr/sap/PX2/SCS62/exe/icman, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:27:35

/usr/sap/PX2/SCS62/exe/sapwebdisp, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:49

/usr/sap/PX2/SCS62/exe/jcontrol, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:17

/usr/sap/PX2/SCS62/exe/jlaunch, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:21

/usr/sap/PX2/SCS62/exe/jstart, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:12

My main local exe is consistent, but SCS and ASCS are at lower level, could this be a reason?

And we restart our JAVA using SMICM only.

Thanks,

[Reagan]

Hello

Could you supply the dev_server0 and std_server0 trace files from the work directory ?

Regards

RB

[Former Member]

Vikas,

You may have something there indeed. However, the SCS and ASCS instances are really only our message server / enqueue server functions, so the sap copy function may not be looking to synch up all of these files in the ASCS / SCS kernel directories.

Yes, you may want to check the hostagent piece under /usr/sap/hostctrl/exe. You can upgrade the host contol agent while the system is running. Check out Note 1031096 - Installing Package SAPHOSTAGENT    for specific information on getting this piece up to date. At the very least, you'll get this piece up to date and  remove it from consideration as a root cause.

I hope this all helps.

Tim

[Former Member]

Hello RB,

dev_server0:

  ********************************************************************************

J  JVMX version - May 30 2013 00:33:03 - 61_REL - optU - linux amd64 - 6 - bas2:197575 (mixed mode)

F

F [Thr 140361962661632] Tue Apr 15 00:33:15 2014

F  [Thr 140361962661632] *** LOG => SfCJavaVm: Java VM started.

F  ********************************************************************************

F  Main method call:

F  com/sap/engine/boot/Start.main()

F    arg[ 0] = -XX:-EnableDebuggingOnDemand

F    arg[ 1] = -DSAPSTART=1

F    arg[ 2] = -DSAPSYSTEM=60

F    arg[ 3] = -DSAPSYSTEMNAME=PX2

F    arg[ 4] = -DSAPMYNAME=a096px2cidb_PX2_60

F    arg[ 5] = -DSAPPROFILE=/usr/sap/PX2/SYS/profile/PX2_DVEBMGS60_a096px2cidb

F    arg[ 6] = -DFRFC_FALLBACK=ON

F    arg[ 7] = -DFRFC_FALLBACK_HOST=localhost

F  ********************************************************************************

J

J Tue Apr 15 00:33:17 2014

J  4.033: [GC 4.033: [ParNew: 786432K->16160K(917504K), 0.0468730 secs] 786825K->16554K(6160384K), 0.0470390 secs] [Times: user=0.26 sys=0.02, real=0.05 secs]

F

F [Thr 140361962661632] Tue Apr 15 00:33:18 2014

F  [Thr 140361962661632] *** LOG => State changed from 0 (Initial) to 1 (Waiting to start).

F  [Thr 140361962661632] *** LOG    state real time: 5.861 CPU time: 1.300 sys, 14.240 usr

F  [Thr 140361962661632] *** LOG    total real time: 5.861 CPU time: 1.300 sys, 14.240 usr

F  [Thr 140361962661632]

F  [Thr 140361962661632] *** LOG => State changed from 1 (Waiting to start) to 2 (Starting framework).

F  [Thr 140361962661632] *** LOG    state real time: 0.000 CPU time: 0.000 sys, 0.000 usr

F  [Thr 140361962661632] *** LOG    total real time: 5.861 CPU time: 1.300 sys, 14.240 usr

F  [Thr 140361962661632]

J

J Tue Apr 15 00:33:22 2014

J  9.522: [GC 9.522: [ParNew: 802592K->43567K(917504K), 0.0473360 secs] 803194K->44169K(6160384K), 0.0475560 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]

J

J Tue Apr 15 00:33:29 2014

J  Heap

J   par new generation   reserved 1048576K, committed 1048576K, used 650769K [0x00000000029a0000, 0x00000000429a0000, 0x00000000429a0000)

J    eden space 786432K,  77% used [0x00000000029a0000, 0x0000000027a98988, 0x00000000329a0000)

J    from space 131072K,  33% used [0x00000000329a0000, 0x000000003542be48, 0x000000003a9a0000)

J    to   space 131072K,   0% used [0x000000003a9a0000, 0x000000003a9a0000, 0x00000000429a0000)

J   concurrent mark-sweep generation reserved 5242880K, committed 5242880K, used 1507K [0x00000000429a0000, 0x00000001829a0000, 0x00000001829a0000)

J   concurrent-mark-sweep perm gen reserved 1048576K, committed 1048576K, used 63929K [0x00000001829a0000, 0x00000001c29a0000, 0x00000001c29a0000)

F

F [Thr 140361448253184] Tue Apr 15 00:33:30 2014

F  [Thr 140361448253184] *** LOG => SfCJavaVm: exit hook is called. (rc = 2150)

F

F Tue Apr 15 00:33:30 2014

F

F  ********************************************************************************

F  *** ERROR => Java node 'server0' terminated with exit code 2150.

F  ***

F  *** Please see section 'Java program exit codes'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

F

F  [Thr 140362446972768] *** LOG => exiting (exitcode 2150, retcode 1).

M  [Thr 140362446972768] CCMS: CCMS Monitoring Cleanup finished successfully.

std_server0:

--------------------------------------------------------------------------------

stdout/stderr redirection

--------------------------------------------------------------------------------

node name   : server0

host name   : sedcaecsp045

system name : PX2

system nr.  : 60

started at  : Tue Apr 15 00:33:11 2014

JVMX version - May 30 2013 00:33:03 - 61_REL - optU - linux amd64 - 6 - bas2:197575 (mixed mode)

The active factory is : com.sap.engine.boot.loader.StandardClassLoaderFactory

Class loading cache information loaded from package index [/usr/sap/PX2/DVEBMGS60/j2ee/cluster/bin/index.list]

Reading manager properties from [kernelProperties.bin] file

   Checked the data for [11] ms

   Objects read for [99] ms

   Substituted the values for [73] ms

Finished reading kernel objects for [172] ms

Startup mode [NORMAL] and action [NONE] read

AS Java version [7.31.3301.349564.20131015072828 SP 09] is starting server process [ID602208850] on host [a096px2cidb.emea.isn.corpintra.net] ...

Loading [LogManager]... [59] ms

Loading [PoolManager]... [3] ms

Loading [ApplicationThreadManager]... 4.033: [GC 4.033: [ParNew: 786432K->16160K(917504K), 0.0468730 secs] 786825K->16554K(6160384K), 0.0470390 secs] [Times: user=0.26 sys=0.02, real=0.05 s

ecs]

[154] ms

Loading [ThreadManager]... [28] ms

Loading [ClassLoaderManager]... [28] ms

Loading [ClusterManager]... [1053] ms

Loading [LockingManager]... [60] ms

Loading [ConfigurationManager]... [1952] ms

Loading [CacheManager]... [24] ms

Loading [DatabaseManager]... [351] ms

Loading [SessionManager]... [341] ms

Loading [LicensingManager]... [19] ms

Loading [ServiceManager]...

Initialize components for : 150 ms.

Resolve components for : 25 ms.

Load components for 221 ms.

Loading services.

9.522: [GC 9.522: [ParNew: 802592K->43567K(917504K), 0.0473360 secs] 803194K->44169K(6160384K), 0.0475560 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]

Calculate startup service set for 1138 ms.

  Service [tc~je~content~classification~handler] started. [8] ms / [2] CPU ms

  Service [security.destination.facade] started. [12] ms / [0] CPU ms

  Service [engine.lm.log.facade] started. [6] ms / [0] CPU ms

  Service [tc~je~sca~ws.plugin] started. [11] ms / [0] CPU ms

  Service [tc~je~security~api] started. [15] ms / [1] CPU ms

  Service [webservices] started. [15] ms / [0] CPU ms

  Service [tc~je~mngt_model~srv] started. [11] ms / [0] CPU ms

  Service [tc~je~cross~api] started. [14] ms / [0] CPU ms

  Service [tc~esi~uddi~content~handler] started. [9] ms / [0] CPU ms

  Service [r3startup] started. [9] ms / [0] CPU ms

  Service [engine.dsr.facade] started. [9] ms / [0] CPU ms

  Service [tc~je~sca~jco.plugin] started. [23] ms / [7] CPU ms

  Service [cluster] started. [17] ms / [5] CPU ms

  Service [runtimeinfo] started. [29] ms / [8] CPU ms

  Service [jmx_notification] started. [108] ms / [36] CPU ms

  Service [userstore] started. [92] ms / [36] CPU ms

  Service [timeout] started. [121] ms / [65] CPU ms

  Service [memory] started. [128] ms / [47] CPU ms

  Service [cross] started. [176] ms / [108] CPU ms

  Service [log_configurator] started. [208] ms / [119] CPU ms

  Service [locking] started. [12] ms / [8] CPU ms

  Service [tc~bl~accounting_infra~srv] started. [291] ms / [190] CPU ms

  Service [tc~sec~destinations~provider] started. [300] ms / [210] CPU ms

  Service [http] started. [293] ms / [263] CPU ms

  Service [schemaprocessor~srv] started. [1093] ms / [1012] CPU ms

  Service [p4] started. [1237] ms / [735] CPU ms

  Service [classload] started. [2] ms / [1] CPU ms

  Service [classload.facade] started. [0] ms / [0] CPU ms

  Service [msp] started. [4] ms / [2] CPU ms

  Service [classpath_resolver] started. [22] ms / [17] CPU ms

  Service [naming] started. [215] ms / [202] CPU ms

  Service [tc~je~sca~ejb.plugin] started. [6] ms / [2] CPU ms

  Service [file] started. [10] ms / [4] CPU ms

  Service [objectanalyzing] started. [18] ms / [11] CPU ms

  Service [javamail] started. [61] ms / [49] CPU ms

  Service [connector] started. [88] ms / [78] CPU ms

  Service [library_container] started. [16] ms / [14] CPU ms

  Service [tc~je~sca~integration] started. [1305] ms / [1226] CPU ms

  Service [deploy] started. [2052] ms / [393] CPU ms

  Service [deploy.facade] started. [0] ms / [0] CPU ms

  Service [metadatarepository] started. [21] ms / [9] CPU ms

  Service [orpersistence] started. [28] ms / [22] CPU ms

  Service [MigrationService] started. [30] ms / [24] CPU ms

  Service [engine.migservice.facade] started. [0] ms / [0] CPU ms

  Service [engine.migration.facade] started. [0] ms / [0] CPU ms

  Service [dbpool] start ================= ERROR =================

  CSN Component [BC-JAS-TRH], DC Name [sap.com/dbpool]

java.lang.NullPointerException

        at java.util.Hashtable.put(Hashtable.java:394)

        at com.sap.engine.services.dbpool.PoolFrame.getComponentsPersistentFiles(PoolFrame.java:405)

        at com.sap.engine.services.dbpool.PoolFrame.start(PoolFrame.java:140)

        at com.sap.engine.core.service630.container.ServiceRunner.startFrame(ServiceRunner.java:165)

        at com.sap.engine.core.service630.container.ServiceRunner.startService(ServiceRunner.java:118)

        at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:65)

        at com.sap.engine.frame.core.thread.Task.run(Task.java:73)

        at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:178)

        at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:316)

FATAL: Critical shutdown with exit code [2150] was requested due to [AS Java cannot be started; core service [dbpool] failed to start]

Heap

par new generation   reserved 1048576K, committed 1048576K, used 650769K [0x00000000029a0000, 0x00000000429a0000, 0x00000000429a0000)

  eden space 786432K,  77% used [0x00000000029a0000, 0x0000000027a98988, 0x00000000329a0000)

  from space 131072K,  33% used [0x00000000329a0000, 0x000000003542be48, 0x000000003a9a0000)

  to   space 131072K,   0% used [0x000000003a9a0000, 0x000000003a9a0000, 0x00000000429a0000)

concurrent mark-sweep generation reserved 5242880K, committed 5242880K, used 1507K [0x00000000429a0000, 0x00000001829a0000, 0x00000001829a0000)

concurrent-mark-sweep perm gen reserved 1048576K, committed 1048576K, used 63929K [0x00000001829a0000, 0x00000001c29a0000, 0x00000001c29a0000)

Let me know if you find something wrong here.

Thanks,

[Former Member]

Sure Tim,

I will definitely fix this first, if nothing else.

may be I will patch complete kernel.

Together, i will try to upgrade HOSTAGENT also.

I will update you once done.

Thanks.

Regards,

[Former Member]

Vikas,

The host control agent should be managing the startup here - could you check this piece first? It's somewhat obvious that there is a conflict at the OS level - it's not clear why. Please check the patch level of the SAP host agent pieces to make sure they are up to date as well. Based on the log file you provided:

Caused by: java.io.IOException: Permission denied

      at java.io.UnixFileSystem.createFileExclusively(Native Method)

      at java.io.File.checkAndCreate(File.java:1715)

      at java.io.File.createTempFile0(File.java:1736)

      at java.io.File.createTempFile(File.java:1813)

com.sap.engine.lib.config.api.exceptions.ClusterConfigurationException: The entry with the specified name containers-info.xml cannot be retrieved in tc~je~sca~integr

ation/descriptors.

It looks like it's a filesystem issue (as opposed to a port lock, or other OS function).

You say you don't see this issue when you restart the J2EE stack by itself - but how do you perform this action - thry the ICM on the ABAP stack? the managment console? or maybe from  a command line? Curious to see if there is a different mechanism in play underneath.

Also check the work directory for the bootstrap out files. How far does the J2EE engine get? Is it a complete failure or is the dispatcher starting but not the server?

Thanks!

[Former Member]

Hello Tim,

I assume, by host control agent, you are pointing to instance specific "sapstartsrv" as this is responsible for startup and sop.

But, yes, I never thought in this direction. Really thankful to you.

And yes there is some ambiguity in EXEs of diff instances.

For CI:

/usr/sap/PX2/DVEBMGS60/exe/sapstartsrv, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:57:04

/usr/sap/PX2/DVEBMGS60/exe/disp+work, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:45:24

/usr/sap/PX2/DVEBMGS60/exe/gwrd, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:46:22

/usr/sap/PX2/DVEBMGS60/exe/msg_server, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:55:18

/usr/sap/PX2/DVEBMGS60/exe/dboraslib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 22 23:38:48

/usr/sap/PX2/DVEBMGS60/exe/dbmssslib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 23 00:02:17

/usr/sap/PX2/DVEBMGS60/exe/dbdb2slib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 22 23:36:58

/usr/sap/PX2/DVEBMGS60/exe/dbdb6slib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 22 23:38:27

/usr/sap/PX2/DVEBMGS60/exe/dbsybslib.so, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 23 00:58:30

/usr/sap/PX2/DVEBMGS60/exe/enserver, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:56:08

/usr/sap/PX2/DVEBMGS60/exe/icman, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:48

/usr/sap/PX2/DVEBMGS60/exe/sapwebdisp, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:49

/usr/sap/PX2/DVEBMGS60/exe/jcontrol, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 11:01:07

/usr/sap/PX2/DVEBMGS60/exe/jlaunch, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 11:01:09

/usr/sap/PX2/DVEBMGS60/exe/jstart, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 11:01:04

For ASCS:

/usr/sap/PX2/ASCS61/exe/sapstartsrv, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:57:04

/usr/sap/PX2/ASCS61/exe/disp+work, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 22 20:17:23

/usr/sap/PX2/ASCS61/exe/gwrd, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:46:22

/usr/sap/PX2/ASCS61/exe/msg_server, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:55:18

/usr/sap/PX2/ASCS61/exe/dboraslib.so, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 21 19:39:19

/usr/sap/PX2/ASCS61/exe/dbmssslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 00:08:44

/usr/sap/PX2/ASCS61/exe/dbdb2slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:07

/usr/sap/PX2/ASCS61/exe/dbdb6slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:17

/usr/sap/PX2/ASCS61/exe/dbsybslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 02:24:54

/usr/sap/PX2/ASCS61/exe/enserver, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:56:08

/usr/sap/PX2/ASCS61/exe/icman, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:27:35

/usr/sap/PX2/ASCS61/exe/sapwebdisp, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:49

/usr/sap/PX2/ASCS61/exe/jcontrol, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:17

/usr/sap/PX2/ASCS61/exe/jlaunch, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:21

/usr/sap/PX2/ASCS61/exe/jstart, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:12

For SCS:

/usr/sap/PX2/SCS62/exe/sapstartsrv, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:57:04

/usr/sap/PX2/SCS62/exe/disp+work, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 22 20:17:23

/usr/sap/PX2/SCS62/exe/gwrd, 721, patch 100, changelist 1396425, optu (Feb 22 2013, 23:56:12), linuxx86_64, 2013 02 28 10:46:22

/usr/sap/PX2/SCS62/exe/msg_server, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:55:18

/usr/sap/PX2/SCS62/exe/dboraslib.so, 721, patch 41, changelist 1388317, optu (Jan 21 2013, 19:57:22), linuxx86_64, 2013 01 21 19:39:19

/usr/sap/PX2/SCS62/exe/dbmssslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 00:08:44

/usr/sap/PX2/SCS62/exe/dbdb2slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:07

/usr/sap/PX2/SCS62/exe/dbdb6slib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 09 23:54:17

/usr/sap/PX2/SCS62/exe/dbsybslib.so, 721, patch 40, changelist 1385685, optu (Jan  9 2013, 23:58:10), linuxx86_64, 2013 01 10 02:24:54

/usr/sap/PX2/SCS62/exe/enserver, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:56:08

/usr/sap/PX2/SCS62/exe/icman, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:27:35

/usr/sap/PX2/SCS62/exe/sapwebdisp, 721, patch 100, changelist 1396425, optu (Feb 23 2013, 03:16:51), linuxx86_64, 2013 02 28 10:54:49

/usr/sap/PX2/SCS62/exe/jcontrol, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:17

/usr/sap/PX2/SCS62/exe/jlaunch, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:21

/usr/sap/PX2/SCS62/exe/jstart, 721, patch 41, changelist 1388317, optu (Jan 22 2013, 04:20:46), linuxx86_64, 2013 01 22 20:37:12

My main local exe is consistent, but SCS and ASCS are at lower level, could this be a reason?

We restart our JAVA using SMICM only.

And, disptacher starts, but server nodes fails just after starting framework phase. Find log below:

J Tue Apr 15 02:56:49 2014

J  [GC 4032K->1388K(9984K), 0.0159590 secs]

J  [GC 5420K->3700K(14016K), 0.0046660 secs]

J  [Full GC 3700K->3551K(21568K), 0.0686880 secs]

J  [GC 11615K->8339K(21568K), 0.0057830 secs]

J  [GC 16403K->12974K(24448K), 0.0085390 secs]

J  [Full GC 12974K->9297K(38400K), 0.0490090 secs]

F

F [Thr 139702947018496] Tue Apr 15 02:56:49 2014

F  [Thr 139702947018496] *** LOG => State changed from 0 (Initial) to 2 (Starting framework).

F  [Thr 139702947018496] *** LOG    state real time: 2.699 CPU time: 0.640 sys, 1.480 usr

F  [Thr 139702947018496] *** LOG    total real time: 2.699 CPU time: 0.640 sys, 1.480 usr

F  [Thr 139702947018496]

J

J Tue Apr 15 02:56:50 2014

J  [GC 20241K->13429K(38400K), 0.0067620 secs]

J  [GC 24373K->18487K(46208K), 0.0084020 secs]

J  [GC 31351K->17232K(46848K), 0.0083040 secs]

J

J Tue Apr 15 02:56:51 2014

J  [GC 30096K->17735K(49920K), 0.0081880 secs]

J  [GC 33863K->18418K(49920K), 0.0092180 secs]

J  [GC 34546K->18756K(53056K), 0.0162340 secs]

J

J Tue Apr 15 02:56:53 2014

J  [GC 37700K->24728K(53312K), 0.0117280 secs]

F

F [Thr 139702947018496] Tue Apr 15 02:56:53 2014

F  [Thr 139702947018496] *** LOG => State changed from 2 (Starting framework) to 3 (Running).

F  [Thr 139702947018496] *** LOG    state real time: 4.242 CPU time: 0.290 sys, 6.200 usr

F  [Thr 139702947018496] *** LOG    total real time: 6.942 CPU time: 0.930 sys, 7.680 usr

F  [Thr 139702947018496]

J

J Tue Apr 15 02:56:55 2014

J  [GC 43672K->28507K(56768K), 0.0179700 secs]

J

J Tue Apr 15 02:56:57 2014

J  [GC 48475K->32421K(58560K), 0.0146960 secs]

F

F [Thr 139702947018496] Tue Apr 15 02:56:57 2014

F  [Thr 139702947018496] *** LOG => State changed from 3 (Running) to 5 (Stopping).

F  [Thr 139702947018496] *** LOG    state real time: 3.252 CPU time: 0.320 sys, 5.360 usr

F  [Thr 139702947018496] *** LOG    total real time: 10.195 CPU time: 1.250 sys, 13.040 usr

F  [Thr 139702947018496]

F  [Thr 139702349141760] *** LOG => SfCJavaVm: exit hook is called. (rc = 0)

F  [Thr 139702349141760] *** LOG => exiting (exitcode 0, retcode 0).

M  [Thr 139702349141760] CCMS: CCMS Monitoring Cleanup finished successfully.

Thanks.