#Critical shutdown was invoked. Reason is: Core service p4 failed. J2EE Engine cannot be started.#

[Former Member]

Hi,

Post DB migration of XI server,we are facing "#Critical shutdown was invoked. Reason is: Core service p4 failed. J2EE Engine cannot be started.#"

In logs it is stating,  "*** ERROR => NiHLGetHostName: NiPGetHostByAddr failed (rc=-1) [nixxhl.cpp  495]"

It cannot resolve hostname,is it?

I have checked my /etc/hosts file which contains the ip against hostname.

We have completed Quality migration and there we did not face such error,

deafultrace,dev_ms,server1.log,dev_jcontrol,stderr1 logs are attached.

Please confirm if more detailed logs are required.

Defaultrace has larger size,thus pasting the contents over here.Sorry for the inconvenience caused.

************************************************************************************

---------------------------------------------------------------

    at com.sap.engine.frame.core.load.ReferencedLoader.loadClass(ReferencedLoader.java:382)

    ... 13 more

#

#1.5 #001A647600F200080000000300005BDF000500FA5B76D0D0#1408450159890#com.sap.engine.core.service630.container.ServiceRunner##com.sap.engine.core.service630.container.ServiceRunner#######SAPEngine_System_Thread[impl:5]_53##0#0#Error#1#/System/Server#Java###Core service p4 failed. J2EE Engine cannot be started.

[EXCEPTION]

{0}#1#java.lang.NoClassDefFoundError: com/sap/engine/transport/ClientPortsManager

    at com.sap.engine.services.rmi_p4.ConnectionManager.<clinit>(ConnectionManager.java:77)

    at com.sap.engine.services.rmi_p4.P4ObjectBrokerClientImpl.postInit(P4ObjectBrokerClientImpl.java:397)

    at com.sap.engine.services.rmi_p4.P4ObjectBroker.loadInstance(P4ObjectBroker.java:279)

    at com.sap.engine.services.rmi_p4.P4ObjectBroker.init(P4ObjectBroker.java:176)

    at com.sap.engine.services.rmi_p4.P4ObjectBroker.init(P4ObjectBroker.java:158)

    at com.sap.engine.services.rmi_p4.server.ClusterOrganizer.<init>(ClusterOrganizer.java:59)

    at com.sap.engine.services.rmi_p4.server.P4SessionProcessor.<init>(P4SessionProcessor.java:81)

    at com.sap.engine.services.rmi_p4.server.P4ServiceFrame.start(P4ServiceFrame.java:82)

    at com.sap.engine.core.service630.container.ServiceRunner.startApplicationServiceFrame(ServiceRunner.java:214)

    at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:144)

    at com.sap.engine.frame.core.thread.Task.run(Task.java:64)

    at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:83)

    at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:109)

Caused by: java.lang.ClassNotFoundException: com.sap.engine.transport.ClientPortsManager

------------------------- Loader Info -------------------------

ClassLoader name: [common:service:iiop;service:naming;service:p4;service:ts]

Parent loader name: [Frame ClassLoader]

References:

  interface:naming

  interface:log

  interface:cross

  interface:security

  interface:appcontext

  interface:shell

  interface:visual_administration

  service:timeout

  library:j2eeca

  library:jts

  library:idl

  interface:resourceset_api

  interface:resourcecontext_api

  interface:transactionext

  library:tc~bl~txmanager~plb

  interface:csiv2

  library:sapxmltoolkit

  library:core_lib

  common:library:IAIKSecurity;library:activation;library:mail;library:tc~sec~ssl

Resources:

  /usr/sap/XID/DVEBMGS02/j2ee/cluster/server0/bin/services/naming/naming.jar

  /usr/sap/XID/DVEBMGS02/j2ee/cluster/server0/bin/services/p4/p4.jar

  /usr/sap/XID/DVEBMGS02/j2ee/cluster/server0/bin/services/iiop/iiop.jar

  /usr/sap/XID/DVEBMGS02/j2ee/cluster/server0/bin/services/ts/ts.jar

Loading model: {parent,local,references}

---------------------------------------------------------------

    at com.sap.engine.frame.core.load.ReferencedLoader.loadClass(ReferencedLoader.java:382)

    ... 13 more

#

#1.5 #001A647600F200080000000500005BDF000500FA5B76D223#1408450159890#com.sap.engine.core.Framework##com.sap.engine.core.Framework#######SAPEngine_System_Thread[impl:5]_53##0#0#Fatal#1#/System/Server/Critical#Plain###Critical shutdown was invoked. Reason is: Core service p4 failed. J2EE Engine cannot be started.#

******************************************************************

stderr1 :

********************************

(21607) New Child Process created.

(21607) Starting local Command:

Command:  dw.sapXID_DVEBMGS02

          pf=/usr/sap/XID/SYS/profile/XID_DVEBMGS02_devbid

Tue Aug 19 17:38:23 2014

DpShmPrfCreate : created shared memory at 7fb9fbe5d000, 1798 entries (1654952 bytes)

Tue Aug 19 17:38:23 2014

ES/TABLE = UNIX_STD

[Thr 139944993081120] Tue Aug 19 17:38:23 2014

[Thr 139944993081120] MtxInit: 30002 0 2

*** ICM up and operational (pid: 21621) ***

Exception in thread "main" java.lang.NoClassDefFoundError: com/sap/engine/offline/OfflineToolStart

Caused by: java.lang.ClassNotFoundException: com.sap.engine.offline.OfflineToolStart

    at java.net.URLClassLoader$1.run(URLClassLoader.java:255)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.net.URLClassLoader.findClass(URLClassLoader.java:243)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:362)

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:318)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:308)

Exception in thread "main" java.lang.NoClassDefFoundError: com/sap/engine/offline/OfflineToolStart

Caused by: java.lang.ClassNotFoundException: com.sap.engine.offline.OfflineToolStart

    at java.net.URLClassLoader$1.run(URLClassLoader.java:255)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.net.URLClassLoader.findClass(URLClassLoader.java:243)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:362)

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:318)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:308)

Exception in thread "main" java.lang.NoClassDefFoundError: com/sap/engine/offline/OfflineToolStart

Caused by: java.lang.ClassNotFoundException: com.sap.engine.offline.OfflineToolStart

    at java.net.URLClassLoader$1.run(URLClassLoader.java:255)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.net.URLClassLoader.findClass(URLClassLoader.java:243)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:362)

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:318)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:308)

Exception in thread "main" java.lang.NoClassDefFoundError: com/sap/engine/offline/OfflineToolStart

Caused by: java.lang.ClassNotFoundException: com.sap.engine.offline.OfflineToolStart

    at java.net.URLClassLoader$1.run(URLClassLoader.java:255)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.net.URLClassLoader.findClass(URLClassLoader.java:243)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:362)

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:318)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:308)

*************************************************8

Regards,

VIneet

dev_ms.txt.zip

server.1.txt.zip

dev_jcontrol.txt.zip

[cathal_ohare · Accepted solution]

Hi,

the problem is that there is a class file missing:

"NoClassDefFoundError: com/sap/engine/transport/ClientPortsManager"

The casue of this is that your components are all on different SP levels and not compatible with each other.

You need to do a full SP upgrade to 7.00 SP31, you can try upgrade each component indivdually to sp 31 with the SDM in standalone mode but you may be better just doing a fresh install at SP 31.

Kind regards,

Cathal

[divyanshu_srivastava3 · Accepted solution]

Yes. perfectly said.

I didn't really look into the SP levels but asked him to compare the stack from market place for correctness

Regards,

Divyanshu

[Former Member]

Hi All,

Thanks for the root cause analysis.

We have started restoration process,once done we will let you know the JAVA status.

Regards,

Vineet

[Former Member]

Hi All,

Post restoration issue got resolved.Thanks for for the help guys.

Regards,

Vineet

[Former Member]

Hi Divyanshu,

element.resynch=force parameter was tried tried earlier.Issue was not resolved.

But at that time serverX was not renamed,currently we are facing some network issues thus servers are not available.Once issue is resolved I will check and revert.

Thanks n Regards,

Vineet

[Former Member]

Hi Divyanshu,

Yes given two notes are checked.Things are in place as per note id.

PFA dispatcher logs.

Regards,

Vineet

dev_dispatcherr.txt.zip

std_dispatcher.txt.zip

[divyanshu_srivastava3]

Hi Vinnet,

The class definition is not found. If this exists in DB, then use below method. It should work if rest of the things are fine.

Can you check with following steps ?

1. stop java engine

3  in bootstrap.properties add element.resynch=force parameter(Refer: 710663)

4. rename /usr/sap/<SID>/<Java Instance>/j2ee/cluster/serverX to serverX_old.

5. restart system

Share the results.

Regards,

Divyanshu

[Former Member]

Hi All,

Checked the note ids,bind variable value is blank only.

And one more observation is that components are on different patch levels.Is it the cause?

Please find below component versions:

SCNAME    SAPRELEASE    SERVICELEVEL    PATCHLEVEL

CORE-TOOLS        7.00    30    0

CAF-UM            7.00    9    0

KM-KW_JIKS        7.00    9    0

LM-TOOLS        7.00    9    3

SAP-JEE            7.00    30    0

JLOGVIEW        7.00    30    0

BI_MMR            7.00    9    0

UMEADMIN        7.00    9    0

BI_UDI            7.00    9    0

JSPM            7.00    30    0

ADSSAP            7.00    9    0

CAF            7.00    9    0

SAP-JEECOR        7.00    31    0

SAP_JTECHS        7.00    9    0

SAP_XITOOL        7.00    9    0

SAP_XIAF        7.00    9    0

SAP_JTECHF        7.00    30    0

BASETABLES        7.00    30    0

SAP-XIAFC        7.00    30    5

SAP-XICONS        7.00    30    0

Regards,

Vineet

[divyanshu_srivastava3]

Hi Vinnet,

You can check and match your current stack from service market place.

Other thing is, did you check above 2 notes ?

If the dispatcher logs are showing those error, you need to fix in that way.

Otherwise, please share dev_dispatcher and std_dispatcher.out

Regards,

Divyanshu

[Former Member]

Hi all,

@James,Divyanshu n Reagan PFA dev_server0 and std_server0

@Divyanshu, I will check the note id and will revert shortly.

@Madasamy, Initially we faced password issue,but resolved it by updating password using configtool.

Regards,

Vineet

std_server0.txt.zip

dev_server0.txt.zip

[divyanshu_srivastava3]

Hi,

Can you also share dev_dispatcher and std_dispatcher.

Check this note: 1658911 - J2EE System fails with "exitcode = -11113"

Regards,

Divyanshu

[madasamy_arunachalam]

Hi

Based on log file server1, there is error while connecting DB

##0#0#Error#1#com.sap.engine.core.configuration#Plain###Error while connecting to DB.
=======================================================
com.sap.engine.frame.core.configuration.ConfigurationException: Error while connecting to DB.
.........
at com.sap.engine.boot.FrameThread.run(FrameThread.java:61)
    at java.lang.Thread.run(Thread.java:708)
Caused by: com.ibm.db2.jcc.am.SqlInvalidAuthorizationSpecException: [jcc][t4][2013][11249][3.67.27] Connection authorization failure occurred.  Reason: User ID or Password invalid. ERRORCODE=-4214, SQLSTATE=28000

Can you check , why this password issue ?, any password got changed during or before your activity ?

As James updated, attached log from server0 also (hopefully it will have same error like log from server1)

regards

Swami

[divyanshu_srivastava3]

Hi Vinnet,

Can you check your 3rd drivers ? Are hey in their place ?

You may have to update them.

1028961 - How to prepare axisproviderlib.sda for Adapter Framework

Regards,

Divyanshu

[divyanshu_srivastava3]

And this is as per your default trace.

Please supply std_server0.out for finding out service failure.

[James_Connolly]

Hi Vineet,

Please attach or paste the contents of the std_server0.out log from the work directory.

regards,

James