Error in bootstrap within patching of NW PI 7.40 AEX (Java Only)

[JaySchwendemann]

Hi all,

we are suffering an error witin patching a NW PI 7.40 AEX (Java Only)

In SUM we get the following error when SUM wants to stop / start system in execution phase

FATAL: Critical shutdown with exit code [2150] was requested due to [AS
Java 
cannot be started; core service [tc~lm~nzdm~crrsrv] failed to
start]

We already checked note http://service.sap.com/sap/support/notes/1713021 for this with no luck.

We are getting the following output in bootstrap

Starting to initialize database connection
Exception occurred for component [internal/unkown] of type [unkown]
com.sap.engine.bootstrap.SynchronizationException: Storage access initialization failed due to OpenSQL error.Check if your db driver file is avail
able on the file system or if the user has correct permissions for that file
        at com.sap.engine.bootstrap.StorageAccess.<init>(StorageAccess.java:146)
        at com.sap.engine.bootstrap.Bootstrap.initializeSynchronizer(Bootstrap.java:79)
        at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:273)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:162)
----------==[ Caused by: ]==----------
com.sap.engine.frame.core.configuration.ConfigurationException: Error while connecting to DB.
        at com.sap.engine.core.configuration.impl.persistence.rdbms.DBConnectionPoolImpl.<init>(DBConnectionPoolImpl.java:171)
        at com.sap.engine.core.configuration.impl.persistence.rdbms.PersistenceHandlerImpl.<init>(PersistenceHandlerImpl.java:50)
        at com.sap.engine.core.configuration.impl.cache.ConfigurationController.<init>(ConfigurationController.java:139)
        at com.sap.engine.core.configuration.bootstrap.ConfigurationManagerBootstrapImpl.init(ConfigurationManagerBootstrapImpl.java:309)
        at com.sap.engine.core.configuration.bootstrap.ConfigurationManagerBootstrapImpl.<init>(ConfigurationManagerBootstrapImpl.java:59)
        at com.sap.engine.bootstrap.StorageAccess.<init>(StorageAccess.java:138)
        at com.sap.engine.bootstrap.Bootstrap.initializeSynchronizer(Bootstrap.java:79)
        at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:273)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:162)
Caused by: com.sap.sql.log.OpenSQLException: Error while accessing secure store: The check record (key "$internal/check") was not found in the sto
re although this store is encrypted..
        at com.sap.sql.log.Syslog.createAndLogOpenSQLException(Syslog.java:104)
        at com.sap.sql.log.Syslog.createAndLogOpenSQLException(Syslog.java:143)
        at com.sap.sql.connect.OpenSQLDataSourceImpl.setDataSourceName(OpenSQLDataSourceImpl.java:143)
        at com.sap.sql.connect.OpenSQLDataSourceImpl.setDataSourceName(OpenSQLDataSourceImpl.java:115)
        at com.sap.engine.core.configuration.impl.persistence.rdbms.DBConnectionPoolImpl.<init>(DBConnectionPoolImpl.java:164)
        ... 12 more
Caused by: com.sap.security.core.server.secstorefs.FileInvalidException: The check record (key "$internal/check") was not found in the store altho
ugh this store is encrypted.
        at com.sap.security.core.server.secstorefs.SecStoreFS.openExistingStore(SecStoreFS.java:2048)
        at com.sap.sql.connect.OpenSQLConnectInfo.getStore(OpenSQLConnectInfo.java:793)
        at com.sap.sql.connect.OpenSQLConnectInfo.lookup(OpenSQLConnectInfo.java:784)
        at com.sap.sql.connect.OpenSQLDataSourceImpl.setDataSourceName(OpenSQLDataSourceImpl.java:128)
        ... 14 more

[Bootstrap]> Error during synchronization. More details: the previous log entries

Do you guys have any clue on this?

Any help would be greatly appreciated.

Many thanks

Jens

[JaySchwendemann]

I'd like to share some updates on this:

We discovered, that the line having "$internal/check" within SecStore.properties was indeed commented out. However we did not do that manually. We removed the comment from that file manually (which might be a mistake) but in doing so we at least got some progress but having some other errors:

We find now following Messages in the dev-jstart-logfile:

* WARNING => SfCEventDispatcher::processQueue: rc 3 (invalid function argument) from SfCJavaProcHandler(server0)([mType=PROC_CMD, mRepl y=TRUE, pca=[cmd=GET_CALLSTACK, procId=2, arg=65], mpEvent= [mType=NODE_CMD, mReply=TRUE, nca=[cmd=GET_CALLSTACK, nodeId=2]]])
[sfxxevt.hpp 3152] F [Thr 01] *** WARNING => SfCNodeManager(server0)::onReply: command 2010 GET_CALLSTACK returned 3 invalid function argument.

Executing sapcontrol -GetProcessList you can see that all precesses are running but jstart is yellow (some processes are running). Checking the message Server logfile you can find:

[Thr 01] Wed Feb 26 14:32:03 2014 [Thr 01] LOGOUT: J2EE106864420 [10.6.1.174] (ENQ) [Thr 01] *** ERROR => MsSSend: NiBufSend (rc=NIECONN_BROKEN) to C1
(JC_rs174QSF_QSF_10) [msxxserv.c 13557] [Thr 01] *** ERROR => MsSSndInfo: cant send info request to C1
(JC_rs174QSF_QSF_10) [msxxserv.c 13778] [Thr 01] LOGOUT: JC_rs174QSF_QSF_10 [10.6.1.174] (DIA )

Any help would really be great.

Thanks and kind regards

Jens