error in downtime phase STOP_ENGINE_DB_DEPLOYMENT (dble-stck upgrade NW 7.3

[rdiger_hckel]

The phases START_SCS and START_J2EE_INITIAL ended with an error but,

after starting the needed instance manually, the upgrade continued.

Now, instance 03 cannot be stopped automatically by the upgrade program.So I stopped the instance 03 manually again (no processes left in

subsystem SAP03BW1) and even did a "cleanipc 03 remove".

So I would like to ignore this error and go to the next phase.

But this requires a password...

Trouble Ticket Report

Upgrade to SAP NetWeaver 730

SID................: BW1

Hostname...........: ap400c

Install directory..: /usr/sap/BW1

Upgrade directory..: /usr/sap/BW1/upg/java

Database...........: DB2 UDB for AS/400

Operating System...: OS400

JDK version........: 1.6.0_07 SAP AG

SAPJup version.....: 2.3.44

Source release.....: 700

Target release.....: 730

Start release SP...: 23

Target release SP..: 5

Current usages.....: AS;BI;EP;EPC;AAS

ABAP stack present.: true

The execution of UPGRADE/DOWNTIME/STOP_ENGINE_DB_DEPLOYMENT ended in error.

Could not stop AS Java instance with name J2EE and number 03 of the standard system.

Could not execute operating system process for action stop on instance 03. Process ID is 744, process name is system.

Return code condition success evaluated to false for process system for action stop.

More information can be found in the log file /usr/sap/BW1/upg/java/log/STOP_ENGINE_DB_DEPLOYMENT_PDD_05.LOG.

I think the problem is that the STOPSAP command uses the parameter

PROFILE (that is not valid anymore).

logfile SYSTEM_STOP_03_PDD_02.ERR:

CPD0043: Schl}sselwort PROFILE f}r diesen Befehl ung}ltig.

CPF0006: Im Befehl ist ein Fehler aufgetreten.

The command looks like this:

Command line: system -e -k RUNLIBCMD LIB('SAPBW1I03') CMD('STOPSAP

SID(BW1) INSTANCE(03) INSTHOST(''ap400c'') WAIT(*YES) WAITTIME(600)

PROFILE(''/usr/sap/BW1/SYS/profile/BW1_DVEBMGS03_ap400c'') TYPE(*DUP) ')

The instance itself has already been stopped manually (no active job in

subsystem SAP03BW1) but due to the 'syntax error' in the command (keyword

PROFILE not valid for this command) the execution stops after one second (without even having tried to stop the instance).

[former_member195641 · Accepted solution]

Hello Rüdiger,

this looks like a bug in the SUM.SAR archive. Please open a customer ticket for the SAP support.

We also need to check why the STARTSAP phases before failed.

Please leave this thread open. I will write down the SUM patch level needed to fix this problem.

Best regards,

René Lang

SAP on IBM i Development

[rdiger_hckel · Accepted solution]

Hello Rene,

I already opened a message:

password for phase STOP_ENGINE_DB_DEPLOYMENT needed ( 1090995 / 2011 )

The two start phases before stopped because the automatic start did not function properly:

At first the command

Command line: system -e -k RUNLIBCMD LIB('SAPBW1I03') CMD('STARTSAP SID(BW1) INSTANCE(03) INSTHOST(''ap400c'') WAIT(YES) WAITTIME(600) PROFILE(''/usr/sap/BW1/SYS/profile/BW1_DVEBMGS03_ap400c'') TYPE(DUP) ')

is tried to be executed.

This fails (I think this might be a kernel/library problem -> see below).

The following command then checks if the instance has come up properly:

Command line: /usr/sap/BW1/DVEBMGS03/exe/sapcontrol -nr 03 -host ap400c -user BW1ADM -function WaitforStarted 3600 120

After having started the instance manually the first command still fails but the sapcontrol command (2nd command) recognizes that the instance is up & running and therefore the upgrade could continue.

This was OK for the START phases but in this STOP phase there is just one command:

Dec 20, 2011 2:59:50 PM : com.sap.sdt.tools.proc.OsProcessController.logProcessInfo(OsProcessController.java:156) [Thread[main,5,main]]: Command line: system -e -k RUNLIBCMD LIB('SAPBW1I03') CMD('STOPSAP SID(BW1) INSTANCE(03) INSTHOST(''ap400c'') WAIT(YES) WAITTIME(600) PROFILE(''/usr/sap/BW1/SYS/profile/BW1_DVEBMGS03_ap400c'') TYPE(DUP) ')

This command just fails and the upgrade stops, no sapcontrol checks the actual state.

Although I have stopped the instance manually this command just fails and brings the upgrade to a halt.

kernel/library problem:

As we can see the command STARTSAP resp. STOPSAP should be executed from the library SAPBW1I03.

This library (called SAP BW (QSECOFR) Objects) has only 14 objects (13x *PGM + 1x *DTAARA) but no *CMD objects at all.

In comparison, the library of the shadow instance (ABAP) named SAPBW1I13 has 230 objects including the STARTSAP and STOPSAP *CMD.

At this point in time of the upgrade the instance 03 is purely considered a JAVA instance (according to the instance profile),

normally instance 03 was a doublestack (ABAP/Java) instance.

SUM.SAR: I can't find this archive so I can't tell you more about it (size + date/timestamp).

Kind regards

Rüdiger

Edited by: Rüdiger Höckel on Dec 21, 2011 10:26 AM

[rdiger_hckel · Accepted solution]

Error cause was that the patch version of sapcpe was too low (which is provided on the SAP upgrade kernel DVDs)

and I had just donwloaded the latest kernel patches for the non-unicode ABAP kernel (and not for the unicode Java/SCS kernel).

So here is the answer from SAP:

Hi Rüdiger,

as discussed via phone,

download latest 720 ile.sar patch (unicode)

sign on as bw1ofr,

run command apysidkrn, provide path to downloaded ile.sar, mode *add,

stop DVEBMGS03 and SCS23

start SCS23, when it is up and running

start DVEBMGS03

check kernel

SAPBW1I23 & SAPBW1I03 whether they contain commands

if yes, continue upgrade, if no return message for further analysis

Thanks and regards Tobias

and here is my answer:

Hello Tobias,

thanks a lot for your call & answers.

In preparation for the next double-stack upgrade on the production

system I'm going to also download the unicode kernel patches (a new

stack.xml has to be made as well).

APYSIDKRN as by adding the followingline (so far no sapcpe command existed):

Execute_00 = immediate $(DIR_CT_RUN)/sapcpe pf=$(_PF)

After executing the start command the sapcpe command built up the

kernellib and then started the instance.

A check with DSPLIB SAPBW1I03 (SAPBW1I23) showed that the library now has

234 objects (instead of just 14 objects).

Right now the upgrade is running again...

UPDATE:

In the next upgrade phase RUN_CONTAINER_MIGRATION the instance 03 should be started (logfile RUN_CONTAINER_MIGRATION_RCM_01.LOG:

Dec 22, 2011 2:54:28 PM :

com.sap.sdt.tools.proc.OsProcessController.logProcessInfo(OsProcessController.java:156)

[Thread[main,5,main]]: Command line: system -e -k RUNLIBCMD

LIB('SAPBW1I03') CMD('STARTSAP SID(BW1) INSTANCE(03) INSTHOST(''ap400c'')

WAIT(*YES) WAITTIME(600)

PROFILE(''/usr/sap/BW1/SYS/profile/BW1_DVEBMGS03_ap400c'') TYPE(*DUP) ')

but the instance doesn't start automatically although the command

STARTSAP is now in the library SAPBW1I03.

So a manual start with type *DUP was done and the

process sapcontrol (to check whether the instance has started) is

satisfied...

[rdiger_hckel]

see my last comment

[rdiger_hckel]

STOP_ENGINE_DB_DEPLOYMENT_PDD_05.LOG:

Dec 20, 2011 2:59:49 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.initialize(AbstractPhaseType.java:751) [Thread[main,5,main]]: Phase UPGRADE/DOWNTIME/STOP_ENGINE_DB_DEPLOYMENT has been started.

Dec 20, 2011 2:59:49 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.initialize(AbstractPhaseType.java:752) [Thread[main,5,main]]: Phase type is com.sap.sdt.j2ee.phases.PhaseTypeControlEngine.

Dec 20, 2011 2:59:49 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.logParameters(AbstractPhaseType.java:403) [Thread[main,5,main]]: Parameter action=stop

Dec 20, 2011 2:59:49 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.logParameters(AbstractPhaseType.java:403) [Thread[main,5,main]]: Parameter instance=J2EE

Dec 20, 2011 2:59:49 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.logParameters(AbstractPhaseType.java:403) [Thread[main,5,main]]: Parameter connect=standard

Dec 20, 2011 2:59:49 PM : com.sap.sdt.j2ee.phases.PhaseTypeControlEngine.checkParameters(PhaseTypeControlEngine.java:522) [Thread[main,5,main]]: Phase parameters have been checked. All required 3 phase parameters have been set.

Dec 20, 2011 2:59:49 PM : com.sap.sdt.j2ee.tools.jprocesspiddetector.JProcessPidDetector.getJProcessPid(JProcessPidDetector.java:84) [Thread[main,5,main]]: Starting to detect the Jcontrol or JStart process ID for instance DVEBMGS03

Dec 20, 2011 2:59:49 PM : com.sap.sdt.j2ee.tools.jprocesspiddetector.JProcessPidDetector.getJProcessPid(JProcessPidDetector.java:94) [Thread[main,5,main]]: Checking file /usr/sap/BW1/DVEBMGS03/work/dev_jstart

Dec 20, 2011 2:59:49 PM : com.sap.sdt.j2ee.tools.jprocesspiddetector.JProcessPidDetector.getJProcessPid(JProcessPidDetector.java:113) [Thread[main,5,main]]: The current release is not controlling the java stack independently from the instance, so no jcontrol/jstart PID is needed here

Dec 20, 2011 2:59:49 PM : com.sap.sdt.j2ee.phases.PhaseTypeControlEngine.getJProcessPid(PhaseTypeControlEngine.java:1255) [Thread[main,5,main]]: The JControl process ID for instance J2EE has been detected.

Dec 20, 2011 2:59:50 PM : com.sap.sdt.tools.proc.OsProcessController.logProcessInfo(OsProcessController.java:154) [Thread[main,5,main]]: Process ID 744, name system has been started.

Dec 20, 2011 2:59:50 PM : com.sap.sdt.tools.proc.OsProcessController.logProcessInfo(OsProcessController.java:156) [Thread[main,5,main]]: Command line: system -e -k RUNLIBCMD LIB('SAPBW1I03') CMD('STOPSAP SID(BW1) INSTANCE(03) INSTHOST(''ap400c'') WAIT(YES) WAITTIME(600) PROFILE(''/usr/sap/BW1/SYS/profile/BW1_DVEBMGS03_ap400c'') TYPE(DUP) ')

Dec 20, 2011 2:59:50 PM : com.sap.sdt.tools.proc.OsProcessController.logProcessInfo(OsProcessController.java:157) [Thread[main,5,main]]: Standard out: /usr/sap/BW1/upg/java/log/SYSTEM_STOP_03_PDD_05.OUT

Dec 20, 2011 2:59:52 PM : com.sap.sdt.tools.proc.OsProcessController.launch(OsProcessController.java:133) [Thread[main,5,main]]: Process ID 744 has been started.

Dec 20, 2011 2:59:52 PM : com.sap.sdt.tools.proc.OsProcessController.waitFor(OsProcessController.java:200) [Thread[main,5,main]]: Waiting for process ID 744, name system to finish.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.tools.proc.OsProcessController.waitFor(OsProcessController.java:235) [Thread[main,5,main]]: Process ID 744, name system has been finished, exit code 255.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.doExecute(AbstractPhaseType.java:883) [Thread[main,5,main]]: Exception has occurred during the execution of the phase.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.tools.procx.DefaultProcessAdapter.checkReturnCode(DefaultProcessAdapter.java:563) [Thread[main,5,main]]: Return code condition success evaluated to false for process system for action stop.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.j2ee.tools.engine.AbstractEngineController.stopEngine(AbstractEngineController.java:842) [Thread[main,5,main]]: Could not execute operating system process for action stop on instance 03. Process ID is 744, process name is system.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.j2ee.phases.PhaseTypeControlEngine.executeAction(PhaseTypeControlEngine.java:607) [Thread[main,5,main]]: Could not stop AS Java instance with name J2EE and number 03 of the standard system.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.cleanup(AbstractPhaseType.java:925) [Thread[main,5,main]]: Phase UPGRADE/DOWNTIME/STOP_ENGINE_DB_DEPLOYMENT has been completed.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.cleanup(AbstractPhaseType.java:926) [Thread[main,5,main]]: Start time: 2011/12/20 14:59:49.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.cleanup(AbstractPhaseType.java:928) [Thread[main,5,main]]: End time: 2011/12/20 14:59:53.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.cleanup(AbstractPhaseType.java:930) [Thread[main,5,main]]: Duration: 0:00:03.711.

Dec 20, 2011 2:59:53 PM : com.sap.sdt.ucp.phases.AbstractPhaseType.cleanup(AbstractPhaseType.java:931) [Thread[main,5,main]]: Phase status is error.

Edited by: Rüdiger Höckel on Dec 20, 2011 4:20 PM