a 'real' procedure for NetWever ABAP stack only KERNEL UPGRADE

[raoul_serra2]

Does SAP make available some where a real procedure

for NetWever 7.0 ABAP stack only KERNEL UPGRADE ?

a lot of people have questions ? answer are even available... but still... even experienced

basis admin having trouble now with Kerne Upgrade

Main line :

stopsap, clean shared memory

stopsap

saposcol -k

ps -ef|grep sapstartsrv

kill <sapstartsrv_pids>

cleanipc <inst_nr>

lsnrctl stop

backup your kernel

cd /sapmnt/<SID>

cp -pr exe exe.001

extract new kernel

cd exe

SAPCAR -xvf <YOUR_DW.SAR>

or

. Login as root

2. Make new backup directory

cd /

mkdir SIDkernel_10102007

3. Create latest kernel backup

cp -pr /sapmnt/SID/exe/* /SIDkernel_10102007

4. Compare the original kernel and the backup

du -sk /sapmnt/SID/exe/  840413 /sapmnt/SID/exe

du -sk /SIDkernel_10102007  841992 /SIDkernel_10102007

5. Make new shadow kernel directory

cd /

mkdir newkernel_175

6. Copy the new downloaded kernel files into /newkernel_175

7. Unpack the new downloaded kernel in folder /newkernel_175

SAPCAR -xvf SAPEXE_175-20000221.SAR

SAPCAR -xvf SAPEXEDB_175-20000223.SAR

8. Login as sidadm

9. Stop the SAP System

stopsap

10. Stop saposcol

saposcol -k

11. Login as root

12. Go to the current kernel directory

cd /sapmnt/SID/exe

13. Copy all extracted support packages into this folder

cp -pr /newkernel_175/* /sapmnt/SID/exe

14. Copy back the original file saproot.sh into this folder.

cp -p /SIDkernel_10102007/saproot.sh /sapmnt/SID/exe

15. Go to the current kernel directory

cd /sapmnt/SID/exe

Then run:

./saproot.sh SID

16. Login as sidadm

17. Start the saposcol service

saposcol

18. Start the SAP system

startsap

AND STILL ... FOLLOWING ALL THE STEPS... I have the work process going down

as soon as i put back old kernel is working fine again

I'm trying to upgrade to :

SAPEXE_179-20000978.SAR

SAPEXEDB_179-20000979.SAR

DBATL700O10_39-20000979.SAR

on an AIX 64 bit with Oracle 10G

Error I'm getting is :

FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

and even the dpmon cannot work

dpmon pf=/usr/sap/H1D/SYS/profile/H1D_DVEBMGS00_dehrdb01

DpMonInit failed - possibly no dispatcher running

So, my QUESTION is : DOES SAP have a detailled procedure to do the Kernel Upgrade on NW7.0

for an ABAP stack only installation ?

[markus_doehr2]

And what I see is:

. Login as root

Then run:

./saproot.sh SID

16. Login as sidadm

17. Start the saposcol service

saposcol

18. Start the SAP system

startsap

It´s clear that it can´t work. You omitted one important step. If you extract/copy files using the user "root" the system can´t open file due to missing permissions.

I would just add an

cd /sapmnt
chown -Rf <sid>adm:sapsys exe

and then run ./saproot.sh .

Markus

[raoul_serra2]

users and group setting was done and ./saproot.sh SID (in our case H1D) did run from user root as well and from the exe path...

by the way... I have 13 years of SAP basis experience... and 1st time I see this

[markus_doehr2]

Error I'm getting is :

FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

What is the full error message from the developer trace?

I´m doing kernel upgrades since more than a decade...

Markus

[raoul_serra2]

Error found in work logs :

06.12.2008 03:06:47

ShmDetach

FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

(2850872) Starting: local ms.sapH1D_DVEBMGS00 pf=/usr/sap/H1D/SYS/profile/H1D_DVEBMGS00_dehrdb01

(2732220) Starting: local dw.sapH1D_DVEBMGS00 pf=/usr/sap/H1D/SYS/profile/H1D_DVEBMGS00_dehrdb01

(868592) Starting: local co.sapH1D_DVEBMGS00 pf=/usr/sap/H1D/SYS/profile/H1D_DVEBMGS00_dehrdb01 -F

(2953220) Starting: local se.sapH1D_DVEBMGS00 pf=/usr/sap/H1D/SYS/profile/H1D_DVEBMGS00_dehrdb01 -F

(2412770) Starting: local ig.sapH1D_DVEBMGS00 -mode=profile pf=/usr/sap/H1D/SYS/profile/H1D_DVEBMGS00_dehrdb01

(2748518) Waiting for Child Processes to terminate.

Instance on host dehrdb01 started

(2748518) **** 2008/12/06 03:08:56 Child 2732220 terminated with Status 0 . ****

(2732220) **** 2008/12/06 03:08:56 No RestartProgram command for program 1 ****

here another one:

M NiWait: sleep (1000ms) ...

M Sat Dec 6 03:08:53 2008

M NiWait: sleep (1000ms) ...

M Sat Dec 6 03:08:54 2008

M DpSigInt: caught signal 2

M SigICallExitRoutine: call exithandler (ThSigHandler)

M ThSigHandler: call ThErrHandle

M in_ThErrHandle: 1

M ThSigHandler: signal (step 1, th_errno 11, action 2)

M ThIErrHandle: new stat of W1 is WP_SHUTDOWN

M ThIErrHandle: I'm during shutdown

M PfStatDisconnect: disconnect statistics

M ThIErrHandle: action changed to 2

M Entering ThSetStatError

M ThShutDownServer: shutdown server

M PfStatIndInit: Initializing Index-Record

M PfWriteIntoFile: copied shared buf (0 bytes) to local buf

M ThIErrHandle: do not call ThrCoreInfo (no_core_info=0, in_dynp_env=0)

M Entering ThReadDetachMode

M call ThrShutDown (1)...

A Saving PXA statistics.

A No programs loaded

B B dbtbxbuf: Saving TBX statistics:

B dbtbxbuf: 0 object entries written.

M ***LOG Q02=> wp_halt, WPStop (Workproc 1 512038) [dpuxtool.c 260]

M Good Bye .....

and another...

[Thr 1] MsHttpOwnDomain: own domain[1] = emea.int.tenneco.com

[Thr 1] ms/icf_info_server : deleted

[Thr 1] *** I listen to port sapmsH1D (3600) ***

[Thr 1] *** I listen to internal port 3900 (3900) ***

[Thr 1] CUSTOMER KEY: >P1592267744<

[Thr 1] Sat Dec 6 03:08:55 2008

[Thr 1] *** ERROR => MsSClientHandle: no server provides service ENQ (4), requested from dehrdb01_H1D_00 [ms

xxserv_mt. 4684]

and this one:

more sapcpe.log

================ >>> main =======================

• INFO: sapcpe version: @(#) $Id: //bas/700_REL/src/ccm/sapstart/sapxxcpe.c#18 $ SAP

=================================================

• INFO: Try to find file table sapcpeft (default:sapcpeft)

• INFO: Try to find file table in the central directory /usr/sap/H1D/SYS/exe/ctrun/sapcpeft

• INFO: not found any file table.

=================================================

Update Local Programs (dir-sync + file-path + ln)

=================================================

Starting at 2008/12/06 03:06:47

• ERROR: can't change to central Executable-Dir /usr/sap/H1D/SYS/exe/ctrun (No such file or directory) [sap

xxcpe.c:1488]

or this one too :

[Thr 1] * LOCATION ICM runing on host dehrdb01

[Thr 1] * ERROR dp lib version mismatch, please check installation

[Thr 1] *

• TIME Sat Dec 6 03:06:52 2008

[Thr 1] * RELEASE 700

[Thr 1] * COMPONENT DP

[Thr 1] * VERSION 241000

[Thr 1] * RC -9999

[Thr 1] * MODULE dpxxtool2_mt.c

[Thr 1] * LINE 688

[Thr 1] * DETAIL version of external and kernel dp lib mismatch (ext lib

• version is 241000 / kernel lib version is 232000)

[Thr 1] * COUNTER 1

[Thr 1] *

[Thr 1] *****************************************************************************

[Thr 1] *** ERROR => DpIPCInit2: DpSysAdmExtCreate [dpxxtool2_mt 369]

[Thr 1] *** ERROR => IcmInit: IcmIPCInit failed (rc=-9999) [icxxman_mt.c 1741]

[Thr 1] *** ICM shutdown completed (pid: 2539700) ***

and sappfpar has no errors

we 1st tried this without success:

SAPEXE_179-20000978.SAR

SAPEXEDB_179-20000979.SAR

DBATL700O10_39-20000979.SAR

the we add the following with same result and revert back to kernel 146 (that one works fine)

SAPCPE_3-20000978.SAR

LIBRFC_41-20000978.SAR

CPICTLIB_11-20000978.SAR

DW_187-20000978.SAR

IGSEXE_13-10003077.SAR

[markus_doehr2]

or this one too :

* LOCATION ICM runing on host dehrdb01

* ERROR dp lib version mismatch, please check installation

*

• TIME Sat Dec 6 03:06:52 2008

* RELEASE 700

* COMPONENT DP

* VERSION 241000

* RC -9999

* MODULE dpxxtool2_mt.c

* LINE 688

* DETAIL version of external and kernel dp lib mismatch (ext lib

• version is 241000 / kernel lib version is 232000)

Does the same error also occur if you run "slibclean" as root just before you start the instance?

If yes then I´d say the kernel is wrong. I would try to uncar

- SAPEXE

- SAPEXEDB

- DBTOOLS*

and then use the latest

DW*

and try again.

Markus

[raoul_serra2]

root> /usr/sbin/slibclean

yes this command was used in all the attempts

this upgrade has been tried with 2 of the differents latest available kernels...

are you suggesting that maybe the problem could be the SAPCAR itself ?

but then someone would have to explain why the backup of the old kernel

made with the same SAPCAR was packed and unpacked again and still works fine

This is what we do with old kernel...

Logon as root

> cd /sapmnt/[SID]/exe

we create an archive with all files and directories from actual directory

into specified path without point files:

SAPCAR -cvf /sapmnt/[SID]/old_SAP_kernel.SAR *

Then as new kernel was not a success we brought back our archive

SAPCAR -xvf /sapmnt/H1D/old_SAP_kernel.SAR

It does not cost a lot to try donwload the latest SAPCAR but I'm not convinced

Kind regards,

R.S.

[raoul_serra2]

Here is the full procedure for SID = H1D

PROCEDURE for system H1D

h1dadm>

stopsap

saposcol -k

ps -ef|grep adm

kill <remaining sap process>

cleanipc <inst_nr>

to revert back to old kernel

cd /sapmnt/H1D/exe

/sapmnt/SAPCAR -xvf /sapmnt/H1D/old_SAP_kernel.SAR

root>

/usr/sbin/slibclean

cd /sapmnt/H1D

cp -pr exe exe.old

OR use SAPCAR to backup old Kernel

[ download latest SAPCAR and copy it to /sapmnt/ ]

cd /sapmnt/H1D/exe

/sapmnt/SAPCAR -cvf /sapmnt/H1D/old_SAP_kernel.SAR *

(extract new kernel - user root)

cd /sapmnt/H1D

mkdir kernel179

cd /sapmnt/H1D/kernel179

/sapmnt/SAPCAR -xvf /sapmnt/SAPEXE_179-20000978.SAR

/sapmnt/SAPCAR -xvf /sapmnt/SAPEXEDB_179-20000979.SAR

/sapmnt/SAPCAR -xvf /sapmnt/DBATL700O10_39-20000979.SAR

/sapmnt/SAPCAR -xvf /sapmnt/SAPCPE_3-20000978.SAR

/sapmnt/SAPCAR -xvf /sapmnt/LIBRFC_41-20000978.SAR

/sapmnt/SAPCAR -xvf /sapmnt/CPICTLIB_11-20000978.SAR

/sapmnt/SAPCAR -xvf /sapmnt/DW_187-20000978.SAR

/sapmnt/SAPCAR -xvf /sapmnt/IGSEXE_13-10003077.SAR

cd ..

chown -Rf h1dadm:sapsys kernel179

cp -pr ./kernel179/* /sapmnt/H1D/exe/

cd /sapmnt/H1D/exe

./saproot.sh H1D

/usr/sbin/slibclean

h1dadm>

saposcol -l

startsap

1. Despite the fact this was done again with the latest SAPCAR available for AIX 64 / Netwearver 7.0

the result is still the same : the work processes (dw.*) are dieing

this makes me MAD the old kenel that still work fine is 146 - the new is 179 plus "dw 187"

Kind regards,

Raoul

[raoul_serra2]

MORE details about the error itself :

The message server disconnect, the error seems to be in the dispatcher :

The Error :

Versions of dplib are different (sys_adm lib= / lib 2410)

dpxxtool 681# lib=241000 krn=232000 DpSysAd bad version

The dplib version defined in the structure sys_adm dplib is different

to your version. This is because the version status of the logged on

program is different to the SAP kernel. Check the version of

the execute programs.

Error (Function,Module,Row) Q0 Z lib=241000 krn=232000 DpSysAdbad verdpxxtool68

[markus_doehr2]

cleanipc <inst_nr>

Just to add: this does nothing. The program must be called as root using

cleanip <systemnumber> remove

in order to really clean up the shared memory segments.

Before you start the new kernel, what´s the output of

ipcs -m

Markus

[raoul_serra2]

ipcs only shows a few root entry, nothing from sidadm, memory was cleared

I created a OSS message and I'm suggesting to focus on the error message

with the dispatcher :

The Error :

Versions of dplib are different (sys_adm lib= / lib 2410)

dpxxtool 681# lib=241000 krn=232000 DpSysAd bad ver

The dplib version defined in the structure sys_adm dplib is different

to your version. This is because the version status of the logged on

program is different to the SAP kernel. Check the version of

the execute programs.

Error (Function,Module,Row) Q0 Z lib=241000 krn=232000 DpSysAd bad ver dpxxtool 681

[raoul_serra2]

Please note that the procedure remains the same for getting back to old kernel 146

and there we have no problems at all

Here answer to your question about memory:

ipcs only shows a few root entry, nothing from sidadm, memory was cleared

I created a OSS message and I'm suggesting to focus on the error message

with the dispatcher :

The Error :

Versions of dplib are different (sys_adm lib= / lib 2410)

dpxxtool 681# lib=241000 krn=232000 DpSysAd bad ver

The dplib version defined in the structure sys_adm dplib is different

to your version. This is because the version status of the logged on

program is different to the SAP kernel. Check the version of

the execute programs.

Error (Function,Module,Row) Q0 Z lib=241000 krn=232000 DpSysAd bad ver dpxxtool 681

[markus_doehr2]

> I created a OSS message and I'm suggesting to focus on the error message

> with the dispatcher :

> The Error :

> Versions of dplib are different (sys_adm lib= / lib 2410)

That is a kernel internal problem. The versions in the kernel itself are not consistent. Only the development will be able to comment on this.

Markus

[raoul_serra2]

Thus, any one who wants to upgrade kernel to 179 will have this problem on AIX 64 / Oracle platform

and as this kernel version is also include in Stack 14... well... SAP will start to have some complains I guess ... and seems tha DW187 is not solving this issue neither

[raoul_serra2]

I've got an answer from SAP to my customer message and what they suggest is this :

cp -r ./kernelNEW/ /sapmnt/[SID]/exe/* (that is what I did till now)

and add the extra step just after :

cp -r ./kernelNEW/ /usr/sap/[SID]/DVEBMGS<NR>/exe*

This is suggesting that sapcpe it's not working or at least does not

do the required copy from /sapmnt/SID/exe into

usr/sap/SID/DVEBMSG<nr>/exe ?

as the only real big difference in the steps is that they did add a

manual copy to usr/sap/SID/DVEBMSG<nr>/exe

I will try this extra manual copy. I see that not all the files

in /sapmnt/SID/exe are duplicated in usr/sap/SID/DVEBMSG<nr>/exe ... is

there any specif rule to be aware ? br* files from DBATOOL are not there

[raoul_serra2]

or... is sapcpe not working because it needs a parameter to know that

he should do that 1st time copy from cdexe into instance/exe ?

[raoul_serra2]

Here we go, now my problem is solved.

Explanation for any one interested in it is :

the SAP instance where I was trying to do a Kernel Upgrade is one that has been upgraded

from 46C to NW 7.0 / ERP 6.0 by some external consultant.

NW 7.0 have a new structure for the EXE directory

a program is supposed to synchronize /sapmnt/SID/exe with /usr/sap/SID/DVEBMGS<num>/exe

the program used for this is "sapcpe" and it needs some extra parameters in the SAP "start profile"

When this is working fine sapcpe at start of SAP will replicate the kernel in the instance directory

this was not happening in my case :

Solution in case of emergency :

just add a manual copy of the kernel files into usr/sap/SID/DVEBMGS<num>/exe

Final fix:

Check OSS Note 1104735, and make sure the sapcpe replicate mechanism works fine in the future

Hope this will help some of you

Kind regards,

Raoul