Cannot start ECC after a dummy change

[Former Member]

We have an ECC6(ehp7) which was running well until yesterday.

What we did was renaming its exe as a prepare to upgrade kernel:

However, we’ve found that rs6000_64 already is the newest kernel, so we immediately reverted the action:

mv  re6000_64.old rs6000_64.

But above “dummy” move makes us unable to start SAP. 

A test of dw shows errors below:

1) We find a thread  but our current is already at the highest SP level:

another match:

but our AIX is version 7.

SAP ECC failed to start:

The links are OK:

Thanks for help.

[Reagan · Accepted solution]

Filename, VersionInfo, Time
/usr/sap/AAA/DVEBMGS00/exe/sapstartsrv, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 16:05:22
/usr/sap/AAA/DVEBMGS00/exe/disp+work, 741, patch 13, changelist 1463568, opt (Dec 18 2013, 11:40:35), rs6000_64, 2014 01 03 16:19:35
/usr/sap/AAA/DVEBMGS00/exe/gwrd, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:38
/usr/sap/AAA/DVEBMGS00/exe/msg_server, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:26
/usr/sap/AAA/DVEBMGS00/exe/dboraslib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:45
/usr/sap/AAA/DVEBMGS00/exe/dbdb2slib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:41
/usr/sap/AAA/DVEBMGS00/exe/dbdb4slib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 16:05:34
/usr/sap/AAA/DVEBMGS00/exe/dbdb6slib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:45
/usr/sap/AAA/DVEBMGS00/exe/dbsybslib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:46
/usr/sap/AAA/DVEBMGS00/exe/enserver, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:15
/usr/sap/AAA/DVEBMGS00/exe/icman, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:46
/usr/sap/AAA/DVEBMGS00/exe/sapwebdisp, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:46
/usr/sap/AAA/DVEBMGS00/exe/jcontrol, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:27:35
/usr/sap/AAA/DVEBMGS00/exe/jlaunch, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:53:09
/usr/sap/AAA/DVEBMGS00/exe/jstart, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 16:05:19

I can see that the disp+work is on 741 where as the others are on 740.

Was it like this before ?

Regards

RB

[Reagan · Accepted solution]

/usr/sap/AAA/DVEBMGS00/exe/disp+work, 741, patch 13, changelist 1463568, opt (Dec 18 2013, 11:40:35), rs6000_64, 2014 01 03 16:19:35

It also looks like you are not using the Unicode version of the kernel.

Make sure all the kernel files are correct.

For me it looks like a compatibility of the kernel files.

Regards

RB

[Former Member · Accepted solution]

Reagan:

You are pretty correct. I will investigate. But might only reply by Monday. Have to run.

Thanks!

[former_member215981 · Accepted solution]

Hello,

it looks like the disp+work's version does match others, i suggest use a completed new kernel to see it helps.

Best Regards,

Yong Luo

[Former Member · Accepted solution]

The fix is to replace the entire exe with the newest download from SMP.

Thanks to Reagan and others.

[Reagan]

Hello

Could you check whether the ownership of the files are correct ?

su - sidadm

cdexe

ls -la

If the permissions are OK then try once again the same command you have issued.

./disp+work -version

Also check the output of this

su - sidadm

cdexe

ldd disp+work

Regards

RB

[Former Member]

aaaadm 1> cdexe

aaaadm 2> ls -al d+work

-rwxr-xr-x    1 aaaadm   sapsys    113877995 Jan  3 16:19 disp+work

aaaadm 3> ldd !$

ldd disp+work

disp+work needs:

/usr/sap/AAA/SYS/exe/run/dw_gui.so

/usr/sap/AAA/SYS/exe/run/dw_mdm.so

/usr/sap/AAA/SYS/exe/run/dw_xml.so

/usr/sap/AAA/SYS/exe/run/dw_xtc.so

/usr/sap/AAA/SYS/exe/run/dw_rndrt.so

/usr/sap/AAA/SYS/exe/run/dw_abp.so

/usr/sap/AAA/SYS/exe/run/dw_parse3.so

/usr/sap/AAA/SYS/exe/run/libregex.so

/usr/sap/AAA/SYS/exe/run/dw_stl.so

         /usr/lib/libperfstat.a(shr_64.o)

         /usr/lib/libwlm.a(shr_64.o)

         /usr/lib/libpthread.a(shr_xpg5_64.o)

         /usr/lib/libC.a(shr_64.o)

         /usr/lib/libC.a(shr2_64.o)

         /usr/lib/libC.a(shr3_64.o)

         /usr/lib/libC.a(ansi_64.o)

         /usr/lib/libc.a(shr_64.o)

         /usr/lib/libbsd.a(shr_64.o)

         /usr/lib/libpthreads.a(shr_xpg5_64.o)

         /usr/lib/libcfg.a(shr_64.o)

         /usr/lib/libodm.a(shr_64.o)

         /unix

         /usr/lib/liblvm.a(shr_64.o)

         /usr/lib/libcorcfg.a(shr_64.o)

         /usr/lib/libC.a(shrcore_64.o)

         /usr/lib/libC.a(ansicore_64.o)

         /usr/lib/libcrypt.a(shr_64.o)

         /usr/lib/libsrc.a(shr_64.o)

aaaadm 4>

I do not see issue.

I hide the true SID with AAA.

Thanks!

[Reagan]

Have you tried starting the SAP system ?

If yes what does the trace files say ?

I believe there was no change in the environment variables and also with the files in the /sapmnt/AAA/exe directory.

Is this an AIX machine ?

What is the version of kernel you have now ?

su - aaaadm

sapcontrol -nr 01 -function GetVersionInfo

Note: Replace 01 with the right system number.

Regards

RB

[Former Member]

Thanks a lot!

1) what we did is merely :

mv rs6000_64 rs6000_64.old

mv rs6000_64.old  rs6000_64

then no matter how many times of startsap would not start SAP.

2)

sapcontrol -nr 00 -function GetVersionInfo

04.01.2014 11:41:17

GetVersionInfo

OK

Filename, VersionInfo, Time

/usr/sap/AAA/DVEBMGS00/exe/sapstartsrv, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 16:05:22

/usr/sap/AAA/DVEBMGS00/exe/disp+work, 741, patch 13, changelist 1463568, opt (Dec 18 2013, 11:40:35), rs6000_64, 2014 01 03 16:19:35

/usr/sap/AAA/DVEBMGS00/exe/gwrd, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:38

/usr/sap/AAA/DVEBMGS00/exe/msg_server, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:26

/usr/sap/AAA/DVEBMGS00/exe/dboraslib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:45

/usr/sap/AAA/DVEBMGS00/exe/dbdb2slib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:41

/usr/sap/AAA/DVEBMGS00/exe/dbdb4slib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 16:05:34

/usr/sap/AAA/DVEBMGS00/exe/dbdb6slib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:45

/usr/sap/AAA/DVEBMGS00/exe/dbsybslib.so, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:23:46

/usr/sap/AAA/DVEBMGS00/exe/enserver, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:15

/usr/sap/AAA/DVEBMGS00/exe/icman, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:46

/usr/sap/AAA/DVEBMGS00/exe/sapwebdisp, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:24:46

/usr/sap/AAA/DVEBMGS00/exe/jcontrol, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:27:35

/usr/sap/AAA/DVEBMGS00/exe/jlaunch, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 15:53:09

/usr/sap/AAA/DVEBMGS00/exe/jstart, 740, patch 12, changelist 1406564, optU (Apr 10 2013, 21:22:54), rs6000_64, 2013 04 10 16:05:19

3) sapstart.log

(14942368) **** 2014/01/03 16:45:23 Child 14221510 terminated with Status 255 . ****

(14221510) **** 2014/01/03 16:45:23 No RestartProgram command for program 0  ****

(14942368) **** 2014/01/03 16:52:46 Caught Signal to Stop all Programs. ****

(14942368) Parent Shutdown at 2014/01/03 16:52:48

(14942368) Stop Child Process: 5570580

(14942368) Wait 300 sec for Process 5570580 terminating...

Execute Post-Shutdown Commands

------------------------------

(14942368) Exiting with Return-Code 0. (Ok)