SAP not starting after crash: Terminateing With error code 12

[Former Member]

Hi,

After system crash SAP is not starting normally. When running startsap under SIDadm puts out: /usr/sap/TST/SYS/exe/run/startdb: Terminateing With error code 12

startdb.log shows that:

/oracle/TST/102_64/bin/sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shares object file: No such file or directory

I have checked - file and directory are present.

However - when manually starting DB under oraSID and then logging in as SIDadm and starting SAP - everything starts normally.

(Allthough SID is has been changed before - system also worked normally after SID change) I have checked userrights and groups , also profile - everything seems to be normal. Any ideas?

Edited by: Kristjan T on Mar 16, 2011 1:16 PM

[Former Member · Accepted solution]

Hi,
> 
> After system crash SAP is not starting normally. When running startsap under SIDadm puts out: /usr/sap/TST/SYS/exe/run/startdb: Terminateing With error code 12
> 
> startdb.log shows that: 
> /oracle/TST/102_64/bin/sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shares object file: No such file or directory
> 
> I have checked - file and directory are present.
> 
> However - when manually starting DB under oraSID and then logging in as SIDadm and starting SAP - everything starts normally.
> (Allthough SID is has been changed before - system also worked normally after SID change) I have checked userrights and groups , also profile - everything seems to be normal. Any ideas?
> 
> Edited by: Kristjan T on Mar 16, 2011 1:16 PM

Hi Kristjan,

Now that you already checked "LD_LIBRARY_PATH", as Markus stated, please consider the items, below;

• Check the location of "libsqlplus.so", by executing "locate libsqlplus.so" to the permissions

• executed the "root.sh"?

• Check permission of "$ORACLE_HOME"

• Under "$ORACLE_HOME/bin" and execute "relink all"

Best regards,

Orkun Gedik

[Former Member · Accepted solution]

Hi,

Sorry for not giving feedback sooner.

"relink all" did the trick

Problem solved - Thank you very much!

[Former Member]

I have noticed that RZ10 shows some old profiles to be active that do not exist in the system nomore. Could this be the issue?

(All config files have been changed in the system however so startsap is including right profiles that are not shown in RZ10)

Br,

Kristjan T.

[volker_borowski2]

Hi,

what UNIX Flavor are we talking about ?

What SAP Kernel Release is in place (may be 6.40 non EX2 ?)

What release is the DB suposed to have?

I see, you are calling an sqlplus from a 10.2 ORACLE_HOME.

Is that the correct HOME the DB is running with ?

Can you start the DB with

brconnect -c -u / -f dbstart

As already requested, what is the output of:

ldd sqlplus

in addition:

env | grep -i ora

env | grep -i nls

env | grep -i lib

Volker

[Former Member]

Hi,

1.what UNIX Flavor are we talking about -> SuSe

2.What SAP Kernel Release is in place (may be 6.40 non EX2 ?) -> 6.40

3.What release is the DB suposed to have? -> 10.2.0.2.0 - 64bit Prod

4.I see, you are calling an sqlplus from a 10.2 ORACLE_HOME.

Is that the correct HOME the DB is running with ? -> Yes

5.Can you start the DB with

brconnect -c -u / -f dbstart -> NO

Error is:

#################################################

:devadm 51> brconnect -c -u / -f dbstart

BR0801I BRCONNECT 7.00 (40)

oracleDEV: error while loading shared libraries: libskgxp10.so: cannot open shared object file: No such file or directory

BR0278E Command output of '/oracle/DEV/102_64/bin/sqlplus /nolog':

/oracle/DEV/102_64/bin/sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory

BR0280I BRCONNECT time stamp: 2011-05-05 15.01.17

BR0279E Return code from '/oracle/DEV/102_64/bin/sqlplus /nolog': 127

BR0302E SQLPLUS call for database instance DEV failed

BR0303E Determination of Oracle version failed

BR0280I BRCONNECT time stamp: 2011-05-05 15.01.17

BR0804I BRCONNECT terminated with errors

#################################################

(Different machine but same situation)

6.As already requested, what is the output of:

6.1.ldd sqlplus

:devadm 55> ldd sqlplus

ldd: ./sqlplus: No such file or directory

in addition:

6.2. env | grep -i ora

devadm 56> env | grep -i ora

PATH=/oracle/DEV/102_64/bin:/opt/IBMJava2-amd64-142/bin:.:/home/devadm:/usr/sap/DEV/SYS/exe/run:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/devadm/bin:/usr/games:/opt/gnome/bin:/opt/kde3/bin:/usr/bin/X11:/usr/lib/mit/bin:/usr/lib/mit/sbin

LD_LIBRARY_PATH=/usr/sap/DEV/SYS/exe/run:/oracle/client/10x_64/instantclient

dbms_type=ORA

dbs_ora_tnsname=DEV

dbs_ora_schema=SAPSR3

ORACLE_SID=DEV

ORACLE_BASE=/oracle

TNS_ADMIN=/usr/sap/DEV/SYS/profile/oracle

ORACLE_HOME=/oracle/DEV/102_64

SAPDATA_HOME=/oracle/DEV

6.3 env | grep -i nls

:devadm 57> env | grep -i nls

XNLSPATH=/usr/X11R6/lib/X11/nls

NLS_LANG=AMERICAN_AMERICA.UTF8

6.4 env | grep -i lib

devadm 58> env | grep -i lib

PATH=/oracle/DEV/102_64/bin:/opt/IBMJava2-amd64-142/bin:.:/home/devadm:/usr/sap/DEV/SYS/exe/run:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/devadm/bin:/usr/games:/opt/gnome/bin:/opt/kde3/bin:/usr/bin/X11:/usr/lib/mit/bin:/usr/lib/mit/sbin

XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB

XNLSPATH=/usr/X11R6/lib/X11/nls

PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig:/opt/kde3/lib64/pkgconfig:/opt/gnome/lib64/pkgconfig:/opt/gnome/lib64/pkgconfig:/opt/gnome/share/pkgconfig

GTK_PATH=/usr/local/lib/gtk-2.0:/opt/gnome/lib/gtk-2.0:/usr/lib/gtk-2.0

GTK_PATH64=/usr/local/lib64/gtk-2.0:/opt/gnome/lib64/gtk-2.0:/usr/lib64/gtk-2.0

DIR_LIBRARY=/usr/sap/DEV/SYS/exe/run

LD_LIBRARY_PATH=/usr/sap/DEV/SYS/exe/run:/oracle/client/10x_64/instantclient

Any ideas???

Br,

Kristjan T.

[Former Member]

I even ran

find . -type l | (while read FN ; do test -e "$FN" || ls -ld "$FN"; done)

to find any broken symlinks concerning SAP or Oracle - no go... Everything seems to be in order...

[volker_borowski2]

> 2.What SAP Kernel Release is in place (may be 6.40 non EX2 ?) -> 6.40
> 3.What release is the DB suposed to have? -> 10.2.0.2.0 - 64bit Prod 
> devadm 56> env | grep -i ora
> PATH=/oracle/DEV/102_64/bin:/opt/IBMJava2-amd64-142/bin:.:/home/devadm:/usr/sap/DEV/SYS/exe/run:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/devadm/bin:/usr/games:/opt/gnome/bin:/opt/kde3/bin:/usr/bin/X11:/usr/lib/mit/bin:/usr/lib/mit/sbin
> LD_LIBRARY_PATH=/usr/sap/DEV/SYS/exe/run:/oracle/client/10x_64/instantclient
> 6.3 env | grep -i nls
> :devadm 57> env | grep -i nls
> XNLSPATH=/usr/X11R6/lib/X11/nls
> NLS_LANG=AMERICAN_AMERICA.UTF8
> Any ideas???
> 
> Br,
> Kristjan T.

Hi,

In fact yes, I have an Idea!

If your Kernel is really 6.40, you environment is wrong.

6.40 is linked against a version 9 oracle client.

So sidadm needs ORA_NLS33 and should NOT use the instantclient Version 10 in the shared libs.

This environment looks like a 6.40 EX2 kernel situation.

The 6.40 EX2 kernel is linked against the insantclient.

Could it be, that the system was running 6.40 EX2 and

was "downgraded" accidently with a non-EX2 Kernelpatch ?

This would explain the observed behavior.

Check out the several SAP notes explaining the different kernel and client situations.

Get yourself the note explaining the 6.40 EX2 implementation and re-process it.

Good hunting

Volker

[markus_doehr2]

> (Allthough SID is has been changed before - system also worked normally after SID change) I have checked userrights and groups , also profile - everything seems to be normal. Any ideas?

How did you do that "SID change"? You're aware that this is not a supported procedure if you don't use the official system copy procedure?

It appears to me that there is LD_LIBRARY_PATH not properly set for ora<SID> and <SID>adm.

Markus

[Former Member]

Hi,

(We found a manual for SID change) After SID change sap has worked normally...

Allthough I can not remember did we start it manually (instance and DB sepperately) or automatically (startsap). Which files should I check?

(It seems that problem is with SIDadm profile....)

Edited by: Kristjan T on Mar 16, 2011 1:48 PM

[markus_doehr2]

> (We found a manual for SID change) After SID change sap has worked normally...

Ok - then you should contact the person who wrote this manual since it seems to be incomplete

> Allthough I can not remember did we start it manually (instance and DB sepperately) or automatically (startsap). Which files should I check?

> (It seems that problem is with SIDadm profile....)

check the "dot"-files

.dbenv.sh|csh
.dbenv_<hostname>.sh|csh
.sapenv.sh|csh
.sapenv_<hostname>.sh|csh

and check the LD_LIBRARY_PATH includes the PATH to libsqlplus.so (and the database client).

Markus

[Former Member]

Hi,

I have checked the backup files.. - that problem dates back to prior SID change. So SID change is not the issue...

Also LD_LIBRARY path is the same in every file. Checked also differences between oraSID and SIDadm profile LD paths - same. Any ideas, what to check next?

Could it be that somehow su - oraSID does not work for SIDadm?

Could it help if i'd assign SIDadm to dba group?

Edited by: Kristjan T on Mar 17, 2011 10:35 AM

[audunlea_hansen]

Hi!

> Could it be that somehow su - oraSID does not work for SIDadm?

Yes, that is possible. I've just been there myself, tried to do a su - <SID>adm from root -> Not working

When I tried a new logon to the server with ssh <SID>adm@host, startsap worked.

I have seen before not all environment are loaded when doing a su to the user.

Regards

Audun

[audunlea_hansen]

Hi!

Check you have permission to access the file(s) as <SID>adm and not only as ora<SID>

Test 'l dd sqlplus ' with both users.

Regards

Audun

DBA

[Former Member]

Hi,

check environment variables too.