SAP System down : not starting up

[Former Member]

Hi Folks ,

We are facing a problem.

Our quality system was down .

When I checked ps-ef | grep sap : the processes are running but it is not allowing us to login.

we tried re-starting the system but that didn't worked.

I checked the processes with dpmon cmd , many process are free, still not sure on why it is not allowing us to login.

Need your guidance in moving further

Thanks

Regards

Sandy

[Former Member]

Sandy ,

Did you check if all the mountpoints has sufficient space to write the logs.Check for mountpoint space and the oraarch .

If oraarch and all mountpoints has sufficient spaces then please paste dev_w0 log.

Thanks..

Mohit

[Former Member]

Please paste the content of dev_w* files from work directory.

If there are no such latest files, you can provide dev_disp file contents.

These are the files where you can start the troubleshooting.

Regards,

rajneesh

[Former Member]

Hi ,

This is what I can see

-

---

trc file: "dev_disp.new", trc level: 1, release: "701"

-

---

sysno 00

sid QA2

systemid 370 (Solaris on SPARCV9 CPU)

relno 7010

patchlevel 0

patchno 32

intno 20020600

make: single threaded, Unicode, 64 bit, optimized

pid 23214

Sun Apr 10 23:46:43 2011

kernel runs with dp version 241000(ext=110000) (@(#) DPLIB-INT-VERSION-241000-UC)

length of sys_adm_ext is 576 bytes

• SWITCH TRC-HIDE on ***

***LOG Q00=> DpSapEnvInit, DPStart (00 23214) [dpxxdisp.c 1286]

shared lib "dw_xml.so" version 32 successfully loaded

shared lib "dw_xtc.so" version 32 successfully loaded

shared lib "dw_stl.so" version 32 successfully loaded

shared lib "dw_gui.so" version 32 successfully loaded

shared lib "dw_mdm.so" version 32 successfully loaded

rdisp/softcancel_sequence : -> 0,5,-1

use internal message server connection to port 13902

Sun Apr 10 23:46:44 2011

MtxInit: 30000 0 0

• ERROR => DpSysAdmExtCreate: ShmCreate (CREATE) [dpxxtool2.c 608]

• ERROR => dispatcher already running ???? [dpxxtool2.c 609]

• ERROR => I better EXIT before I do any (more) damage [dpxxtool2.c 611]

[Former Member]

Sandy ,

Please paste dev_w0 logs.

Thanks..

Mohit

[Former Member]

Hi Mohit,

I checked all the mount points and they are fine .

Please find the below dev_w0 log

Sat Apr 9 02:13:11 2011

C Logon as OPS$-user to get SAPR3's password

C Connecting as /@QA2 on connection 1 (nls_hdl 0) ... (dbsl 700 151208)

C Nls CharacterSet NationalCharSet C EnvHp ErrHp ErrHpBatch

C 0 UTF8 0 0x1073cf120 0x1073d6c30 0x1073e78

C Starting user session: OCISessionBegin(con_hdl=1, usr='/',svchp=0x10b1de0b0, srvhp=0x1073ea2c8, )

C OCI-call failed with -1=OCI_ERROR

C SQL error 942: 'ORA-00942: table or view does not exist'

C Now '/@QA2' is connected: con_hdl=1, nls_hdl=0, session_id=-1.

C Got SAPR3's password from OPS$-user

C Disconnecting from connection 1 ...

C Closing user session (con_hdl=1,svchp=0x10b1de0b0,usrhp=0x108c3a4d8)

C Now I'm disconnected from ORACLE

C Connecting as SAPR3/<pwd>@QA2 on connection 1 (nls_hdl 0) ... (dbsl 700 151208)

C Nls CharacterSet NationalCharSet C EnvHp ErrHp ErrHpBatch

C 0 UTF8 0 0x1073cf120 0x1073d6c30 0x1073e78

C Starting user session: OCISessionBegin(con_hdl=1, usr=SAPR3/<pwd>, svchp=0x10b1de0b0, srvhp=0x10)

C Now 'SAPR3/<pwd>@QA2' is connected: con_hdl=1, nls_hdl=0, session_id=525.

C DB instance QA2 is running on sapqa2 with ORACLE version 10.2.0.4.0 since JAN 31, 2011, 11:09:13

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20110323 143908 sapqa2

B 000 001 R/3*INACT_PACK 000000227 ACTIVE NO NO NO 004 255 255 20110409 021311 sapqa2

B 000 002 R/3*EXPRESSPOPUP 000000211 DISCONNECTED NO NO NO 000 255 255 20110406 170059 sapqa2

B 000 003 R/3*ABAP_DEBUGGE 000000220 DISCONNECTED NO NO NO 004 255 255 20110408 001610 sapqa2

C

C Sat Apr 9 02:16:06 2011

C CbApplInfoGet() failed! Ignore, but uninstall callback to avoid more erroneous calls

C

C Sat Apr 9 02:31:06 2011

C Disconnecting from connection 1 ...

C Closing user session (con_hdl=1,svchp=0x10b1de0b0,usrhp=0x108c3a4d8)

C Now I'm disconnected from ORACLE

B Disconnected from connection con_da={R/3*INACT_PACK,227,1}

N

N Sat Apr 9 03:01:26 2011

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

C

C Sat Apr 9 03:44:29 2011

C build_stmt: reallocating stmt buffer: 3072 -> 4074 characters

Let me know if you need any more details

[Former Member]

Hi Sandy ,

It seems that there is some issue with DB connection.Please run R3trans -d .Check the error code of same and if its failing with any error code other than 0 then please paste trans.log to assist you further.

Thanks..

Mohit

[Former Member]

Hi Mohit ,

I already checked that and it is fine.

This is R3trans version 6.14 (release 701 - 23.07.10 - 09:23:00).

unicode enabled version

R3trans finished (0000).

[Former Member]

N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23
N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23
N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23
N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23
N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23
N Ssf_GetOwnCertificate: SsfOpenProfile failed with rc=23

Are you running system with ABAP and java as addin.It seems there is some issue with certificate.You need to look at if certificatte is expired.Also please let us know what was last acitvity performed before system went down?

Thanks..

Mohit

[Former Member]

use internal message server connection to port 13902

Could you please check why message server is connecting to 13902 port.It should connect to to 36XX where xx is your instance no.

Thanks..

Mohit

[Former Member]

Hi Mohit and All,

Thanks for your valuable help.

The issue was resolved now .

It was due to shared memory (semaphore). We cleaned that and the system is up and running

Regards

Sandy

[Former Member]

Resolved