Unable to start the SAP instance

[former_member199632]

Hi,

in DEV server, when login to SAP it gave this error! and unable to login to the system

sql error in the database when accessing a table sap

Then restarted the server and sap instance

when performing the restart it indicates DB and R3 instance started up correctly.

But it doesn't start the SAP instance.

What would be the reason for this.

erpdev:devadm 51> startsap

Checking DEV Database
------------------------------
 ABAP Database is running

Starting SAP-Collector Daemon
------------------------------
12:19:16 17.03.2015   LOG: Effective User Id is root
***********************************************************************
* This is Saposcol Version COLL 20.95 700 - v2.16, AMD/Intel x86_64 with Linux, 2008/08/28
* Usage:  saposcol -l: Start OS Collector
*         saposcol -k: Stop  OS Collector
*         saposcol -d: OS Collector Dialog Mode
*         saposcol -s: OS Collector Status
* The OS Collector (PID 6086) is already running .....
************************************************************************
 saposcol already running

Starting SAP Instance DVEBMGS00
------------------------------
 Startup-Log is written to /home/devadm/startsap_DVEBMGS00.log
 Instance Service on host erpdev started
 Instance on host erpdev started

Even its indicate as Instance on host erpdev started , it is NOT started.

erpdev:devadm 52> ps -aef | grep dw
devadm   11443 11420  1 12:19 ?        00:00:00 dw.sapDEV_DVEBMGS00 pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev
(for few seconds , it shows above line and disappears)


erpdev:devadm 54> ps -aef | grep dw
devadm   11625 10952  0 12:20 pts/6    00:00:00 grep dw

It starts the DB, and shows oracle processes are running

Any ideas, how to solve this!

regards,

Zerandib

[alwina_enns · Accepted solution]

Hello Zerandib,

what is logged in the trace /home/devadm/startsap_DVEBMGS00.log?

Regards,
Alwina

[former_member199632 · Accepted solution]

Hi Alwina,

pl check.

i have attached startsap_DVEBMGS00.log

Trace of system startup/check of SAP System DEV on Tue Mar 17 12:00:42 IST 2015


Called command: /usr/sap/DEV/SYS/exe/run/startsap DVEBMGS00

Starting SAP Instance DVEBMGS00
------------------------------
Instance Service on host erpdev started

SAP-R/3-Startup Program Rel 700 V1.8 (2003/04/24)
-------------------------------------------------

Starting at 2015/03/17 12:00:42
Startup Profile: "/usr/sap/DEV/SYS/profile/START_DVEBMGS00_erpdev"

Setup Environment Variables
---------------------------
(9909) SETENV LD_LIBRARY_PATH=/usr/sap/DEV/DVEBMGS00/exe:/usr/sap/DEV/DVEBMGS00/exe:/usr/sap/DEV/SYS/exe/run:/oracle/client/10x_64/instantclient
(9909) SETENV SHLIB_PATH=/usr/sap/DEV/DVEBMGS00/exe:
(9909) SETENV LIBPATH=/usr/sap/DEV/DVEBMGS00/exe:

Update local Kernel Files
-------------------------
(9910) Local: /usr/sap/DEV/SYS/exe/run/sapcpe name=DEV
(9909) system(/usr/sap/DEV/SYS/exe/run/sapcpe name=DEV) returns 1
(9909) Return-Code 1 in Local-Kernel-Update. See sapcpe.log.

Execute Pre-Startup Commands
----------------------------
(9912) Local: /usr/sap/DEV/SYS/exe/run/sapcpe pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev
(9909) system(/usr/sap/DEV/SYS/exe/run/sapcpe pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev) returns 4
(9918) Local: /usr/sap/DEV/DVEBMGS00/exe/sapmscsa pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev -n
/usr/sap/DEV/DVEBMGS00/exe/sapmscsa: SCSA is attached and useable.
/usr/sap/DEV/DVEBMGS00/exe/sapmscsa: finished.
(9919) Local: rm -f ms.sapDEV_DVEBMGS00
(9920) Local: ln -s -f /usr/sap/DEV/DVEBMGS00/exe/msg_server ms.sapDEV_DVEBMGS00
(9921) Local: rm -f dw.sapDEV_DVEBMGS00
(9922) Local: ln -s -f /usr/sap/DEV/DVEBMGS00/exe/disp+work dw.sapDEV_DVEBMGS00
(9923) Local: rm -f co.sapDEV_DVEBMGS00
(9924) Local: ln -s -f /usr/sap/DEV/DVEBMGS00/exe/rslgcoll co.sapDEV_DVEBMGS00
(9925) Local: rm -f se.sapDEV_DVEBMGS00
(9926) Local: ln -s -f /usr/sap/DEV/DVEBMGS00/exe/rslgsend se.sapDEV_DVEBMGS00
(9927) Local: rm -f ig.sapDEV_DVEBMGS00
(9928) Local: ln -s -f /usr/sap/DEV/DVEBMGS00/exe/igswd_mt ig.sapDEV_DVEBMGS00

Starting Programs
-----------------

17.03.2015 12:00:42
ShmDetach
OK
(9931) Starting: local ms.sapDEV_DVEBMGS00 pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev
(9932) Starting: local dw.sapDEV_DVEBMGS00 pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev
(9933) Starting: local co.sapDEV_DVEBMGS00 pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev -F
(9934) Starting: local se.sapDEV_DVEBMGS00 pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev -F
(9935) Starting: local ig.sapDEV_DVEBMGS00 -mode=profile pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00_erpdev
(9909) Waiting for Child Processes to terminate.
Instance on host erpdev started
(9909) **** 2015/03/17 12:01:18 Child 9932 terminated with Status 0 . ****
(9932) **** 2015/03/17 12:01:18 No RestartProgram command for program 1  ****

Regards,

zerandib

[former_member185954 · Accepted solution]

Hello Zerandib

Ensuring your database is up and running, can you execute the command R3trans -d and attach the content of trans.log that is created in the same directory.

Regards,

Siddhesh

[alwina_enns · Accepted solution]

Hello Zerandib,

this process has terminated:

9932) Starting: local dw.sapDEV_DVEBMGS00 pf=/usr/sap/DEV/SYS/profile/DEV_DVEBMGS00

..................................................

9909) **** 2015/03/17 12:01:18 Child 9932 terminated with Status 0 . ****

This is the dispatcher, could you please attach dev_disp from the working directory of the instance (DIR_HOME)?

Regards,
Alwina

[Reagan · Accepted solution]

Stop all processes running under <SID>adm from the OS level and run cleanipc.

Start SAP and if it fails then check the logs (dev-disp, dev_w0) in the /usr/sap/DEV/DVEBMGS00/work directory.

[former_member199632 · Accepted solution]

Hi Alwina,

I have attached dev_disp log

zerandib

dev_disp.txt.zip

[former_member199632 · Accepted solution]

Hi Reagan,

i hv attached dev_disp & dev_w0

regards,

zerandib

dev_w0.txt.zip

dev_disp.txt.zip

[Former Member · Accepted solution]

The log file says: a datafile is made unavailable(OFFLINE/RECOVER) for the system.

The below will display the status of datafile.

If the datafile status is OFFLINE, make it ONLINE.

SQL> SELECT FILE#, STATUS FROM V$DATAFILE;

Do check the below note for the same.

328785 - ORA-00376: File cannot be read at this time

regards,

pavan

[Former Member · Accepted solution]

Hi,

It seems that datafile data15 @/oracle/DEV/sapdata3/sr3_15/ location is either corrupt or SAP is not able to access it.

Please check if the permissions and groups are proper for the file.

Also check the below sap note:

328785 - ORA-00376: File cannot be read at this time

Regards,

Prithviraj.

[Sriram2009]

Hello Zerandib

Is this any disk space issue? could you check the DB or SAP USR folder free space?

BR

SS

[former_member199632]

Hi SS,

There was 1 harddisk failure in the server..... (Its in RAID 5) ; O/S is working fine. DB also starts up correctly

But will that be the issue for this, as server is up and running

spacewise enough space is there;

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p2      99G   22G   72G  24% /
udev                   47G  204K   47G   1% /dev
/dev/cciss/c0d0p1    1012M   42M  919M   5% /boot

/dev/mapper/vg_erpqas-lv_oracle
                      296G  207G   74G  74% /oracle

/dev/mapper/vg_erpqas-lv_sapmt_qas
                      5.0G 1019M  3.7G  22% /sapmnt

/dev/mapper/vg_erpqas-lv_usr_sap
                      5.0G  1.4G  3.4G  29% /usr/sap

/dev/mapper/vg_erpqas-lv_usr_sap_trans
                       20G  1.7G   18G   9% /usr/sap/trans

[former_member185954]

Hello,

If the hardware issue was the only change that caused the server to stop, rebooting the server once RAID rebuild is completed might help clearing of shared memory segments that may have been corrupted because of the server crash.

or you can manually do this by using cleanipc, here is a document , although its for AIX, its more or less similar on linux.

Regards,

Siddhesh

[Former Member]

Hi Zerandib,

As you mentioned the dw process get stop after few second. So can you please paste the dev_w0 and the dev_w* latest log and dev_disp log.

Also what was the activity you have performed was there any changes happened in system before restart.

Let us know the details.

Regards,

Ram