Regarding SAP startup failed on HACMP

[Former Member]

Dear All,

I have install DBCI on one node and APPs on another node, both are working fine, without IBM HACMP Cluster.We are working on IBM AIX/SAP ECC 6.0/Oracle 10g.

We have sucessfully install HACMP on both nodes, that is also working nicly.

We are able to startup and shutdown sap/database from both node using hacmp cluster commands.

Incase of failover and takeover, i am facing little bit problem, when failover occure on DBCI node then all resources and instances takeover on Apps server, in this case

Dataase is sucessfully takeover on Apps Server, but SAP instance is not going to start and giving me error, it is showing me to check Starsap_D01.log inside

/home/<sid>,

I have given startsap_D01.log/and/dev_sapstart log file for your reference.

***************startsap_D01.log

Trace of system startup/check of SAP System CLU on Sat Mar 10 23:35:33 IST 2007

Called command: /home/cluadm/startsap D01

Starting SAP Instance D01

-

---

Instance Service on host cluapp1-CO started

SAP-R/3-Startup Program Rel 700 V1.8 (2003/04/24)

-

---

Starting at 2007/03/10 23:35:33

Startup Profile: "/usr/sap/CLU/SYS/profile/START_D01_cluapp1-CO"

Setup Environment Variables

-

---

(889018) SETENV LD_LIBRARY_PATH=/usr/sap/CLU/D01/exe:

(889018) SETENV SHLIB_PATH=/usr/sap/CLU/D01/exe:

(889018) SETENV LIBPATH=/usr/sap/CLU/D01/exe:/usr/sap/CLU/SYS/exe/run:/sapmnt/CLU/exe:/usr/sap/CLU/SYS/exe/run:/oracle/client/10x_64/instantclient

Update local Kernel Files

-

---

(889018) Local: /usr/sap/CLU/SYS/exe/run/sapcpe name=CLU

(889018) system(/usr/sap/CLU/SYS/exe/run/sapcpe name=CLU) returns 1

(889018) Return-Code 1 in Local-Kernel-Update. See sapcpe.log.

Execute Pre-Startup Commands

-

---

(889018) Local: /usr/sap/CLU/SYS/exe/run/sapcpe pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO

(889018) system(/usr/sap/CLU/SYS/exe/run/sapcpe pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO) returns 2

(889018) Local: /usr/sap/CLU/D01/exe/sapmscsa pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO -n

sh: /usr/sap/CLU/D01/exe/sapmscsa: not found.

(889018) system(/usr/sap/CLU/D01/exe/sapmscsa pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO -n) returns 127

(889018) Local: rm -f dw.sapCLU_D01

(889018) Local: ln -s -f /usr/sap/CLU/D01/exe/disp+work dw.sapCLU_D01

(889018) Local: rm -f se.sapCLU_D01

(889018) Local: ln -s -f /usr/sap/CLU/D01/exe/rslgsend se.sapCLU_D01

(889018) Local: rm -f ig.sapCLU_D01

(889018) Local: ln -s -f /usr/sap/CLU/D01/exe/igswd_mt ig.sapCLU_D01

Starting Programs

-

---

(819394) Starting: local dw.sapCLU_D01 pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO

(987146) Starting: local se.sapCLU_D01 pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO -F

(847880) Starting: local ig.sapCLU_D01 -mode=profile pf=/sapmnt/CLU/profile/CLU_D01_cluapp1-CO

(889018) Waiting for Child Processes to terminate.

(889018) **** 2007/03/10 23:35:33 Child 987146 terminated with Status 150 . ****

(889018) **** 2007/03/10 23:35:33 Child 819394 terminated with Status 150 . ****

(889018) **** 2007/03/10 23:35:33 Child 847880 terminated with Status 150 . ****

(889018) **** No more Child Processes to wait for.

(889018) Parent Shutdown at 2007/03/10 23:35:33

Execute Post-Shutdown Commands

-

---

(889018) Exiting with Return-Code 3. (No more child processes)

Startup of Instance failed

****************************************************************************************************************************************************************

dev_sapstart Log File

-

---

trc file: "dev_sapstart", trc level: 1, release: "700"

-

---

Sat Mar 10 23:35:33 2007

SigISetDefaultAction : default handling for signal 20

Waiting for Your Kind response.

Thanks and Regards

K R Singh

[Former Member]

<b>Hi Maurice Sens,

Thanks a lot for your response.

I have resolve the issue, now i can takeover DB+CI server from one node to another node on APPlication server, but my problem is that when i am stoping cluster on Takeovered node (Application Server), then SAP and database is sucessfully shutingdown but /sapmnt/sid/exe,global and profile is not able to unmount and throwiing me some NFS realated errors,

Errors are

(1) NFS lookup failed for server <DBCIHostname> :error 3 (RPC: 1832-006 unable to send )

(2) umount: error unmounting /dev/sapexelv : Device busy (in this case i am not able to see any SAP processes are running to takeover node).

After this error my Takeovered node (Application Server is going to hung).

Please do the needful.

Thanks & Regards,

K R Singh</b>

[former_member1198651]

Hi,

have you solve your problem with unmounting you /sapmnt ?

We're currently facing the same problem and we can't find any process that prevent us from unmount the filesystem.

Any help will be great.

Regards

Nicolas LOUIS

[former_member383291]

Hi K R Singh,

We also got the same error in our test system during the kernel upgrade. we are also getting the same error.

child 3760276 terminated with status 150

child 4935928 terminated with status 255

we are struck with this error from past few weeks and unable to proceed further further with procudtion patching. Kindly let us know if you can look into this.

Suvarna

[Former Member]

To get quick response you should have opened another thread as this thread is marked as answered.

have you checked the logs from dev_disp, dev_ms and dev_w0?

[Former Member]

Supply dev_w0 and dev_disp log files.