Shutting down database hangs

[Former Member]

Dear Gurus,

I m facing a database hang issue while performing the scheduled offline backup.I m able to take the online backup.When the backup starts , the database starts to shutdown.When checked in the log of backup it shows in tail as shutting down database.When checked for user trace , it shows message as "Attempting to kill pid xxxxxxx" and keeps on retrying for n no. of times.Then I have to manually kill the backup and restart the application.Then again try for an offline backup.This time it happens smoothly.This activity we did for couple of weeks .But now from past 3 days even after restarting the backup, it goes to hang mode.In other words unable to perform the backup activity.

The message had been raised to SAP from almost 15 days at very high.They every time ask the logs , and say that the data in log is nt providing much info for analysing the issue or cause of hang.Also enabled the traces of 10400 for trace level 1 ,10 some more traces also.But still the output is not helping even them to analyse the root cause.

Its my observation that this issue is occuring only on ABAP+JAVA systems(CRM &BIW)not in ECC.We are on oracle database 10.2.0.0.4 with latest patch set as suggested by SAP for the same issue.

This issue is h

Please suggest me at the earliest.As need to resolve this at the earliest.

Bye.

Amit

[Former Member]

Dear all,

I had started with an alternate method where script is written for lsnrctl stop and start.So in case if a db hangs it continues after a few mins as lisnter gets stopped temporarily.

This solution can be used as an wayout method.

Regards,

Amit

[Former Member]

Thanks for info

[sunny_pahuja2]

Hi,

Have you applied database sap note 830576 ?

Thanks

Sunny

[Former Member]

Hi Sunny,

Thanks for the immediate reply.Yes we had set the parameters as per the below mentioned note.

Bye

[Former Member]

I thnk this probelm not related to parameter because its working fine previously .

U have to check Inactive session by ps -aef|grep LOCAL=NO

Check alert<SID>.log file is there "archive process shutdown avoided"

Kindly inform

[Former Member]

Hi,

No "archive process shutdown avoided" message is there in the log file.

In usertrace i get messages as mentioned below.

bslcrmprddb #more bcp_ora_1257502.trc

Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_1257502.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

With the Partitioning, Data Mining and Real Application Testing options

ORACLE_HOME = /oracle/BCP/102_64

System name: AIX

Node name: bslcrmprddb

Release: 3

Version: 5

Machine: 00C6C4024C00

Instance name: BCP

Redo thread mounted by this instance: 1

Oracle process number: 0

Unix process pid: 1257502, image: oracle@bslcrmprddb

• 2009-06-05 08:01:36.875

Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown

Failed to acquire instance startup/shutdown serialization primitive

bslcrmprddb #more bcp_ora_2453676.trc

Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_2453676.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

With the Partitioning, Data Mining and Real Application Testing options

ORACLE_HOME = /oracle/BCP/102_64

System name: AIX

Node name: bslcrmprddb

Release: 3

Version: 5

Machine: 00C6C4024C00

Instance name: BCP

Redo thread mounted by this instance: 0 <none>

Oracle process number: 0

Unix process pid: 2453676, image: oracle@bslcrmprddb

• 2009-06-06 07:04:22.494

Dynamic strand is set to TRUE

Running with 2 shared and 88 private strand(s). Zero-copy redo is FALSE

bslcrmprddb #more bcp_ora_1495266.trc

Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_1495266.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

With the Partitioning, Data Mining and Real Application Testing options

ORACLE_HOME = /oracle/BCP/102_64

System name: AIX

Node name: bslcrmprddb

Release: 3

Version: 5

Machine: 00C6C4024C00

Instance name: BCP

Redo thread mounted by this instance: 1

Oracle process number: 213

Unix process pid: 1495266, image: oracle@bslcrmprddb (TNS V1-V3)

• ACTION NAME:() 2009-06-06 03:00:22.223

• MODULE NAME:(sqlplus@bslcrmprddb (TNS V1-V3)) 2009-06-06 03:00:22.223

• SERVICE NAME:(SYS$USERS) 2009-06-06 03:00:22.223

• SESSION ID:(517.13229) 2009-06-06 03:00:22.223

ksukia: Starting kill, force = 0

ksukia: killed 274 out of 274 processes.

ksukia: Starting kill, force = 0

ksukia: killed 0 out of 2 processes.

ksukia: Starting kill, force = 0

ksukia: killed 0 out of 1 processes.

ksukia: Starting kill, force = 0

ksukia: killed 0 out of 1 processes.

ksukia: Starting kill, force = 0

Please suggest.

Regards,

Amit

[markus_doehr2]

I did a quick shot at metalink and found

375935.1 What To Do and Not To Do When 'shutdown immediate' Hangs

Maybe with the help of that note you can try to figure out why the database is not coming down.

Do you see any CPU activity when this occurs (using e. g. topas)?

Markus