SAP ERP 6.0 EHP 5 TO EHP7 UPGRADE-Shadow instance is not starting up

[Former Member]

Hi

We have system with ERP 6.0  ehp 5  Kernel 721_EXT,OS LINUX RHEL 6.0,ORACLE 11.2.0.4

while upgrading to EHP 7 we have created mopz xml with kernel 741_Rel. Now at phase MAIN_SHDPREPUT/START_SHDI_PREPUT  we are stuck as shadow instance is not starting up and when checked we found dispatcher is not running.

We have already set ssfs for the system.

Let us know as soon as possible as we can not proceed further

we are getting below error in  DEVTRACE.LOG

I  MtxInit: 0 0 0

I  *** ERROR => e=28 semget(20127,1,2016) (28: No space left on device) [semux.c      508]

M  *** ERROR => ThrRegisterSem: SemInit SEM_SAPCOM_1 [thxxrun1.c   318]

M  *** ERROR => ThCallHooks: event handler ThrRegisterSem for event BEFORE_DB_CONNECT failed (-1) [thSos.c      2084]

M  in_ThErrHandle: 1

M  *** ERROR => ThCallHooks: hook failed (step TH_INIT, thRc ERROR-CORE-INTERNAL_ERROR, action STOP_WP, level 1) [thxxhead.c   2280]

M

[Former Member · Accepted solution]

Hello ,

the issue has been resolved we raised call to SAP. SAP recommended below notes

0000021893   R/3 does not start,dev_disp:fork(11: No more proc) 

0000941735   SAP memory management for 64-bit Linux systems 

0001584228   Internal error in secure store/Semaphore run out on a system.

    We have made changes as per note  0001584228-->1048303 Red Hat Enterprise Linux 5.x: Installation and upgrade

We  have changed linux parameter as below in etc/sysctl.conf

kernel.msgmni=1024
kernel.sem=1250 256000 100 1024

and repeated the phase. it worked!!

Thank you all for your quick support and responses.

[Former Member]

HI,

Please refer to this article.

http://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=375030464

• semget: points to semaphore

Kind Regards,

Johan

[Former Member]

dev_disp

[...]

[MODERATOR MOVED LOG TO ATTACHMENT]

message-log.txt.zip

[Former Member]

Please stop SAP application and database.

stopeed shadow instance

Execute cleanipc <NR> remove

and then started  It did not work..

server reboot is not done..

Filesystem

Size  Used Avail Use% Mounted on

	9.9G  3.7G  5.7G  40% /
	5.9G  100K  5.9G   1% /dev/shm
	485M   54M  406M  12% /boot
	7.9G  189M  7.3G   3% /home

9.9G  169M  9.2G   2% /opt

9.9G  154M  9.2G   2% /tmp

15G  5.1G  9.0G  36% /usr

20G   11G  8.6G  55% /var

5.0G  3.4G  1.4G  72% /sapmnt/SID

24G  5.5G   17G  25% /usr/sap/SID

74G   42G   29G  60% /usr/sap/trans

1008M  128M  830M  14% /usr/sap/hostctrl

3.0G   69M  2.8G   3% /usr/sap/SDA

7.9G  6.0G  1.6G  80% /oracle/stage

2.0G  468M  1.5G  25% /oracle/client

2.0G  338M  1.6G  18% /oracle/SID

9.9G  9.3G  118M  99% /oracle/SID/112_64

9.9G  4.7G  4.7G  51% /oracle/SID/11204

1008M  150M  808M  16% /oracle/SID/origlogA

1008M  150M  808M  16% /oracle/SID/origlogB

1008M   34M  924M   4% /oracle/SID/mirrlogA

1008M  234M  724M  25% /oracle/SID/mirrlogB

5.0G  349M  4.4G   8% /oracle/SID/oraarch

1008M  949M  8.6M 100% /oracle/SID/sapreorg

50G   44G  3.3G  94% /oracle/SID/sapdata1

50G   47G   10M 100% /oracle/SID/sapdata2

50G   46G  1.1G  98% /oracle/SID/sapdata3

50G   46G  1.4G  98% /oracle/SID/sapdata4

50G   47G   32K 100% /oracle/SID/sapdata5

50G   46G  1.1G  98% /oracle/SID/sapdata6

/

79G   62G   14G  83% /oracle/SID/sapdata7

/d

50G   45G  2.6G  95% /oracle/SID/sapdata8

50G   45G  2.6G  95% /oracle/SID/sapdata9

59G   50G  7.0G  88% /oracle/SID/sapdata10

138G  118G   14G  90% /oracle/SID/sapdata11

50G   34G   14G  71% /media

[Former Member]

Hi,

The sapdata's should not necessarily cause problems as the tablespace can be extended via new files to other sapdata directories.

I also see the 1008M  949M  8.6M 100% /oracle/SID/sapreorg is full this system writes files there during backups/space activities please check your oracle alerT_SID.log for which filesystem is complaining about if your DB is not starting up.

The control file must be available for writing by the Oracle Database server whenever the database is open. Without the control file, the database cannot be mounted and recovery is difficult.

Also if your cleanup parameters are maintained properly run brconnect -c -u / -f cleanup

Kind Regards,

Johan

[Former Member]

My syestem is alreadu up and running..But shadow instance can not

[former_member182657]

Hi,

You could also refer SAP Note 546006 - Problems with Oracle due to operating system errors

Regards,

Gaurav

[former_member182657]

In addition follow SAP Note   1473193 - Unknown error (No space left on device (errno:28))

Hope this will resolve your issue.

Regards,

Gaurav

[former_member188883]

Hi Spr,

I  MtxInit: 0 0 0
I  *** ERROR => e=28 semget(20127,1,2016) (28: No space left on device) [semux.c      508]
M  *** ERROR => ThrRegisterSem: SemInit SEM_SAPCOM_1 [thxxrun1.c   318]
M  *** ERROR => ThCallHooks: event handler ThrRegisterSem for event BEFORE_DB_CONNECT failed (-1) [thSos.c      2084]
M  in_ThErrHandle: 1
M  *** ERROR => ThCallHooks: hook failed (step TH_INIT, thRc ERROR-CORE-INTERNAL_ERROR, action STOP_WP, level 1) [thxxhead.c   2280]

Please stop SAP application and database.

Execute cleanipc <NR> remove

Or

Reboot the server

Then start SAP and database.

It should work.

Regards,

Deepak Kori

[Former Member]

Yes I have perforemed it ..still the error

also we have enough space

Filesystem

Size  Used Avail Use% Mounted on

	9.9G  3.7G  5.7G  40% /
	5.9G  100K  5.9G   1% /dev/shm
	485M   54M  406M  12% /boot
	7.9G  189M  7.3G   3% /home

9.9G  169M  9.2G   2% /opt

9.9G  154M  9.2G   2% /tmp

15G  5.1G  9.0G  36% /usr

20G   11G  8.6G  55% /var

5.0G  3.4G  1.4G  72% /sapmnt/SID

24G  5.5G   17G  25% /usr/sap/SID

74G   42G   29G  60% /usr/sap/trans

1008M  128M  830M  14% /usr/sap/hostctrl

3.0G   69M  2.8G   3% /usr/sap/SDA

7.9G  6.0G  1.6G  80% /oracle/stage

2.0G  468M  1.5G  25% /oracle/client

2.0G  338M  1.6G  18% /oracle/SID

9.9G  9.3G  118M  99% /oracle/SID/112_64

9.9G  4.7G  4.7G  51% /oracle/SID/11204

1008M  150M  808M  16% /oracle/SID/origlogA

1008M  150M  808M  16% /oracle/SID/origlogB

1008M   34M  924M   4% /oracle/SID/mirrlogA

1008M  234M  724M  25% /oracle/SID/mirrlogB

5.0G  349M  4.4G   8% /oracle/SID/oraarch

1008M  949M  8.6M 100% /oracle/SID/sapreorg

50G   44G  3.3G  94% /oracle/SID/sapdata1

50G   47G   10M 100% /oracle/SID/sapdata2

50G   46G  1.1G  98% /oracle/SID/sapdata3

50G   46G  1.4G  98% /oracle/SID/sapdata4

50G   47G   32K 100% /oracle/SID/sapdata5

50G   46G  1.1G  98% /oracle/SID/sapdata6

/

79G   62G   14G  83% /oracle/SID/sapdata7

/d

50G   45G  2.6G  95% /oracle/SID/sapdata8

50G   45G  2.6G  95% /oracle/SID/sapdata9

59G   50G  7.0G  88% /oracle/SID/sapdata10

138G  118G   14G  90% /oracle/SID/sapdata11

50G   34G   14G  71% /media

[divyanshu_srivastava3]

SAPDATA2, SAPDATA5, /oracle/SID/112_64 all are full.

Divyanshu

[former_member188883]

Hi Spr,

If possible stop the main SAP instance as well along with shadow instance and then execute cleanipc <Nr> remove

From the error message while starting shadow instance it is not able to allocate required shared memory segments

you can execute sappfpar check pf=<path of shadow instance profile>

ensure that there are no errors here.

Regards,

Deepak Kori

[Former Member]

yes I got error for shared memory

***WARNING: Unexpected parameter: SETENV_00 =DIR_LIBRARY=/oracle/SID/sapdata7/SUM/abap/exe

***WARNING: Unexpected parameter: ZTTA/DEFTITLE =SAP Web Application Server

***WARNING: Unexpected parameter: abap/preput_target_release =740_SP05

***WARNING: Unexpected parameter: rdisp/bgrfc_watchdog_time =0

***WARNING: Unexpected parameter: rdisp/system_needs_spool =false

***ERROR: Size of shared memory pool 40 too small

================================================================

SOLUTIONS: (1) Locate shared memory segments outside of pool 40

               with parameters like: ipc/shm_psize_<key> =0

SOLUTION: Increase size of shared memory pool 40

          with parameter: ipc/shm_psize_40 =486000000

--> So i need to set the parameter

ipc/shm_psize_<key> =0

ipc/shm_psize_40 =486000000 to the shadow instance right?

I am little confuse.

[former_member188883]

Hi Spr,

***ERROR: Size of shared memory pool 40 too small
================================================================
SOLUTIONS: (1) Locate shared memory segments outside of pool 40
               with parameters like: ipc/shm_psize_<key> =0

SOLUTION: Increase size of shared memory pool 40
          with parameter: ipc/shm_psize_40 =486000000

--> So i need to set the parameter
ipc/shm_psize_<key> =0
ipc/shm_psize_40 =486000000 to the shadow instance right?

Yes you need to set and adjust the value of ipc/shm_psize_10 & 40 as suggested in the response of sappfpar command for shadow instance.

Post this restart shadow instance and it should help.

Regards,

Deepak Kori

[Former Member]

The error for sappfpar check has gone but instance still not starting up

Thank you

[divyanshu_srivastava3]

I just saw you /dev/shm is almost full.

You OS shared memory seems to be full.

If es/implementation = map is set.

Can you increase it ?

Refer:

1674325 - DEV/SHM directory is used by mulitple instances and prevents startup

1623979 - SAP start fails on Linux

Divyanshu

[Former Member]

Dear

It is the issue with SPACE

make necessary space in file system

/usr/sap/trans

/usr/sap/SID/sum

/oracle/SID/oraarch

necessary space in data files and add some space in temporary table

Regards

[former_member182657]

Hi,

28: No space left on device

Confirm first for any space related issue on the system.

Regards,

Gaurav