Error when applying EHP6

[symon_braunbaer]

Hello,

I am facing the following error during the phase MAIN_SHDRUN/DDIC_UPG (from SAPup.ECO):

SAPup> Starting subprocess with PID 26258 in phase 'DDIC_UPG' at 20140410162631

    ENV: DIR_LIBRARY=/usr/sap/trans/SUM_<SID>/abap/exe

    ENV: LD_LIBRARY_PATH=/usr/sap/trans/SUM_<SID>/abap/exe:/usr/sap/trans/SUM_<SID>/jvm/jre/lib/amd64/server:/usr/sap/trans/SUM_<SID>/jvm/jre/lib/amd64:/usr/sap/trans/SUM_<SID>/jvm/jre/../lib/amd64:/usr/sap/<SID>/SYS/exe/run:/usr/sap/<SID>/SYS/exe/nuc/sunx86_64:/oracle/client/11x_64/instantclient

    ENV: NLS_LANG=AMERICAN_AMERICA.WE8DEC

    ENV: ORACLE_BASE=/oracle

    ENV: ORACLE_SID=<SID>

    ENV: PATH=/usr/sap/trans/SUM_<SID>/abap/exe:/usr/bin:/bin:/usr/sap/<SID>/SYS/exe/nuc/sunx86_64:/usr/sap/<SID>/SYS/exe/run:/home/<SID>adm:.:/usr/ccs/bin:/usr/ucb

    ENV: SAPDATA_HOME=/oracle/<SID>

    ENV: SAPSYSTEMNAME=<SID>

    ENV: auth_shadow_upgrade=1

    ENV: dbms_type=ORA

    ENV: dbs_ora_schema=SAPSR3

    ENV: dbs_ora_tnsname=<SID>

EXECUTING /usr/sap/trans/SUM_<SID>/abap/exe/tp pf=/usr/sap/trans/SUM_<SID>/abap/var/DDICUPG.TPP put <SID> -Dmainimp_proc=1 -Dparallel=6

This is tp version 380.08.46 (release 721)

Looking for effective putsteps ... ... ready (looking for putsteps)

Changing parameter 'MAINIMP_PROC' from '1' to '2'

ERROR: /usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID> is already in use. Giving up after (31) tries because MAX_SEMAPHORE_WAIT is set. My name: pid 26258 on wsps451 (<SID>adm)

ERROR: EXIT(16) -> process ID is: 26258

HALT 20140410162803

tp returncode summary:

TOOLS: Highest return code of single steps was: 16

ERRORS: Highest tp internal error was: 0212

tp finished with return code: 212

meaning:

  could not access file as supposed (see dev_tp or SLOG731 for more details)

SAPup> Process with PID 26258 terminated with status 212 at 20140410162811!

And here is an excerpt from SLOG731:

### Phase MAIN_SHDRUN/DDIC_UPG:

START SET STOPMARK         <SID>        20140410162403              <SID>adm       wsps451 2014041016240100636a

STOP  SET STOPMARK         <SID>        20140410162403              <SID>adm       wsps451 2014041016240100636a

START put                  <SID>        20140410162404              <SID>adm       wsps451 2014041016240400637a

START SET STOPMARK         <SID>        20140410162405              <SID>adm       wsps451 2014041016240400637a

STOP  SET STOPMARK         <SID>        20140410162405              <SID>adm       wsps451 2014041016240400637a

LIST  put                  <SID>

START DD IMPORT            <SID> H      20140410162405              <SID>adm       wsps451 2014041016240400637a

STOP  DD IMPORT            <SID> H      20140410162405              <SID>adm       wsps451 2014041016240400637a

STOP  put                  <SID>   0000 20140410162416              <SID>adm       wsps451 2014041016240400637a

START SET STOPMARK         <SID>        20140410162424              <SID>adm       wsps451 20140410162423006398

STOP  SET STOPMARK         <SID>        20140410162424              <SID>adm       wsps451 20140410162423006398

START put                  <SID>        20140410162425              <SID>adm       wsps451 2014041016242500639d

START SET STOPMARK         <SID>        20140410162427              <SID>adm       wsps451 2014041016242500639d

STOP  SET STOPMARK         <SID>        20140410162427              <SID>adm       wsps451 2014041016242500639d

LIST  put                  <SID> H

START DD IMPORT            <SID> H      20140410162427              <SID>adm       wsps451 2014041016242500639d

WARNING: /usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID> is already in use (10), I'm waiting 1 sec (20140410162451). My name: pid 25501 on wsps451 (<SID>adm)

WARNING: /usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID> is already in use (20), I'm waiting 4 sec (20140410162519). My name: pid 25501 on wsps451 (<SID>adm)

WARNING: /usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID> is already in use (30), I'm waiting 4 sec (20140410162552). My name: pid 25501 on wsps451 (<SID>adm)

HALT 20140410162556

ERROR: /usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID> is already in use. Giving up after (31) tries because MAX_SEMAPHORE_WAIT is set. My name: pid 25501 on wsps451 (<SID>adm)

ERROR: EXIT(16) -> process ID is: 25501

STOP  put                  <SID>   0016 20140410162556              <SID>adm       wsps451 2014041016242500639d

This is a distributed environment - an AS and a DB server.

We are running on Oracle 11 and Solaris 10.

The SUM directory and the EHP6 packages are in subdirectories of /usr/sap/trans

PLEASE help!! This is becoming very urgent now!

[Reagan · Accepted solution]

ERROR: /usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID> is already in use. Giving up after (31) tries because MAX_SEMAPHORE_WAIT is set. My name: pid 25501 on wsps451 (<SID>adm)

Before renaming this file I would check wthere there are tp and R3trans processes running at the OS level and whether the file is getting updated.

If yes then wait until it finishes. Else I would terminate SUM and restart it.

Regards

RB

[symon_braunbaer · Accepted solution]

Hi Reagan,

I have done all this (no running tp or R3trans) and still getting the error, even after restarting SUM... Any other ideas please ?

[Reagan · Accepted solution]

Hello Symon

In that I would stop SAP and the SUM process and do a clean restart and see if that helps.

The SUM tool should move the files from the tmp to the log directory once the activity has been completed. Unfortunately this is not happening in your case and most probably due to some permissions issues.

Restart all and let us know the outcome.

Regards

RB

[symon_braunbaer · Accepted solution]

Hello Reagan,

many thanks for your suggestion, which I have followed. Unfortunately after restarting everything, the system came back with exactly the same error 😞

Could you pls kindly suggest something else ?

[Reagan · Accepted solution]

Hello Symon

Do you have sufficient space in the file system (/usr/sap/trans/SUM_<SID>) used for the upgrade ?

Can you provide the output of these commands please?

ls -latr /usr/sap/trans/SUM_<SID>/abap/log | tail -15

ls -latr /usr/sap/trans/SUM_<SID>/abap/tmp

Regards

RB

[symon_braunbaer · Accepted solution]

Hi Reagan and many thanks for the quick reply!! SAP did not yet give any answer to the OSS message...

Here is the output of the commands you requested... And one more thing, I have also tried to replace tp and R3trans in the SUM directory with the newest versions I could get (2nd of April), the problem is still persisting.

# ls -latr /usr/sap/trans/SUM_T07/abap/log | tail -15

-rwxrwxrwx   1 t07adm   sapsys     15862 Apr 12 00:35 SAPupStat.log

-rwxrwxrwx   1 t07adm   sapsys    744731 Apr 12 00:35 TP.SAV

-rw-r--r--   1 t07adm   sapsys      1089 Apr 12 00:35 TP.ECO

-rwxrwxrwx   1 t07adm   sapsys    529289 Apr 12 00:35 ULOG14_2

-rwxrwxrwx   1 t07adm   sapsys    199314 Apr 12 00:37 SLOG731

-rwxrwxrwx   1 t07adm   sapsys    894501 Apr 12 00:37 SAPup.ECO

-rw-r--r--   1 t07adm   sapsys     36504 Apr 12 00:37 PHASES.LOG

-rwxrwxrwx   1 t07adm   sapsys      1774 Apr 12 00:37 SAPup_troubleticket.log

-rwxrwxrwx   1 t07adm   sapsys    425718 Apr 12 00:37 SAPupConsole.log

-rwxrwxrwx   1 t07adm   sapsys     25304 Apr 12 00:37 TROUBLE_SAVELOGS.LOG

-rwxrwxrwx   1 t07adm   sapsys   3228486 Apr 12 00:37 SAPup_troubleticket_logs.sar

-rwxrwxrwx   1 t07adm   sapsys   9567116 Apr 12 00:37 TPCHECKBUFFER.SAV

drwxrwxrwx   2 t07adm   sapsys      2162 Apr 12 00:37 .

-rw-r--r--   1 t07adm   sapsys    479448 Apr 12 00:37 TPCHECKBUFFER.ECO

-rwxrwxrwx   1 t07adm   sapsys    236217 Apr 12 00:37 SAPup.log

# ls -latr /usr/sap/trans/SUM_T07/abap/tmp

total 3819

-rwxrwxrwx   1 t07adm   sapsys         0 Nov 20  1976 SAPUSER.LOT.old

-rwxrwxrwx   1 t07adm   sapsys         0 Nov 20  1976 SAPUSER.LOT

drwxrwxrwx   2 t07adm   sapsys         2 Apr  7 10:56 igs

drwxrwxrwx   2 t07adm   sapsys         3 Apr  7 11:28 dbs

-rwxrwxrwx   1 t07adm   sapsys      1576 Apr  7 11:28 SYSSTATUS.LOG

-rwxrwxrwx   1 t07adm   sapsys      3041 Apr  7 15:51 R3trans.out

-rwxrwxrwx   1 t07adm   sapsys    583129 Apr  7 16:38 cenoupyc.sta

-rwxrwxrwx   1 t07adm   sapsys       205 Apr  7 17:05 connT07.log

-rwxrwxrwx   1 t07adm   sapsys   1095074 Apr  7 17:05 cenouqwg.sta

-rwxrwxrwx   1 t07adm   sapsys       450 Apr 10 12:29 dev_rfc.trc

-rwxrwxrwx   1 t07adm   sapsys         0 Apr 10 13:45 dev_hwid

-rwxrwxrwx   1 t07adm   sapsys      2327 Apr 10 13:45 dev_likey

-rwxrwxrwx   1 t07adm   sapsys       460 Apr 10 13:51 dev_evt

-rwxrwxrwx   1 t07adm   sapsys      4362 Apr 10 13:59 rfc03232_00001.trc

-rwxrwxrwx   1 t07adm   sapsys       552 Apr 10 15:05 SAPKK-60011INFINBASIS.T07.bak

drwxrwxrwx  23 t07adm   sapsys        26 Apr 10 15:57 ..

-rw-rw-r--   1 t07adm   sapsys       532 Apr 10 16:03 SAPKKGPFD08.T07.bak

-rw-rw-r--   1 t07adm   sapsys       532 Apr 12 00:31 SAPKKGPFD08.T07

-rw-rw-r--   1 t07adm   sapsys     68960 Apr 12 00:31 SAPHGPFD08.T07

-rw-r--r--   1 t07adm   sapsys       863 Apr 12 00:37 SAPupDialog.txt

drwxrwxrwx   4 t07adm   sapsys        22 Apr 12 00:37 .

-rw-r--r--   1 t07adm   sapsys        28 Apr 12 00:37 upalert.log

Many many thanks for your effort!

[Reagan · Accepted solution]

Hello Symon

Thanks.

# ls -latr /usr/sap/trans/SUM_T07/abap/log
TP.SAV
TP.ECO
ULOG14_2
SLOG731
SAPup.ECO
TPCHECKBUFFER.SAV
TPCHECKBUFFER.ECO
SAPup.log
# ls -latr /usr/sap/trans/SUM_T07/abap/tmp
SAPKKGPFD08.T07
SAPHGPFD08.T07

Could you supply these logs or check whether there are some errors ?

Regards

RB

[symon_braunbaer · Accepted solution]

Hi Reagan,

just to let you know: I solved it by constantly watching SLOG731 and as soon

as it would show the first warning (I am waiting...), I immediately went to another

open window and deleted the file for which it was complaining and this way it

went on and I could finish the upgrade last week.

I think this could be a bug in SUM SP10. I did the previous system with SP9

and didn't have this problem. Now I just started the next system in the landscape,

using again the SP9 SUM, so we'll see

Thanks for your help !!

[former_member188883]

Hi Symon,

Please stop the upgrade.

Rename the file /usr/sap/trans/SUM_<SID>/abap/tmp/<filename>.LOB

Start the upgrade again.

Hope this helps.

Regards,

Deepak Kori

[symon_braunbaer]

Hello Deepak,

thanks for the reply, but I do not have any files, named .LOB in the mentioned location.

(You do not mean .LOG, do you ?)

[former_member188883]

Hi Symon,

Looks like some SUM process is still holding the file

/usr/sap/trans/SUM_<SID>/abap/tmp/SAPKKGPFD08.<SID>

Please stop SUM and check for any zombie SUM processes. Also try renaming above file.

Hope this helps.

Regards,

Deepak Kori

[symon_braunbaer]

Hi Deepak,

I have tried stopping and restarting SUM, the error is still persisting. We will reboot the Solaris zones this evening and retry...