SUM tool ends with error in phase MAIN_SHDCRE/SUBMOD_SHDDBCLONE/DBCLONE!

[Former Member]

Hi Experts,

We are currently performing a stack upgrade on SAP ECC 6.0 EHP6 to SP11 and got struck in pre-processing phase MAIN_SHDCRE/SUBMOD_SHDDBCLONE/DBCLONE! because DB is not available :

the DBCLONE log shows :

hnpclecd:ecdadm 91> tail DBCLONE.LOG

2 ETQ399 2014/04/13 02:09:59: sleeping 30 seconds

1 ETQ359 RFC Login to: System="ECD", AsHost="hnpclecd" Nr="00", GwHost="hnpclecd", GwService="sapgw00"

2EETQ231 RFC Login failed

3EETQ399 Checking job #0 'RSUPG_CLONE_TABLES' failed with RFC login to system ECD ashost hnpclecd nr 00 gwhost hnpclecd gwservice sapgw00 failed with code 2 key RFC_LOGON_FAILURE: 00024No connect to database, session terminated

2 ETQ399 2014/04/13 02:10:40: sleeping 30 seconds

1 ETQ359 RFC Login to: System="ECD", AsHost="hnpclecd" Nr="00", GwHost="hnpclecd", GwService="sapgw00"

2EETQ231 RFC Login failed

3EETQ399 Checking job #0 'RSUPG_CLONE_TABLES' failed with RFC login to system ECD ashost hnpclecd nr 00 gwhost hnpclecd gwservice sapgw00 failed with code 2 key RFC_LOGON_FAILURE: 00024No connect to database, session terminated

1EETQ399 Last error code set is: Batch jobs (DBCLONE1) may still be running: started 6, finished/aborted/running 0/0/6, check SM37 and SM21 for details

1EETQ204 Upgrade phase "DBCLONE" aborted with severe errors ("20140413021120")

R3trans returns the below error:

hnpclecd:ecdadm 94> R3trans -d

This is R3trans version 6.24 (release 721 - 26.09.13 - 20:13:07 ).

unicode enabled version

2EETW169 no connect possible: "DBMS = SYBASE --- "

R3trans finished (0012).

hnpclecd:ecdadm 95> tail trans*

4 ETW000                                                                                               39826  0.051507

4 ETW000  [     dev trc,00000]  (30012) [08001] [Sybase][ODBC Driver]Client unable to establish a connection

4 ETW000                                                                                                  14  0.051521

4 ETW000  [     dev trc,00000]  syb_glob.dbsl_ca.errclass set to DBSL_EC_RESTART                          14  0.051535

4 ETW000  [     dev trc,00000]  Connection 0 failed using DRIVER={Adaptive Server Enterprise};server=hnpclecd;port=4901;uid=SAPSR3;db=ECD;pwd=xxxxxxxxxxxxx;TextSize=2147483647;FetchArraySize=1000;ReleaseLocksOnCursorClose=1;DynamicPrepare=1;UseCursor=2;homogeneousbatch=1;charset=utf8;EncryptPassword=1;EnableLOBLocator=0;app=R3 000 00 comm rd ODBC;logintimeout=3;ProtocolCapture=cap_p3255_comm_rd;supresstdscontroltokens=1;SuppressRowFormat=1;SuppressParamFormat=1;

4 ETW000                                                                                                  23  0.051558

4 ETW000  [    dblink  ,00000]  ***LOG BY2=>sql error 30012  performing CON                              114  0.051672

4 ETW000  [    dblink  ,00000]  ***LOG BY0=>[ASE Error SQL30012:08001]Client unable to establish a connection

4 ETW000                                                                                                  13  0.051685

2EETW169 no connect possible: "DBMS = SYBASE --- "

hnpclecd:ecdadm 96>

I have shutdown the sybase server and started sap, then R3trans failed with below error :

***LOG BY0=>[ASE Error SQL1105]Can't allocate space for object 'syslogs' in database 'ECD' because 'logsegment' segment is full/has no free extents. If you ran out of space in syslogs, dump the transaction log. Otherwise, use ALTER DATABASE to increase the size of the segment.

Then after adding some log segments then we were able to establish the connection between SAP and DB. However when we retry in SUM it again fails with the same error !!!!

This happened thrice, clueless why this is happening when I initiate SUM tool. Reduced the R3load, R3trans processes but still the same error.

Any idea what could be the reason behind ??

We have upgraded the kenel and DBSL file is also upgraded to latest version.

Thanks !!

Regards,

Srikanth G

[former_member182657 · Accepted solution]

Hi,

Please follow SCN thread How to improve performance in ACT_UPG - ERP Upgrade - SCN Wiki

Regards,

Gaurav

[Former Member]

Hi,

I have started the reset of upgrade yesterday and its more than 24Hrs and its still running..

---------------------------------------------------------------------------------------------------------------------------------------

>> 2014/04/16 01:04:39  START OF PHASE REV_PREPROC/RUN_RSUPGREV_SHDT

Starting of Deletes shadow tables in batch (job RSUPGREV)

----------------------------------------------------------------------------------------------------------------------------------------

The only thing it is updating is :

hnpclecd:ecdadm 170> pwd

/sapdump/SUM/SUM/abap/log

hnpclecd:ecdadm 171> tail RSUPGREV_SHDT.LOG

2 ETQ399 'SUBST_CHECK_BATCHJOB' for JOBNAME='RSUPGREV' JOBCOUNT='01044100' returns 'R'.

2 ETQ399 'SUBST_CHECK_BATCHJOB' for JOBNAME='RSUPGREV' JOBCOUNT='01044100' returns 'R'.

2 ETQ399 'SUBST_CHECK_BATCHJOB' for JOBNAME='RSUPGREV' JOBCOUNT='01044100' returns 'R'.

2 ETQ399 'SUBST_CHECK_BATCHJOB' for JOBNAME='RSUPGREV' JOBCOUNT='01044100' returns 'R'.

Is that normal, and wait for it to complete ?

Regards,

Srikanth G

[Reagan]

Tx SM37 and check whether the job RSUPGREV is running.

Do a "Check Status" for this job.

Regards

RB

[Former Member]

It is active from 98.026 sec... and did a check status

1 jobs were checked and 0 jobs were corrected

hnpclecd:ecdadm 172> ps -ef|grep SUM

ecdadm    5628 20863  0 Apr16 pts/4    00:00:01 ./SAPup reset prepare rootdir=/sapdump/SUM/SUM/abap restart=3

ecdadm   18836  9838  0 04:22 pts/5    00:00:00 grep SUM

Regards,

Srikanth

[former_member188883]

Hi Srikanth,

Do you have a backup just before starting the upgrade?

If yes, please restore the same .

Cleanup the upgrade directory and startup fresh.

Hope this helps.

Regards,

Deepak Kori

[Former Member]

Hi Deepak,

We don't have a proper backup to restore, this is the first system installed and and never tried restore in this landscape. Is there any other way ? can I do a restart of SAP and continue the reset ? as I have a parameter change..

Regards,

Srikanth G

[former_member182657]

as I have a parameter change..

Which parameter you changed ? Please share.

We don't have a proper backup to restore, this is the first system installed and and never tried restore in this landscape.

If you're testing it on a sandbox system then you can restore it from your DEV or PRD database again as per standard methods and then start a fresh upgrade with use of SUM 1.0 SP09 .

Regards,

Gaurav

[Former Member]

Hi Gaurav,

We don't have sandbox system and upgrade is initiated on Dev where we have some development done by our consultants. Took a backup by 'dump database' and 'dump transaction' but we haven't tried restoration before in this landscape so little concerned.

Quality and Prod or not yet built.

Regards,

Sri

[Former Member]

Hi,

SAP has replied that there is a bug in the SP10 SUM tool and suggested to reset the upgrade and start with SP9 SUM. We are currently resetting the upgrade.

Will keep you informed on the status

Regards,

Srikanth

[former_member182657]

Hi,

Share the SUM details first before any restart of SUM tool.

Regards,

Gaurav

[Former Member]

Hi,

SL controller start screen shows the below error:

Can I close the SUM tool now and restart it  ?

Regards,

Srikanth G

[Reagan]

What do you have on the SUM console ?

Regards

RB

[former_member182657]

Please share the values of Batch process, maximum uptime process and R3trans process during configuration phase.It's totally depends on the hardware & on distribution of processes.Just wait for a good finish.If any issue occurs during this please share with us. 

Regards,

Gaurav

[Former Member]

Hi Gaurav,

There is no log file named SHDALLIMP.ELG .

Regards,

Srikanth G

[former_member182657]

Just wait for until phase finish.No manual interventions required now up till any issue occurs.So please stay clam and cool.

Thanks,

Gaurav

[former_member182657]

Hi Srikanth,

Could you share hardware details ,distribution of R3load and other parameters details as well for the upgrade.Nothing to worry,usually depends on memory parameters & on hardware config.

Thanks,

Gaurav

[Former Member]

Hi Gaurav,

There is log file named log file named SHDALLIMP.ELG .

The processes set for SUM  are :

=========== Background Processing and Update Processes ===========

Enter the maximum number of background processes during the update:

BATCH PROCESSES (UPTIME) [3]:

BATCH PROCESSES (DOWNTIME) [3]:

Enter the maximum number of parallel SQL processes:

SQL PROCESSES (UPTIME) [3]:

SQL PROCESSES (DOWNTIME) [4]:

Enter the maximum number of parallel import processes:

R3TRANS PROCESSES (UPTIME) [3]:

R3TRANS PROCESSES (DOWNTIME) [4]:

Enter the maximum number of parallel table import processes:

R3LOAD PROCESSES (UPTIME) [5]:

R3LOAD PROCESSES (DOWNTIME) [10]:

Enter the maximum number of parallel phases to run:

PARALLEL PHASES (UPTIME) [3]:

PARALLEL PHASES (DOWNTIME) [3]:

Select the instance of your system that shall be used as background server:

Note: If it is different from the central instance (primary application server

instance) host, /sapdump/SUM/SUM/abap must be mounted on the background host.

Also note that during the system downtime, all background jobs are started on

your central instance (primary application server instance) while the

background server has to be shut down.

01)  -  Automated batch job distribution (system will decide where to run the jobs)

02)  *  hnpclecd_ECD_00 (3 batchprocesses located)

[hnpclecd_ECD_00 (3 batchprocesses located)]:

4 ETQ399 Dialog finished at 20140415023340.

*** Parameter(s) updated ***

RAM is 16GB   / 8 Core CPU

[root@hnpclecd ~]# grep MemTotal /proc/meminfo

MemTotal:       16303940 kB

[root@hnpclecd ~]# uname -a

Linux hnpclecd 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Dec 13 06:58:20 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@hnpclecd ~]# grep "model name" /proc/cpuinfo

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

Regards,

Srikanth

[former_member182657]

Hi Srikanth,

Is there any log file named SHDALLIMP.ELG present,if yes please share.

Thanks,

Gaurav

[Former Member]

Hi Gaurav,

We have gone past the activation phase currently

ABAP: Phase MAIN_SHDIMP/SUBMOD_SHDIMP/SHADOW_IMPORT_INC is running , from the past 2 hours  it is not updating any logs and when I could see the shadow instance is also down.

When I tried starting the shadow instance with './SAPup startshd' it says :

------------------------------------------------------------

hnpclecd:ecdadm 156> ./SAPup startshd

At this time of the procedure, the shadow instance does not exist anymore since you are passed phase STOP_SHDI_*!

------------------------------------------------------------

But I could still see some shadow import phases pending..

Is something wrong !!!

Regards,

Srikanth G

[Former Member]

Hi,

I could see the STOP_SHDI* phase at the end of pre-processing, though it is not is executed my shadow instance got deleted !!!!!!

Clueless now

Regards,

Srikanth

[Reagan]

The upgrade is running so far without any issues. You need to wait until it finishes or moves into the next phase or when it ends with an error.

There are certain phases where parallelism can be influenced. But it depends on the hardware resources. If you don't have sufficient resources then you have to wait.

Check the progress of the upgrade activity in the logs at /sapdump/SUM/SUM/abap/tmp and /sapdump/SUM/SUM/abap/log directories

Regards

RB

[Former Member]

Hi RB,

Nothing is getting updated in the the tmp or log directory. None of workproccess are handling any task.

No shadow instance related file system/ directories..

hnpclecd:ecdadm 101> ps -ef|grep SUM

ecdadm   15630 20646  0 00:12 pts/1    00:00:04 tp pf=/sapdump/SUM/SUM/abap/var/SHDALLIMP.TPP put ECD -Dmainimp_proc=3 -Dparallel=no

ecdadm   17350 15630  0 00:25 pts/1    00:00:00 /sapdump/SUM/SUM/abap/exe/R3trans -u 26 -w /sapdump/SUM/SUM/abap/tmp/SAPK-60026INISH.ECD /sapdump/SUM/SUM/abap/tmp/SAPKK-60026INISH.ECD

ecdadm   20563 32117  0 Apr13 pts/1    00:01:01 /sapdump/SUM/SUM/sdt/../jvm/jre/bin/java -classpath /sapdump/SUM/SUM/sdt/lib/sdt_bootstrap.jar -ea -Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Djava.system.class.loader=com.sap.sdt.bootstrap.SystemClassLoader com.sap.sdt.bootstrap.Startup rootdir=/sapdump/SUM/SUM/sdt

ecdadm   20646 20563  0 Apr13 pts/1    00:03:41 /sapdump/SUM/SUM/abap/bin/SAPup stdout=/sapdump/SUM/SUM/abap/log/SAPup.out rootdir=/sapdump/SUM/SUM/abap uaport=4240 restart=3

ecdadm   21920 20863  0 04:53 pts/4    00:00:00 grep SUM

Regards,

Srikanth G

[Reagan]

What do you mean by "Nothing is getting updated in the the tmp or log directory. None of workproccess are handling any task." ?

I can clearly see that there is an R3trans is under execution and the trace file for that is /sapdump/SUM/SUM/abap/tmp/SAPK-60026INISH.ECD

Have you checked that trace log ?

Looks like you are new to upgrades and on top of that you have no patience at all. No offence.

You need to wait until it finishes each activity.

If there is a manual intervention required then SUM will prompt you for that.

If the log file is getting updated just wait rather than interrupting the upgrade.

Regards

RB

[Former Member]

Hi RB,

Apologies, could be because of the pressure here..

My main concern was the shadow instance, as I couldn't see the shadow instance related directories /sapdump/SUM/SUM/abap/<SID> , and log/tmp directories are not getting updated.

The package handled by R3trans process is of around 20MB and running from past 5hrs, and also as per the  configuration 3 R3trans processes are allocated for uptime.

Anyways will monitor till it throws some error. Thanks

Regards,

Srikanth

[Reagan]

Just before shadow instance creation SUM prompted for 'lock the development phase'. So can our consultant's work ?

After this phase you will not be able to bring in any modifications (transport requests) to this system.

What I see is that you are nearing the "Downtime Phase". The upgrade tool will stop the shadow instance before the down time transfer. This is normal and you do not need to start it if you see it not running. Also the main system is still available and only with the above restriction.

There is no need to be under pressure as this is how the upgrade works and you just need to stay calm.

Regards

RB

[Former Member]

Hi,

Currently we are at ACT_UPG phase on shadow instance. However it is

running for a very long time. Could you please suggest any work around to improve hte performance of

the phase as I could see only one package has been processed in 4hrs !!!!

Also I have locked all the users from logging onto the system. Could

you please suggest as the upgrade is in progress on shadow instance can

our consultants work on the system till the execution phase starts ??

If yes do we have any limitation ?

Just before shadow instance creation SUM prompted for 'lock the

development phase'. So can our consultant's work ?

Please advice

Regards,

Srikanth G

[Former Member]

Hi ,

After dumping the transaction log we were able to continue SUM and got

past the phase which was ending with error earlier. But we got error in

phase MAIN_SHDINST/SUBMOD_SHDALIASCRE/SCEXEC_GRA_ORG!

Last error code set: 1 processes failed, check 'SCEXEC_G.*' for details

When checked the SCEXEC_G.001 log says :

hnpclecd:ecdadm 81> cat SCEXEC_G.001

SAPup> Starting subprocess with PID 11501 in phase 'SCEXEC_GRA_ORG' at

20140414114523

ENV: DIR_LIBRARY=/sapdump/SUM/SUM/abap/bin

ENV:

LD_LIBRARY_PATH=/sapdump/SUM/SUM/abap/bin:/sapdump/SUM/SUM/jvm/jre/lib/am

d64/server:/sapdump/SUM/SUM/jvm/jre/lib/amd64:/sapdump/SUM/SUM/jvm/jre/..

/lib/amd64:/usr/sap/ECD/SYS/global/syb/linuxx86_64/sybodbc:/sybase/ECD/AS

E-15_0/lib:/sybase/ECD/OCS-15_0/lib:/sybase/ECD/OCS-

15_0/lib3p64:/sybase/ECD/OCS-

15_0/lib3p:/usr/sap/ECD/SYS/exe/run:/usr/sap/ECD/SYS/exe/uc/linuxx86_64

ENV: PATH=/sapdump/SUM/SUM/abap/bin:/sybase/ECD/ASE-

15_0/jobscheduler/bin:/sybase/ECD/ASE-15_0/bin:/sybase/ECD/ASE-

15_0/install:/sybase/ECD/DBISQL/bin:/sybase/ECD/OCS-

15_0/bin:/usr/lib64/qt-

3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/sap/ECD/SYS/exe/uc/linuxx86_64:

/usr/sap/ECD/SYS/exe/run:/home/ecdadm:.

ENV: SAPSYSTEMNAME=ECD

ENV: dbms_type=syb

ENV: dbs_syb_dbname=ECD

ENV: dbs_syb_port=4901

ENV: dbs_syb_schema=SAPSR3

ENV: dbs_syb_server=hnpclecd

EXECUTING /sapdump/SUM/SUM/abap/bin/SAPuptool execdb ../var/PTALORGR.XQLLogging to '/sapdump/SUM/SUM/abap/log/PTALORGR.LOG'

FATAL ERROR: DBSL error 99: Connect failed

[ASE Error SQL30274:08S01]Error setting up connection

SAPup> Process with PID 11501 terminated with status 20 at

20140414114529!

Could someone advice what could be the issue..

Regards,

Srikanth

[Former Member]

Hi ,

Just retried and it ran without any issue.

Regards,

Srikanth G

[former_member182657]

Hi,

Please follow SCN link

Regards,

Gaurav

[Former Member]

Hi ,

As the system hangs due to R3trans issue we wcouldn't check the SM21/SM37 logs at that time.

For resolving hte R3trans issue I had to restart the system. After system is accessible :

In SM21:

SQL error 30012 occurred (possible network problems?). Work process is stopped

Database error 30012 at CON

> [ASE Error SQL30012:08001]Client unable to establish a

> connection

Initialization DB-Connect Failed, Return Code 000256

In SM37

Jobs fails as system is shutdown..

I have extended the database and transaction log also before starting SUM, but again it ends up with the same error.

Please advice !!

Regards,

Srikanth G

[Former Member]

Hi,

After increasing the user connections from 200 to 400, R3trans was returning 0000 code.

But after retrying SUM it executed for some time and again throwed the same error and user connections got exhausted.

Could you please advice how many user connections is recommended ?

Regards,
Srikanth G

[former_member188883]

Hi Srikanth,

Refer SCN thread where similar issues is discussed and resolved.

Problem seems to be with dbsl patch.

Hope this helps.

Regards,

Deepak Kori

[former_member188883]

Hi Srikanth,

Additionally please check SAP note 1749935

1749935 - SYB: Configuration Guide for SAP ASE 15.7

For below Sybase ASE configuration settings

The following settings affect memory usage of SAP ASE. The values given are required to allow SAP applications to run on SAP ASE:

> number of open objects >= 40000 [REQ]

> number of open indexes >= 30000 [REQ]

> number of open partitions >= 30000 [REQ]

Hope this helps.

Regards,

Deepak Kori

[former_member185239]

Hi Srikanth,

The Error was

"[ASE Error SQL1105]Can't allocate space for object 'syslogs' in database 'ECD' because 'logsegment' segment is full/has no free extents. If you ran out of space in syslogs, dump the transaction log. Otherwise, use ALTER DATABASE to increase the size of the segmen"

Since you have shutdown the Sybase , it will do crash recovery and as the sysbase is not able to recover since logsegment is getting full. so you need to expand the transaction log segment.

Follow the below sapnotes

1802842 - SYB: Error 1105 and Error 3475 in ASE error log

1810223 - SYB: Recovery of a database fails with error 3475

With Regards

Ashutosh Chaturvedi

[former_member185239]

Hi Srikanth,

There is some more information on sybase logsegment.

Sybase error 1105 - logsegment is full Sybase error 1105 logsegment is full

With Regards

Ashutosh Chaturvedi

[former_member182657]

Hi,

It would be nice to add logs from SM21,ST22 (Any dump creation during the phase) & SM37 job logs.If possible try to verify for any space related issue ( transaction log file size,tablespace full ).

Regards,

Gaurav

[Former Member]

Hi,

I have retried after updating tp, R3load, R3trans, dbsybslib.so, sybcntrl, startdb and stopdb files with the latest version. All the DBCLONE jobs got cancelled and restarted the SUM tool but still the same error

Please help !!!

Regards,

Srikanth

[Reagan]

Tx SM37 - DBCLONE* - F8

Provide the job log.

Regards

RB

[Reagan]

If you have corrected the issue at the database side then restart SUM.

Also check whether the DBCLONE* job is running in the primary system.

If the job is under execution then you will need to wait until the job finishes.

Regards

RB

[ashish_vikas]

Hi,

did DBCLONE jobs started/cancelled in SM37 ?

SUM directory should be accessible from all application server if you have more than one app server and job is getting started there.

best regards

ashish

[former_member182657]

Hi,

Try to use latest database binaries,tp,R3trans versions .If possible go with kernel upgrade & retry .

Regards,

Gaurav