HANA DB is not comin UP

[Mofizur]

Hi,

I was doing a migration using DMO from SQl Server to HANA (Rev 91).

During phase EU_CLONE_MIG_DT_RUN target HANA DB crashed.Now when I try to bring it UP it does not come up.It is already 50 minutes now.Seeing the below log it seems it is trying to do a recovery.Any idea please..

tail -6 indexserver_<hostname>.30003.003.trc

[22045]{-1}[-1/-1] 2015-05-04 05:36:44.459951 i Logger           RecoveryHandlerImpl.cpp(00673) : Redo done up to position: 2120659904 and time: 2015-05-04 03:29:40.249577-04:00 (81%)

[22045]{-1}[-1/-1] 2015-05-04 05:46:54.939218 i Logger           RecoveryHandlerImpl.cpp(00673) : Redo done up to position: 2120659904 and time: 2015-05-04 03:29:40.249577-04:00 (81%)

[22045]{-1}[-1/-1] 2015-05-04 05:57:05.346442 i Logger           RecoveryHandlerImpl.cpp(00673) : Redo done up to position: 2137429952 and time: 2015-05-04 03:30:22.722735-04:00 (83%)

[22045]{-1}[-1/-1] 2015-05-04 06:07:15.829694 i Logger           RecoveryHandlerImpl.cpp(00673) : Redo done up to position: 2137429952 and time: 2015-05-04 03:30:22.722735-04:00 (83%)

[22045]{-1}[-1/-1] 2015-05-04 06:17:26.256954 i Logger           RecoveryHandlerImpl.cpp(00673) : Redo done up to position: 2137429952 and time: 2015-05-04 03:30:22.722735-04:00 (83%)

[22045]{-1}[-1/-1] 2015-05-04 06:27:36.720224 i Logger           RecoveryHandlerImpl.cpp(00673) : Redo done up to position: 2137429952 and time: 2015-05-04 03:30:22.722735-04:00 (83%)

Thanks,

Mofizur

[Former Member]

Hello ,

First look gives me a idea that your hana db has crashed .

Could you please share the detail indexserver trace - additionally please check if you have OOM trace files - share them too.

Which version of SUM are you using ??

Thanks

Dev

[Mofizur]

Hi Dev, 

How to share them?Can I em zip them and upload here?

Thanks,

Mofizur

[Former Member]

Yeah sure - just zip thm and attach.

Garbage collection would have taken time as you are in middle of migration and many threads would be running.

[Mofizur]

Hi Dev,

Trace files attached.Other traces are greater than 1 MB .Not sure how to send them.
Thanks,

Mofizur

indexserver_goazr1db144d.30003.rtedump.20150504-033242.005713.after_oom_cleanup.trc.txt.zip

[Former Member]

Looks like its an OoM issue - may be you can analyse more by using the following note  :

1840954 - Alerts related to HANA memory consumption

Could you also share OoM trace also ? and Indexserver trace ?

In your primary application server you can see which r3load is stuck or running? , from there you can find out which table in getting imported which would help to anylyze if table is to big to be handled y HANA.

Thanks

Dev

[Mofizur]

Hi Dev,

My main problem is the startup process does not fail..It is still running.I cancelled the earlier one and tries to start a new it says the same thing..still trying to recover..

Do we have any other option left?

Thanks,

Mofizur

indexserver_goazr1db144d.30003.003.trc.txt.zip

[Mofizur]

We contacted SAP Support and as per their suggestion after applying Revision 95 DB came up.
But now when I resume my migration I am getting below error

D:\SUM\abap\migrate_dt\MIGRATE_DT_CNT_00015_SXMSCLUP_IMP.LOG:
2 ETQ399 Connecting to database 'HDB'.
3EETQ008 Error message: DBSL error 99 (db code -1): Connect failed
4EETQ399 1 [dbhdbsql.c, 293]:
Try to connect via secure store (DEFAULT) on connection 0 ...
4EETQ399 0 [dbhdbsql.c, 355]: Connect to database failed, rc=1, rcSQL=591 (internal error)

I have already upgraded HDBCLIENT to Rev 95 on the server where DMO is running.Seems it is related to DBSL file.Guidance please.

Thanks,
Mofizur

[Former Member]

I am not 100% sure why this is happening but can you try to run following command under your hdbclient :

hdbuserstore LIST <sidadm>

hdbuserstore LIST SAPService<sid>

Thanks

Dev

[Mofizur]

Hi Dev,

We got it resolved now.SAP Support replied below..

"please upgarde to Hana revision 95 and try to startup Hana afterwards.The bug which causing the crash is fixed since then."

After that it went fine.But currently MAIN_POSTP/SUBMOD_MIG_BI_TASKS_POSTP1/RUN_CUBE_CALC_VIEW_CREATE is

running for 12.197 seconds.Is this a normal runtime?

I can see a blocked transaction in DB02.

goazr1db144d

Thanks,

Mofizur

[Former Member]

Hi Mofizur,

If I understand correctly your HANA System is not coming up currently right? did you manage to bring it up?

Did SAP support indicate any note for the issue?

Sunil

[Mofizur]

Hi,

That issue got resolved once we upgarded to Hana revision 95.But currently MAIN_POSTP/SUBMOD_MIG_BI_TASKS_POSTP1/RUN_CUBE_CALC_VIEW_CREATE is running very long.This actually triggers a background job at SAP level which is kind of stuck.Doing nothing due to lock at DB level.

dbslsdb.cpp  9778]

C  SQLCODE    : 131

C  SQLERRTEXT : transaction rolled back by lock wait timeout: Lock timeout occurs while waiting RECORD_LOCK of mode \

C               EXCLUSIVE(TRANSACTION_ID=81, UPDATE_TRANSACTION_ID=3732299)

C  sc_p=00000000195C7248,no=601,idc_p=0000000019627F10,con=1,act=1,slen=167,smax=256,#vars=5,stmt=000000001DD04E90,table=NRIV                         

C  stmtid = <0/SAPLRRBA                                /106956758/20150502230825>

C  SELECT TOP 1 "FROMNUMBER" , "TONUMBER" , "NRLEVEL" FROM "NRIV" WHERE "CLIENT" = ? AND "OBJECT" = ? A\

C  ND "SUBOBJECT" = ? AND "NRRANGENR" = ? AND "TOYEAR" = ? FOR UPDATE ;

B  ***LOG BY4=> sql error 131    performing FET on table NRIV [dbdbslst     3751]

B  ***LOG BY0=> transaction rolled back by lock wait timeout: Lock timeout occurs while waiting RECORD_LOCK of mode EXCLUSIVE(TRANSACTION_ID=81, UPDATE_TRANSACTION_ID=3732299) [dbdbslst     3751]

I have tried to manually kill the session but no success.

Thanks,

Mofizur