Help required! Cannot bring database into admin state

[Former Member]

Hi,

tonight our database crashed again due to a bad index problem and cannot be brought into admin state. I can't do a check and/or recovery either, because it's not in state admin.

Any suggestions???

Thanks,

Arne

This is the KNLDIAG.ERR extract:

2007-12-11 05:32:14 151 ERR 51080 SYSERROR -9049 BD Primkey from inv in primtree no

2007-12-11 05:32:14 151 ERR 53419 B*TREE BD600: predsep.k > sep.k: 0

2007-12-11 05:32:14 151 ERR 53000 B*TREE 07010000000000018EBE000000000000

2007-12-11 05:32:14 151 ERR 53000 B*TREE Index Root 834277

2007-12-11 05:32:14 151 ERR 53000 B*TREE bd600BuildSeparator: lef: 383217

2007-12-11 05:32:14 151 ERR 53000 B*TREE 07010000000000018EBE000000000000

2007-12-11 05:32:14 151 ERR 53000 B*TREE Index Root 834277

2007-12-11 05:32:14 151 ERR 53000 B*TREE bd600BuildSeparator: rig: 153800

2007-12-11 05:32:14 151 ERR 53250 INDEX Bad Index 834277 (Root)

2007-12-11 05:32:14 151 ERR 53250 INDEX Reason "System error: BD Invalid leaves"

2007-12-11 05:32:14 151 ERR 51080 SYSERROR -9049 BD Primkey from inv in primtree no

2007-12-11 05:33:29 158 ERR 53093 DATACACH Occup PNO 1245190 PTR C88162B72A000000

2007-12-11 05:33:29 158 ERR 53000 B*TREE 07040000000000018AFA000000000000

2007-12-11 05:33:29 158 ERR 53000 B*TREE Index Root 48642

2007-12-11 05:33:29 158 ERR 3 Admin Kernel_Administration.cpp:606

2007-12-11 05:33:29 158 ERR 3 Admin 2007-12-11 05:33:29 Admin Error 3

2007-12-11 05:33:29 158 ERR 3 Admin Database state: OFFLINE

2007-12-11 05:33:29 158 ERR 3 Admin + vgg999.cpp:153

2007-12-11 05:33:29 158 ERR 3 Admin + 2007-12-11 05:33:29 Admin Error 2

2007-12-11 05:33:29 158 ERR 3 Admin - A fatal error caused EMERGENCY SHUTDOWN. core is written

2007-12-11 05:33:29 158 ERR 11196 DBCRASH vabort:Emergency Shutdown, Kernel_Administration.cpp: 606

2007-12-11 05:33:29 158 ERR 11599 BTRACE -

---

> Emergency Stack Back Trace <----

2007-12-11 05:33:29 158 ERR 11599 BTRACE (0):0xddcc95

2007-12-11 05:33:29 158 ERR 11599 BTRACE (1):0xddccf9

2007-12-11 05:33:29 158 ERR 11599 BTRACE (2):0xddce42

2007-12-11 05:33:29 158 ERR 11599 BTRACE (3):0xddce77

2007-12-11 05:33:29 158 ERR 11599 BTRACE (4):0xdfeed7

2007-12-11 05:33:29 158 ERR 11599 BTRACE (5):0xd1bbdd

2007-12-11 05:33:29 158 ERR 11599 BTRACE (6):0xce28ab

2007-12-11 05:33:29 158 ERR 11599 BTRACE (7):0xa25e4f

2007-12-11 05:33:29 158 ERR 11599 BTRACE (8):0x91343b

2007-12-11 05:33:29 158 ERR 11599 BTRACE (9):0x914814

2007-12-11 05:33:29 158 ERR 11599 BTRACE (10):0x94d8ba

2007-12-11 05:33:29 158 ERR 11599 BTRACE (11):0x9d81a4

2007-12-11 05:33:29 158 ERR 11599 BTRACE (12):0x9b88e0

2007-12-11 05:33:29 158 ERR 11599 BTRACE (13):0x93c360

2007-12-11 05:33:29 158 ERR 11599 BTRACE (14):0x84e68f

2007-12-11 05:33:29 158 ERR 11599 BTRACE (15):0x8439a4

2007-12-11 05:33:29 158 ERR 11599 BTRACE (16):0x844455

2007-12-11 05:33:29 158 ERR 11599 BTRACE (17):0x84c43d

2007-12-11 05:33:29 158 ERR 11599 BTRACE (18):0x84af19

2007-12-11 05:33:29 158 ERR 11599 BTRACE (19):0x8350fb

2007-12-11 05:33:29 158 ERR 11599 BTRACE (20):0x80ba7f

2007-12-11 05:33:29 158 ERR 11599 BTRACE (21):0x80e28f

2007-12-11 05:33:29 158 ERR 11599 BTRACE (22):0x74195f

2007-12-11 05:33:29 158 ERR 11599 BTRACE (23):0x742ac0

2007-12-11 05:33:29 158 ERR 11599 BTRACE (24):0x737886

2007-12-11 05:33:29 158 ERR 11599 BTRACE (25):0x738750

2007-12-11 05:33:29 158 ERR 11599 BTRACE (26):0x7f150c

2007-12-11 05:33:29 158 ERR 11599 BTRACE (27):0x7f200d

2007-12-11 05:33:29 158 ERR 11599 BTRACE (28):0x7f3047

2007-12-11 05:33:29 158 ERR 11599 BTRACE (29):0x7f5d88

2007-12-11 05:33:29 158 ERR 11599 BTRACE (30):0xcf39a4

2007-12-11 05:33:29 158 ERR 11599 BTRACE (31):0xcf3b83

2007-12-11 05:33:29 158 ERR 11599 BTRACE (32):0xd7a03f

2007-12-11 05:33:29 158 ERR 11599 BTRACE (33):0xe23dec

2007-12-11 05:33:29 158 ERR 11599 BTRACE (34):0xe2405a

2007-12-11 05:33:29 158 ERR 11599 BTRACE (35):0xdd0b21

2007-12-11 05:33:29 158 ERR 11599 BTRACE (36):0x2a95673aff

2007-12-11 05:33:29 158 ERR 11599 BTRACE (37):0x2a95e884b3

2007-12-11 05:33:29 158 ERR 11599 BTRACE -

---

> Module List <----

2007-12-11 05:33:29 158 ERR 11599 BTRACE |.text Start |.text End | Module File Name

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000000000400000|0x0000000000fc4000| /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95556000|0x0000002a9556b000| /lib64/ld-2.3.3.so

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a9566d000|0x0000002a9567b000| /lib64/tls/libpthread.so.0

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95782000|0x0000002a95784000| /lib64/libdl.so.2

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95885000|0x0000002a9588c000| /lib64/tls/librt.so.1

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a9598d000|0x0000002a95a3b000| /usr/lib64/libstdc++.so.5.0.6

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95b6b000|0x0000002a95bbf000| /lib64/tls/libm.so.6

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95cc3000|0x0000002a95ccd000| /lib64/libgcc_s.so.1

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95dce000|0x0000002a95ed9000| /lib64/tls/libc.so.6

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a9610e000|0x0000002a96117000| /lib64/libnss_files.so.2

2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002ab96da000|0x0000002ab9904000| /opt/sdb/7600/lib/liboms.so

2007-12-11 05:33:29 158 ERR 11599 BTRACE

2007-12-11 05:33:29 158 ERR 11599 BTRACE -

---

> Symbolic Stack Back Trace <----

2007-12-11 05:33:29 158 ERR 11599 BTRACE 0: 0x0000000000ddc32d eo670_UnixTraceStack +0x01ad

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 1: 0x0000000000ddccf9 eo670_CTraceContextStackOCB +0x0009

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 2: 0x0000000000ddce42 eo670_CTraceStackOCB +0x0032

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 3: 0x0000000000ddce77 eo670_CTraceStack +0x0017

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 4: 0x0000000000dfeed7 vabort +0x0027

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 5: 0x0000000000d1bbdd Z9RTECrashRK20SAPDBErr_MessageList +0x008d

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 6: 0x0000000000ce28ab ZN21KernelAdministration7OfflineERK20SAPDBErr_MessageListbb +0x034b

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 7: 0x0000000000a25e4f gg999Abort +0x031f

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 8: 0x000000000091343b bd20_Abort +0x003b

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 9: 0x0000000000914814 bd20RReleasePage +0x0124

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 10: 0x000000000094d8ba b13r_release_node +0x00ba

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 11: 0x00000000009d81a4 ZN11cbd600Node12bd600ReleaseEb +0x0044

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 12: 0x00000000009b88e0 Z35bd402SearchIndexForQualifiedRecordsR17cbd300InvCurrentR12tgg00_FileIdPhRsS3_iS3_iS3_i

2007-12-11 05:33:29 158 ERR 11599 BTRACE S3_S4_iP10tsp00_VoidR23tgg00_SelectFieldsParamR15tgg00_StackDescR10tsp00_EnumI22tgg00_Lock

2007-12-11 05:33:29 158 ERR 11599 BTRACE ReqMode_EnumLi7EhERb +0x0770

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 13: 0x000000000093c360 b03select_invrec +0x0510

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 14: 0x000000000084e68f ZN22JoinInvAccessOperator13SelectRecordsEv +0x011f

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 15: 0x00000000008439a4 ZN19JoinAccessOperator4NextERP9tgg00_Rec +0x00a4

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 16: 0x0000000000844455 ZN20JoinIAccessOperator4nextEPP9tgg00_Rec +0x0025

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 17: 0x000000000084c43d ZN18JoinJoinOperator24nextEPP9tgg00_Rec +0x00fd

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 18: 0x000000000084af19 ZN18JoinJoinOperator24NextERP9tgg00_Rec +0x0059

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 19: 0x00000000008350fb a686_create_join_result +0x073b

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 20: 0x000000000080ba7f a682_execute_join_operator +0x036f

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 21: 0x000000000080e28f a682_only_ex_join +0x020f

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 22: 0x000000000074195f a505most_execute +0x0bff

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 23: 0x0000000000742ac0 a505loop_most_execute +0x0360

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 24: 0x0000000000737886 a501exec_with_change_rec +0x05b6

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 25: 0x0000000000738750 a501execute +0x0510

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 26: 0x00000000007f150c ak92analyze_messagetype +0x042c

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 27: 0x00000000007f200d a92_mode_analyzer +0x019d

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 28: 0x00000000007f3047 ak93one_command +0x0477

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 29: 0x00000000007f5d88 a93_user_commands +0x0528

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 30: 0x0000000000cf39a4 Z7SQLTaskR20takall_command_globRhb +0x0074

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 31: 0x0000000000cf3b83 Z11KernelMainR13RTETask_ITask +0x01b3

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 32: 0x0000000000d7a03f RTETask_TaskMain +0x005f

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 33: 0x0000000000e23dec Z23en88CallKernelTaskMainP9TASK_TYPE +0x009c

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE 34: 0x0000000000e2405a en88_RunPthreadTask +0x003a

2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel

2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]

2007-12-11 05:33:29 158 ERR 11599 BTRACE -

---

> End of Stack Back Trace <----

2007-12-11 05:33:30 158 ERR 7 Messages Msg_List.cpp:3617

2007-12-11 05:33:30 158 ERR 7 Messages 2007-12-11 05:33:30 Messages Error 7

2007-12-11 05:33:30 158 ERR 7 Messages Begin of dump of registered messages

2007-12-11 05:33:30 158 ERR 2 Admin vgg999.cpp:153

2007-12-11 05:33:30 158 ERR 2 Admin 2007-12-11 05:33:29 Admin Error 2

2007-12-11 05:33:30 158 ERR 2 Admin A fatal error caused EMERGENCY SHUTDOWN. core is written

2007-12-11 05:33:30 158 ERR 2 Admin vgg999.cpp:153

2007-12-11 05:33:30 158 ERR 2 Admin 2007-12-11 05:33:29 Admin Error 2

2007-12-11 05:33:30 158 ERR 2 Admin A fatal error caused EMERGENCY SHUTDOWN. core is written

2007-12-11 05:33:30 158 ERR 8 Messages Msg_List.cpp:3636

2007-12-11 05:33:30 158 ERR 8 Messages 2007-12-11 05:33:30 Messages Error 8

2007-12-11 05:33:30 158 ERR 8 Messages End of the message list registry dump

2007-12-11 05:33:30 158 ERR 11196 DBCRASH vabort:Emergency Shutdown, Task: 141

2007-12-11 05:33:37 0 ERR 12006 DBCRASH Kernel exited without core and exit status 0x200

2007-12-11 05:33:37 0 ERR 12010 DBCRASH Kernel exited exit code 2

2007-12-11 05:33:38 ___ Stopping GMT 2007-12-11 04:33:38 7.6.00 Build 016-123-109-428

[Former Member]

Hi,

look at following link might help you

h***://ftp.support.veritas.com/pub/support/products/NetBackup_Enterprise_Server/276336.pdf

(change *** to ttp)

regards,

kaushal