on 06-19-2008 6:45 PM
I am running an application on Windows 2008 Server Enterprise Edition that uses a sapdb 7.4.3 database. It intermittently crashes, and I get the following in knldiag.err:
2008-06-19 12:16:32 0x2D4 ERR 18006 EXCEPT EXCEPTION:0xc0000005 Addr:0x557dd4 ( 0:0x504d4e62:0:0 )
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE SymbolSearchPath: c:\program files\longitude\sapdb\indep_data\wrk\FZEDB1;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\p
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE gm;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\symbols;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\sap;C
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE :\Windows;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\sap\
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE -
> Register Dump <----
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE Eax=0x504d4e53 Ebx=0x0719e860 Ecx=0x504d4e53 Edx=0x504d4e53 Esi=0x7ddb8038 Edi=0x00000004
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE Eip=0x00557dd4 Esp=0x07187bd0 Ebp=0x07187be0
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE Cs=0x001b Ss=0x0023 Ds=0x0023 Es=0x0023 Fs=0x003b Gs=0x0000 Efl=0x10206
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE -
> Module List <----
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE |.text Start |.text End | Module File Name
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x00400000 | 0x0092d000 | C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\pgm\kernel.exe
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x01d10000 | 0x01d22000 | c:\program files\longitude\sapdb\depend\pgm\dbpinstall.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x06ca0000 | 0x06d55000 | C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\pgm\dbghelp.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x10000000 | 0x10056000 | C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\pgm\liboms.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x74330000 | 0x74336000 | C:\Windows\system32\rasadhlp.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x74340000 | 0x74379000 | C:\Windows\system32\OLEACC.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x74af0000 | 0x74c8e000 | C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6001.17051
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE : : : none5ce0931e083514f9\comctl32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x755c0000 | 0x755c8000 | C:\Windows\system32\VERSION.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x75700000 | 0x75735000 | C:\Windows\system32\dhcpcsvc.DLL
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x75740000 | 0x75761000 | C:\Windows\system32\dhcpcsvc6.DLL
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x757d0000 | 0x757e9000 | C:\Windows\system32\IPHLPAPI.DLL
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x759b0000 | 0x759b7000 | C:\Windows\system32\WINNSI.DLL
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x759e0000 | 0x75a0c000 | C:\Windows\system32\DNSAPI.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x75dd0000 | 0x75de4000 | C:\Windows\system32\Secur32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x76000000 | 0x7600d000 | c:\program files\longitude\sapdb\indep_prog\pgm\sqltcp.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x76010000 | 0x760d6000 | C:\Windows\system32\ADVAPI32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x760e0000 | 0x761bb000 | C:\Windows\system32\kernel32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x761c0000 | 0x7623d000 | C:\Windows\system32\USP10.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x76240000 | 0x76304000 | C:\Windows\system32\RPCRT4.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x763e0000 | 0x7648a000 | C:\Windows\system32\msvcrt.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x76490000 | 0x76496000 | C:\Windows\system32\NSI.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x764a0000 | 0x764eb000 | C:\Windows\system32\GDI32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x764f0000 | 0x77000000 | C:\Windows\system32\SHELL32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77000000 | 0x770c8000 | C:\Windows\system32\MSCTF.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77260000 | 0x7727e000 | C:\Windows\system32\IMM32.DLL
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x773b0000 | 0x774f4000 | C:\Windows\system32\ole32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77500000 | 0x7759d000 | C:\Windows\system32\USER32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x775f0000 | 0x77648000 | C:\Windows\system32\SHLWAPI.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77650000 | 0x7767d000 | C:\Windows\system32\WS2_32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77680000 | 0x776a9000 | C:\Windows\system32\imagehlp.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x776c0000 | 0x777e7000 | C:\Windows\system32\ntdll.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77880000 | 0x7790d000 | C:\Windows\system32\OLEAUT32.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x77910000 | 0x77919000 | C:\Windows\system32\LPK.DLL
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x7c000000 | 0x7c054000 | C:\Windows\system32\MSVCR70.dll
2008-06-19 12:16:32 0x2D4 ERR 19999 BTRACE | 0x7c140000 | 0x7c22e000 | C:\Windows\system32\MFC70.DLL
This is an existing software application that uses the sapdb database, and has been around for a few years. We would like it to run on WIndows 2008 as well as Windows 2003.
Here is the output from param_directgetall:
OK
KERNELVERSION KERNEL 7.4.3 BUILD 032-123-068-842
INSTANCE_TYPE OLTP
MCOD NO
RESTART_SHUTDOWN AUTO
SERVERDBFOR_SAP YES
_UNICODE NO
DEFAULT_CODE ASCII
DATE_TIME_FORMAT INTERNAL
CONTROLUSERID DBM
CONTROLPASSWORD
MAXARCHIVELOGS 2
MAXDATADEVSPACES 11
LOG_BACKUP_TO_PIPE NO
MAXBACKUPDEVS 2
BACKUP_BLOCK_CNT 8
LOG_MIRRORED NO
LOG_MODE SINGLE
_MAXDEVSPACES 14
MULTIO_BLOCK_CNT 4
DELAYLOGWRITER 0
LOG_IO_QUEUE 50
RESTARTTIME 600
MAXCPU 1
MAXUSERTASKS 45
TREERGNS 8
TRANSRGNS 8
TABRGNS 8
OMSREGIONS 0
OMSRGNS 25
OMS_HEAP_LIMIT 0
OMS_HEAP_COUNT 1
OMS_HEAP_BLOCKSIZE 10000
OMS_HEAP_THRESHOLD 100
OMS_VERS_THRESHOLD 2097152
HEAP_CHECK_LEVEL 0
ROWRGNS 8
MINSERVER_DESC 21
MAXSERVERTASKS 21
_MAXTRANS 272
MAXLOCKS 2720
LOCKSUPPLY_BLOCK 100
DEADLOCK_DETECTION 4
SESSION_TIMEOUT 900
OMS_STREAM_TIMEOUT 30
REQUEST_TIMEOUT 180
USEASYNC_IO YES
IOPROCSPER_DEV 1
IOPROCSFOR_PRIO 1
USEIOPROCS_ONLY NO
IOPROCSSWITCH 2
LRU_FOR_SCAN NO
PAGESIZE 8192
PACKETSIZE 36864
MINREPLYSIZE 4096
MBLOCKDATA_SIZE 32768
MBLOCKQUAL_SIZE 8192
MBLOCKSTACK_SIZE 16384
WORKSTACKSIZE 8192
WORKDATASIZE 8192
CATCACHE_MINSIZE 262144
CAT_CACHE_SUPPLY 1472
INIT_ALLOCATORSIZE 237568
TASKCLUSTER01 tw;al;ut;2000sv,100bup;10ev,10gc;
TASKCLUSTER02 ti,100dw;30000us;
TASKCLUSTER03 compress
MPRGN_QUEUE YES
MPRGN_DIRTY_READ NO
MPRGN_BUSY_WAIT NO
MPDISP_LOOPS 1
MPDISP_PRIO NO
XP_MP_RGN_LOOP 0
MP_RGN_LOOP 0
MPRGN_PRIO NO
MAXRGN_REQUEST 300
PRIOBASE_U2U 100
PRIOBASE_IOC 80
PRIOBASE_RAV 80
PRIOBASE_REX 40
PRIOBASE_COM 10
PRIOFACTOR 80
DELAYCOMMIT NO
SVP1_CONV_FLUSH NO
MAXGARBAGECOLL 0
MAXTASKSTACK 1024
DWIO_AREA_SIZE 50
DWIO_AREA_FLUSH 50
FBMDEVSP_COMPR 50
FBMDEVSP_BALANCE 10
FBMLOW_IO_RATE 10
CACHE_SIZE 25600
DWLRU_TAIL_FLUSH 25
XP_DATA_CACHE_RGNS 0
DATACACHE_RGNS 16
CONVERTER_REGIONS 8
_MAXDATAWRITER 16
SEQUENCE_CACHE 1
IDXFILELIST_SIZE 2048
SERVERDESC_CACHE 69
SERVERCMD_CACHE 22
DEVNO_BIT_COUNT 8
OPTIM_MAX_MERGE 500
OPTIM_INV_ONLY YES
OPTIM_CACHE NO
OPTIM_JOIN_FETCH 0
JOIN_SEARCH_LEVEL 0
JOIN_MAXTAB_LEVEL4 16
JOIN_MAXTAB_LEVEL9 5
READAHEADBLOBS 25
RUNDIRECTORY c:\program files\longitude\sapdb\indep_data\wrk\FZEDB1
_KERNELDIAGFILE knldiag
KERNELDIAGSIZE 800
_EVENTFILE knldiag.evt
_EVENTSIZE 0
_MAXEVENTTASKS 1
_MAXEVENTS 100
_KERNELTRACEFILE knltrace
TRACE_PAGES_TI 2
TRACE_PAGES_GC 0
TRACE_PAGES_AL 5
TRACE_PAGES_DW 3
TRACE_PAGES_US 10
TRACE_PAGES_UT 5
TRACE_PAGES_SV 5
TRACE_PAGES_EV 2
TRACE_PAGES_BUP 0
KERNELTRACESIZE 618
AKDUMP_ALLOWED YES
_KERNELDUMPFILE knldump
_RTEDUMPFILE rtedump
UTILITYPROTFILE dbm.utl
UTILITY_PROTSIZE 100
BACKUPHISTFILE dbm.knl
BACKUPMED_DEF dbm.mdf
MAXMESSAGE_FILES 0
EVENTALIVE_CYCLE 0
_SHAREDDYNDATA 25887
_SHAREDDYNPOOL 6453
ARCHIVE_LOG_001 LOG_0001
ARCHIVE_TYPE_001 F
ARCHIVE_SIZE_001 5000
DATADEV_0001 DATA0001
DATA_TYPE_0001 F
DATA_SIZE_0001 393216
USE_MEM_ENHANCE NO
MEM_ENHANCE_LIMIT 0
__PARAM_CHANGED___ 0
__PARAM_VERIFIED__ 2008-06-19 16:41:37
DIAG_HISTORY_NUM 2
DIAG_HISTORY_PATH c:\program files\longitude\sapdb\indep_data\wrk\FZEDB1\DIAGHISTORY
DIAGSEM 1
SHOW_MAX_STACK_USE NO
LOG_SEGMENT_SIZE 1666
SUPPRESS_CORE YES
FORMATTING_MODE PARALLEL
FORMAT_DATAVOLUME YES
HIRES_TIMER_TYPE CPU
LOAD_BALANCING_CHK 0
LOAD_BALANCING_DIF 10
LOAD_BALANCING_EQ 5
EXPAND_COM_TRACE NO
SYMBOL_DEMANGLING NO
USE_OPEN_DIRECT NO
USE_COROUTINES YES
MIN_RETENTION_TIME 60
MAX_RETENTION_TIME 480
CNSTVIEW_RW_SYNC NO
Here is the full contexts of knldiag.err:
-
Date Time TID(hex) Typ MsgID Label Message-Text
-
2008-06-19 16:40:29 --- Starting GMT 2008-06-19 16:40:29 7.4.3 Build 032-123-068-842
2008-06-19 16:41:48 --- Starting GMT 2008-06-19 16:41:48 7.4.3 Build 032-123-068-842
2008-06-20 06:31:30 0x2B8 ERR 18006 EXCEPT EXCEPTION:0xc0000005 Addr:0x557dd4 ( 0:0x2020202f:0:0 )
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE SymbolSearchPath: c:\program files\longitude\sapdb\indep_data\wrk\FZEDB1;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\p
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE gm;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\symbols;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\sap;C
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE :\Windows;C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\sap\
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE -
> Register Dump <----
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE Eax=0x20202020 Ebx=0x0687e860 Ecx=0x20202020 Edx=0x20202020 Esi=0x7de08038 Edi=0x00000004
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE Eip=0x00557dd4 Esp=0x06867bd0 Ebp=0x06867be0
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE Cs=0x001b Ss=0x0023 Ds=0x0023 Es=0x0023 Fs=0x003b Gs=0x0000 Efl=0x10202
2008-06-20 06:31:30 0x2B8 ERR 19999 BTRACE -
> Module List <----
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE |.text Start |.text End | Module File Name
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x003c0000 | 0x003d2000 | c:\program files\longitude\sapdb\depend\pgm\dbpinstall.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x00400000 | 0x0092d000 | C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\pgm\kernel.exe
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x01ae0000 | 0x01b95000 | C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\pgm\dbghelp.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x10000000 | 0x10056000 | C:\PROGRAM FILES\LONGITUDE\SAPDB\DEPEND\pgm\liboms.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x74390000 | 0x74396000 | C:\Windows\system32\rasadhlp.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x743b0000 | 0x743e9000 | C:\Windows\system32\OLEACC.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x74b70000 | 0x74d0e000 | C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6001.18000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE : : : none5cdbaa5a083979cc\comctl32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x754b0000 | 0x754b8000 | C:\Windows\system32\VERSION.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x756a0000 | 0x756d5000 | C:\Windows\system32\dhcpcsvc.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x757f0000 | 0x75811000 | C:\Windows\system32\dhcpcsvc6.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x75960000 | 0x75979000 | C:\Windows\system32\IPHLPAPI.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x759e0000 | 0x759e7000 | C:\Windows\system32\WINNSI.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x75a70000 | 0x75a9c000 | C:\Windows\system32\DNSAPI.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x75e60000 | 0x75e74000 | C:\Windows\system32\Secur32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76000000 | 0x7600d000 | c:\program files\longitude\sapdb\indep_prog\pgm\sqltcp.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76090000 | 0x760bd000 | C:\Windows\system32\WS2_32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x760c0000 | 0x760e9000 | C:\Windows\system32\imagehlp.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x760f0000 | 0x76234000 | C:\Windows\system32\ole32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76240000 | 0x7631b000 | C:\Windows\system32\kernel32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76320000 | 0x763bd000 | C:\Windows\system32\USER32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x763c0000 | 0x7643d000 | C:\Windows\system32\USP10.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76650000 | 0x766dd000 | C:\Windows\system32\OLEAUT32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x766e0000 | 0x7672b000 | C:\Windows\system32\GDI32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76730000 | 0x767f8000 | C:\Windows\system32\MSCTF.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76800000 | 0x76858000 | C:\Windows\system32\SHLWAPI.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x768b0000 | 0x7695a000 | C:\Windows\system32\msvcrt.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76960000 | 0x76a23000 | C:\Windows\system32\RPCRT4.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x76bc0000 | 0x776cf000 | C:\Windows\system32\SHELL32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x77750000 | 0x77877000 | C:\Windows\system32\ntdll.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x77890000 | 0x778ae000 | C:\Windows\system32\IMM32.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x778c0000 | 0x778c9000 | C:\Windows\system32\LPK.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x778d0000 | 0x778d6000 | C:\Windows\system32\NSI.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x778e0000 | 0x779a6000 | C:\Windows\system32\ADVAPI32.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x7c000000 | 0x7c054000 | C:\Windows\system32\MSVCR70.dll
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE | 0x7c140000 | 0x7c22e000 | C:\Windows\system32\MFC70.DLL
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE -
> Stack Back Trace <----
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak562long_type_st_entries_attach + 116 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:00557dd4 RA:00559458 FP:06867be0 SP:06867be8
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0687e860 02:0686ccac 03:0686d244 04:7df1032c
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak562.c ( Line: 7208 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak562ins_sel_buffer_build + 1384 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:00559458 RA:0055a047 FP:06869c1c SP:06869c24
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0687e860 02:0686ccac 03:06869c44 04:00000001
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak562.c ( Line: 5991 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak562do_select_insert + 2711 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:0055a047 RA:0055a0bb FP:0686cc98 SP:0686cca0
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:01000002 02:0686ccac 03:0687e860 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak562.c ( Line: 5734 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!a562_select_insert + 75 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:0055a0bb RA:004ba775 FP:0687e004 SP:0687e00c
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0287e860 02:0687eb53 03:0687e860 04:0687e860
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak562.c ( Line: 7116 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak35call_semantik + 757 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:004ba775 RA:004bbed9 FP:0687e038 SP:0687e040
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0087e860 02:00000000 03:00000000 04:0687e860
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak35.c ( Line: 6266 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!a35_asql_statement + 3705 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:004bbed9 RA:0056fbf0 FP:0687e088 SP:0687e090
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0087e860 02:00000000 03:00000001 04:0687e860
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak35.c ( Line: 5486 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak92not_only_execute + 1232 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:0056fbf0 RA:00571453 FP:0687e1d4 SP:0687e1dc
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0087e860 02:0687e2e4 03:0687e2fa 04:0687e2c4
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak92.c ( Line: 5141 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak92analyze_messagetype + 1203 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:00571453 RA:0057169d FP:0687e2ec SP:0687e2f4
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0687e860 02:00000000 03:00000000 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak92.c ( Line: 5512 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!a92_mode_analyzer + 205 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:0057169d RA:00585856 FP:0687e338 SP:0687e340
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0287e860 02:00000000 03:00000000 04:00000001
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak92.c ( Line: 6153 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak93one_command + 998 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:00585856 RA:005865e6 FP:0687e3a0 SP:0687e3a8
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0187e860 02:00000001 03:019f01c8 04:00000001
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak93.c ( Line: 6435 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!a93_user_commands + 758 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:005865e6 RA:004374ce FP:0687e42c SP:0687e434
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0087e860 02:0687e84f 03:0687e85b 04:0687ff18
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak93.c ( Line: 5566 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak91run_sql_process + 94 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:004374ce RA:004375e1 FP:0687e850 SP:0687e858
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0087e860 02:0687ff18 03:019f01a8 04:019f0390
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak91.c ( Line: 5293 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!ak91run_user_process + 225 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:004375e1 RA:00437629 FP:0687fe90 SP:0687fe98
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0687ff18 02:00000003 03:7ff793f8 04:0687ff00
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak91.c ( Line: 5338 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!a91mainprogam_with_allocator + 57 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:00437629 RA:006aada3 FP:0687fea4 SP:0687feac
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0387ff18 02:00000000 03:7fed97a0 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak91.c ( Line: 5364 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!gg941CreateAllocatorAndCallMainprog + 211 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:006aada3 RA:00437658 FP:0687ff00 SP:0687ff08
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:0687ff03 02:0687ff60 03:0000000c 04:0687ff18
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vgg941.cpp ( Line: 138 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!a91mainprogram + 40 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:00437658 RA:006fce0f FP:0687ff7c SP:0687ff84
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:00000000 02:00000000 03:7ff330c8 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vak91.c ( Line: 5351 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel!sql88k_kernel_fiber + 191 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:006fce0f RA:762520e6 FP:0687ffb8 SP:0687ffc0
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:7ff330c8 02:e6a723c7 03:00000000 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Source: vos88k.c ( Line: 435 )
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE kernel32!CreateFiberEx + 514 bytes
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:762520e6 RA:ffffffff FP:0687fff0 SP:0687ffc0
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:777f75de 02:00000000 03:00000000 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE ntdll!ffffffff <nosymbols>
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SFrame: IP:ffffffff RA:00000000 FP:0687fffc SP:0687ffc0
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE Params: 01:00000000 02:00000000 03:00000000 04:00000000
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE SymGetLineFromAddr() failed, rc = 126
2008-06-20 06:31:31 0x2B8 ERR 19999 BTRACE No crash dump file 'knlcrash.dmp' written
I'm not sure about the memory use, but I'm checking that now.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
> This is an existing software application that uses the sapdb database, and has been around for a few years.
Yes - this is ok and understandable - but you also never update your own software since a certain version?
> We would like it to run on WIndows 2008 as well as Windows 2003.
If your software didn't work under Windows 2008, wouldn't you adapt it so it does and create a new version?
I still don't get why you don't try with an actual build, the database version used is secondary - as long as it does work, isn't it?
Markus
Hi Dick,
looks like you're hitting bug http://maxdb.sap.com/webpts?wptsdetail=yes&ErrorType=0&ErrorID=1116809 .
An upgrade would be a solution for that...
KR Lars
I am in the process of updating our application to use MaxDB 7.6. The testing up to now indicates that this is no longer a problem, which is great.
We have run into a different problem running the older version of our application on Windows 2003 server that still uses SAPDB 7.4. (The new version is not yet in production.) The database intermittently crashes on a few machines, then restarts a few hours later. The knldiag.err file contains:
2008-08-12 12:39:06 0x148B0 ERR 18006 EXCEPT EXCEPTION:0xc0000005 Addr:0x30322068 ( 0:0x30322068:0:0 )
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE SymbolSearchPath: f:\longitude\sapdb\indep_data\wrk\FZEDB1;F:\LONGITUDE\SAPDB\DEPEND\pgm;F:\LONGITUDE\SAPDB\DEPEND
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE \symbols;F:\LONGITUDE\SAPDB\DEPEND\sap;C:\WINDOWS;F:\LONGITUDE\SAPDB\DEPEND\sap\
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE -
> Register Dump <----
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE Eax=0x076f0d00 Ebx=0x00000020 Ecx=0x076ff6f4 Edx=0x076da208 Esi=0x076fea98 Edi=0x076ff6f4
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE Eip=0x30322068 Esp=0x076d9738 Ebp=0x2414c600
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE Cs=0x001b Ss=0x0023 Ds=0x0023 Es=0x0023 Fs=0x003b Gs=0x0000 Efl=0x10246
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE -
> Module List <----
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE |.text Start |.text End | Module File Name
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x00400000 | 0x0092d000 | F:\LONGITUDE\SAPDB\DEPEND\pgm\kernel.exe
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x07a30000 | 0x07ae5000 | F:\LONGITUDE\SAPDB\DEPEND\pgm\dbghelp.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x10000000 | 0x10056000 | F:\LONGITUDE\SAPDB\DEPEND\pgm\liboms.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x71640000 | 0x7180d000 | C:\WINDOWS\AppPatch\AcGenral.DLL
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x71af0000 | 0x71b12000 | C:\WINDOWS\system32\ShimEng.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x71b70000 | 0x71ba6000 | C:\WINDOWS\system32\UxTheme.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x71bf0000 | 0x71bf8000 | C:\WINDOWS\system32\WS2HELP.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x71c00000 | 0x71c17000 | C:\WINDOWS\system32\WS2_32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x75e60000 | 0x75e87000 | C:\WINDOWS\system32\apphelp.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76000000 | 0x7600d000 | f:\longitude\sapdb\indep_prog\pgm\sqltcp.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76920000 | 0x769e4000 | C:\WINDOWS\system32\USERENV.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76aa0000 | 0x76acd000 | C:\WINDOWS\system32\WINMM.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76c10000 | 0x76c39000 | C:\WINDOWS\system32\imagehlp.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76ed0000 | 0x76ef9000 | C:\WINDOWS\system32\DNSAPI.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76f50000 | 0x76f63000 | C:\WINDOWS\system32\Secur32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x76f80000 | 0x76f88000 | C:\WINDOWS\system32\rasadhlp.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77380000 | 0x77412000 | C:\WINDOWS\system32\USER32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77420000 | 0x77523000 | C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.2778_
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE : : : x-ww_A8F04F11\comctl32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77670000 | 0x777a4000 | C:\WINDOWS\system32\ole32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77b70000 | 0x77b84000 | C:\WINDOWS\system32\MSACM32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77b90000 | 0x77b98000 | C:\WINDOWS\system32\VERSION.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77ba0000 | 0x77bfa000 | C:\WINDOWS\system32\msvcrt.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77c00000 | 0x77c49000 | C:\WINDOWS\system32\GDI32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77c50000 | 0x77cf0000 | C:\WINDOWS\system32\RPCRT4.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77d00000 | 0x77d8c000 | C:\WINDOWS\system32\OLEAUT32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77da0000 | 0x77df2000 | C:\WINDOWS\system32\SHLWAPI.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77e40000 | 0x77f42000 | C:\WINDOWS\system32\kernel32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x77f50000 | 0x77fec000 | C:\WINDOWS\system32\ADVAPI32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x7c000000 | 0x7c054000 | C:\WINDOWS\system32\MSVCR70.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x7c800000 | 0x7c8c0000 | C:\WINDOWS\system32\ntdll.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE | 0x7c8d0000 | 0x7d0d4000 | C:\WINDOWS\system32\SHELL32.dll
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE -
> Stack Back Trace <----
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE SymGetModuleInfo() failed, rc = 126
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE ?!30322068 <nosymbols>
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE SFrame: IP:30322068 RA:38303830 FP:076d9734 SP:076d9738
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE Params: 01:32313231 02:34353833 03:30303030 04:c5003030
2008-08-12 12:39:06 0x148B0 ERR 19999 BTRACE SymGetLineFromAddr() failed, rc = 126
2008-08-12 12:39:07 0x148B0 ERR 19999 BTRACE SymGetModuleInfo() failed, rc = 126
2008-08-12 12:39:07 0x148B0 ERR 19999 BTRACE ?!38303830 <nosymbols>
2008-08-12 12:39:07 0x148B0 ERR 19999 BTRACE SFrame: IP:38303830 RA:00000000 FP:2414c600 SP:076d9738
2008-08-12 12:39:07 0x148B0 ERR 19999 BTRACE Params: 01:00000000 02:00000000 03:00000000 04:00000000
2008-08-12 12:39:07 0x148B0 ERR 19999 BTRACE SymGetLineFromAddr() failed, rc = 126
2008-08-12 12:39:07 0x148B0 ERR 19999 BTRACE No crash dump file 'knlcrash.dmp' written
Can you tell me if this something that is also fixed in MaxDB 7.6? Can you tell me anything else about what might have caused this, such as whether it is related to something written to one of the tables?
Thank you.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Dick,
without the stackback trace nobody knows, which function failed here.
Unfortunately it's exactly that what also does not work:
SymGetLineFromAddr() failed, rc = 126
SymGetModuleInfo() failed, rc = 126
According to "xsysrc 126" the error# means:
>Errortext for errorcode 126:
>'The specified module could not be found.'
So, as a wild guess I would say: something with your SAP DB installation is not OK.
The [SymGetLineFromAddr|http://msdn.microsoft.com/en-us/library/ms681323(VS.85).aspx] function is part of the DbgHelp.dll - and this DLL is distributed with SAP DB (check it's location <dependend_path>/pgm).
Once this is fixed and there is a stack backtrace - we can check whether the bug is known and fixed.
BTW: It might be easier for everybody to follow if you just open a new thread for this new issue.
regards,
Lars
Thank you.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello Dick,
1) As I already wrote you that the SAPDB database version 7.4.3 software was not released on Windows 2008. And it's out of maintenance.
As of MAXDB version 7.5 the database software is planning to be released on Windows 2008.
Please pay attention that 7.6 MAXDB version is planning to be released on Windows 2008 for the customers & it is still testing on the local servers.
2) "looks like you're hitting bug http://maxdb.sap.com/webpts?wptsdetail=yes&ErrorType=0&ErrorID=1116809 ."
The database crashed during the 'ak562long_type_st_entries_attach' run according the back
stack information from you & if you will search at www.sapdb.org/webpts - Advanced search with
Reported 7.4.03.*
Search in Texts ak562long_type_st_entries_attach
=> You will see PTS 1116809
PTS 1116809: as 7.4.03 build 3 "ak562long_type_st_entries_attach now checks for all LONG columns of a table whether they can use the short col file or not (it only checked columns that were explicitly changed until now)" & you have the database version 7.4.3 BUILD 032.
3) It is good to know what SQL statement you was running at the time when the crash occurred.
Was the problem reproduced on Windows 2003 server?
As you could see in the posted back stack of the database crash, the problem occurred when insert...select is done and a long-column may be in one of the inserted columns ::
2008-06-20 06:31:31 0x2B8 ERR 19999
BTRACE -
> Stack Back Trace <----
kernel!ak562long_type_st_entries_attach + 116 bytes
Source: vak562.c ( Line: 7208 )
kernel!ak562ins_sel_buffer_build + 1384 bytes
Source: vak562.c ( Line: 5991 )
kernel!ak562do_select_insert + 2711 bytes
Source: vak562.c ( Line: 5734 )
kernel!a562_select_insert + 75 bytes
u2026
and there are some fixes/improvements done in MAXDB version 7.6. in vak562.c , for example, see PTS 1136344 at www.sapdb.org/webpts .
I recommend you to run the database update & see if the case could be reproduced with database version 7.6.03 Build 15, the database version 7.4.3 is out of maintenance.
Thank you and best regards, Natalia Khlopina
Thank you for your help. Is the most recent version I can update to 7.6 or is there something newer?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
We want this software to run on both Windows 2003 and Windows 2008. We've asked Microsoft about this, and they directed us to SAP.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello Dick,
1) The SAPDB database version 7.4.3 software was not released on Windows 2008.
And it's out of maintenance. As of MAXDB version 7.5 the database software
is planning to be released on Windows 2008.
Why did you install the database an application on Windows 2008 Server Enterprise Edition?
2) "Typically, the product runs correctly and then after about 3 hours, the database crashes with this error."
Please post all database parameter settings < run the dbm command 'param_directgetall' >.
Are you running out of memory?
3) You posted the knldiag.err file without the back stack trace information.
Could you please post the complete knldiag file to the thread.
< If the database was restarted once after the crash, then knldiag.old.
Check the subdirectory in the DIAGHISTORY of the database RUNDIRECTORY for the rtedump & knltrace and knldiag files. >
Thank you and best regards, Natalia Khlopina
It's an intermittent problem. It occurs when running our software application that uses the sapdb database. Typically, the product runs correctly and then after about 3 hours, the database crashes with this error. If we restart the database, it again runs correctly for a while before crashing again. It is not predictable when it will crash, and we have not seen this happen when running on Windows 2003.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I understand that this is an old version, but I can't upgrade at this time. Any information you can give me will be appreciated. Thank you.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
> I am running an application on Windows 2008 Server Enterprise Edition that uses a sapdb 7.4.3 database. It intermittently crashes, and I get the following in knldiag.err:
7.4.3 is LOOOONG out of support and I doubt it will work on a new operating system like Windows 2008. Any reason why you can't use 7.6?
Markus
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
87 | |
10 | |
10 | |
10 | |
7 | |
6 | |
6 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.