Dispatcher won't stay up. (disp+work.exe)

[Former Member]

I have an SAP BW 3.5 system on WIndows 2003 with an Oracle database.

It was working fine for some time. Now the system won't come online. When I start the system everything goes green temporarily and then shuts down as the Dispatcher stops and won't come back.

I can't figure out how to fix this. I have a trace file from the dev_disp log that I will paste below. Would really appreciate someone with some basis experience to walk me through how to fix this. Your help is greatly appreciated. Thank you in advance.

-start of log--

-

---

trc file: "dev_disp", trc level: 1, release: "640"

-

---

Thu Jan 18 10:26:14 2007

kernel runs with dp version 128(ext=102) (@(#) DPLIB-INT-VERSION-128)

length of sys_adm_ext is 312 bytes

systemid 560 (PC with Windows NT)

relno 6400

patchlevel 0

patchno 21

intno 20020600

make: multithreaded, ASCII

pid 3816

***LOG Q00=> DpSapEnvInit, DPStart (02 3816) [dpxxdisp.c 1100]

shared lib "dw_xml.dll" version 21 successfully loaded

shared lib "dw_xtc.dll" version 21 successfully loaded

shared lib "dw_stl.dll" version 21 successfully loaded

shared lib "dw_gui.dll" version 21 successfully loaded

Thu Jan 18 10:26:18 2007

• WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 4 seconds

***LOG GZZ=> 1 possible network problems detected - check tracefile and

adjust the DNS settings [dpxxtool2.c 3886]

MtxInit: -2 0 0

DpSysAdmExtInit: ABAP is active

DpSysAdmExtInit: JAVA is not active

DpShMCreate: sizeof(wp_adm) 6624 (828)

DpShMCreate: sizeof(tm_adm) 2219848 (11044)

DpShMCreate: sizeof(wp_ca_adm) 18000 (60)

DpShMCreate: sizeof(appc_ca_adm) 6000 (60)

DpShMCreate: sizeof(comm_adm) 192000 (384)

DpShMCreate: sizeof(vmc_adm) 0 (320)

DpShMCreate: sizeof(wall_adm) (22440/34344/56/100)

DpShMCreate: SHM_DP_ADM_KEY (addr: 04960040, size: 2505416)

DpShMCreate: allocated sys_adm at 04960040

DpShMCreate: allocated wp_adm at 049617A8

DpShMCreate: allocated tm_adm_list at 04963188

DpShMCreate: allocated tm_adm at 049631B0

DpShMCreate: allocated wp_ca_adm at 04B810F8

DpShMCreate: allocated appc_ca_adm at 04B85748

DpShMCreate: allocated comm_adm_list at 04B86EB8

DpShMCreate: allocated comm_adm at 04B86ED0

DpShMCreate: allocated vmc_adm_list at 04BB5CD0

DpShMCreate: system runs without vmc_adm

DpShMCreate: allocated ca_info at 04BB5CF8

DpShMCreate: allocated wall_adm at 04BB5D00

MBUF state OFF

Thu Jan 18 10:26:19 2007

EmInit: MmSetImplementation( 2 ).

<ES> client 0 initializing ....

<ES> InitFreeList

<ES> block size is 1024 kByte.

Using implementation std

<EsNT> Memory Reset enabled as NT default

<EsNT> EsIUnamFileMapInit: Initialize the memory 500 MB

<ES> 499 blocks reserved for free list.

ES initialized.

rdisp/http_min_wait_dia_wp : 1 -> 1

***LOG Q0K=> DpMsAttach, mscon ( SAPSERV-2) [dpxxdisp.c 9719]

CCMS: Initalizing shared memory of size 40000000 for monitoring segment.

Thu Jan 18 10:26:20 2007

CCMS: start to initalize 3.X shared alert area (first segment).

DpMsgAdmin: Set release to 6400, patchlevel 0

MBUF state PREPARED

MBUF component UP

DpMBufHwIdSet: set Hardware-ID

***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1025]

DpMsgAdmin: Set patchno for this platform to 21

Release check o.K.

Thu Jan 18 10:26:59 2007

• ERROR => W0 (pid 2544) died [dpxxdisp.c 12170]

• ERROR => W1 (pid 4040) died [dpxxdisp.c 12170]

my types changed after wp death/restart 0xbf --> 0xbe

• ERROR => W2 (pid 3848) died [dpxxdisp.c 12170]

my types changed after wp death/restart 0xbe --> 0xbc

• ERROR => W3 (pid 3308) died [dpxxdisp.c 12170]

my types changed after wp death/restart 0xbc --> 0xb8

• ERROR => W4 (pid 3996) died [dpxxdisp.c 12170]

• ERROR => W5 (pid 2964) died [dpxxdisp.c 12170]

my types changed after wp death/restart 0xb8 --> 0xb0

• ERROR => W6 (pid 2700) died [dpxxdisp.c 12170]

my types changed after wp death/restart 0xb0 --> 0xa0

• ERROR => W7 (pid 3260) died [dpxxdisp.c 12170]

my types changed after wp death/restart 0xa0 --> 0x80

• DP_FATAL_ERROR => DpWPCheck: no more work processes

• DISPATCHER EMERGENCY SHUTDOWN ***

increase tracelevel of WPs

killing W0-2544 (SIGUSR2)

• ERROR => DpWpKill(2544, SIGUSR2) failed [dpxxtool.c 2468]

killing W1-4040 (SIGUSR2)

• ERROR => DpWpKill(4040, SIGUSR2) failed [dpxxtool.c 2468]

killing W2-3848 (SIGUSR2)

• ERROR => DpWpKill(3848, SIGUSR2) failed [dpxxtool.c 2468]

killing W3-3308 (SIGUSR2)

• ERROR => DpWpKill(3308, SIGUSR2) failed [dpxxtool.c 2468]

killing W4-3996 (SIGUSR2)

• ERROR => DpWpKill(3996, SIGUSR2) failed [dpxxtool.c 2468]

killing W5-2964 (SIGUSR2)

• ERROR => DpWpKill(2964, SIGUSR2) failed [dpxxtool.c 2468]

killing W6-2700 (SIGUSR2)

• ERROR => DpWpKill(2700, SIGUSR2) failed [dpxxtool.c 2468]

killing W7-3260 (SIGUSR2)

• ERROR => DpWpKill(3260, SIGUSR2) failed [dpxxtool.c 2468]

NiWait: sleep (10000 msecs) ...

NiISelect: timeout 10000 ms

NiISelect: maximum fd=1637

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Thu Jan 18 10:27:09 2007

NiISelect: TIMEOUT occured (10000 ms)

dump system status

Workprocess Table (long) Thu Jan 18 15:27:09 2007

========================

No Ty. Pid Status Cause Start Err Sem CPU Time Program Cl User

Action Table

-

---

0 DIA 2544 Ended no 1 0 0

1 DIA 4040 Ended no 1 0 0

2 UPD 3848 Ended no 1 0 0

3 ENQ 3308 Ended no 1 0 0

4 BTC 3996 Ended no 1 0 0

5 BTC 2964 Ended no 1 0 0

6 SPO 2700 Ended no 1 0 0

7 UP2 3260 Ended no 1 0 0

Dispatcher Queue Statistics Thu Jan 18 15:27:09 2007

===========================

--------

---

+

---

+

---

+

---

+--

---

+

Typ

now

high

max

writes

reads

--------

---

+

---

+

---

+

---

+--

---

+

NOWP

0

3

2000

10

10

--------

---

+

---

+

---

+

---

+--

---

+

DIA

6

6

2000

6

0

--------

---

+

---

+

---

+

---

+--

---

+

UPD

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

ENQ

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

BTC

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

SPO

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

UP2

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

max_rq_id 14

wake_evt_udp_now 0

wake events total 9, udp 8 ( 88%), shm 1 ( 11%)

since last update total 9, udp 8 ( 88%), shm 1 ( 11%)

Dump of tm_adm structure: Thu Jan 18 15:27:09 2007

=========================

Term uid man user term lastop mod wp ta a/i (modes)

RM-T8, U9, , bigdaddy, 10:26:49, M0, W-1, , 1/0

Workprocess Comm. Area Blocks Thu Jan 18 15:27:09 2007

=============================

Slots: 300, Used: 2, Max: 1

--------

---

+

---

+--

---

+

id

owner

pid

eyecatcher

--------

---

+

---

+--

---

+

0	DISPATCHER	-1	WPCAAD000
1	DISPATCHER	-1	WPCAAD001

NiWait: sleep (5000 msecs) ...

NiISelect: timeout 5000 ms

NiISelect: maximum fd=1637

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Thu Jan 18 10:27:14 2007

NiISelect: TIMEOUT occured (5000 ms)

Shutdown server ...

DpModState: buffer in state MBUF_PREPARED

NiBufSend starting

NiIWrite: write 110, 1 packs, MESG_IO, hdl 3, data complete

MsINiWrite: sent 110 bytes

MsIModState: change state to SHUTDOWN

DpModState: change server state from STARTING to SHUTDOWN

Switch off Shared memory profiling

ShmProtect( 57, 3 )

ShmProtect(SHM_PROFILE, SHM_PROT_RW

ShmProtect( 57, 1 )

ShmProtect(SHM_PROFILE, SHM_PROT_RD

DpWakeUpWps: wake up all wp's

Stop work processes...

Stop gateway

killing process (2952) (SOFT_KILL)

Stop icman

killing process (3548) (SOFT_KILL)

Terminate gui connections

send SHUTDOWN to REM TM 8

blks_in_queue/wp_ca_blk_no/wp_max_no = 2/300/8

LOCK WP ca_blk 2

return errno (-17) to T8

errormsg without overhead: take mode 0

NiBufIAlloc: malloc NiBufadm, to 0 bytes

try to send 14 to M0

NiBufSend starting

NiIWrite: write 14, 1 packs, MESG_IO, hdl 7, data complete

REL WP ca_blk 2

set status of T8 to TM_DISCONNECTED

NiSelClear: removed hdl 7 from selectset

DpDelSocketInfo: del info for socket 7 (type=3)

NiBufClose: clear extensions for hdl 7

NiBufSetStat: bufstat of hdl 7 changed from OK to OFF

NiICloseHandle: shutdown and close hdl 7 / socket 1468

dp_tm_adm[8].stat = DP_SLOT_FREE

DpGetSchedule: next schedule 1169134039

DpGetSchedule: no schedule found

DpITmSlotRelease: release slot 8

DpListInsert: insert elem 8 into tmadm_free_list (at begin)

DpListInsert: 193 elems in tmadm_free_list

DpListRemove: remove elem 8 from tmadm_inuse_list

DpListRemove: 8 elems in tmadm_inuse_list

[DpProcDied] Process lives (PID:2952 HANDLE:1616)

waiting for termination of gateway

NiWait: sleep (1000 msecs) ...

NiISelect: timeout 1000 ms

NiISelect: maximum fd=1637

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Thu Jan 18 10:27:15 2007

NiISelect: TIMEOUT occured (1000 ms)

[DpProcDied] Process died (PID:2952 HANDLE:1616)

[DpProcDied] Process died (PID:3548 HANDLE:1608)

DpHalt: cancel all lcom connections

MPI CancelAll 2 -> 0

MPI DeleteAll 2 -> 0

NiIMyHostName: hostname = 'SAPSERV-2'

AdGetSelfIdentRecord: >

<

AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0

AdCvtRecToExt: opcode 4 (AD_STARTSTOP), ser 0, ex 0, errno 0

DpConvertRequest: net size = 163 bytes

NiBufSend starting

NiIWrite: write 562, 1 packs, MESG_IO, hdl 3, data complete

MsINiWrite: sent 562 bytes

send msg (len 110+452) to name -, type 4, key -

detach from message server

***LOG Q0M=> DpMsDetach, ms_detach () [dpxxdisp.c 9945]

NiBufSend starting

NiIWrite: write 110, 1 packs, MESG_IO, hdl 3, data complete

MsINiWrite: sent 110 bytes

MsIDetach: send logout to msg_server

MsIDetach: call exit function

DpMsShutdownHook called

NiSelClear: removed hdl 3 from selectset

MBUF state OFF

AdGetSelfIdentRecord: >

<

AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0

AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0

AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0

blks_in_queue/wp_ca_blk_no/wp_max_no = 2/300/8

LOCK WP ca_blk 2

make DISP owner of wp_ca_blk 2

DpRqPutIntoQueue: put request into queue (reqtype 1, prio LOW, rq_id 20)

MBUF component DOWN

NiBufClose: clear extensions for hdl 3

NiBufSetStat: bufstat of hdl 3 changed from OK to OFF

NiICloseHandle: shutdown and close hdl 3 / socket 1620

MsIDetach: detach MS-system

EsCleanup ....

***LOG Q05=> DpHalt, DPStop ( 3816) [dpxxdisp.c 8478]

Good Bye .....

[Former Member]

HI,

FIRST TRY THIS

CONN TO YR DB IN NOLOG MODE AS SYSDBA

SHUTDOWN IMMEDIATE

AND AGAIN START THE DB BY

STARTUP OPEN

IT SEEMS THERE IS A PROM IN THE LISTNER CONFIG

THIS MAY RESET IT UP.

OTHERWISE IT HAS TO BE RECONFIGURED MANUALLY.

REGARDS

[former_member185954]

Hi Peter,

Can you post dev_w0 trace file, it should contain details of why the process got killed.

Regards,

Siddhesh

[Former Member]

Hi Siddhesh,

Thanks for trying to help. I have found the dev_w0 trace file and pasted it below:

-

---

start file ---

-

---

trc file: "dev_w0", trc level: 1, release: "640"

-

---

*

• ACTIVE TRACE LEVEL 1

• ACTIVE TRACE COMPONENTS all, M

*

B

B Thu Jan 18 12:30:04 2007

B create_con (con_name=R/3)

B Loading DB library 'C:\usr\sap\B01\SYS\exe\run\dboraslib.dll' ...

B Library 'C:\usr\sap\B01\SYS\exe\run\dboraslib.dll' loaded

B Version of 'C:\usr\sap\B01\SYS\exe\run\dboraslib.dll' is "640.00",

patchlevel (0.19)

B New connection 0 created

M systemid 560 (PC with Windows NT)

M relno 6400

M patchlevel 0

M patchno 21

M intno 20020600

M make: multithreaded, ASCII

M pid 3724

M

M ***LOG Q0Q=> tskh_init, WPStart (Workproc 0 3724) [dpxxdisp.c 1160]

I MtxInit: -2 0 0

M DpSysAdmExtCreate: ABAP is active

M DpSysAdmExtCreate: JAVA is not active

M DpShMCreate: sizeof(wp_adm) 6624 (828)

M DpShMCreate: sizeof(tm_adm) 2219848 (11044)

M DpShMCreate: sizeof(wp_ca_adm) 18000 (60)

M DpShMCreate: sizeof(appc_ca_adm) 6000 (60)

M DpShMCreate: sizeof(comm_adm) 192000 (384)

M DpShMCreate: sizeof(vmc_adm) 0 (320)

M DpShMCreate: sizeof(wall_adm) (22440/34344/56/100)

M DpShMCreate: SHM_DP_ADM_KEY (addr: 05160040, size: 2505416)

M DpShMCreate: allocated sys_adm at 05160040

M DpShMCreate: allocated wp_adm at 051617A8

M DpShMCreate: allocated tm_adm_list at 05163188

M DpShMCreate: allocated tm_adm at 051631B0

M DpShMCreate: allocated wp_ca_adm at 053810F8

M DpShMCreate: allocated appc_ca_adm at 05385748

M DpShMCreate: allocated comm_adm_list at 05386EB8

M DpShMCreate: allocated comm_adm at 05386ED0

M DpShMCreate: allocated vmc_adm_list at 053B5CD0

M DpShMCreate: system runs without vmc_adm

M DpShMCreate: allocated ca_info at 053B5CF8

M DpShMCreate: allocated wall_adm at 053B5D00

X EmInit: MmSetImplementation( 2 ).

X <ES> client 0 initializing ....

X Using implementation std

M <EsNT> Memory Reset enabled as NT default

X ES initialized.

M

M Thu Jan 18 12:30:05 2007

M calling db_connect ...

C Got ORACLE_HOME=E:\oracle\ora92 from environment

C Client NLS settings: AMERICAN_AMERICA.WE8DEC

C Logon as OPS$-user to get SAPB01's password

C Connecting as /@S01 on connection 0 ...

C Attaching to DB Server S01 (con_hdl=0,svchp=06E14A4C,svrhp=037C27C4)

C Starting user session

(con_hdl=0,svchp=06E14A4C,srvhp=037C27C4,usrhp=037CAFA4)

C *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033

[dboci.c 3721]

C Detaching from DB Server (con_hdl=0,svchp=06E14A4C,srvhp=037C27C4)

C *** ERROR => CONNECT failed with sql error '1033'

[dbsloci.c 9753]

C Try to connect with default password

C Connecting as SAPB01/<pwd>@S01 on connection 0 ...

C Attaching to DB Server S01 (con_hdl=0,svchp=06E14A4C,svrhp=037C27C4)

C Starting user session

(con_hdl=0,svchp=06E14A4C,srvhp=037C27C4,usrhp=037CAFA4)

C *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033

[dboci.c 3721]

C Detaching from DB Server (con_hdl=0,svchp=06E14A4C,srvhp=037C27C4)

C *** ERROR => CONNECT failed with sql error '1033'

[dbsloci.c 9753]

B ***LOG BV3=> severe db error 1033 ; work process is stopped [dbsh#2

@ 1195] [dbsh 1195 ]

B ***LOG BY2=> sql error 1033 performing CON [dblink#1 @ 419] [dblink

0419 ]

B ***LOG BY0=> ORA-01033: ORACLE initialization or shutdown in progress

[dblink#1 @ 419] [dblink 0419 ]

M ***LOG R19=> tskh_init, db_connect ( DB-Connect 000256) [thxxhead.c

1271]

M in_ThErrHandle: 1

M *** ERROR => tskh_init: db_connect (step 1, th_errno 13, action 3, level

1) [thxxhead.c 9379]

M

M Info for wp 0

M

M stat = 4

M reqtype = 1

M act_reqtype = -1

M rq_info = 0

M tid = -1

M mode = 255

M len = -1

M rq_id = 65535

M rq_source = 255

M last_tid = 0

M last_mode = 0

M int_checked_resource(RFC) = 0

M ext_checked_resource(RFC) = 0

M int_checked_resource(HTTP) = 0

M ext_checked_resource(HTTP) = 0

M report = > <

M action = 0

M tab_name = > <

M

M

*****************************************************************************

M *

M * LOCATION SAP-Server SAPSERV-2_B01_02 on host SAPSERV-2 (wp 0)

M * ERROR tskh_init: db_connect

M *

M * TIME Thu Jan 18 12:30:05 2007

M * RELEASE 640

M * COMPONENT Taskhandler

M * VERSION 1

M * RC 13

M * MODULE thxxhead.c

M * LINE 9555

M * COUNTER 1

M *

M

*****************************************************************************

M

M PfStatDisconnect: disconnect statistics

M Entering TH_CALLHOOKS

M ThCallHooks: call hook >ThrSaveSPAFields< for event BEFORE_DUMP

M *** ERROR => ThrSaveSPAFields: no valid thr_wpadm [thxxrun1.c 730]

M *** ERROR => ThCallHooks: event handler ThrSaveSPAFields for event

BEFORE_DUMP failed [thxxtool3.c 246]

M Entering ThSetStatError

M Entering ThReadDetachMode

M call ThrShutDown (1)...

M ***LOG Q02=> wp_halt, WPStop (Workproc 0 3724) [dpnttool.c 357]

[former_member185954]

Hi Peter,

You are facing an ORA 1033 error, I found a link on SDN itself.

Check this forum link:

The above forum post, talks about

SAP Note 903904 - DB Connect: SAP system does not start after OS reboot.

Regards,

Siddhesh

Message was edited by:

Siddhesh Ghag

[former_member185954]

Hi Peter,

It could be that your Oracle is stuck while starting.. check the alert<SID>.log and find out what is happening.

Regards,

Siddhesh

[Former Member]

I tried changing the registry entry and now I get a 1034 error saying the database isn't available. Any other ideas on what to try next? I still cannot get the Dispatcher online.

[former_member185954]

Hi Peter,

There are few steps you can try to check if your R/3 is able to connect to your oracle.

*Check listener trace/log files to see if there are any errors.

*Check if your oracle is up and running, check whether you can login to ORACLE database via sqlplus.

*Check if R3trans -d -v returns 0, if it cannot connect the trans.log created by it will show you what the real problem is.

Regards,

Siddhesh

[Former Member]

have a look at stderr0 or stderr1 in \usr\sap\<sid>\dvebmgsXX\work.

this one containes the output of strdbs.cmd which should normally startup your database.

if this contains any errors you may also have a look at

\oracle\<sid>\saptrace\usertrace\alert*.log

This should contain more detailed information at the end of the file.

peter

[Former Member]

i am also facing the same ora 1034,have got any sollution please help me.

[andreas_herzog]

-> any log files or traces you could provide?

GreetZ, AH