dispatcher is not alive

[nikhil_bose]

hi experts,

Please have a look at this developer trace:

-

---

trc file: "dev_disp", trc level: 1, release: "700"

-

---

sysno 00

sid <SID>

systemid 560 (PC with Windows NT)

relno 7000

patchlevel 0

patchno 95

intno 20050900

make: multithreaded, ASCII, optimized

pid 392

Fri Mar 14 17:20:08 2008

kernel runs with dp version 224(ext=109) (@(#) DPLIB-INT-VERSION-224)

length of sys_adm_ext is 360 bytes

• SWITCH TRC-HIDE on ***

***LOG Q00=> DpSapEnvInit, DPStart (00 392) [dpxxdisp.c 1239]

shared lib "dw_xml.dll" version 95 successfully loaded

shared lib "dw_xtc.dll" version 95 successfully loaded

shared lib "dw_stl.dll" version 95 successfully loaded

shared lib "dw_gui.dll" version 95 successfully loaded

shared lib "dw_mdm.dll" version 95 successfully loaded

rdisp/softcancel_sequence : -> 0,5,-1

use internal message server connection to port 3900

Fri Mar 14 17:20:11 2008

• WARNING => DpNetCheck: NiHostToAddr(www.doesnotexist0103.qqq.nxst) took 3 seconds

Fri Mar 14 17:20:15 2008

• WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 4 seconds

Fri Mar 14 17:20:19 2008

• WARNING => DpNetCheck: NiServToNo(http) took 4 seconds

***LOG GZZ=> 3 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 5355]

MtxInit: 30000 0 0

DpSysAdmExtInit: ABAP is active

DpSysAdmExtInit: VMC (JAVA VM in WP) is not active

DpIPCInit2: start server >server_<SID>_00 <

DpShMCreate: sizeof(wp_adm) 6328 (904)

DpShMCreate: sizeof(tm_adm) 3605136 (17936)

DpShMCreate: sizeof(wp_ca_adm) 1200 (60)

DpShMCreate: sizeof(appc_ca_adm) 1200 (60)

DpCommTableSize: max/headSize/ftSize/tableSize=500/8/528040/528048

DpShMCreate: sizeof(comm_adm) 528048 (1048)

DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=0/0/0/0/0

DpShMCreate: sizeof(slock_adm) 0 (96)

DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0

DpShMCreate: sizeof(file_adm) 0 (72)

DpShMCreate: sizeof(vmc_adm) 0 (1280)

DpShMCreate: sizeof(wall_adm) (22440/34344/56/100)

DpShMCreate: sizeof(gw_adm) 48

DpShMCreate: SHM_DP_ADM_KEY (addr: 05730040, size: 4205552)

DpShMCreate: allocated sys_adm at 05730040

DpShMCreate: allocated wp_adm at 05731A28

DpShMCreate: allocated tm_adm_list at 057332E0

DpShMCreate: allocated tm_adm at 05733310

DpShMCreate: allocated wp_ca_adm at 05AA35A0

DpShMCreate: allocated appc_ca_adm at 05AA3A50

DpShMCreate: allocated comm_adm at 05AA3F00

DpShMCreate: system runs without slock table

DpShMCreate: system runs without file table

DpShMCreate: allocated vmc_adm_list at 05B24DB0

DpShMCreate: allocated gw_adm at 05B24DF0

DpShMCreate: system runs without vmc_adm

DpShMCreate: allocated ca_info at 05B24E20

DpShMCreate: allocated wall_adm at 05B24E28

MBUF state OFF

DpCommInitTable: init table for 500 entries

EmInit: MmSetImplementation( 2 ).

MM global diagnostic options set: 0

<ES> client 0 initializing ....

<ES> InitFreeList

<ES> block size is 1024 kByte.

Using implementation view

<EsNT> Memory Reset disabled as NT default

<ES> 127 blocks reserved for free list.

ES initialized.

rdisp/http_min_wait_dia_wp : 1 -> 1

***LOG Q0K=> DpMsAttach, mscon ( server) [dpxxdisp.c 11663]

DpStartStopMsg: send start message (myname is >server_<SID>_00 <)

DpStartStopMsg: start msg sent

CCMS: alert/MONI_SEGM_SIZE = 0 monitoring and alerting switched off.

DpMsgAdmin: Set release to 7000, patchlevel 0

MBUF state PREPARED

MBUF component UP

DpMBufHwIdSet: set Hardware-ID

***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1050]

DpMsgAdmin: Set patchno for this platform to 95

Release check o.K.

Fri Mar 14 17:20:28 2008

*

• SWITCH TRC-LEVEL from 1 TO 0

*

Fri Mar 14 17:20:59 2008

• DP_FATAL_ERROR => DpWPCheck: no more work processes

• DISPATCHER EMERGENCY SHUTDOWN ***

increase tracelevel of WPs

NiWait: sleep (10000ms) ...

NiISelect: timeout 10000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:09 2008

NiISelect: TIMEOUT occured (10000ms)

dump system status

Workprocess Table (long) Sat Mar 15 01:21:09 2008

========================

No Ty. Pid Status Cause Start Err Sem CPU Time Program Cl User Action Table

-

---

0 DIA 3104 Ended no 1 0 0

1 DIA 3852 Ended no 1 0 0

2 DIA 2472 Ended no 1 0 0

3 UPD 556 Ended no 1 0 0

4 ENQ 228 Ended no 1 0 0

5 BTC 2488 Ended no 1 0 0

6 SPO 4036 Ended no 1 0 0

Dispatcher Queue Statistics Sat Mar 15 01:21:09 2008

===========================

--------

---

+

---

+

---

+

---

+--

---

+

Typ

now

high

max

writes

reads

--------

---

+

---

+

---

+

---

+--

---

+

NOWP

0

2

2000

6

6

--------

---

+

---

+

---

+

---

+--

---

+

DIA

4

4

2000

4

0

--------

---

+

---

+

---

+

---

+--

---

+

UPD

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

ENQ

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

BTC

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

SPO

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

UP2

0

0

2000

0

0

--------

---

+

---

+

---

+

---

+--

---

+

max_rq_id 11

wake_evt_udp_now 0

wake events total 7, udp 7 (100%), shm 0 ( 0%)

since last update total 7, udp 7 (100%), shm 0 ( 0%)

Dump of tm_adm structure: Sat Mar 15 01:21:09 2008

=========================

Term uid man user term lastop mod wp ta a/i (modes)

Workprocess Comm. Area Blocks Sat Mar 15 01:21:09 2008

=============================

Slots: 20, Used: 1, Max: 0

--------

---

+

---

+--

---

+

id

owner

pid

eyecatcher

--------

---

+

---

+--

---

+

0

DISPATCHER

-1

WPCAAD000

NiWait: sleep (5000ms) ...

NiISelect: timeout 5000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:14 2008

NiISelect: TIMEOUT occured (5000ms)

DpHalt: shutdown server >server_<SID>_00 < (normal)

DpJ2eeDisableRestart

DpModState: buffer in state MBUF_PREPARED

NiBufSend starting

NiIWrite: hdl 2 sent data (wrt=110,pac=1,MESG_IO)

MsINiWrite: sent 110 bytes

MsIModState: change state to SHUTDOWN

DpModState: change server state from STARTING to SHUTDOWN

Switch off Shared memory profiling

ShmProtect( 57, 3 )

ShmProtect(SHM_PROFILE, SHM_PROT_RW

ShmProtect( 57, 1 )

ShmProtect(SHM_PROFILE, SHM_PROT_RD

DpWakeUpWps: wake up all wp's

Stop work processes

Stop gateway

killing process (3424) (SOFT_KILL)

Stop icman

killing process (1720) (SOFT_KILL)

Terminate gui connections

wait for end of work processes

wait for end of gateway

[DpProcDied] Process lives (PID:3424 HANDLE:1584)

waiting for termination of gateway ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:15 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process died (PID:3424 HANDLE:1584)

wait for end of icman

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:16 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:17 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:18 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:19 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:20 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:21 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:22 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:23 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:24 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:25 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process lives (PID:1720 HANDLE:1588)

waiting for termination of icman ...

NiWait: sleep (1000ms) ...

NiISelect: timeout 1000ms

NiISelect: maximum fd=1613

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Mar 14 17:21:26 2008

NiISelect: TIMEOUT occured (1000ms)

[DpProcDied] Process died (PID:1720 HANDLE:1588)

DpStartStopMsg: send stop message (myname is >server_<SID>_00 <)

NiIMyHostName: hostname = 'server'

AdGetSelfIdentRecord: > <

AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0

AdCvtRecToExt: opcode 4 (AD_STARTSTOP), ser 0, ex 0, errno 0

DpConvertRequest: net size = 189 bytes

NiBufSend starting

NiIWrite: hdl 2 sent data (wrt=562,pac=1,MESG_IO)

MsINiWrite: sent 562 bytes

send msg (len 110+452) to name -, type 4, key -

DpStartStopMsg: stop msg sent

NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)

NiBufIIn: NIBUF len=274

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 274 bytes

MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -

DpHalt: received 164 bytes from message server

NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)

NiBufIIn: NIBUF len=274

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 274 bytes

MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -

DpHalt: received 164 bytes from message server

NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)

NiBufIIn: NIBUF len=274

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 274 bytes

MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -

DpHalt: received 164 bytes from message server

NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)

NiBufIIn: NIBUF len=274

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 274 bytes

MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -

DpHalt: received 164 bytes from message server

NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)

NiBufIIn: NIBUF len=274

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 274 bytes

MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -

DpHalt: received 164 bytes from message server

NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)

NiBufIIn: NIBUF len=274

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 274 bytes

MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -

DpHalt: received 164 bytes from message server

NiIRead: hdl 2 recv would block (errno=EAGAIN)

NiIRead: read for hdl 2 timed out (0ms)

DpHalt: no more messages from the message server

DpHalt: send keepalive to synchronize with the message server

NiBufSend starting

NiIWrite: hdl 2 sent data (wrt=114,pac=1,MESG_IO)

MsINiWrite: sent 114 bytes

send msg (len 110+4) to name MSG_SERVER, type 0, key -

MsSndName: MS_NOOP ok

Send 4 bytes to MSG_SERVER

NiIRead: hdl 2 received data (rcd=114,pac=1,MESG_IO)

NiBufIIn: NIBUF len=114

NiBufIIn: packet complete for hdl 2

NiBufReceive starting

MsINiRead: received 114 bytes

MSG received, len 110+4, flag 3, from MSG_SERVER , typ 0, key -

Received 4 bytes from MSG_SERVER

Received opcode MS_NOOP from msg_server, reply MSOP_OK

MsOpReceive: ok

MsSendKeepalive : keepalive sent to message server

NiIRead: hdl 2 recv would block (errno=EAGAIN)

Fri Mar 14 17:21:27 2008

NiIPeek: peek for hdl 2 timed out (r; 1000ms)

NiIRead: read for hdl 2 timed out (1000ms)

DpHalt: no more messages from the message server

DpHalt: sync with message server o.k.

detach from message server

***LOG Q0M=> DpMsDetach, ms_detach () [dpxxdisp.c 11976]

NiBufSend starting

NiIWrite: hdl 2 sent data (wrt=110,pac=1,MESG_IO)

MsINiWrite: sent 110 bytes

MsIDetach: send logout to msg_server

MsIDetach: call exit function

DpMsShutdownHook called

NiBufISelUpdate: new MODE -- (r-) for hdl 2 in set0

SiSelNSet: set events of sock 1548 to: ---

NiBufISelRemove: remove hdl 2 from set0

SiSelNRemove: removed sock 1548 (pos=2)

SiSelNRemove: removed sock 1548

NiSelIRemove: removed hdl 2

MBUF state OFF

AdGetSelfIdentRecord: > <

AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0

AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0

AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0

blks_in_queue/wp_ca_blk_no/wp_max_no = 1/20/7

LOCK WP ca_blk 1

make DISP owner of wp_ca_blk 1

DpRqPutIntoQueue: put request into queue (reqtype 1, prio LOW, rq_id 14)

MBUF component DOWN

NiICloseHandle: shutdown and close hdl 2 / sock 1548

NiBufIClose: clear extension for hdl 2

MsIDetach: detach MS-system

cleanup EM

EsCleanup ....

EmCleanup() -> 0

Es2Cleanup: Cleanup ES2

***LOG Q05=> DpHalt, DPStop ( 392) [dpxxdisp.c 10333]

Good Bye .....

tell me where is the bug

regards,

nikhil

[Former Member]

Can you send dev_w0? The work processes are failing so the dev trace for one of these would provide more detail.

Thanks,

J. Haynes