disp+work.EXE dispatcher stopped

[former_member649905]

Hi Experts

I am not able to log on into my Sap system, I am getting error message dispwork.EXE dispatcher running but dialog queue standstill, J2EE status unavailable followed by dispwork.EXE dispatcher stopped

I reviews all the threads posted so far but still not able to fix that issue

The trace:

kernel runs with dp version 210000(ext=109000) (@(#) DPLIB-INT-VERSION-210000-UC)

length of sys_adm_ext is 572 bytes

• SWITCH TRC-HIDE on ***

***LOG Q00=> DpSapEnvInit, DPStart (01 3552) [dpxxdisp.c 1231]

shared lib "dw_xml.dll" version 52 successfully loaded

shared lib "dw_xtc.dll" version 52 successfully loaded

shared lib "dw_stl.dll" version 52 successfully loaded

shared lib "dw_gui.dll" version 52 successfully loaded

rdisp/softcancel_sequence : -> 0,5,-1

use internal message server connection to port 3901

Wed Apr 20 12:55:04 2011

• WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 5 seconds

***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 5233]

MtxInit: 30000 0 0

DpSysAdmExtInit: ABAP is active

DpSysAdmExtInit: VMC (JAVA VM in WP) is not active

DpIPCInit2: start server >sapserver_DM0_01 <

DpShMCreate: sizeof(wp_adm) 12672 (1408)

DpShMCreate: sizeof(tm_adm) 3954072 (19672)

DpShMCreate: sizeof(wp_ca_adm) 24000 (80)

DpShMCreate: sizeof(appc_ca_adm) 8000 (80)

DpCommTableSize: max/headSize/ftSize/tableSize=500/8/528056/528064

DpShMCreate: sizeof(comm_adm) 528064 (1048)

DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0

DpShMCreate: sizeof(file_adm) 0 (72)

DpShMCreate: sizeof(vmc_adm) 0 (1452)

DpShMCreate: sizeof(wall_adm) (38456/34360/64/184)

DpShMCreate: sizeof(gw_adm) 48

DpShMCreate: SHM_DP_ADM_KEY (addr: 05E00040, size: 4607512)

DpShMCreate: allocated sys_adm at 05E00040

DpShMCreate: allocated wp_adm at 05E01E28

DpShMCreate: allocated tm_adm_list at 05E04FA8

DpShMCreate: allocated tm_adm at 05E04FD8

DpShMCreate: allocated wp_ca_adm at 061CA570

DpShMCreate: allocated appc_ca_adm at 061D0330

DpShMCreate: allocated comm_adm at 061D2270

DpShMCreate: system runs without file table

DpShMCreate: allocated vmc_adm_list at 06253130

DpShMCreate: allocated gw_adm at 06253170

DpShMCreate: system runs without vmc_adm

DpShMCreate: allocated ca_info at 062531A0

DpShMCreate: allocated wall_adm at 062531A8

MBUF state OFF

EmInit: MmSetImplementation( 2 ).

MM diagnostic options set: 0

<ES> client 0 initializing ....

<ES> InitFreeList

<ES> block size is 1024 kByte.

Using implementation flat

<EsNT> Memory Reset disabled as NT default

<ES> 511 blocks reserved for free list.

ES initialized.

J2EE server info

start = TRUE

state = STARTED

pid = 2276

argv[0] = D:\usr\sap\DM0\DVEBMGS01\exe\jcontrol.EXE

argv[1] = D:\usr\sap\DM0\DVEBMGS01\exe\jcontrol.EXE

argv[2] = pf=D:\usr\sap\DM0\SYS\profile\DM0_DVEBMGS01_sapserver

argv[3] = -DSAPSTART=1

argv[4] = -DCONNECT_PORT=4850

argv[5] = -DSAPSYSTEM=01

argv[6] = -DSAPSYSTEMNAME=DM0

argv[7] = -DSAPMYNAME=sapserver_DM0_01

argv[8] = -DSAPPROFILE=D:\usr\sap\DM0\SYS\profile\DM0_DVEBMGS01_sapserver

argv[9] = -DFRFC_FALLBACK=ON

argv[10] = -DFRFC_FALLBACK_HOST=localhost

start_lazy = 0

start_control = SAP J2EE startup framework

DpJ2eeStart: j2ee state = STARTED

rdisp/http_min_wait_dia_wp : 1 -> 1

***LOG CPS=> DpLoopInit, ICU ( 3.0 3.0 4.0.1) [dpxxdisp.c 1617]

***LOG Q0K=> DpMsAttach, mscon ( sapserver) [dpxxdisp.c 11414]

DpStartStopMsg: send start message (myname is >sapserver_DM0_01 <)

DpStartStopMsg: start msg sent

CCMS: AlInitGlobals : alert/use_sema_lock = TRUE.

CCMS: Initalizing shared memory of size 60000000 for monitoring segment.

CCMS: start to initalize 3.X shared alert area (first segment).

DpMsgAdmin: Set release to 7000, patchlevel 0

MBUF state PREPARED

MBUF component UP

DpMBufHwIdSet: set Hardware-ID

***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1050]

DpMsgAdmin: Set patchno for this platform to 52

Release check o.K.

DpJ2eeLogin: j2ee state = CONNECTED

Wed Apr 20 12:55:09 2011

***LOG Q0I=> NiIRead: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 4235]

• ERROR => NiIRead: SiRecv failed for hdl 4 / sock 1536*

(SI_ECONN_BROKEN; I4; ST; 127.0.0.1:4852) [nixxi.cpp 4235]

DpJ2eeMsgProcess: j2ee state = CONNECTED (NIECONN_BROKEN)

DpIJ2eeShutdown: send SIGINT to SAP J2EE startup framework (pid=2276)

• ERROR => DpProcKill: kill failed [dpntdisp.c 371]*

DpIJ2eeShutdown: j2ee state = SHUTDOWN

Wed Apr 20 12:55:44 2011

• ERROR => W0 (pid 1400) died [dpxxdisp.c 14021]*

• ERROR => W1 (pid 3576) died [dpxxdisp.c 14021]*

• ERROR => W2 (pid 3480) died [dpxxdisp.c 14021]*

• ERROR => W3 (pid 3984) died [dpxxdisp.c 14021]*

my types changed after wp death/restart 0xbf --> 0xbe

• ERROR => W4 (pid 1160) died [dpxxdisp.c 14021]*

my types changed after wp death/restart 0xbe --> 0xbc

• ERROR => W5 (pid 3492) died [dpxxdisp.c 14021]*

my types changed after wp death/restart 0xbc --> 0xb8

• ERROR => W6 (pid 1296) died [dpxxdisp.c 14021]

my types changed after wp death/restart 0xb8 --> 0xb0

• ERROR => W7 (pid 2360) died [dpxxdisp.c 14021]

my types changed after wp death/restart 0xb0 --> 0xa0

• ERROR => W8 (pid 2760) died [dpxxdisp.c 14021]

my types changed after wp death/restart 0xa0 --> 0x80

• DP_FATAL_ERROR => DpWPCheck: no more work processes

• DISPATCHER EMERGENCY SHUTDOWN ***

increase tracelevel of WPs

NiWait: sleep (10000ms) ...

NiISelect: timeout 10000ms

NiISelect: maximum fd=1645

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Wed Apr 20 12:55:54 2011

NiISelect: TIMEOUT occured (10000ms)

dump system status

Workprocess Table (long) Wed Apr 20 16:55:54 2011

Please help,

Regards

Edited by: Blaiso on Apr 20, 2011 7:20 PM

Edited by: Blaiso on Apr 20, 2011 7:21 PM

Edited by: Blaiso on Apr 20, 2011 7:21 PM

[paul_power]

Hi Blaiso,

It seems all the work processes are dying off.

Please check them dev_w0.log and you will find the reason for the work process stopping, please post the dev_w0 errors here. Check the other dev_w*.logs but likely all of the work processes are dying with the same errors in all of them.

Regards,

Paul

[former_member649905]

Hi Paul,

Thanks for replying,

I was able to deep into that issue, the dev_w0 shows

Mon Apr 25 11:54:15 2011

C Starting user session (con_hdl=0,svchp=05BC4288,srvhp=05BD5B54,usrhp=05C22A78)

C *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033

[dboci.c 4345]

C Detaching from DB Server (con_hdl=0,svchp=05BC4288,srvhp=05BD5B54)

C *** ERROR => CONNECT failed with sql error '1033'

[dbsloci.c 10704]

B ***LOG BV3=> severe db error 1033 ; work process is stopped [dbsh#2 @ 1199] [dbsh 1199 ]

B ***LOG BY2=> sql error 1033 performing CON [dblink#3 @ 431] [dblink 0431 ]

B ***LOG BY0=> ORA-01033: ORACLE initialization or shutdown in progress [dblink#3 @ 431] [dblink 0431 ]

M ***LOG R19=> ThInit, db_connect ( DB-Connect 000256) [thxxhead.c 1403]

M in_ThErrHandle: 1

M *** ERROR => ThInit: db_connect (step 1, th_errno 13, action 3, level 1) [thxxhead.c 10019]

As well as dev_w1u2026

-

---

Based on Note 655240, I checked D:\oracle\DM0\saptrace\background and below the alert_dm0 error:

Errors in file d:\oracle\dm0\saptrace\usertrace\dm0_ora_3340.trc:

ORA-00202: control file: 'D:\ORACLE\DM0\ORIGLOGB\CNTRL\CNTRLDM0.DBF'

ORA-27091: unable to queue I/O

ORA-27070: async read/write failed

OSD-04006: ReadFile() failure, unable to read from file

O/S-Error: (OS 23) Data error (cyclic redundancy check).

-

---

Below the trace of dm0_ora_3340.trc:

Dump file d:\oracle\dm0\saptrace\usertrace\dm0_ora_3340.trc

Mon Apr 25 11:52:22 2011

ORACLE V10.2.0.2.0 - Production vsnsta=0

vsnsql=14 vsnxtr=3

Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production

With the Partitioning, OLAP and Data Mining options

Windows Server 2003 Version V5.2 Service Pack 1

CPU : 2 - type 586, 1 Physical Cores

Process Affinity : 0x00000000

Memory (Avail/Total): Ph:1550M/2046M, Ph+PgF:8231M/9899M, VA:615M/2047M

Instance name: dm0

Redo thread mounted by this instance: 0 <none>

Oracle process number: 13

Windows thread id: 3340, image: ORACLE.EXE (SHAD)

• SERVICE NAME:() 2011-04-25 11:52:22.781

• SESSION ID:(85.1) 2011-04-25 11:52:22.781

kccsga_update_ckpt: num_1 = 50, num_2 = 0, num_3 = 0, lbn_2 = 0, lbn_3 = 0

• 2011-04-25 11:54:00.656

ksedmp: internal or fatal error

ORA-00202: control file: 'D:\ORACLE\DM0\ORIGLOGB\CNTRL\CNTRLDM0.DBF'

ORA-27091: unable to queue I/O

ORA-27070: async read/write failed

OSD-04006: ReadFile() failure, unable to read from file

O/S-Error: (OS 23) Data error (cyclic redundancy check).

Current SQL statement for this session:

ALTER DATABASE MOUNT

Pl adviced

[Former Member]

Hello

Looks like your operating system and Oracle are unable to read the file system mentioned.

Kindly check once on that.

Kindly shutdown the database once and try mounting it back.

Try running R3trans -x to check if the database connects to Application.

If that succeeds kindly stop the database and start SAP along with database using startsap command.

Regards

Vivek

[former_member208104]

Hi,

From error looks like your oracle control file is currpt. You need to fix the currpt control file issue.

You can take help from dba as well.

Here are the genral steps to fix currpt control file.

1. connect to database using sqlplus with SYS as SYSDBA

2. shutdown immediate the database.

3. create a backup copy of all control file

4. rename CNTRLDM0.DB' (corrupt control file) to CNTRLDM0.DBF.bak

5. rename one of working control file to become CNTRLDM0.DBF

6.then startup the database via sqlplus using SYS as SYSDBA

-Sanjay

Edited by: Sanjay Hanspal on Apr 25, 2011 9:09 PM

[Former Member]

Hello

Kindly stop whole of the instance and restart the system.

All the workprocesses are in hung status and will not process any requests.

Kindly go ahead and stop the instace clear all the share memory related to that SID and start the instance.

This will fix the issue.

Regards

Vivek