disp+work.exe Stops

[Former Member]

I have a system that has stopped working recently. the disp+work stops after about 1 minute. See the logs below:

-

---

trc file: "sapstartsrv.log", trc level: 0, release: "700"

-

---

Mon Apr 11 10:46:40 2011

<<- ERROR: SapSSLInit(read_profile=1)==SSSLERR_LIB_NOT_FOUND

• dev.disp Log*

Mon Apr 11 11:44:15 2011

***LOG Q0I=> NiIRead: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 4235]

• ERROR => NiIRead: SiRecv failed for hdl 4 / sock 1456

(SI_ECONN_BROKEN; I4; ST; 127.0.0.1:1605) [nixxi.cpp 4235]

DpJ2eeMsgProcess: j2ee state = CONNECTED (NIECONN_BROKEN)

DpIJ2eeShutdown: send SIGINT to SAP J2EE startup framework (pid=3108)

• ERROR => DpProcKill: kill failed [dpntdisp.c 371]

DpIJ2eeShutdown: j2ee state = SHUTDOWN

Mon Apr 11 11:44:49 2011

• ERROR => W0 (pid 3064) died [dpxxdisp.c 14241]

• ERROR => W1 (pid 3152) died [dpxxdisp.c 14241]

• ERROR => W2 (pid 3748) died [dpxxdisp.c 14241]

• ERROR => W3 (pid 1292) died [dpxxdisp.c 14241]

• ERROR => W4 (pid 3956) died [dpxxdisp.c 14241]

• ERROR => W5 (pid 3156) died [dpxxdisp.c 14241]

my types changed after wp death/restart 0xbf --> 0xbe

• ERROR => W6 (pid 2168) died [dpxxdisp.c 14241]

my types changed after wp death/restart 0xbe --> 0xbc

• ERROR => W7 (pid 924) died [dpxxdisp.c 14241]

my types changed after wp death/restart 0xbc --> 0xb8

• ERROR => W8 (pid 1980) died [dpxxdisp.c 14241]

• ERROR => W9 (pid 2256) died [dpxxdisp.c 14241]

• ERROR => W10 (pid 520) died [dpxxdisp.c 14241]

my types changed after wp death/restart 0xb8 --> 0xb0

• ERROR => W11 (pid 1260) died [dpxxdisp.c 14241]

my types changed after wp death/restart 0xb0 --> 0xa0

• ERROR => W12 (pid 432) died [dpxxdisp.c 14241]

my types changed after wp death/restart 0xa0 --> 0x80

• DP_FATAL_ERROR => DpWPCheck: no more work processes

• DISPATCHER EMERGENCY SHUTDOWN ***

increase tracelevel of WPs

NiWait: sleep (10000ms) ...

NiISelect: timeout 10000ms

NiISelect: maximum fd=1585

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Mon Apr 11 11:44:59 2011

NiISelect: TIMEOUT occured (10000ms)

dump system status

Workprocess Table (long) Mon Apr 11 15:44:59 2011

========================

*dev_w0

C Mon Apr 11 11:44:14 2011

C Starting user session (con_hdl=0,svchp=05BACDB8,srvhp=05BBE684,usrhp=05C09D68)

C *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033

[dboci.c 4391]

C Detaching from DB Server (con_hdl=0,svchp=05BACDB8,srvhp=05BBE684)

C *** ERROR => CONNECT failed with sql error '1033'

[dbsloci.c 10933]

C Try to connect with default password

C Connecting as SAPSR3/<pwd>@W11 on connection 0 (nls_hdl 0) ... (dbsl 700 110706)

C Nls CharacterSet NationalCharSet C EnvHp ErrHp ErrHpBatch

C 0 UTF8 1 05BA8048 05BAD5E4 05BACE6C

C Attaching to DB Server W11 (con_hdl=0,svchp=05BACDB8,svrhp=05BBE684)

C Starting user session (con_hdl=0,svchp=05BACDB8,srvhp=05BBE684,usrhp=05C09D68)

C *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033

[dboci.c 4391]

C Detaching from DB Server (con_hdl=0,svchp=05BACDB8,srvhp=05BBE684)

C *** ERROR => CONNECT failed with sql error '1033'

[dbsloci.c 10933]

B ***LOG BV3=> severe db error 1033 ; work process is stopped [dbsh#2 @ 1199] [dbsh 1199 ]

B ***LOG BY2=> sql error 1033 performing CON [dblink#3 @ 431] [dblink 0431 ]

B ***LOG BY0=> ORA-01033: ORACLE initialization or shutdown in progress [dblink#3 @ 431] [dblink 0431 ]

M ***LOG R19=> ThInit, db_connect ( DB-Connect 000256) [thxxhead.c 1411]

M in_ThErrHandle: 1

M *** ERROR => ThInit: db_connect (step 1, th_errno 13, action 3, level 1) [thxxhead.c 10156]

M

M Info for wp 0

M

M stat = 4

M reqtype = 1

M act_reqtype = -1

M rq_info = 0

M tid = -1

M mode = 255

M len = -1

M rq_id = 65535

M rq_source = 255

M last_tid = 0

M last_mode = 0

M semaphore = 0

M act_cs_count = 0

M control_flag = 0

M int_checked_resource(RFC) = 0

M ext_checked_resource(RFC) = 0

M int_checked_resource(HTTP) = 0

M ext_checked_resource(HTTP) = 0

M report = > <

M action = 0

M tab_name = > <

M vm = V-1

M

M *****************************************************************************

M *

M * LOCATION SAP-Server sman_W11_00 on host sman (wp 0)

M * ERROR ThInit: db_connect

M *

M * TIME Mon Apr 11 11:44:14 2011

M * RELEASE 700

M * COMPONENT Taskhandler

M * VERSION 1

M * RC 13

M * MODULE thxxhead.c

M * LINE 10354

M * COUNTER 1

M *

M *****************************************************************************

M

M PfStatDisconnect: disconnect statistics

M Entering TH_CALLHOOKS

M ThCallHooks: call hook >ThrSaveSPAFields< for event BEFORE_DUMP

M *** ERROR => ThrSaveSPAFields: no valid thr_wpadm [thxxrun1.c 720]

M *** ERROR => ThCallHooks: event handler ThrSaveSPAFields for event BEFORE_DUMP failed [thxxtool3.c 260]

M Entering ThSetStatError

M ThIErrHandle: do not call ThrCoreInfo (no_core_info=0, in_dynp_env=0)

M Entering ThReadDetachMode

M call ThrShutDown (1)...

M ***LOG Q02=> wp_halt, WPStop (Workproc 0 3064) [dpnttool.c 327]

Any ideas on what could be occurring here?

[Former Member]

I have had this problem before and one of these things has worked for me int he past.

1. /etc/Services file is wrong, doesnu2019t include gateway.

2. SNC parameters are set up but SNC not being used.

Hope this helps.

[sunny_pahuja2]

Hi,

> C Mon Apr 11 11:44:14 2011

> C Starting user session (con_hdl=0,svchp=05BACDB8,srvhp=05BBE684,usrhp=05C09D68)

> C *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033

> [dboci.c 4391]

> C Detaching from DB Server (con_hdl=0,svchp=05BACDB8,srvhp=05BBE684)

> C *** ERROR => CONNECT failed with sql error '1033'

> [dbsloci.c 10933]

> C Try to connect with default password

You are getting above error. Please check SAP note 655240.

Thanks

Sunny

[Former Member]

Thanks for the response. Per this note it says to check the Alert log under $ORACLE_HOME/saptrace/backgroun. Here are my findings:

Mon Apr 11 10:48:43 2011
ALTER DATABASE   MOUNT
Mon Apr 11 10:50:16 2011
Errors in file c:\oracle\w11\saptrace\usertrace\w11_ora_3716.trc:
ORA-00202: control file: 'C:\ORACLE\W11\ORIGLOGB\CNTRL\CNTRLW11.DBF'
ORA-27091: unable to queue I/O
ORA-27070: async read/write failed
OSD-04006: ReadFile() failure, unable to read from file
O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.

[sunny_pahuja2]

Hi,

It looks like that some file is corrupted. Please check SAP note 546006.

Thanks

Sunny

[Former Member]

Post this error file here:

"c:\oracle\w11\saptrace\usertrace\w11_ora_3716.trc"

Also try to set FILESYSTEMIO_OPTIONS = NONE in init file and startup db using pfile=init file location and see if it mounts this time.

Regards

SM

[Former Member]

Thanks again for the help. Here is the first part of the log. Let me know if you need to see any more of this:

Dump file c:\oracle\w11\saptrace\usertrace\w11_ora_3716.trc
Mon Apr 11 10:48:44 2011
ORACLE V10.2.0.2.0 - Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
Windows NT Version V5.2 Service Pack 2
CPU                 : 2 - type 586, 1 Physical Cores
Process Affinity    : 0x00000000
Memory (Avail/Total): Ph:3401M/4029M, Ph+PgF:10983M/12044M, VA:1273M/2047M
Instance name: w11

Redo thread mounted by this instance: 0 <none>

Oracle process number: 79

Windows thread id: 3716, image: ORACLE.EXE (SHAD)


*** SERVICE NAME:() 2011-04-11 10:48:44.203
*** SESSION ID:(85.1) 2011-04-11 10:48:44.203
kccsga_update_ckpt: num_1 = 50, num_2 = 0, num_3 = 0, lbn_2 = 0, lbn_3 = 0
*** 2011-04-11 10:50:16.671
ksedmp: internal or fatal error
ORA-00202: control file: 'C:\ORACLE\W11\ORIGLOGB\CNTRL\CNTRLW11.DBF'
ORA-27091: unable to queue I/O
ORA-27070: async read/write failed
OSD-04006: ReadFile() failure, unable to read from file
O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.
Current SQL statement for this session:
ALTER DATABASE   MOUNT
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
_ksedst+38           CALLrel  _ksedst1+0           0 1
_ksedmp+898          CALLrel  _ksedst+0            0
_kcccsi+61           CALLrel  _ksedmp+0            
__VInfreq__kccgcpa_  CALLrel  _kcccsi+0            CC 3688E644 3 32 6BF7E00
ext+231                                            
_kccgcpa+234         CALLrel  _kccgcpa_ext+0       
_kccsbck+492         CALLrel  _kccgcpa+0           1 4D65F6C 32
_kccchb+816          CALLrel  _kccsbck+0           1
_kccocx+852          CALLrel  _kccchb+0            6DB9A60 4C23120 1 0
_kccocf+104          CALLrel  _kccocx+0            6DB9A60 0 1 1
_kcfcmb+773          CALLrel  _kccocf+0            6DB9A60 1 0 0
_kcfmdb+27           CALLrel  _kcfcmb+0            0 35BDA39E 3 0 0 0 0 0 0 0 0
                                                   0 0 0
_adbdrv+40737        CALLrel  _kcfmdb+0            35BDA39E 3 0 0
_opiexe+11908        CALLrel  _adbdrv+0            
_opiosq0+6112        CALLrel  _opiexe+0            4 0 6DBD864
_kpooprx+232         CALLrel  _opiosq0+0           3 E 6DBD97C A4
_kpoal8+775          CALLrel  _kpooprx+0           6DBF69C 6DBE220 16 1 0 A4
_opiodr+1099         CALLreg  00000000             5E 17 6DBF698
_ttcpip+1273         CALLreg  00000000             5E 17 6DBF698 0
_opitsk+1017         CALL???  00000000             
_opiino+1087         CALLrel  _opitsk+0            0 0
_opiodr+1099         CALLreg  00000000             3C 4 6DBFC30
_opidrv+819          CALLrel  _opiodr+0            3C 4 6DBFC30 0
_sou2o+45            CALLrel  _opidrv+0            3C 4 6DBFC30
_opimai_real+112     CALLrel  _sou2o+0             6DBFC24 3C 4 6DBFC30
_opimai+92           CALLrel  _opimai_real+0       2 6DBFC5C
_OracleThreadStart@  CALLrel  _opimai+0            
4+708                                              
77E6482C             CALLreg  00000000             
 
--------------------- Binary Stack Dump ---------------------

[Former Member]

Hello Jeff

As I see the log also shows this.

O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.

Current SQL statement for this session:

Kindly check on which volume the datafile which you are mentioning is there and whether that is being properly called by OS or not.

I hope this is mount point or disk issue which is causing your databse executable is unable to read the volume on which the database files are there.

Regards

Vivek

[Former Member]

I beleive that Oracle cannot access 'C:\ORACLE\W11\ORIGLOGB\CNTRL\CNTRLW11.DBF'

First check the directory permissions are set accordigly as below:

http://help.sap.com/saphelp_nw70/helpdata/en/8b/76be382776c252e10000009b38f889/frameset.htm

And If you have a back up restore it to the locations specified in INITSID.ORA file in $ORACLE_HOME/dbs/

You'll find a parameter like

*.control_files='/oracle/DV2/origlogA/cntrl/cntrlDV2.dbf','/oracle/DV2/sapdata1/system_1/cntrl/cntrlDV2.dbf','/oracle/DV2/saparch/cntrl/cntrlDV2.dbf'

If you don't have any backup then you have to recreate the control file during nomount stage.

Let me know.

Regards

SM

[Former Member]

Hi

We had similar problems last few days ago

The reason why it happened is not known and UNIx team working on that

Our WP were in reconnect status and got normal when access was ok within no time

But the problem with WP is due to problem with file access on database

In your case it seems there is some error caused on C drive

Get in touch with windows storage team

We can update here the further details for knowledge of all

Success!!

Ganesh