Hi ,
Following is the current Configuration:
- Oracle 10G RAC on 4-node cluster environement.
- ASCS01 ( Standalone Enqueue server ) configured to failover on all nodes.
-ERS01 (Replication server) configured to failover on all nodes.
- D03 ( application serveR) configured to failover on all nodes.
- OCFS2 Filesystem
- Linux( RedHat 5.3)
- SAP WebAS 640_EXT2 kernel.
I am done with configuring this setup. Now the following are the issues I am facing during testing:
1. When I try to restart the Application server with sapctl, it actually fails to the node defined in the sapctl command. But after failing over to the defined node,
1- Enqueue server stops and fails over where replication server was ruuning.
2- replication server stops and fails over where the enqueue server stopped.
3. Application server (D03) can not connect to the message server even after the enqueue server was started.
After looking at the crsstat , there was a loop generated which keeps on failing enqueue and replication server on each other.
After a while the application server goes down.
Attached are logs from Enqueue server, Replication serer and Application Sever:
Enqueue server dev_enqsrv:
-
trc file: "dev_enqsrv", trc level: 1, release: "640"
-
sysno 00
sid RQ1
systemid 390 (AMD/Intel x86_64 with Linux)
relno 6400
patchlevel 0
patchno 291
intno 20020600
make: multithreaded, ASCII, 64 bit
pid 4047
[Thr 47070374262896] Wed Oct 7 18:31:32 2009
[Thr 47070374262896] profile /usr/sap/RQ1/SYS/profile/RQ1_ASCS01_inysapdb02
[Thr 47070374262896] hostname inysapdb02
[Thr 47070374262896] Wed Oct 7 18:31:33 2009
[Thr 47070374262896] EnRepClass::getReplicaData: found old replication table with the following data:
[Thr 47070374262896] Line size:388, Line count: 40429, Failover Count: 16
[Thr 47070374262896] EnqId: 1254952889/28890, last stamp: 1/254954677/3000
[Thr 47070374262896] Byte order tags: int:9600753 char:Z
[Thr 47070374262896] Enqueue server runs with SAP codepage 1100
[Thr 47070374262896] Enqueue: EnqMemStartupAction Utc=1254954693
[Thr 47070374262896] Enqueue Info: replication enabled
[Thr 47070374262896] Enqueue Info: enque/replication_dll not set
[Thr 47070374262896] Wed Oct 7 18:31:34 2009
[Thr 47070374262896] ShadowTable:destroy: ShmCleanup( SHM_ENQ_REP_SHADOW_TBL)
[Thr 47070374262896] enque/backup_file disabled in enserver environment
[Thr 47070374262896] Wed Oct 7 18:31:35 2009
[Thr 47070374262896] ***LOG GEZ=> Server start [encllog.cpp 493]
[Thr 47070374262896] Wed Oct 7 18:33:05 2009
[Thr 47070374262896] calling doAsyncSignal ( 2 ) (SigThrDefaultHandler, 0x476380)
[Thr 47070374262896] caught SIGINT (2)
[Thr 47070374262896] stopAllThreads: stop Thread worker thread ...
Replication sever:
systemid 390 (AMD/Intel x86_64 with Linux)
relno 6400
patchlevel 0
patchno 291
intno 20020600
make: multithreaded, ASCII, 64 bit
pid 5513
[Thr 47541016516720] Wed Oct 7 18:33:57 2009
[Thr 47541016516720] profile /usr/sap/RQ1/SYS/profile/RQ1_ERS01_inysapdb02
[Thr 47541016516720] hostname inysapdb02
[Thr 47541016516720] Wed Oct 7 18:33:59 2009
[Thr 47541016516720] Replication server start with instance number 00
[Thr 47541016516720] Connected to Enqueue Server and created repl. table with 40429 lines
[Thr 47541016516720] Wed Oct 7 18:34:52 2009
[Thr 47541016516720] RepServer: main: fatal error on enqueue connection
[Thr 47541016516720] RepServer: main: disconnect from enqueue server
[Thr 47541016516720] *** ERROR => EncNiConnect: invalid state 2 [encomi.c 516]
[Thr 47541016516720] *** ERROR => NiCloseHandle: invalid hdl 1 [nixx_mt.c 666]
[Thr 47541016516720] *** ERROR => RepServer: main: connect failed with return code -1 [enrepserv.cp 587]
[Thr 47541016516720] Wed Oct 7 18:34:53 2009
[Thr 47541016516720] ***LOG Q0I=> NiPConnect2: SiPeekPendConn (111: Connection refused) [nixxi_r_mt.cpp 8786]
[Thr 47541016516720] *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
[Thr 47541016516720] *** ERROR => RepServer: main: connect failed with return code -7 [enrepserv.cp 587]
[Thr 47541016516720] Wed Oct 7 18:34:54 2009
[Thr 47541016516720] *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
[Thr 47541016516720] *** ERROR => RepServer: main: connect failed with return code -7 [enrepserv.cp 587]
[Thr 47541016516720] Wed Oct 7 18:34:55 2009
[Thr 47541016516720] *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
[Thr 47541016516720] *** ERROR => RepServer: main: connect failed with return code -7 [enrepserv.cp 587]
[Thr 47541016516720] Wed Oct 7 18:34:58 2009
[Thr 47541016516720] *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
[Thr 47541016516720] *** ERROR => RepServer: main: connect failed with return code -7 [enrepserv.cp 587]
[Thr 47541016516720] Wed Oct 7 18:35:08 2009
[Thr 47541016516720] *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
[Thr 47541016516720] *** ERROR => RepServer: main: connect failed with return code -7 [enrepserv.cp 587]
[Thr 47541016516720] Wed Oct 7 18:35:21 2009
[Thr 47541016516720] EnEnqToRep::checkEnserverAttached: the enqueue server has attached to the replicationtable
[Thr 47541016516720] checkActive: Enqueue server has attached my repl. table
[Thr 47541016516720] checkActive: shutdown ...
Application Server: dev_disp:
Wed Oct 7 18:17:12 2009
MBUF state OFF
MBUF component DOWN
ERROR => DpMsgProcess: MsReceive (NIECONN_BROKEN) [dpxxdisp.c 2316]
ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c 2383]
ERROR => DpRqCheck: T25 in stat TM_SLOT_FREE [dpxxdisp.c 6064]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 4842]
-IN-- sender_id DISPATCHER tid 25 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 29 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len -1 rq_id 4030
-IN-- req_info MS_ERROR
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 4842]
-IN-- sender_id DISPATCHER tid 26 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 30 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len -1 rq_id 4031
-IN-- req_info MS_ERROR
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c 4842]
-IN-- sender_id DISPATCHER tid 28 wp_ca_blk -1 wp_id -1
-IN-- action SEND_TO_WP uid 32 appc_ca_blk -1 type DIA
-IN-- new_stat NO_CHANGE mode 0 len -1 rq_id 4032
-IN-- req_info MS_ERROR
***LOG Q0I=> NiPConnect2: SiPeekPendConn (111: Connection refused) [nixxi.cpp 8786]
***LOG Q0N=> DpRqNoWpHandle, MsSndName () [dpxxdisp.c 4315]
***LOG Q0N=> DpRqNoWpHandle, MsSndName () [dpxxdisp.c 4315]
Wed Oct 7 18:17:25 2009
***LOG Q0K=> DpMsAttach, mscon ( abapvip) [dpxxdisp.c 10563]
***LOG Q0I=> NiPWrite: writev (32: Broken pipe) [niuxi.c 1174]
DpMsgAdmin: Set release to 6400, patchlevel 0
MBUF state PREPARED
MBUF component UP
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1025]
DpMsgAdmin: Set patchno for this platform to 302
Release check o.K.
MBUF state ACTIVE
Dev_w0
dev_w0M Wed Oct 7 18:18:57 2009
M *** WARNING => EncCliReq: close connection (broken) [enccli.c 677]
E *** ERROR => EnsaCliDoRequest: DoRequest failed (rc=-1): 752 bytes on con. 0 in layer 0 (EncCli) (1/1) [ensacli.c 406]
E ENSA_DoRequest (): Reconnect
M EnsaCliRegisterCom: found layer EncCli
M EnsaCliRegisterCom: found layer EncTh
M ***LOG Q0I=> NiPConnect2: SiPeekPendConn (111: Connection refused) [nixxi.cpp 8786]
M *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
M
M Wed Oct 7 18:19:02 2009
M *** ERROR => EncNiConnect: unable to connect (NIECONN_REFUSED) [encomi.c 445]
M
M Wed Oct 7 18:19:12 2009
M *** ERROR => EncNiConnect: unable to connect (NIECONN_PENDING) [encomi.c 438]
M
M Wed Oct 7 18:19:17 2009
M *** ERROR => NiWriteOK: invalid hdl 1 [nixxi.cpp 694]
M *** ERROR => EncNiConnect: NiWriteOK failed (NIEINVAL) [encomi.c 473]
M *** ERROR => NiCloseHandle: invalid hdl 1 [nixx.c 666]
E
"dev_w0" 620L, 33283C
Edited by: Rajesh Bhatt on Oct 8, 2009 12:38 AM
Edited by: Rajesh Bhatt on Oct 8, 2009 12:39 AM
Edited by: Rajesh Bhatt on Oct 8, 2009 12:39 AM
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.