App Servers's SM21: Cannot connect to Central lock server

[Former Member]

Hello experts,

We recently had production issue where App Servers couldn't connect to Enqueue Service,

but when I tried it through CI it was successful.

Error in SM21:

Error calling the central lock handler

I have checked nfile limit = 12000/65000 limit which is apparently underutilized and diagnosied SM12--> 'Diagnosis in Update' but I didn't find root cause.

I suspect Network bottleneck but not sure at this point.

I'm looking for only expert level answers.

Devloper trace from App Server:

B Tue Apr 19 17:02:29 2011

B dbmyclu : info : my major identification is 3960736784, minor one

1003.

B dbmyclu : info : Time Reference is 1.12.2001 01:00:00h GMT.

B dbmyclu : info : my initial uuid is E06AD0B3A7FCBAF185CB001CC4E592FC.B dbmyclu : info : current optimistic cluster level: 0

B dbmyclu : info : pessimistic reads set to 2.

M

M Tue Apr 19 17:02:43 2011

M ThIUsrDel: th_rollback_usrdelentry = 1

E

E Tue Apr 19 17:03:20 2011

E Profile-Parameter: enque/deque_wait_answer = FALSE

M

M Tue Apr 19 17:06:42 2011

M Deactivate ASTAT hyper index locking

S

S Tue Apr 19 17:09:38 2011

S found spool memory service RSPO-ACTIONS at 000000006567AFB0

M

M Tue Apr 19 17:09:43 2011

M *** WARNING => ThSndDelUser: delete client/user 200/M001272

(T165) on server XXXXX_PRD_03

M

M Tue Apr 19 17:09:54 2011

M *** WARNING => ThSndDelUser: delete client/user 200/EMASALSK

(T176) on server XXXX_PRD_02

M

M Tue Apr 19 17:30:24 2011

M *** WARNING => PfStatWrite: missing STAT_END for opcode STAT_ENQ

[pfxxstat.c 4149]

M

Thanks

SM

Edited by: SM on Apr 25, 2011 8:48 PM

Edited by: SM on Apr 25, 2011 9:45 PM

[Former Member · Accepted solution]

Hi SM,

Have you checked in ST22 , If not check the Dump once. Can you tell me what message is there in ST22 ?

Regards,

Marasa

[Former Member · Accepted solution]

Thanks for trying to answer,

No dumps related to the issue,

as I said I'm looking for expert answers.

Thanks

SM

Edited by: SM on Apr 26, 2011 3:47 PM

[mercedes_fernandez · Accepted solution]

hello,

if you get in the system log:

Error in SM21:

Error calling the central lock handler

you need to check the dispatcher trace (dev_disp) of the instance where you get this messaget and the dev_ms , dev_disp of the central instance (or the work directory of the ASCS instance if your system is running standalone enqueue server) at the same time.

kind regards,

Mercedes

[Former Member · Accepted solution]

dev_disp trace error started at 5:00:47 but the issue first happend at 5:00:32, I don't know if this is related to the issue

Dev_disp:

• ERROR => DpRqCheck: T120 in state TM_SLOT_FREE [dpxxdisp.c 7419]

***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 5694]

• ERROR => BAD REQUEST - Reason: DpRqCheck failed (line 3918): [dpxxdisp.c 5696]

-IN sender_id APPC_SERVER tid 120 wp_ca_blk -1 wp_id -1

-IN action SEND_TO_WP uid 6655 appc_ca_blk 30 type NOWP

-IN new_stat NO_CHANGE mode 0 len 80 rq_id 55254

-IN forward NOWP

-IN msgkey 00050006000007D4

• ERROR => release_ca_blk: I (1-10550) tried to rel a free blk (30) [dpxxca.c 615]

• ERROR => DpRelAppcBlk: blk 30 not released [dpxxca.c 504]

DEV_MS: 5:10:00

[Thr 1] *** ERROR => MsSAddCon: FiInsert(00B3243103000001_0_000007_000000), rc=3 [msxxserv_mt. 6157]

[Thr 1] dump of MSADM table

1. NR ADDR >server name < TYP STA FTNR FIHDL REQLSTREPLST

[Thr 1] # 0 0x60000000005d4d00 > XXXX_PRD_01 3 153 7 48992 0x6000000000648f70 (nil)

[Thr 1] # 1 0x60000000005d47a0 > XXXX_PRD_02 3 153 6 49040 0x6000000000648cf0 (nil)

[Thr 1] # 2 0x60000000005d4240 > XXXX_PRD_03 3 153 5 49088 0x6000000000648e30 (nil)

[Thr 1] # 3 0x60000000005d3ce0 > - 3 0 4 0 (nil) (nil)

[Thr 1] # 4 0x60000000005d3780 > - 3 0 3 0 (nil) (nil)

[Thr 1] # 5 0x60000000005d3220 > - 3 0 2 0 (nil) (nil)

[Thr 1] # 6 0x60000000005d2cc0 > - 3 0 1 0 (nil) (nil)

[Thr 1] # 7 0x60000000005d2760 > XXXX_PRD_00 3 191 0 49136 (nil) 0x6000000000648f70

[Thr 1] #entries = 8

[Thr 1] dump of mscon table

NR ADDRESS > Unique key FIHDL NEXTREQ NEXTREP

[Thr 1] # 0 0x6000000000649010 > 00B3243103000001_0_000007_000000 0 (nil) (nil)

[Thr 1] # 1 0x6000000000648f70 > 00D2292D00000162_0_000007_000000 1557088 0x6000000000648ed0 0x6000000000648ed0

[Thr 1] # 2 0x6000000000648ed0 > 00822AE501000007_0_000007_000000 1557136 0x6000000000648750 0x6000000000648e30

[Thr 1] # 3 0x6000000000648e30 > 009326F400000001_0_000005_000000 1557184 0x6000000000648d90 0x6000000000648d90

[Former Member · Accepted solution]

In SHort

IF the error is this:

.

• ERROR => DpRqCheck: T120 in state TM_SLOT_FREE [dpxxdisp.c 7419]

***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c 5694]

• ERROR => BAD REQUEST - Reason: DpRqCheck failed (line 3918): [dpxxdisp.c 5696]

-IN-- sender_id APPC_SERVER tid 120 wp_ca_blk -1 wp_id -1

-IN-- action SEND_TO_WP uid 6655 appc_ca_blk 30 type NOWP

-IN-- new_stat NO_CHANGE mode 0 len 80 rq_id 55254

-IN-- forward NOWP

-IN-- msgkey 00050006000007D4

• ERROR => release_ca_blk: I (1-10550) tried to rel a free blk (30) [dpxxca.c 615]

• ERROR => DpRelAppcBlk: blk 30 not released [dpxxca.c 504]

Then bump this value:rdisp/wp_ca_blk_no and rdisp/appc_ca_blk_no accordingly.

Thanks

SM

Edited by: SM on May 3, 2011 4:26 PM

[Former Member]

rdisp/appc_ca_blk_no

Edited by: SM on May 3, 2011 4:43 PM