1 Instance's Gateway unreachable

[Former Member]

Dear Experts,

We have faced this issue twice in same instance (02) in last 2 months. The instance 02 gateway becomes unreachable and impacts performance system-wide. To quickly fix the issue, we need to restart this instance. Following are the system details and log info:

OS: AIX 6.1

DB: Oracle 11g

ECC6.0 EHP5

HA configured

SSO configured

CI & DB on one server, 3 apps on different server

Attached are the gateway trace and workprocess trace files. Attaching dev_w30 trace file also because this work process was the one which goes in the hang state first then rest of the work process, look at the time in following screenshot (1426 sec):

Following is a SM21 screenshot:

No short dumps were generated during the issue.

Let me know if further details are required.

Appreciate your time and help in advance!

Thanks,

Hardeep

dev_w30.txt.zip

dev_rd.txt.zip

dev_w0.txt.zip

[Former Member · Accepted solution]

Hi All,

Thank you for your help on this issue.

Problem is with the Kernal patch, to resolve it I have to set following parameter:

rdisp/check_master_slave_active = 3

see note for more information: 1309072 - System standstill: MtxLock EMADM

Closing this thread.

Thanks,

Hardeep

[Former Member]

Try to update your kernel to latest PL available.

[Former Member]

Hi Roman,

Thanks for your reply. This issue is not reproducible in non-prod environments hence I can not varify if updating patch resolved the issue

Trying to find the root cause from the wp traces and log files so workaround can be found for this issue.

Thanks,

Hardeep

[Former Member]

Hi Hardeep,

Not sure whether you checked the below note.

3223 - Number of WP and APPC blocks

URGENT: "NO APCC block received" after Tcode FNM1 | SCN

Regards,

Raja. G

[Former Member]

Thanks Raja for your prompt reply.

I looked into the note, we have enough value given to these parameters per note.

FYI, error started around 9:05am and ended around 9:30am when instance restarted.

"No block received" error log from 8:30am, so this root cause is ruled out.

If you look at the attached traces you will find more logs during issue occurrence.

Only dev_w30 will have more information about memory stack whereas rest of all wp's have "EMADM deadlock"related info.

Thanks again, let me know if more details are required.

Hardeep