Portal - App Nodes Restarted Automatically due to No Notification from message server

[Former Member]

Hi Experts,

Our Portal Application servers nodes gettings rebooted automatically every wednesday at 7Pm with following error.

**********************************
[Framework -> criticalShutdown] No notification from message server in 540000 ms. Triggering reboot of node.
Aug 29, 2012 6:53:33 PM             com.sap.engine.core.Framework [SAPEngine_System_Thread[impl:5]_14] Fatal: Critical shutdown was invoked. Reason is: No notification from message server in 540000 ms. Triggering reboot of node.
***********************************

I have been seeing this problem for last two years with multiple clients, but end of the day we close this issue stating Network and increase the time out parameters and decrease keepalive and left as it is...But did not find any root cause for this.

Last couple of weeks, We are facing similar issue with one of our client and we have included the network experts and got into details of all network devices to analyse where the issue in network side...but when we look at from network side, we haven't observed any abnormal symptoms. Ping response, switch where servers connected and also the switch ports, we are not finding any problem. All are fine...Even during the time the portal nodes were restarted automatically, but my ping response between all App servers to CI / DB shows perfect...No latency and Request timed out. Also we have many other applications apart from SAP, but we haven't faced any issue with them as all are there in same location.

We have 14 Apps servers and CI/DB running on single node. From Hardware perspective, it is high end configuration servers.

I have gone thru the below wiki post and did the changes wrt parameters like increased the reconnect timeout 540000 and keepalive 20000 which we do usually for any client, even still the app nodes were restarted automatically.

http://wiki.sdn.sap.com/wiki/display/JSTSG/(JSTSG)(Kernel)Cluster-MSConnectionProblems

We have raised support message, but we have got response stating to look at network layer, but no evident that its a network problem.

Can somebody throw some light on it....If you have experienced or how to identify the exact issue. Appreciate your guidence on this.

Regards,

Iyyappan

[Former Member]

Hi

check the load on the servers at Wed - 7 PM and if you have access to OS level then please record the load on the server at them . if you find extensive load then please try to remove the application running at that time. because here problem is timout  after certain time.

Thanks

Dishant pathak

[anja_engelhardt2]

moved to SAP NetWeaver Application Server by moderator

[Former Member]

Hello Iyyappan

An important point to check can be execution of any resource intensive process on the java server that might lead to slow response from java engine. If that were the case, java server node might crash for want of connection to message server (it sometimes happen during high jvm heap memory usage).

Thanks

Tapan

[junwu]

all your 14 server get restarted at that time?

[Former Member]

Hi, First time about 6 out of 14 applications servers's nodes got restarted..2nd time, all 14 application server's nodes got restarted....

[Former Member]

Dear All,

We have got the same situation where all the server nodes got restarted alike at the same time. Kindly suggest on the same.

[anja_engelhardt2]

Hi,

did you check the log files of message server. the error message you posted shows a problem with central services.

Regards,

Anja