Two app servers of EP crashed

[Former Member]

Hi Experts,

We had an issue today in Enterprise Portal system.

The production system has 4 application servers and CI instance.

2 of the application servers got crashed today and other 2 were running fine with no issues at all.

After the restart of dispatcher and the server node on these 2 app servers through jcmon the system is back to normal.

However we are trying to get the root cause of the issue.

We checked the default traces and work folder logs,

We found few exceptions as follows

Would you please help in getting the root cause of the issue happned today.

Thanks and Regards,

Girish

Exceptions are as follows

Exception I

Exception while loading class: com.sapportals.supportplatform.plugins.portal.connectors.ConnectorGatewayServiceCheck

java.lang.NoClassDefFoundError: com/sapportals/portal/prt/resource/ResourceException

Exception II

java.util.zip.ZipException: error in opening zip file

at java.util.zip.ZipFile.open(Native Method)

at java.util.zip.ZipFile.<init>(ZipFile.java:123)

at java.util.jar.JarFile.<init>(JarFile.java:127)

at java.util.jar.JarFile.<init>(JarFile.java:65)

Exception III

[meta_plugin_v2_config://local/system/deployment_(w)] error looking for source synchronizer for authority [local]

java.lang.IllegalArgumentException: Timeout Service Synchronous Internal Thread : the calling thread is not application thead !

Exception IV

java.lang.NullPointerException

at com.sap.portal.spider.SpiderUtils.getLastUpdatedFromDB(SpiderUtils.java:153)

Exception V

setting initial ACL on /reporting_backend/reports/System Administration/CM Store/cm.crawlcontent - com.sap.security.api.NoSuchRoleException: Role with uniqueName system_admin_role not found!

at com.sap.security.core.imp.RoleFactory.getRoleByUniqueName(RoleFactory.java:1783)

at com.sapportals.wcm.repository.manager.reporting.RPRepositoryManager.getRoles(RPRepositoryManager.java:1474)

Exception VI

java.lang.ClassNotFoundException: com.sap.ip.bi.km.filter.BIContentFilterManager

at com.sapportals.wcm.crt.CrtClassLoaderRegistry.findClass(CrtClassLoaderRegistry.java:176)

Exception VII

[PortalServiceItem.startServices] service initialisation failed:com.sap.ip.bi.webdynpro.alv.pdf|exportpdfservice

[EXCEPTION]

com.sapportals.portal.prt.core.broker.PortalServiceInstantiationException

Caused by: com.sapportals.portal.prt.core.broker.PortalApplicationNotFoundException: Could not find portal application com.sap.ip.bi.service.generic

Edited by: girish_gmail on Mar 11, 2010 8:45 AM

Edited by: girish_gmail on Mar 11, 2010 8:46 AM

[Former Member]

Hello,

When a server restarts, its usually an outofmemory error or in some cases a jdk crash leading to server crash. Please check in your <j2ee engine>/work directory dev_server0 and std_server0.out files if you see errors like OutOfMemory!

Please ensure the JVM settings are configured as per the note 723909 and the OS Specific note within.

Please also ensure that you have heapdump settings configured as per following notes based on which OS you have:

1004255 Sun jdk

1267126 IBM i: How to get a heapdump which can by analyzed with MAT

1265455 How can you generate a system dump for analysis using MAT? (LNX)

1263258 LXN / 64

1259465 How to get a heapdump which can by analyzed with MAT (AIX)

(1053495 heapdump settings on AIX)

Thye heapdump which is .hprof file on sun jdk and on ibm its .phd/.dmp file is necessary for the analysis of the outofmemory crash if the crash is not occuring due to JVM misconfiguration but due to some memory leak.

The heapdumps can be analyzed by MAT Tool which can be downloaded from www.ecllipse.org/mat

Hope the above helps.

Best Regards,

Snehal Bhaidasna

SAP Labs, Palo Alto, CA(USA)

[Former Member]

Hi ,

Can you post the dev_server0 and DEV_DISPATCHER Logs

[Former Member]

Hi Experts,

We had an issue today in Enterprise Portal system.

The production system has 4 application servers and CI instance.

2 of the application servers got crashed today and other 2 were running fine with no issues at all.

After the restart of dispatcher and the server node on these 2 app servers through jcmon the system is back to normal.

However we are trying to get the root cause of the issue. 

We checked the default traces and work folder logs,

We found few exceptions as follows 


Would you please help in getting the root cause of the issue happned today.

Thanks and Regards,
Girish


Exceptions are as follows 
*Exception I*
Exception while loading class: com.sapportals.supportplatform.plugins.portal.connectors.ConnectorGatewayServiceCheck 
java.lang.NoClassDefFoundError: com/sapportals/portal/prt/resource/ResourceException


*Exception II*
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:123)
	at java.util.jar.JarFile.<init>(JarFile.java:127)
	at java.util.jar.JarFile.<init>(JarFile.java:65)


*Exception III*
[meta_plugin_v2_config://local/system/deployment_(w)] error looking for source synchronizer for authority [local]
java.lang.IllegalArgumentException: Timeout Service Synchronous Internal Thread : the calling thread is not application thead !

*Exception IV*
java.lang.NullPointerException
	at com.sap.portal.spider.SpiderUtils.getLastUpdatedFromDB(SpiderUtils.java:153)

*Exception V*
setting initial ACL on /reporting_backend/reports/System Administration/CM Store/cm.crawlcontent - com.sap.security.api.NoSuchRoleException: Role with uniqueName system_admin_role not found!
	at com.sap.security.core.imp.RoleFactory.getRoleByUniqueName(RoleFactory.java:1783)
	at com.sapportals.wcm.repository.manager.reporting.RPRepositoryManager.getRoles(RPRepositoryManager.java:1474)

*Exception VI*
java.lang.ClassNotFoundException: com.sap.ip.bi.km.filter.BIContentFilterManager
	at com.sapportals.wcm.crt.CrtClassLoaderRegistry.findClass(CrtClassLoaderRegistry.java:176)


*Exception VII*
[PortalServiceItem.startServices] service initialisation failed:com.sap.ip.bi.webdynpro.alv.pdf|exportpdfservice 
[EXCEPTION]
 com.sapportals.portal.prt.core.broker.PortalServiceInstantiationException
Caused by: com.sapportals.portal.prt.core.broker.PortalApplicationNotFoundException: Could not find portal application com.sap.ip.bi.service.generic