J2EE server node crashes / .hotspot_compiler
I'm trying to install a NW AS Java + usage type DI 7.0 SR3 on W2K3 R2 x64 SP2 with MS SQL Server 2005 and Java HotSpot(TM) 64-Bit Server VM (build 1.4.2_18-b06, mixed mode). During sapinst the server node was shut down by the program but doesn't come up anymore. It crashes constantly and is directly beeing restarted.
A look in std_server0.out gave me the following insight (example, added lf):
Login :33.902: [ParNew 218352K->59053K(966656K), 0.0923355 secs] 39.132: [ParNew 222893K->62535K(966656K), 0.1131081 secs] 44.103: [ParNew 226375K->65891K(966656K), 0.0794097 secs] ### Excluding compile: com.sap.engine.services. webservices.jaxrpc.encoding.TypeMappingImpl::initializeRelations 48.296: [ParNew 229731K->66972K(966656K), 0.0845196 secs] ### Excluding compile: com.sap.engine.services. webservices.jaxrpc.encoding.InstanceBuilder::readElement 52.107: [ParNew 230812K->71757K(966656K), 0.0862884 secs] ### Excluding compile: com.sap.engine.services. webservices.jaxrpc.encoding.GeneratedComplexType::_loadInto 56.691: [ParNew 235597K->75601K(966656K), 0.0875517 secs] An unrecoverable stack overflow has occurred. # # An unexpected error has been detected by HotSpot Virtual Machine: # # EXCEPTION_STACK_OVERFLOW (0xc00000fd) at pc=0x00000000080e3dd6, pid=4460, tid=5532 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.4.2_18-b06 mixed mode) # Problematic frame: # V [jvm.dll+0xe3dd6] # # An error report file with more information is saved as hs_err_pid4460.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # -------------------------------------------------------------------------------- stdout/stderr redirect -------------------------------------------------------------------------------- node name : server0 pid : 3924 system name : DID system nr. : 00 started at : Wed Sep 24 12:27:17 2008
As you can see the VM crashes. A look in the corresponding log-file (hs_err_pid4460.log) gave me the following insight (added lf):
Current CompileTask: opto:1295 ! com.sap.engine.core.cluster.impl6.ms.MSRawConnection.sendMessage( Lcom/sap/engine/core/cluster/impl6/ms/MSMessageObjectImpl; Lcom/sap/engine/core/cluster/impl6/ms/MSRegistrable; [BIIZ)Lcom/sap/engine/frame/cluster/message/MessageAnswer; (1477 bytes)
It seems that the CompileTask for the Hotspot VM always crashes when trying to compile the MSRawConnection class natively (reproducable). Sun describes a workaround for this type of problem, http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/gbyzo.html#gbyzd . The workaround is to place a .hotspot_compiler file in the working directory of the application with an exclusion of the method. This will advice the VM whenever it decides to natively compile a certain bit of code first to check this file for exclusions. If the method identified by the VM to be compiled is exluded in this file, the compilation will be skipped. Therefore it would have been worth a try to exclude the above mentioned method sendMessage of class MSRawConnection for hotspot compilation. But when I do a quick search for .hotspot_compiler in my sap folder I find four of them, it seems SAP is already making heavy use of this "workaround" instead of reporting a bug. These files are located under:
D:\usr\sap\DID\JC00\j2ee\cluster D:\usr\sap\DID\JC00\SDM\program D:\usr\sap\DID\JC00\j2ee\cluster\dispatcher D:\usr\sap\DID\JC00\j2ee\cluster\server0
They all contain the same exclusions, listed here (added lf):
## This file contains a list of methods which are going to be excluded from JIT compilation on server start ## The format of the file is as follows ## ## exclude package/subpackage1/subpackage2/../subpackageN/<Class_name> <method_to_exclude> ## ## Each line of the file describes only one method ## <method_to_exclude> is method name that will not be compiled with JIT ## ## package/subpackage1/subpackage2/../subpackageN/<Class_name> is the name of the class with the packages containing <method_to_exclude> ## ## Example: ## exclude com/sap/engine/boot/Start main ## ## will not compile with JIT the main method of com.sap.engine.boot.Start class ## ## To enter a list of methods to exclude from JIT compilation write them after this line exclude com/sapportals/portal/pb/layout/taglib/ContainerTag addIviewResources exclude com/sap/engine/services/keystore/impl/security/CodeBasedSecurityConnector getApplicationDomain exclude com/sap/engine/services/rmi_p4/P4StubSkeletonGenerator generateStub exclude com/sapportals/portal/prt/util/StringUtils escapeToJS exclude com/sapportals/portal/prt/core/broker/PortalServiceItem startServices exclude com/sap/engine/services/webservices/server/deploy/WSConfigurationHandler downloadFile exclude com/sapportals/portal/prt/jndisupport/util/AbstractHierarchicalContext lookup exclude com/sapportals/portal/navigation/cache/CacheNavigationNode getAttributeValue exclude com/sapportals/portal/navigation/TopLevelNavigationiView PrintNode exclude com/sapportals/wcm/service/ice/wcm/ICEPropertiesCoder encode exclude com/sap/lcr/pers/delta/importing/ObjectLoader loadObjects exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readElement exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readSequence exclude com/sap/engine/services/webservices/jaxrpc/encoding/TypeMappingImpl initializeRelations exclude com/sap/engine/services/webservices/jaxrpc/encoding/GeneratedComplexType _loadInto
I thought the D:\usr\sap\DID\JC00\j2ee\cluster\server0\.hotspot_compiler file must be the right file, but however to be sure (test) I added the following line to all four files after having shutdown the instance:
exclude com/sap/engine/core/cluster/impl6/ms/MSRawConnection sendMessage
When I start the engine again, the content of every file gets overwritten by the original content, therefore lacking my new line. So it seems to me that the content is somehow hardcoded or contained in the db. If it is in the db, is it possible to change the content via config tool? It also seems that this jdk is a beta version since it reports itself with the version string 1.4.2_18-b06. This is the one officially delivered by Sun on the [SAP download page|http://java.sun.com/j2se/1.4.2/SAPsite/download.html], as mentioned in [SAP Note 941595|https://websmp130.sap-ag.de/sap(bD1kZSZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=941595]. Can you please provide me a solution to add an exclusion to the .hotspot_compiler file or workaround for the above mentioned problem. As a last option I will deinstall the system and reinstall it with another jdk (e.g. J2SE v 1.4.2_17 x64 SDK), but first I want to try to exclude the method/class from compilation. Thanks for your help!