Solved: BO4 - Tomcat service crashed couple of times in la...

Hi,

We have experienced a couple of Tomcat service crashes in last couple of days.

Nothing changed system-wise.

Running BO4 SP08.

How can we determine what is causing this crashing? - Had a look in the logs but nothing obvious – please see attached with a view to preventing further re-occurences.

Thanks

SAP Managed Tags:
SAP BusinessObjects Business Intelligence platform

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Accepted Solutions (1)

Hi Philip,

The Tomcat service might crash if it reaches maximum memory and crashes with out of memory errors

Can you please try below steps.

Open Task Manager and monitor how much memory tomcat takes when issue occurs. If it has reached maximum memory threshold then you need to increase Tomcat maximum memory pool.
It can be increased from Tomcat Configuration Java tab. The XMX parameter can be set depending upon total RAM on your server.
Open stderr log file from Tomcat directory and look for any 'Severe' error messages.
Also check if there is any recent change in user activities such as number of users are increased or they are running any long running reports.

Please refer below KBAs as well to troubleshoot further.

http://service.sap.com/sap/support/notes/2075671

http://service.sap.com/sap/support/notes/1637342

http://service.sap.com/sap/support/notes/1982241

http://service.sap.com/sap/support/notes/1906557

Regards,

Hrishikesh

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi Hrishikesh,

Thanks for detailed and useful reply.

Looking in the hspid file says 86% memory used so looks like not hitting threshold when crashed.

Can see following in stderr log file but not sure what means.

Users same, report usage should be fairly statdic.

19-Oct-2015 10:23:24 org.apache.catalina.core.StandardWrapperValve invoke

SEVERE: Servlet.service() for servlet equinoxbridgeservlet threw exception

java.lang.RuntimeException: java.lang.IllegalStateException

at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:254)

at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.service

Thanks

Philip

Hi Philip,

We need to search more in stderr file for error messages, probably for out of memory errors. Can you please attach the file here if possible.

Also if it is taking 86% of memory then there is high possibility for out of memory. Can you please share initial and maximum memory pool values as well.

I would suggest try using JVisulavm as per KBA 2075671 and analyze the Tomcat memory usage.

Regards,

Hrishikesh

Hi Hrishikesh

Thanks for reply.

4096MB for initial and maximun memory pool - been this way for a couple of years.

To use JVisulavm, do you need to install it on the bo server itself? Any dnager with this - notice from sap note an oracle rthare than sap product?

Please see content of sterr file below.

Tried attaching file but only looks to be able to attach hostred files.

2015-10-19 04:00:05 Commons Daemon procrun stderr initialized

19-Oct-2015 04:00:07 org.apache.catalina.core.AprLifecycleListener init

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Windows\SysWOW64\;C:\bo4\SAP BusinessObjects Enterprise XI 4.0\win64_x64\

19-Oct-2015 04:00:07 org.apache.coyote.http11.Http11Protocol init

INFO: Initializing Coyote HTTP/1.1 on http-80

19-Oct-2015 04:00:07 org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 955 ms

19-Oct-2015 04:00:07 org.apache.catalina.core.StandardService start

INFO: Starting service Catalina

19-Oct-2015 04:00:07 org.apache.catalina.core.StandardEngine start

INFO: Starting Servlet Engine: Apache Tomcat/6.0.35

19-Oct-2015 04:00:07 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor AdminTools.xml

19-Oct-2015 04:00:08 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor BOE.xml

19-Oct-2015 04:02:12 com.sun.faces.config.ConfigureListener contextInitialized

INFO: Initializing Mojarra (1.2_12-b01-FCS) for context '/BOE'

19-Oct-2015 04:02:13 com.sun.faces.spi.InjectionProviderFactory createInstance

INFO: JSF1048: PostConstruct/PreDestroy annotations present. ManagedBeans methods marked with these annotations will have said annotations processed.

19-Oct-2015 04:02:17 org.apache.cxf.transport.servlet.CXFNonSpringServlet loadBusNoConfig

INFO: Load the bus without application context

19-Oct-2015 04:02:18 org.apache.cxf.transport.servlet.AbstractCXFServlet replaceDestinationFactory

INFO: Replaced the http destination factory with servlet transport factory

19-Oct-2015 04:02:18 org.apache.cxf.endpoint.ServerImpl initDestination

INFO: Setting the server's publish address to be /

19-Oct-2015 04:02:19 org.apache.cxf.transport.servlet.CXFNonSpringServlet loadBusNoConfig

INFO: Load the bus without application context

19-Oct-2015 04:02:19 org.apache.cxf.transport.servlet.AbstractCXFServlet replaceDestinationFactory

INFO: Replaced the http destination factory with servlet transport factory

19-Oct-2015 04:02:21 org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass

INFO: Creating Service {urn:services-businessobjects-com:coresdk:wsgateway}Gateway from class com.businessobjects.services.coresdk.wsgateway.GatewayPort

19-Oct-2015 04:02:22 org.apache.cxf.endpoint.ServerImpl initDestination

INFO: Setting the server's publish address to be /Gateway

19-Oct-2015 04:02:22 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor BusinessProcessBI.xml

19-Oct-2015 04:02:22 org.apache.catalina.startup.ContextConfig validateSecurityRoles

INFO: WARNING: Security role name noaccess used in an <auth-constraint> without being defined in a <security-role>

19-Oct-2015 04:02:35 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor clientapi.xml

19-Oct-2015 04:02:35 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor dswsbobje.xml

19-Oct-2015 04:02:36 org.apache.catalina.startup.ContextConfig validateSecurityRoles

INFO: WARNING: Security role name noaccess used in an <auth-constraint> without being defined in a <security-role>

19-Oct-2015 04:02:36 org.apache.catalina.startup.ContextConfig validateSecurityRoles

INFO: WARNING: Security role name SAPRole used in an <auth-constraint> without being defined in a <security-role>

19-Oct-2015 04:02:51 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor host-manager.xml

19-Oct-2015 04:02:51 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor manager.xml

19-Oct-2015 04:02:51 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor MobileBIService.xml

19-Oct-2015 04:02:57 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor MOBIServer.xml

19-Oct-2015 04:02:57 org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory docs

19-Oct-2015 04:02:57 org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory ROOT

19-Oct-2015 04:02:57 org.apache.coyote.http11.Http11Protocol start

INFO: Starting Coyote HTTP/1.1 on http-80

19-Oct-2015 04:02:57 org.apache.catalina.startup.Catalina start

INFO: Server startup in 169390 ms

19-Oct-2015 09:07:55 com.sap.tc.ls.styles.laf.LAFService registerTheme

INFO: Theme "sap_tradeshow_plus" has been registered on the LS-LAFService. Name is "SAP Tradeshow Plus", version is "7.33.3.9.0", the theme is a a SAP template theme.

19-Oct-2015 09:07:55 com.sap.tc.ls.styles.laf.LAFService <init>

INFO: LS-LAFService has been initialized. Default theme is "sap_tradeshow_plus". The folder that represents the themes root on the file system is "bundleentry://106.fwk378423301/web/themes". The context path is "../..".

19-Oct-2015 09:07:55 com.sap.tc.ls.styles.laf.LAFService registerTheme

INFO: Theme "sap_platinum" has been registered on the LS-LAFService. Name is "SAP Platinum", version is "7.33.3.9.0", the theme is a a SAP template theme.

19-Oct-2015 09:18:10 org.apache.catalina.core.StandardWrapperValve invoke

SEVERE: Servlet.service() for servlet equinoxbridgeservlet threw exception

java.lang.RuntimeException: java.lang.IllegalStateException

at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:254)

at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.service(BundlePathAwareServiceHandler.java:197)

at com.businessobjects.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:248)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:220)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.businessobjects.pinger.TimeoutManagerFilter.doFilter(TimeoutManagerFilter.java:166)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.IllegalStateException

at org.apache.catalina.connector.ResponseFacade.reset(ResponseFacade.java:310)

at com.businessobjects.adv_ivcdzview.DownloadPDForXLS.doPost(Unknown Source)

at com.businessobjects.adv_ivcdzview.DownloadPDForXLS.doGet(Unknown Source)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at com.businessobjects.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:110)

at com.businessobjects.http.servlet.internal.ServletLastFilterChainElement.service(ServletLastFilterChainElement.java:30)

at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:46)

at com.businessobjects.webutil.boetrustguard.BOETrustPrepareFilter.doFilter(BOETrustPrepareFilter.java:32)

at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)

at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)

at com.businessobjects.swd.shared.tracelog.TraceLogScopeFilter.doFilter(TraceLogScopeFilter.java:38)

at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)

at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)

at com.businessobjects.sdk.actionfilter.WorkflowFilter.doFilter(WorkflowFilter.java:45)

at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)

at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)

at com.businessobjects.swd.appcontext.RequestInitFilter.doFilter(RequestInitFilter.java:26)

at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)

at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)

at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:235)

... 20 more

19-Oct-2015 09:26:51 org.apache.catalina.core.StandardWrapperValve invoke