cancel
Showing results for 
Search instead for 
Did you mean: 

BO4 - Tomcat service stopped unexpectedly 20 minutes apart

former_member272336
Participant
0 Kudos

Hi,

Today a couple of times 20 minutes aprt our bo4 tomcat service stopped unexpectedly.

Couple of days ago we increased the heap size to 4gb from 2gb as advised by our implemenattaion partners but ok for 2 days since and don't think this a cuase.


Trying to find why happpened and if we can prevent.


Using standrad bo4 sp08 with bundled Tomcat.


Ran catalina run form the command prompt - output below.


Any ideas/thoughts what can cause tomcat to crash and how we can see what cause it - nothing in logs obvious and how we can prevent in future.


Thanks

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyPrecompiledJsps(BundlePathAwareServiceHandler.java:326)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.registerResources(BundlePathAwareServiceHandler.java:319)

        at com.businessobjects.http.servlet.internal.ProxyServlet.registerIntern

al(ProxyServlet.java:399)

        at com.businessobjects.http.servlet.internal.ProxyServlet.register(Proxy

Servlet.java:317)

        at com.businessobjects.http.servlet.config.WebXmlConfigurator.register(W

ebXmlConfigurator.java:60)

        at com.businessobjects.webpath.WebXmlActivator.start(WebXmlActivator.jav

a:83)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(Bund

leContextImpl.java:782)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActiv

ator(BundleContextImpl.java:773)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(Bund

leContextImpl.java:754)

        at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(Bundl

eHost.java:352)

        at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(Abstrac

tBundle.java:280)

        at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(Abstrac

tBundle.java:272)

        at com.businessobjects.http.servlet.Activator.startBundle(Activator.java

:129)

        at com.businessobjects.http.servlet.Activator.start(Activator.java:116)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(Bund

leContextImpl.java:782)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActiv

ator(BundleContextImpl.java:773)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(Bund

leContextImpl.java:754)

        at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(Bundl

eHost.java:352)

        at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(Abstra

ctBundle.java:370)

        at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Frame

work.java:1068)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl.resumeBundl

es(PackageAdminImpl.java:305)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl.processDelt

a(PackageAdminImpl.java:549)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl.doResolveBu

ndles(PackageAdminImpl.java:241)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl$1.run(Packa

geAdminImpl.java:176)

        at java.lang.Thread.run(Thread.java:662)

java.io.FileNotFoundException: C:\bo4\Tomcat6\work\Catalina\localhost\BOE\proxyt

emp\Xcelsius\org\apache\jsp\jsp\Flash_005fView\view_jsp.class (Access is denied)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:194)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:145)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyInternal(BundlePathAwareServiceHandler.java:356)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyInternal(BundlePathAwareServiceHandler.java:341)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyInternal(BundlePathAwareServiceHandler.java:341)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyInternal(BundlePathAwareServiceHandler.java:341)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyInternal(BundlePathAwareServiceHandler.java:341)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyInternal(BundlePathAwareServiceHandler.java:341)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.copyPrecompiledJsps(BundlePathAwareServiceHandler.java:326)

        at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandl

er.registerResources(BundlePathAwareServiceHandler.java:319)

        at com.businessobjects.http.servlet.internal.ProxyServlet.registerIntern

al(ProxyServlet.java:399)

        at com.businessobjects.http.servlet.internal.ProxyServlet.register(Proxy

Servlet.java:317)

        at com.businessobjects.http.servlet.config.WebXmlConfigurator.register(W

ebXmlConfigurator.java:60)

        at com.businessobjects.webpath.WebXmlActivator.start(WebXmlActivator.jav

a:83)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(Bund

leContextImpl.java:782)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActiv

ator(BundleContextImpl.java:773)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(Bund

leContextImpl.java:754)

        at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(Bundl

eHost.java:352)

        at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(Abstrac

tBundle.java:280)

        at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(Abstrac

tBundle.java:272)

        at com.businessobjects.http.servlet.Activator.startBundle(Activator.java

:129)

        at com.businessobjects.http.servlet.Activator.start(Activator.java:116)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(Bund

leContextImpl.java:782)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActiv

ator(BundleContextImpl.java:773)

        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(Bund

leContextImpl.java:754)

        at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(Bundl

eHost.java:352)

        at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(Abstra

ctBundle.java:370)

        at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Frame

work.java:1068)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl.resumeBundl

es(PackageAdminImpl.java:305)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl.processDelt

a(PackageAdminImpl.java:549)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl.doResolveBu

ndles(PackageAdminImpl.java:241)

        at org.eclipse.osgi.framework.internal.core.PackageAdminImpl$1.run(Packa

geAdminImpl.java:176)

        at java.lang.Thread.run(Thread.java:662)

07-Mar-2014 14:49:50 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor BusinessProcessBI.xml

07-Mar-2014 14:49:51 org.apache.catalina.startup.ContextConfig validateSecurityR

oles

INFO: WARNING: Security role name noaccess used in an <auth-constraint> without

being defined in a <security-role>

log4j:WARN No appenders could be found for logger (org.apache.axis2.deployment.W

arBasedAxisConfigurator).

log4j:WARN Please initialize the log4j system properly.

07-Mar-2014 14:49:53 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor clientapi.xml

07-Mar-2014 14:49:53 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor dswsbobje.xml

07-Mar-2014 14:49:55 org.apache.catalina.startup.ContextConfig validateSecurityR

oles

INFO: WARNING: Security role name noaccess used in an <auth-constraint> without

being defined in a <security-role>

07-Mar-2014 14:49:55 org.apache.catalina.startup.ContextConfig validateSecurityR

oles

INFO: WARNING: Security role name SAPRole used in an <auth-constraint> without b

eing defined in a <security-role>

07-Mar-2014 14:50:03 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor host-manager.xml

07-Mar-2014 14:50:03 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor manager.xml

07-Mar-2014 14:50:03 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor MobileBIService.xml

07-Mar-2014 14:50:12 org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor MOBIServer.xml

07-Mar-2014 14:50:12 org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory docs

07-Mar-2014 14:50:12 org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory ROOT

07-Mar-2014 14:50:12 org.apache.coyote.http11.Http11AprProtocol start

SEVERE: Error starting endpoint

java.lang.Exception: Socket bind failed: [730048] Only one usage of each socket

address (protocol/network address/port) is normally permitted.

        at org.apache.tomcat.util.net.AprEndpoint.init(AprEndpoint.java:649)

        at org.apache.tomcat.util.net.AprEndpoint.start(AprEndpoint.java:766)

        at org.apache.coyote.http11.Http11AprProtocol.start(Http11AprProtocol.ja

va:137)

        at org.apache.catalina.connector.Connector.start(Connector.java:1122)

        at org.apache.catalina.core.StandardService.start(StandardService.java:5

40)

        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754

)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.

java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces

sorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

07-Mar-2014 14:50:12 org.apache.catalina.core.StandardService start

SEVERE: Failed to start connector [Connector[HTTP/1.1-80]]

LifecycleException:  service.getName(): "Catalina";  Protocol handler start fail

ed: java.lang.Exception: Socket bind failed: [730048] Only one usage of each soc

ket address (protocol/network address/port) is normally permitted.

        at org.apache.catalina.connector.Connector.start(Connector.java:1129)

        at org.apache.catalina.core.StandardService.start(StandardService.java:5

40)

        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754

)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.

java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces

sorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

07-Mar-2014 14:50:12 org.apache.catalina.startup.Catalina start

INFO: Server startup in 113034 ms

07-Mar-2014 14:50:12 org.apache.catalina.core.StandardServer await

SEVERE: StandardServer.await: create[8005]:

java.net.BindException: Address already in use: JVM_Bind

        at java.net.PlainSocketImpl.socketBind(Native Method)

        at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)

        at java.net.ServerSocket.bind(ServerSocket.java:328)

        at java.net.ServerSocket.<init>(ServerSocket.java:194)

        at org.apache.catalina.core.StandardServer.await(StandardServer.java:406

)

        at org.apache.catalina.startup.Catalina.await(Catalina.java:676)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:628)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.

java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces

sorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

07-Mar-2014 14:50:12 org.apache.coyote.http11.Http11AprProtocol pause

INFO: Pausing Coyote HTTP/1.1 on http-80

07-Mar-2014 14:50:13 org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina

07-Mar-2014 14:50:13 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/BusinessProcessBI] created a ThreadLocal with key

of type [org.apache.axiom.util.UIDGenerator$1] (value [org.apache.axiom.util.UID

Generator$1@3ce4de50]) and a value of type [long[]] (value [[J@55bf756b]) but fa

iled to remove it when the web application was stopped. This is very likely to c

reate a memory leak.

07-Mar-2014 14:50:15 org.apache.catalina.loader.WebappClassLoader clearReference

sThreads

SEVERE: The web application [/dswsbobje] appears to have started a thread named

[Business Objects - Sessions Clean up] but has failed to stop it. This is very l

ikely to create a memory leak.

07-Mar-2014 14:50:15 org.apache.catalina.loader.WebappClassLoader clearReference

sThreads

SEVERE: The web application [/dswsbobje] appears to have started a thread named

[Timer-1] but has failed to stop it. This is very likely to create a memory leak

.

07-Mar-2014 14:50:15 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/dswsbobje] created a ThreadLocal with key of type

[org.apache.axiom.util.UIDGenerator$1] (value [org.apache.axiom.util.UIDGenerato

r$1@e805af4]) and a value of type [long[]] (value [[J@cd3509c]) but failed to re

move it when the web application was stopped. This is very likely to create a me

mory leak.

07-Mar-2014 14:50:15 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/dswsbobje] created a ThreadLocal with key of type

[org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImp

l] (value [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$Thread

CounterImpl@174a1c22]) and a value of type [org.aspectj.runtime.internal.cflowst

ack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter] (value [org.aspectj.runtim

e.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@27d0820])

but failed to remove it when the web application was stopped. This is very like

ly to create a memory leak.

07-Mar-2014 14:50:16 org.apache.catalina.loader.WebappClassLoader clearReference

sThreads

SEVERE: The web application [/MobileBIService] appears to have started a thread

named [Business Objects - Sessions Clean up] but has failed to stop it. This is

very likely to create a memory leak.

07-Mar-2014 14:50:16 org.apache.catalina.loader.WebappClassLoader clearReference

sThreads

SEVERE: The web application [/MobileBIService] appears to have started a thread

named [Thread-9] but has failed to stop it. This is very likely to create a memo

ry leak.

07-Mar-2014 14:50:16 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/MobileBIService] created a ThreadLocal with key of

type [null] (value [null]) and a value of type [com.google.inject.InternalConte

xt[]] (value [[Lcom.google.inject.InternalContext;@736afade]) but failed to remo

ve it when the web application was stopped. This is very likely to create a memo

ry leak.

07-Mar-2014 14:50:16 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/MobileBIService] created a ThreadLocal with key of

type [com.google.inject.InjectorImpl$10] (value [com.google.inject.InjectorImpl

$10@747c4004]) and a value of type [com.google.inject.InternalContext[]] (value

[[Lcom.google.inject.InternalContext;@94f025b]) but failed to remove it when the

web application was stopped. This is very likely to create a memory leak.

07-Mar-2014 14:50:16 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/MobileBIService] created a ThreadLocal with key of

type [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCoun

terImpl] (value [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$

ThreadCounterImpl@2223a923]) and a value of type [org.aspectj.runtime.internal.c

flowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter] (value [org.aspectj.

runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@533

54a57]) but failed to remove it when the web application was stopped. This is ve

ry likely to create a memory leak.

07-Mar-2014 14:50:16 org.apache.catalina.loader.WebappClassLoader clearThreadLoc

alMap

SEVERE: The web application [/MobileBIService] created a ThreadLocal with key of

type [com.google.inject.spi.SourceProviders$1] (value [com.google.inject.spi.So

urceProviders$1@4fa3c06e]) and a value of type [com.google.inject.spi.SourceProv

ider[]] (value [[Lcom.google.inject.spi.SourceProvider;@7ae9e9b]) but failed to

remove it when the web application was stopped. This is very likely to create a

memory leak.

could not load dll 'TraceLog-4-0' because: null and Native Library C:\bo4\SAP Bu

sinessObjects Enterprise XI 4.0\win64_x64\TraceLog-4-0.dll already loaded in ano

ther classloader

07-Mar-2014 14:50:18 org.apache.coyote.http11.Http11AprProtocol destroy

INFO: Stopping Coyote HTTP/1.1 on http-80

C:\bo4\Tomcat6\bin>

Accepted Solutions (1)

Accepted Solutions (1)

julian_jimenez
Active Contributor
0 Kudos

Philip,

It seems that you have defined two connectors on port 80 or there is already a server listening on that port.

Regards,

Julian

former_member272336
Participant
0 Kudos

Hi,

Thanks for reply.

Only change we made and that was at very start was to chnage port number form 8080 to 80 so that didn't need to have 8080 in the url.

Only one connector as we haven't added any connectors to teh server.xml.

How are you determining a server already listening on that port and how can we determine what else is listening on that port?

Thanks

0 Kudos

Hi,

type "netstat -a" if you running on Windows.

If BI4 is running on Windows, and you changed your Tomcat Port to 80, i could imagine that the IIS on that Windows Server is causing the problem.

Please check in the Services that the "World Wide Web Publishing Service" is stopped and disabled.

Regards

-Seb.

former_member272336
Participant
0 Kudos

Hi,

Thanks for reply - will check this out.

Thanks

former_member272336
Participant
0 Kudos

Hi,

Had a look in services and cannnot see a world wide web publisng service.

Where would I see this/iis  if running?

Using MS Windows 2008 server.

Thanks

Former Member
0 Kudos

You can find it in Windows service.

Just run services.msc in the run prompt and you have all the windows services.

just check for "World Wide Web Publishing Service"

0 Kudos

Hi,

so if it is not there it is not installed with the OS. Now i would check via "netstat -a" which Services are using port 80.

Regards

-Seb.

former_member272336
Participant
0 Kudos

Hi,

Did a netstat -a and shows

Cannot see the application name using it.

Reason we changed tomcat to use port 80 is to stop us having to put 8080 in the url.

How can we see the application using it?

Have used Tomcat for months without any such stopping.

Doesn't even show Tomcat name.

TCP    [::]:80                <server_name>              LISTENING

Thanks

Former Member
0 Kudos

Hi Philip,

Just execute the below command in the CMD prompt.

netstat -aon | findstr 0.0:80

then you will have the PID of the service.

later check the PID in the task manager of the server.

Below is the screenshot for the 8080 port. I am using this port for tomcat7.

former_member272336
Participant
0 Kudos

Hi,

Thanks for reply.

Tried this and only shows tomcat running on that port.

This only happens occassionally - but happened 3 times in the last week.

If another service was running looking for port 80 would we not see 2 entries?

Thanks

Former Member
0 Kudos

When ever your tomcat is down, i would request you to check the same to see which service is occupying the ports.

julian_jimenez
Active Contributor
0 Kudos

Philip,

Probably, Tomcat was still running when the new instance is tryring to start.

Regards,
Julian

former_member272336
Participant
0 Kudos

Thanks again for replies.

One of server team advised dfs file replication listeneing on port 80.  He is going to look to reconfigure to avoid port 80.

However, dfs file replication has been running for over a year and not seen such crashes before.

Only change we  made was to increase the memory pool sizes on tomcat from 2gb to 4gb and since then we have seen this instability

Doees anyone know if this could be causing such insatbility?  4 crashes in a week.

Also someone mentioned possibitlity of setting compression=off in the tomcat connector as when we get the errors we get mention of the zip.dll files.

Any thoughts much apprecaited.

denis_konovalov
Active Contributor
0 Kudos

If tomcat is crashing with zip.dll related errors -- it is a tomcat bug and the only known solution is to disable compression in server.xml (which could affect web apps front end performance)
or disable comprtession and place Apache web server with enabled compression in front of tomcat .

former_member272336
Participant
0 Kudos

Thanks for advice.

Also saw sap note below which mentions disabling compression as well.

However, we have been using tomact for montsh with no problems until chnage the memory settings so not sure why suddenly seeing these errors now unless bug only manifest itself when memory pool settings higher than ones that ship out of the box.

1906557 - Tomcat crash due to EXCEPTION_ACCESS_VIOLATION in the zip.dll with Java_java_util_zip_ZipE...

denis_konovalov
Active Contributor
0 Kudos

it is un-clear what triggers this tomcat bug.

former_member272336
Participant
0 Kudos

Thanks for update

Answers (0)