Tomcat stops with EXCEPTION_ACCESS_VIOLATION (0xc0000005)

[Former Member]

Hello,

We are using BusinessObjects 4.0. Sometimes our tomcat stops with the following message in stdout.log

# A fatal error has been detected by the Java Runtime Environment:

#

#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006d8734ea, pid=8548, tid=2716

#

# JRE version: 6.0_37-b06

# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.12-b01 mixed mode windows-amd64 compressed oops)

# Problematic frame:

# C  [zip.dll+0x34ea]

#

---------------  T H R E A D  ---------------

Current thread (0x000000000bdec000):  JavaThread "http-80-32" daemon [_thread_in_native, id=2716, stack(0x0000000019570000,0x0000000019670000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x000000000750b000

I have to start it and then it works fine.

Any idea what the error could mean?

thanks!

[jmsrpp · Accepted solution]

Hello,

The stack trace indicates a problem with gzip compression:

Java frames: (J=compiled Java code, j=interpreted, V=VM code (C/C++), v=VM code (generated))

J  java.util.zip.Deflater.deflateBytes(J[BIII)I (sp=0x0000000020b2ef90) (pc=0x0000000009576afe)

J  java.util.zip.Deflater.deflate([BII)I (sp=0x0000000020b2f010) (pc=0x000000000a00d80c)

J  java.util.zip.GZIPOutputStream.finish()V (sp=0x0000000020b2f080) (pc=0x000000000a60da40)

J  org.apache.coyote.http11.filters.FlushableGZIPOutputStream.finish()V (sp=0x0000000020b2f0b0) (pc=0x0000000009f70fbc)

J  org.apache.coyote.http11.filters.GzipOutputFilter.end()J (sp=0x0000000020b2f0e0) (pc=0x000000000b4a566c)

J  org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (sp=0x0000000020b2f110) (pc=0x000000000aaff63c)

J  org.apache.catalina.connector.OutputBuffer.close()V (sp=0x0000000020b2f1b0) (pc=0x000000000b165b44)

J  org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (sp=0x0000000020b2f200) (pc=0x000000000b0c51d0)

J  org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (sp=0x0000000020b2f2a0) (pc=0x000000000b16ae88)

J  org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (sp=0x0000000020b2f310) (pc=0x000000000a650794)

J  org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run()V (sp=0x0000000020b2f3c0) (pc=0x000000000b116c94)

J  java.util.concurrent.ThreadPoolExecutor$Worker.run()V (sp=0x0000000020b2f450) (pc=0x0000000009a5a090)

j  java.lang.Thread.run()V - Thread.java:743 (bcx=0x000000018004c74b) (sp=0x0000000020b2f4d0) (pc=0x0000000008d0a4f4)

v  ~StubRoutines::call_stub (sp=0x0000000020b2f538) (pc=0x0000000008d02a3a)

Here are a couple of links that suggest it is a bug in the JVM:

http://stackoverflow.com/questions/9742785/tomcat-7-0-25-apr-jre-crash

http://serverfault.com/questions/506420/tomcat-is-closing-randomly

If the problem is occuring in the SAPJVM then AGS should escalate internally.

As a workaround you could:

1. Disable compression in the server.xml

2. Leverage Apache with mod_deflate + mod+jk using the instructions here.  Since compression would now be handled by the webserver you can disable it from Tomcat and still reap the benefits.

3. Disable APR as the stack suggests a JNI problem (since the stack references a DLL it means Tomcat is accessing the filesystem via Apache Portable Runtime using JNI)

To disable APR, rename/delete the tc-native.dll file from the Tomcat bin directory and restart Tomcat.  While APR is intended to improve performance in Tomcat I have not noticed appreciable differences with it enabled or disabled.

So, in short ... AGS should escalate this with the appropriate stack traces and you can select one of the workarounds above in the meantime.  I personally suggest option 2 as Apache compression is ~25% more efficient than Tomcat and you should see improvement to overall performance.

[Former Member]

Hi ALL,

we disabled compression in the server.xml.

Since 2 days no breakdowns anymore.

Thx for all replies

Holger

[Former Member]

Hello,

I tried to make some researches on this error type before.

Please refer to this KBA note:

http://service.sap.com/sap/support/notes/1906557

I hope this can help you in order to resolve your issue.

Kindest regards.

Mehrez Ben Ouali

[Former Member]

Hi Hans,

Check the JAVA_OPTS memory in tomcat and increase the values.

Check the JAVA version in tomcat and also check Java version in BO.

Thanks & Regards,

V Srinivasan

[Former Member]

Hii Hans ,

Tomcat Server usually stops due to Heavy Load on Server hence either create a cluster Server or

try to Upgrade the hardware specially RAM .

Regards

Jeetan

[Former Member]

Hello,

We are also using BusinessObjects 4.0 and every day between 01:00 and 04:00 PM our tomcat stops with exactly the same message!

After that we start tomcat and it works fine till the next day and then it crashes again 😞

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006d8734ea, pid=13308, tid=15488
#
# JRE version: 6.0_32-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.7-b02 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [zip.dll+0x34ea]
#

---------------  T H R E A D  ---------------

Current thread (0x000000000bd75800):  JavaThread "http-8080-1" daemon [_thread_in_native, id=15488, stack(0x000000000f830000,0x000000000f930000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x000000000c7f5000

------------------------------------------------------------------------------------------------------------------------------

We have updated to SP7 but the error is still there.

We have opened a SAP Call but all solution till now are not successful.

(Downgrade to JRE version 6.0.32 and redeploy tomcat6 is not the solution 😞  )

Nuria, do you found something in the firewall confihuration?

Jin-Chong, what's about your solution? Is it like the SAP solution to Downgrade?

Has anyone a solution for us ?????

Holger

[Former Member]

Hello,

I have the same issue with Tomcat/6.0.35. If anyone has an idea to solve the problem ??

Jarek.

[Henry_Banks]

Please contact Support : www.service.sap.com/support

regards,

H

[Former Member]

Hi Henry,

we did a SAP OSS ticket nearly 2 weeks ago....up to now we got no solution to fix this problem....

we did a lot of trials baesd on this ticket...nothing helps....so we try to find other customers with the same problem and hopfully a final solution .....SAP support could not help us up to now !

regards

Holger

[Former Member]

Hi Holger,

This will be a hard issue to determine. However, you at least have a point in time when the error occurs. Thus, for starters, I'd suggest determining what Tomcat is being used for around 1 - 4PM, before/during the error (Best place to look is the Tomcat Manager). This will give you a good idea on what could potentially be causing the error and narrow down your scope.

Next, it looks like your JVM is having an issue. Have you tried upgrading to the latest version of JVM? This is different than JRE, mind you.

Moreover, I see you're using a 64bit JRE. Have you tried increasing the heap & PermGen sizes? I know this is based upon your resources, so I'm not sure how much you currently have on your system, but that could be something to look into.

One last thing, what could, potentially, be happening is the garbage collection might not be doing it's job properly. You can assist it by adding the following two lines into the 'Java Options' section of Tomcat:

-XX:+CMSClassUnloadingEnabled

-XX:+CMSPermGenSweepingEnabled

Essentially, this will help clear out any chunks of memory, that's just lingering out in the ether, unused, and it can help make it accessible again. That could also be potentially your issue where it's not releasing that chunk of memory and you're getting the error because of it.

Hopefully one of the above can help resolve your issue.

-Victor

[cbethune]

This may seem elementary but have you checked your heap size to make sure the max (Maximum Memory Pool on the Java Tab on Tomcat Config) is >= to the min (Initial Memory Pool on the Java Tab on Tomcat Config)?  Setting them equal will often help with performance. 

I have noticed recently in some installs and patching I have done that upon restart the max heap number is reset the value specified in the setenv.bat file in %Install Dir%\tomcat\bin and as a result was being set to a number lower than the minimum.  I changed the value in setenv.bat and no more issues.  I have also observed similar issues if running low on drive space and the work directory cannot finish building.  Make sure you leave some overhead in terms of space on your install drive and that you have also that you have adequate RAM to support your max heap size, the OS, and any custom sizing you have done (APS, CR servers, dashboard servers, Explorer etc.)

[former_member191664]

Hi,

Judging from your tomcat log, it seems your BI4.0/Tomcat is running on a Windows platform.

# JRE version: 6.0_37-b06

# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.12-b01 mixed mode windows-amd64 compressed oops)

# Problematic frame:

1. Do you have Data Execution Prevention (DEP) on the BI4.0 Windows server?  If not, please turn it on and reboot the server,
2. The default Tomcat shipped with BI4.0 is Tomcat 6.0.24.  You might consider to upgrade it to Tomcat 6.0.35 or above for the memory leak fixes. See http://tomcat.apache.org/tomcat-6.0-doc/changelog.html on "Fix memory leak of servlet instances when running with a SecurityManager and either init() or destroy() methods fail or the servlet is a SingleThreadModel one, and of filter instances if their destroy() method fails with an Error. (kkolinko)", etc.

The steps to replace the default Tomcat6 with new Tomcat 6.0.37 is

1. Stop tomcat
2. Move BI4.0InstallDrive:\Program Files (x86)\SAP BusinessObjects\Tomcat6 to BI4.0InstallDrive:\Program Files (x86)\SAP BusinessObjects\Tomcat6.0.24
3. Download and unpack apache-tomcat-6.0.37-windows-x64.zip to BI4.0InstallDrive:\Program Files (x86)\SAP BusinessObjects\Tomcat6 folder.
4. Run Tomcat configuration and rename it to "Apache Tomcat 6.0.37" in the display name.
5. Ensure Java virtual machine is intact as

E:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\jdk\\jre\bin\server\jvm.dll

f. Ensure Java class path is intact as

E:\Program Files (x86)\SAP BusinessObjects\Tomcat6\bin\bootstrap.jar;E:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\jdk\lib\tools.jar

g. Ensure all Java_Option is intact as

-Dsun.lang.ClassLoader.allowArraySyntax=true

-Djava.library.path=C:\Windows\SysWOW64\;E:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\

-Djava.library.path=C:\Windows\SysWOW64\;E:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\

-Dcatalina.base=E:\Program Files (x86)\SAP BusinessObjects\Tomcat6\

-Dcatalina.home=E:\Program Files (x86)\SAP BusinessObjects\Tomcat6\

-Djava.endorsed.dirs=E:\Program Files (x86)\SAP BusinessObjects\Tomcat6\common\endorsed\

-Dbobj.enterprise.home=E:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\

-Xrs

-XX:MaxPermSize=384M

-Dbusinessobjects.olap.bin=

-Dbusinessobjects.olap.stylesheets=

-Djava.awt.headless=true

h. Max memory pool could be 5120 MB or higher,

i. Copy all BI4.0 war web.xml from BI4.0InstallDrive:\Program Files (x86)\SAP BusinessObjects\Tomcat6.0.24\conf\Catalina\localhost\*.xml to BI4.0InstallDrive:\Program Files (x86)\SAP BusinessObjects\Tomcat6\conf\Catalina\localhost folder,

j. Start Tomcat service in CCM and watch the BI4.0 war are been deployed properly.

Hope this helps,

Jin-Chong

[Former Member]

We are facing the same problem! We are working around the firewall configuration. I'll let you know if we find out something

[Former Member]

Hi Nuria,

we have also the same problem!

Have you a solution for me?

Thanks, Holger