Java.exe Crash

[Former Member]

I am running xMII 11.5 SR4. I have an intermitant problem where the JAVA VM running xMII crashes. The Servelet-exec service is still up and running. The problem occurs approximately monthly. I have reviewed the Log files under \LIGHTHAMMER\LOGS and under \SERVELETEXEC AS\se-XMII\. I cannot find errors in any file that would indicate what the issue is. I have also checked the windows event log and have not found anything out of the ordinary.

Does anybody have any ideas? Are there any parameters that I can set to increase logging?

[Former Member]

We have tested special characters (hyphans, single quotes, etc) and not seen any issues.

The crashes are not specific to a time of day or month.

[agentry_src]

Dave,

Probably the most important question to answer is what build of Java are you using on the MII server? Please check on this and let us know.

Thanks,

Mike

[Former Member]

Currently on 1.4.2_12 and we are in the progress of upgrading to 1.4.2_18.

[agentry_src]

Dave,

Please check the installation instructions. I believe 11.5.4 is supposed to be 1.4.2_13 at a minimum. Once you are up to 18, you may find the problem disappears. How soon are you doing the upgrade? Can you post an update to this when you determine whether the problem has gone away for everyone's benefit?

Thanks,

Mike

[Former Member]

We have it in QC for a sanity test. I will promote it next week (probably). I will post the final resolution once we have determioned it.

[Former Member]

Another reason Java WILL crash, is posting bad data back to SAP over and over and over again.

we use SQL server and it allows ' to be part of a field, SAP does not, sometimes a user manages to add a ' to the field and the interface will hang, if we don't get to it quick enough it will crash Java.

Andy

[agentry_src]

Dave,

What build of Java are you running? Have you googled that build for your problem?

Do you have the Java Runtime Status window running continuously? If so, were you able to correlate the status shown to the times the JVM crashed?

When you say approximately monthly, can you give more specifics on time of the month? Also what other processes/jobs are running that are not MII related which would affect the JVM?

Thanks,

Mike

[Former Member]

The memory setting in the Startup.Bat file is -Xmx1536m. This is also reflected in the Java Runtime Status window under System Management. Other stats from the Status Window are:

• MaxMemory 1,598,226

• Total Memory 93,659

• FreeMemory 56,426

• UnusedMemory 37,233

• Processors 4

• ActiveThreads 50

• ActiveGroups 1

Free and Unused fluctuate. I have seen Free as low as 4,126.

What is the difference between Max and Total memory?

[Former Member]

For JVM memory setting it is important to know which OS you are running.

Is it a 64Bit system or 32Bit? How big is the memory of the OS?

[Former Member]

Windows Server 2003 x32

8GB RAM

[Former Member]

Hi Dave,

ok, then the situation is clear. In a 32Bit system you cant give the JVM 1.5GB. That is because there are only 2GB for a

OS process and Windows is only able to to give something near 1.5GB contigues memory for a process. From what

I see of your listing it seems that exactly the limit has been entered (did someone play with this trying how much RAM

could be given? If so, this is a very bad solution. The Problem then

is that there are there is not much memory left for the OS in Handling the threads. JVM does not take any special care about

PAE. So this is true even you have 8GB of RAM. I could bet that you will see something like "OutOfMemoryException"

and "could not create native Thread". From your stats list you can see that you reserved 1.5GB for the JVM as virtual

maximum memory (but it is reserved). This is what is specified with -Xmx1536m. From the stats one can easily see that

your JVM uses nearly nothing from the Memory (also thread count is very low):

• MaxMemory 1,598,226

• Total Memory 93,659

• FreeMemory 56,426

• UnusedMemory 37,233

• Processors 4

• ActiveThreads 50

• ActiveGroups 1

Total Memory is the RAM that the JVM used as its maximum.

I suggest you set 1GB for the JVM with "-Xmx1024m". This will leave enough space for the OS too. If there is shortage in

RAM one can go further and reduce the RAM that is reserved for one thread. The JVM maps a java-thread to a native

thread on Windows, reserving 1M for one thread (though with your low count of threads this does not seem to be a problem

on your side). One can then use the "-Xss" switch to reduce it to 512m. The you can reserve more for the JVM memory.

Best solution would then be to use more JVM processes (each reserving 1GB) or migrate to a 64Bit OS. But all of this

only if there is a memory shortage which is regarding your stats definitely not the case on your side.

Good luck,

Matthias

[Former Member]

Matthias -

Thanks for the feedback. Great Information!!!

Befor I close this out, a couple of comments and questions...

• The JVM was configured for 1.5GB since install.

• I have reviewed all of the logs and not found any "OutOfMemoryException", "could not create native Thread", or any error containing THREAD, MEMORY, or EXCEPTION. Not sure if this changes things.

• Is there some rule of thumb that tells me how much memory I should have per thread?

• Do you know of a good resource for JVM tuning??

[Former Member]

Hi Dave,

for good resource please look for Java and Garbage Collection. There are also some papers from sun.

I must correct my statement a bit. The total memory in your stats is the current maximum

that the JVM uses. Used and free sums up to this value. Memory usage then follows a

a tooth saw curve linear growing in used memory up to the total memory (this is when you see

very low free memory) where a full garbage collection happens and memory drops nearly instantly in

a large amount. If the JVM detects that it can't free much memory with a full garbage collection

(the tooth saw becomes smaller and smaller) it increases the total memory. The range then for

the total memory is the virtual memory assigned to it via -Xmx.

Strange that you did not see any exception. It is also possible that you have a memory leak that

is growing very slowly. Best is here to monitor the stats regularly together with the Windows Performance

counters regarding thread count, handles, private bytes, virtual bytes, available memory and working set.

If you see a contigues increase in total memory then its very likely to be a memory leak. But, as you said,

you did not find any exception. Such an exception would be vital for diagnosis If you monitor also the

Performance Counters from Windows then you maybe have an indication if the problem comes in from

another process.

Regards,

Matthias

[Former Member]

Dave,

I don't think you have a configuration issue. You are already configured to use the max memory allowed. If you still have to the issue then you need to find out where the memory is getting hogged.

See if you are running any transaction that loads data that may grow over time. Like Production for the current month. At the beginning of the month maybe it is a small file but near the end it grows huge. Also, within a transaction Action blocks don't really free memory while running, so if you load some data with one block, and then load another and another with other blocks, the data is still in memory even if you aren't using the data in those blocks anymore, so you may have to "clean up" at various points within a transaction if you are dealing with large chunks of data.

[Former Member]

Christian,

remember, Dave was not able to see any OutOfMemoryExceptions...If he sets up a monitoring properly he will spot the problem.

Thanks for the information regarding the action blocks. Can you explain how to do the "clean up" exactly? How is this done during a transaction?

This is a very good hint and I would really like to know how to get rid of any DOMs in action blocks during a run of a longer transaction. Most of the

time it will be quite difficult because other action blocks are based on data of action blocks executed before.

[Former Member]

Thanks guys. Once I figure out what I need to set my memory to, does anybody know where to set it?

[Former Member]

C:\ServletExec AS\se-xMII\StartServletExec.bat

[Former Member]

But do you think increasing the Servlet Exec memory help because the issue seems to be with the JVM rather than the ServletExec?

[Former Member]

Hi,

Search in the help file with JVM, you will get the details...

For your purpose..

Changing the Java Virtual Machine Memory Size

1.Stop the ServletExec-[InstanceName] service.

2.Close the Services window.

3.From a DOS window, run c:\ServletExec AS\bin\DeleteNTService u2013n u201C<service name>u201D.

4.Edit the memory size in the C:\ServletExec AS\se-\StartServletExec.bat. The memory setting appears as u2013Xmx256m. You only need to change the number value.

5.From the DOS window, run C:\ServletExec AS\bin\InstallNTService u2013n u201C<service name>u201D.

6.Open the Services window, and start the service. Select a Startup Type of Automatic.

7.Check the ServletExec.log file located in C:\ServletExec AS\se-[InstanceName] to make sure everything initializes okay.

Thi is available in help also...

Make sure that you have entered the right number...

Regards,

Kishore

[Former Member]

Check the system stats log to see if you are running out of memory.

[Former Member]

Try [JVM Tuning|http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp] .

Also google JVM Tuning or JVM Out of Memory etc. to get more hints.