xMII 11.5 ServletExec not responsive

[Former Member]

I have a client that having problems with their ServletExec "locking up" in where the only resolution is to kill java.exe and restart the ServletExec. There is nothing in the windows event viewer or the xMII General log that is pointing to an issue. ServletExec is set to 1536 MB of memory.

The system log right before the last lock up showed:

[MaxMem=1598226 Kb [1598 Mb],TotMem=1598226 Kb [1598 Mb],FreeMem=1128949 Kb [1128 Mb],AvailProc=4,Threads=49,ThreadGrps=1]

I had heard that some HP server monitoring tools can cause conflicts with ServletExec. I told them to kill any uncessary HP monitoring tools. They did this 2 weeks ago and the machine ran fine until yesterday when they had a lock up and they rebooted the computer. Now its locking up every few hours. BUT the monitoring tools were back up (apparently they did not disable the service). I have had them shut them down again. My question is, are there any documented cases of this happening in other places?

Background info.

Server: HP ProLient DL380 G5

Server OS Windows Server 2003 Standard Edition R2 SP2 4 GB RAM. 3.25 showing in windows system information.

xMII version 11.5.6 b73

[jcgood25]

Any new scheduled jobs added recently that might be chewing up all the memory with large XML handling?

[Former Member]

No, nothing new, the system hasn't change significantly since go live over 4 weeks ago. We had an issue with ServletExec running out of memory, the server was set up for only 1024 orginally, however its been moved up to 1538 and worked fine for two weeks. The old issue would happen around shift change where the system would become highly loaded, now it just seems to happen at random times. I'm looking to see if they did any windows updates (of course the claim is "nothing's changed" etc.) etc. I have process explorer monitoring java.exe and its hovering around 1.6 GB.

Servlet exec log is not showing anything significant either. It's very perplexing.

Also running on this machine is SQL server 2005, and Loftware Print service for SAP

[Former Member]

While it might be impactful for a while, perhaps turning on INFO or DEBUG mode for a period of time until you can "catch" the lock-up might be helpful, to see if there was a specific time of day, query, or BLS transaction that caused the lock-up.

Also, is there any chance that there is some type of system management software that is re-installing or re-enabling the HP monitoring software?

Lastly, are there any "odd" data sources that you're connecting to?

[Former Member]

I've put the log in debug mode. Right now the only HP services running are :

HP ProLiant System Shutdown Service - Thermal protection

HP Smart Array SAS/SATA Event Notification Service - Drive monitoring

HP Insight NIC Agents Network card monitoring. I'd shut it down but I'm not really sure what this does so I'm waiting to hear from their IT folks

The only connector I'm using is TdsDriver connecting to SQL 2005 database. I have 3 other servers configured this way, basically all are duplicates of each other, the only difference in this one has much more load on it (larger plant, more users)

[Former Member]

So it seems right before my last crash (Java.exe pegs out 100%) I was getting several errors in the general log.

At 2009-02-06 09:18:21,678

	
The content-type supplied is not supported at this time. com.lighthammer.Illuminator.logging.LHException: The content-type supplied is not supported at this time. at com.lighthammer.Illuminator.system.QueryData.createBuilder(Unknown Source) at com.lighthammer.Illuminator.system.QueryData.setOutputStream(Unknown Source) at com.lighthammer.Illuminator.system.QueryData.<init>(Unknown Source) at com.lighthammer.Illuminator.services.ServiceManager.createData(Unknown Source) at com.lighthammer.Illuminator.servlet.Illuminator.service(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.lighthammer.Illuminator.servlet.ServletRunner.run(Unknown Source) at com.lighthammer.cms.common.RequestRunner.run(Unknown Source) at java.lang.Thread.run Thread.java:534

At 2009-02-06 09:18:21,694

Illuminator	Illuminator Error [-100] com.lighthammer.Illuminator.logging.LHException: The content-type supplied is not supported at this time. at com.lighthammer.Illuminator.system.QueryData.createBuilder(Unknown Source) at com.lighthammer.Illuminator.system.QueryData.setOutputStream(Unknown Source) at com.lighthammer.Illuminator.system.QueryData.<init>(Unknown Source) at com.lighthammer.Illuminator.services.ServiceManager.createData(Unknown Source) at com.lighthammer.Illuminator.servlet.Illuminator.service(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.lighthammer.Illuminator.servlet.ServletRunner.run(Unknown Source) at com.lighthammer.cms.common.RequestRunner.run(Unknown Source) at java.lang.Thread.run Thread.java:534

The thing that gets my attention is "The content-type supplied is not supported at this time" Soon after this I get

java.lang.OutOfMemoryError java.lang.OutOfMemoryError
*** MISSING RESOURCE: Unable to localize java.lang.OutOfMemoryError

Then

 
java.lang.OutOfMemoryError java.lang.OutOfMemoryError

Looks like this is happending about the same time we are doing JCO calls for piece count BAPI_GOODSMVT_CREATE and good movements BAPI_PRODORDCONF_CREATE_TT

I'm wondering if we are having some hang ups in the JCO calls. The problem is we run these all the time, and usually have no issues. Then suddenly my JVM used memory will just ramp up and it will die. Running garbage collect seems to help things somewhat, but still not fool proof.

They are on an old version of java, i'm looking to upgrade them. to 1.4.2_18.

[Former Member]

The content type issue would only occur if some external application (or perhaps a BLS call or XMLQuery attempting web access to a local MII) had an invalid URL containing an invalid content-type. There's pretty much no way it could happen otherwise. Do you have anything like this in your application?

[Former Member]

Rick

After doing more digging I think that error i described is a symptom java crashing, not the cause. After putting in a butt load of log tracers and such. What seems to be happening is in a JCO block where I call BAPI_PRODORD_GET_DETAIL. It looks I get the problem when someone calls this BAPI on a workorder that doesn't' exist. I should get a return saying "WO doesn't exist" but instead, sometime (not sure why) I don't' get anything at all, it just hangs out in space and I don't get a return at all. Of course the end user doesn't get a response either so they just run it again (compounding the problem) Java.exe blows to 100% CPU, and xMII becomes unresponsive. Sometimes, it recovers, i can do a garbage collect and it keeps running, other times the only solution at that time is to restart servletexec service.

Not sure why I'm not getting a response, I have lots of BAPI calls in this application but this one seems to be the bad actor (I'm making sure of this before I make my case to the ERP guys)

[Former Member]

Just a thought, but what if the following is actually happening: The parameters you are passing in to the BAPI method are actually querying for the detail for ALL or a huge # of production orders (perhaps you're using a range selection rather than an exact match?) If so, this could explain everything...

[Former Member]

I'm just sending it a workorderID, and asking for Header, Operations, Components and Positions.

The issue happens when someone puts in a workorder that doesn't exist. Like if someone fat fingers

a workorder etc.

On my development machine (different xMII machine and different SAP server), it acts as it should, I get "Workorder Does not exist" return from the BAPI,

on the production machine, it will literally kill my xMII server. If I run BLS manually I get an error

"Status (500): Internal Server Error"

And of course, when I run the BAPI from SE37, it works as it should on the production machine

Edited by: Doug Holtke on Feb 9, 2009 4:54 PM

Edited by: Doug Holtke on Feb 9, 2009 6:33 PM

[Former Member]

Update.

I have found the issue, however I don't know the cause. When an invalid workorder number is sent BAPI_PRODORD_GET_DETAIL, against my production instance of SAP, my connection is reset, cause a spike in java used memory and often (not always) causing a crash. When calling the same BAPI in the development instance of SAP, I get a t-code error back saying workorder does not exist. (as I should) and no spike and no error.

The SAP instances are identical, except for the fact one is in production the other is development data, its a standard BAPI, so the question becomes. Why does one kill my java memory, and the other acts correctly? All I have to do is change the IP and client number from dev to prod, its not like i'm doing a lot of configuration on my end either.

The SAP guys I'm working with keep blaming xMII, but I can't prove what the problem really is. Worse yet, I don't get any error codes, (other than java running out of memory).

I have a workaround, calling BAPI_PRODORD_EXIST_CHECK which filters out bad workorders, however I still think that SAP / JCO block should handle this better.

[agentry_src]

Hi Doug,

I am a little confused by the 11.5.6 b73. I show that as version 11.5.5. Probably no impact whatsoever, but just wanted to get it straight.

What version of the JRE are you running?

Also have you checked the log for error messages:

C:\ServletExec AS\se-xMII\ServletExec.log

Thanks,

Mike

Edited by: Michael Appleby on Feb 3, 2009 9:05 PM

[Former Member]

The "About" screen shows Product version 11.5.6 b73

The JRE is 1.4.2_07

I ran the "about" screen locally from the server.

[agentry_src]

Hi Doug,

Interesting. Sounds like it might be a typo for that last SP. I did a search on "Problem" "ServletExec", but opened up the timeframe to all messages. There is a lot of stuff there, particularly dealing with the various logs. Since 11.5 has been out for two years or so, a most of the issue resolutions go back further than the 90 day default time window. Try the search with the opened up timeframe and see if any of the addtitional posts are more suited to your situation.

Good luck,

Mike

[jcgood25]

I don't think it's a typo - for no apparent reason the build number for 11.5.5 and 11.5.6 are the same.