High CPU - jlaunch

[Former Member]

Hi,

I have a java application (PDFManip+XmlForm) that runs on the Java stack of SAP (Netweaver 7.0 SPS14). Today this program has now reached about 600 users and in topas of SO (AIX) the jlaunch (of server0)(we have 1 dispatcher and 2 server) is almost always fixed on 97% and causing major delays until you get to the detention of the application.

I was thinking of the parameters is not correct, but do not know what can be.

I can help.

Thank you.

Best regards,

Diego.

Edited by: Diego Garofalo on Mar 8, 2010 9:25 AM

[Former Member]

Hello Diego,

The high cpu usage could occur due to various reasons:

- unreasonably high load on the system and not enough resources to handle the same.

- frequent long running garbage collections

- some bottleneck in form of thread blockage at DB, File I/O or app level.

1) To ensure you have the correct configuration, please first make sure you have correct JVM settings and jdk as recommended by AIX/J2EE colleagues in notes 723909, 1024539, 1008311. Any changes need a system restart.

2) Please check note 866482 for paging and AIX.

3) During the time of high cpu, please follow note 743191. Take a ps output during high cpu, immediately followed by a threaddump. Wait for a minute and repeat the process. Do that for 5 times at least. Then you can follow the steps in the note to figure out which thread is consistently occupying the high cpu. That's the way to find whats causing high cpu. Please paste that thread here if you can't decide based on that whats the culprit.

4) The note below is also quite informative for analyzing issues on AIX JDK : 1055461

Hope the above helps.

Best Regards,

Snehal Bhaidasna

SAP Labs, Palo Alto, CA(USA)

[Former Member]

Hello,

I checked all the notes indicated. paging does not seem to have problems, the parameters of JVM are OK and the thread dump indicates a distributed CPU consumption and not on individual threads.

I updated the kernel and applied the patch mentioned in the note 1131157. Currently, the consumption of CPU is distributed among jlaunch, and XmlForm PDFManip.

Obviously first appearance to say that the problem has been resolved.

Just one question. in controlling the thread dump (more than one) should, if this is the problem, find a thread which consumes, only you, the more CPU. Right?

Thank you for claims that have been very useful.

Best regards,

Diego Garofalo.

[Former Member]

Hello Diego,

To find the high cpu consuming threads, go to the ps output and check for Runnable threads that have the highest CPU indicated. So, basically, you'll ignore all the other threads that have status not equal to 'R'.

E.g. if you find 3 threads with R with 50, 70 and 100 for cpu then you start with the one that has 100 and check that first. However, you need to confirm that this particular thread is consistently occupying the high cpu and that's why you need to check atleast 5 ps outputs along with 5 corresponding thread dumps taken over 5-10 mins of time.

I hope it helps.

Best REgards,

Snehal

[Former Member]

Hi Snehal,

I checked the threads as you have indicated when the system has very high CPU (97/98%) and there are more threads Running with CPU 7/10.

I updated the kernel and applied the patch:

SAPJTECHF14P_22-10003468.SCA

SAPJTECHS14P_23-10003467.SCA

Now the CPU consumption is still quite high, but divided between 2jlaunch, and PDFManip XmlForm.

But it is normal PDFManip and XmlForm consume so much CPU?

Best regards,

Diego.

[Former Member]

Hello Diego,

No, it is not. You should open an SAP Message in BC-SRV-FP component. They can provide you inputs about if there was any issue and/or any patch available.

Regards,

Snehal

[Former Member]

Hi,

I opened the message and posted the thread dump, but SAP did not find anything. I noticed that when the system hangs there in netstat-a 700/800 connections in CLOSE_WAIT.

There is something in mind?

Regards,

Diego.

[Former Member]

Hello Diego,

Just to confirm, one threaddump doesn't give any idea about the high cpu occupying threads. You need to have atleast 5-6 ps outputs at os level and corresponding threaddumps that can be analyzed to find the culprit. The way you do is, first take ps output immediately followed by a threaddump on the same process, wait for 2 mins and then repeat. Do this again 5-6 times.

Regarding CLOSE_WAIT connections, it is possible that there is a connections leak. You need to figure out what kind of connections are these. Please make sure you have the latest http.sda/kernel.sda for that release implemented. You should check for notes with keywords 'known problems SP<no>' and check if that release had any such issue and if the correction is provided.

Please also refer to the link:

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/7fdca26e-0601-0010-369d-b3fc87d3a...

Hope the above helps.

Regards,

Snehal

[Former Member]

Hi,

The problem at present seems resolved.

We increased the CPU.

Best regards,

Diego.

[markus_doehr2]

If you have 600 users working concurrently it's not uncommon that the system will consume CPU

You may take several thread dumps and check what the system is actually doing.

Also check if all parameters are set according to

Note 1024539 - Recommended Settings for NW04 >= SP14, NW 7.0 >= SP6 on AIX

Markus