Portal is slow even after Java Parameters Tune

[Former Member]

Hi Experts,

Your expertise are needed. I'm in a process to tune our portal systems. Currently I'm working on a sandbox. It has 8 GB RAM. It has been running slow so we added Memory. I have added additional server node. The following parameters are in Server_0 and Server_1. These parameters are the SAP recommended ones as per the note # 723909.

Dispatcher Parameters

-XX:NewSize=56M  >>    -XX:MaxNewSize=56M
-Xmx256M     >>     -Xms256M
-XX:+DisableExplicitGC
-verbose:gc
-Xloggc:GC.log
-XX:+PrintGCTimeStamps
-XX:+JavaMonitorsInStackTrace
-XX:+PrintGCDetails

Instance ID

-Djava.security.policy=./java.policy
-Djava.security.egd=file:/dev/urandom
-Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy
-Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy
-Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy
-Dorg.xml.sax.driver=com.sap.engine.lib.xml.parser.SAXParser
-Djco.jarm=1
-Dsun.io.useCanonCaches=false
-XX:NewSize=170m
-XX:MaxNewSize=170m
-XX:MaxPermSize=256m
-Djava.awt.headless=true
-XX:PermSize=256m
-XX:SurvivorRatio=2
-XX:TargetSurvivorRatio=90
-XX:SoftRefLRUPolicyMSPerMB=1
-Xms1024m
-verbose:gc
-XX:+DisableExplicitGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+UseParNewGC
-XX:+JavaMonitorsInStackTrace

Server Node 0 & 1

-Djava.security.policy=./java.policy
-Djava.security.egd=file:/dev/urandom
-Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy
-Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy
-Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy
-Dorg.xml.sax.driver=com.sap.engine.lib.xml.parser.SAXParser
-Djco.jarm=1
-Dsun.io.useCanonCaches=false
-Djava.awt.headless=true
-Xms2048m
-XX:NewSize=200m             Old value was >> 341
-XX:MaxNewSize=200m      Old value was >> 341
-XX:PermSize=300m            Old Value was >> 512
-XX:MaxPermSize=300m     Old Value was >> 512
-XX:SurvivorRatio=2
-XX:TargetSurvivorRatio=90
-XX:SoftRefLRUPolicyMSPerMB=1
-verbose:gc
-XX:+DisableExplicitGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+UseParNewGC
-XX:+HandlePromotionFailure
-XX:SoftRefLRUPolicyMSPerMB=1

I have tried with different value of NewSize, PermSize.and heap. It was not helpful.

Your help is appreciated!

Thanks,

[Former Member · Accepted solution]

Hello Afi,

If this is 700 system (solaris), your parameters are not adhering to the note 723909. Heap recommended in that case is 2G atleast for the server processes.

The best way to figure out what is causing the slow behaviour is by studying garbage collection/swapping @ OS level. Please check the GC.log for the long running or very frequent back to back GCs. If you see any of those or very long running GC (check swapping @OS level). Also, ensure that you have enough threads maintained. We recommend having 200+ MaxThreadCount on server nodes. To resolve the issue, please identify what is causing the slow behaviour and the steps need to be taken accordinly. Is there any particular applicaiton that is running slowly? Is j2ee homepage coming up fast enough? If the response is very slow, you could also take threaddumps on server process during certain execution and check the threads using the threaddump viewer to look for any suspects like DB access, file system read/write issues etc.

Please remove -Xloggc:GC.log from the JVM parameters. If there are memory issues and the system restarts//crashes due to OOM, you'll lose all the GC information as this file will get overwritten by the restart and the previous data will get lost. In absence of this parameter, the GC output is written to std_server* files of which the the backup is taken everytime is restarted (for 700 system). For 6.40 system you need to have a note 1062511 implemented. (needs a restart after the change).

Hope the above helps.

Regards,

Snehal Bhaidasna

SAP Labs, Palo Alto, CA(USA)

[Former Member · Accepted solution]

Hi Snehal,

I have removed -Xloggc:GC.log from Java parameters. I'm on 7.00 so I didn't have to add paramete to instance profile.

I have captued few things.

1. My portal page (Logon page/j2ee home page) is slow.

2. When I browse for anything such as System Admin, User admin etc.. It takes long time to load. I have another portal which has 12 GB RAM on AIX, it does not have any server node, but it works way faster compare to this one. The new portal does not have Newpage size or perm size defined. Do you think if I take out perm size and newsize, it may help?

I was monitoring prstat -a command in Solaris. Memory were used about 70 % (Max) when I end up creating 3 Server Nodes. With 2 server nodes, memory utilization was 35 % used including all other processes. Swapig is used

about little over 10 GB (With 3 sever noes).

I have not seen CPU going higher than 5 %. Most of the time CPU utilization is 0.4 %.

I am currently using Java build 1.4.2_18-b06, mixed mode

Your help is appreciated!

Thanks,

Afi

[Former Member · Accepted solution]

Can you try decreasing to only one server node? That's a fast way to tell whether this is lack of physical memory... Like you said, on another host with 12GB RAM that instance with less server node runs faster.

[Former Member · Accepted solution]

Hello Afi,

1)If you take out Newsize/MaxNewSize/MaxPermSize/PermSize from the specific server JVM settings then the one in the General server settings will be used. If you take out from there as well, the default values which are very low will be used which can crash the instance. So, the right thing to do is adhere to and explicitly define the JVM parameters for 700 system running on 64 bit as per note 723909.

2) You mentioned that the swapping is used with 3 server nodes, so you should definitely not have 3 server nodes then. If you do need 3, you need higher memory. As per the formulae in 723909, you can have 4 nodes. Howver, we also need to take in to account other apps running on the machine. So, it seems your system starts swapping even with 3 server nodes.

I'm not clear from your response if swapping is done with 2 server nodes as well. If that is the case, you should find out what is making the system swap? Since in this case j2ee processes are only consunming around 2.5*2 = 5G.

JDK 1.4.2_18 was known to cause issues. Please refer to note 716604. I suggest having the latest recommended jdk within.

However, I think here, the system is slow due to swapping as per your last inputs.

Also, please check GC output. Do you see very long running GCs? Or very frequet Full GCs?

Long running GC (full GC running higher than 10 secs or minor GC running higher than 2-3 seconds each)= Cause is swapping most of the cases. (very rarely non)

Frequent Full GCs = means you need to tweak memory settings based on what is causing the GC to take place. Minor GCs, tweak NewSize/MaxNewSize. For Full GCs if perm exhaustion is the cause, please modify the Perm/MaxPerm, if tenured is the cause, you can tweak HeapSize a bit higher.

Hope this helps.

Best Regards,

Snehal

[Former Member · Accepted solution]

Hi Snehal and Effan,

My portal is much better now, but still need more tuning. I'm not sure which of the following changes made big diference.

1. I have deleted 2 server nodes. So I have one now.

2. Instance ID heap: Increased to 1024 from 256M.

3. Server Node_1: Chaged NewSize/MaxNewSize=300 from 200.

4. Server Node _1: Changed Permsize/MaxPermSize=600 from 300.

- Swap is stll being used upto 4 GB. I'm not sure If I'm reading right value so I'm pasting the result of prstat -a commandin Solrais: We have about 21 GB SWAP space.

NPROC        USERNAME          SWAP   RSS        MEMORY         TIME      CPU
    24        <sid>adm          4286M   1989M    25%        0:30:44        0.2%
    55        root              435M      384M   4.8%      12:25:29        0.0%
    73        db2<sid>          2428M   1163M    14%       0:00:50         0.0%
     7         daemon           10M      7600K    0.1%       0:05:37        0.0%
     1         noaccess         86M      88M      1.1%       3:22:00        0.0%
     1          smmsp            1552K    7032K    0.1%       0:00:48        0.0%

- GC > The load average is different than with 3 server nodes. For Ex: ServiceManager Started in 88793 ms with 3 server nodes whereas it took 64830 ms with new settins and parameters.

- Samething with FRAmework: It started in 107451 ms whreas it took 64830 wth new settings.

I will be upgrading to Java 1.4.2_25 (latest) soon. I hoping differences. At this point, what do you guys suggest? Should I increase more PermSize?

Both of your help is apprecated!

Thanks,

Afi

Edited by: Afi C on Mar 2, 2010 6:20 PM

[Former Member · Accepted solution]

That's good news indeed. Please refer to Snehal's suggestion earlier and note 716604 to do a further fine tuning, however, I don't think it could bring you a significant improvement further since the bottleneck is the physical memory of 8GB, which is insufficient even for a DEV Java instance on 64bit.

Also be carefully when you change existing JDK, do a full backup and schedule a sufficient down time, and then find details in note 718901.

Regards,

Effan

[Former Member · Accepted solution]

Hi Effan,

I have 8 GB in Sandbox & Dev.

Thanks,

Afi

[Former Member · Accepted solution]

Sorry, seems my memory didn't serve well......

With 8GB it's a lot better, it depends on how many active users and how much activities in this instance to determine the quantities of server nodes. So it's a balance assessment between Java and OracleDB for you, less DB activities spare more memory for server nodes. Basically one server node takes approximately 2.5GB if it's well tuned as 716604, 0.8-1GB left for OS kernel. Have fun with the math. ^_

Effan

[Former Member · Accepted solution]

Hello Afi,

What does it mean by :

2. Instance ID heap: Increased to 1024 from 256M.

If you have a 700 system on 64 bit then the recommended heapsize is 2G per server process. So your instance server settings should have MaxHeapSize = 2G = -Xms in the JVM settings. Let me know if I'm not getting your point 2 clearly. But this is what the value should be at the least.

MaxNewSize and NewSize should be 1/5th of the MaxHeapSize which in this case should be 400M. Note 723909 explains this in detail.

It gives me a feeling that somewhere your JVM settings are incorrect. Could you please paste the dev_server0 excerpt from the latest restart (after ensuring the above heap/newsize settings)? The part that I want to see is the one that shows all the parameters that are taken into account.

Also, the startup times for the service manager you mentioned is very high. I'm thinking this could be due to the low heap value of server node (256 or even 1024 is low for 700). However, I do need to know if you are seeing back to back full GCs or long running GCs. Could you please clarify that?

Unfortunately, I don't know how swapping on Solaris works so I can't comment effectively on your swap values.

Best Regards,

Snehal

[Former Member · Accepted solution]

Hello,

In addition to the above suggestions, please also check if there trace activated for any of the applications. Trace can also cause the portal to run slow.

Thanks & Regards,

Denish

[Former Member · Accepted solution]

Hi Snehal,

On point 2, you are right that it should use 2 GB Heap and 2.5 GB for BI Java/portal, But if you look at the my first post with Instance ID, heap was defined very low. As per the note 723909, I didn't understand if we have to update heap size for instance ID. I thought it is only for Server Nodes & Dispatcher. Initially it was 256, now it is 2560 heap size as per the note since I have BI Java + portal on this box.

So, Server node & Instance ID have 2560 heap size now.

Logs: dev_server0

-JStartupIReadSection: read node properties (ID14367050)

-> node name : server0

-> node type : server

-> node execute : yes

-> jlaunch parameters :

-> java path : /usr/j2se

-> java parameters : -Djava.security.policy=./java.policy -Djava.security.egd=file:/dev/urandom -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy -Dorg.xml.sax.driver=com.sap.engine.lib.xml.parser.SAXParser -Djco.jarm=1 -Dsun.io.useCanonCaches=false -Djava.awt.headless=true -XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=800m -XX:MaxPermSize=800m -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=90 -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:DisableExplicitGC -XX:PrintGCDetails -XX:PrintGCTimeStamps -XX:UseParNewGC -XX:+HandlePromotionFailure -XX:SoftRefLRUPolicyMSPerMB=1

-> java vm version : 1.4.2_12-b03

-> java vm vendor : Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)

-> java vm type : server

-> java vm cpu : sparcv9

-> heap size : 2560M

-> init heap size : 2560M

-> root path : /usr/sap/<SID>/JC01/j2ee/cluster/server0

-> class path : ./bin/boot/boot.jar:./bin/boot/jaas.jar:./bin/system/bytecode.jar:.

-> OS libs path : /usr/sap/<SID>/JC01/j2ee/os_libs

-> main class : com.sap.engine.boot.Start

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : /usr/sap/<SID>/JC01/exe/jstartup.jar:/usr/sap/<SID>/JC01/exe/jvmx.jar

-> shutdown class : com.sap.engine.boot.Start

-> parameters :

-> debuggable : no

-> debug mode : no

-> debug port : 50121

-> shutdown timeout : 120000

-**********************************************************************

-(Thr 1) JLaunchISetDebugMode: set debug mode (no)

-(Thr 4) JLaunchIStartFunc: Thread 4 started as Java VM thread.

-**********************************************************************

-JHVM_LoadJavaVM: VM Arguments of node (server0)

-> stack : 1048576 Bytes

-> arg( 0): exit

-> arg( 1): abort

-> arg( 2): vfprintf

-> arg( 3): -Denv.class.path=:/db2/<SID>/sqllib/java/sqlj.zip:/db2/<SID>/sqllib/java/db2java.zip:/db2/<SID>/sqllib/java/runtime.zip:.:/db2/<SID>/sqllib/java/sqlj.zip:/db-2/<SID>/sqllib/java/db2java.zip:/db2/<SID>/sqllib/java/runtime.zip:.:/db2/<SID>/sqllib/java/sqlj.zip:/db2/<SID>/sqllib/java/db2java.zip:/db2/<SID>/sqllib/ja-va/runtime.zip:.

-> arg( 4): -Djava.security.policy=./java.policy

-> arg( 5): -Djava.security.egd=file:/dev/urandom

-> arg( 6): -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy

-> arg( 7): -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy

-> arg( 8): -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy

-> arg( 9): -Dorg.xml.sax.driver=com.sap.engine.lib.xml.parser.SAXParser

-> arg( 10): -Djco.jarm=1

-> arg( 11): -Dsun.io.useCanonCaches=false

-> arg( 12): -Djava.awt.headless=true

-> arg( 13): -XX:NewSize=512m

-> arg( 14): -XX:MaxNewSize=512m

-> arg( 15): -XX:PermSize=800m

-> arg( 16): -XX:MaxPermSize=800m

-> arg( 17): -XX:SurvivorRatio=2

-> arg( 18): -XX:TargetSurvivorRatio=90

-> arg( 19): -XX:SoftRefLRUPolicyMSPerMB=1

-> arg( 20): -verbose:gc

-> arg( 21): -XX:+DisableExplicitGC

-> arg( 22): -XX:+PrintGCDetails

-> arg( 23): -XX:+PrintGCTimeStamps

-> arg( 24): -XX:+UseParNewGC

-> arg( 25): -XX:+HandlePromotionFailure

-> arg( 26): -XX:SoftRefLRUPolicyMSPerMB=1

-> arg( 27): -Dsys.global.dir=/usr/sap/<SID>/SYS/global

-> arg( 28): -Dapplication.home=/usr/sap/<SID>/JC01/exe

-> arg( 29): -Djava.class.path=/usr/sap/<SID>/JC01/exe/jstartup.jar:/usr/sap/<SID>/JC01/exe/jvmx.jar:./bin/boot/boot.jar:./bin/boot/jaas.jar:./bin/system/bytecode.jar:.

-> arg( 30): -Djava.library.path=/usr/j2se/jre/lib/sparcv9/server:/usr/j2se/jre/lib/sparcv9:/usr/j2se/jre/../lib/sparcv9:/usr/sap/<SID>/JC01/exe:/usr/sap/<SID>/JC01/exe:/-usr/sap/<SID>/SYS/exe/run:/db2/<SID>/sqllib/lib:/usr/lib:/usr/sap/<SID>/JC01/j2ee/os_libs:/usr/sap/<SID>/JC01/exe:/usr/sap/<SID>/JC01/exe:/usr/sap/<SID>/SYS-/exe/run:/db2/<SID>/sqllib/lib

-> arg( 31): -Dmemory.manager=2560M

-> arg( 32): -Xmx2560M

-> arg( 33): -Xms2560M

-> arg( 34): -DLoadBalanceRestricted=no

-> arg( 35): -Djstartup.mode=JCONTROL

-> arg( 36): -Djstartup.ownProcessId=19530

-> arg( 37): -Djstartup.ownHardwareId=X0473126417

-> arg( 38): -Djstartup.whoami=server

-> arg( 39): -Djstartup.debuggable=no

-> arg( 40): -DSAPINFO=<SID>_01_server

-> arg( 41): -DSAPSTARTUP=1

-> arg( 42): -DSAPSYSTEM=01

-> arg( 43): -DSAPSYSTEMNAME=<SID>

-> arg( 44): -DSAPMYNAME=sap-<SID>1_<SID>_01

-> arg( 45): -DSAPDBHOST=sap-<SID>1

-> arg( 46): -Dj2ee.dbhost=sap-<SID>1

**********************************************************************

GC: I'm not too sure about GC logs. I really don't know how to check GC logs. My understanding is that std_server0 would show GC, but I don't know how to read them, but I was comparing std_server0 file everytime I restart SAP. I started seeing differences when I deleted 2 server nodes and added more heap to Instance ID. Please correct me If i'm am looking at the right file for GC (std_server0).

[Former Member · Accepted solution]

Hi Snehal,

After updating parameters, here are two lines from std_server0.

ServiceManager started for 63106 ms.

Framework started for 77711 ms...

It takes about 14 minutes to start the portal from startsap script including JCMON status running.

Thanks,

Afi

Edited by: Afi C on Mar 3, 2010 12:32 AM

[Former Member · Accepted solution]

Hi Snehal,

Accordin to Note 716604, I have to add the following Parameters for Sun Systems since we are on UtraSPARC T2.

1. -XX:+UseConcMarkSweepGC

2. -XX:+CMSPermGenSweepingEnabled

3. -XX:+CMSClassUnloadingEnabled

4. -XX:CMSInitiatingOccupancyFraction=60

5. -XX:+UseCMSInitiatingOccupancyOnly

6. -XX:ParallelGCThreads=<number of cores>

7. -XX:+UseParNewGC

Q1. What value do I need to provide for number 6 (-XX:ParallelGCThreads=<number of cores>)?

Q2. (I suppose) I should update those pram. in Instance ID, Server Node, and Dispatcher. Please confirm!

Your help is reall appreciated!

Thanks,

Afi

[Former Member · Accepted solution]

Hello Afi,

1)I checked the dev_server0 file and your parameters (heap et all) look okay.

2) Ok, I understand your confusion. Instance level means you go to configtool and click on Instance_<id>, goto Servers General and maintain parameters here. These will be taken into effect if any parameters are not specified at the server process level.

Normally, we directly go to Instance_<id> -> server<id> and make modifications there. This is what I meant for JVM settings update. These take the highest priority if maintained correctly. If you've maintained heap as 2G here but in Instance -> server general as 256M, the value taken into effect will be 2G.

3)14 mins startup time is fine, not bad at all.

4) The value of -XX:+UseParNewGC should be the no. of core processors your machine has.

5) Finally, I'd support Denish's suggestion as well. The best way to reset the log settings is to go to Visual Administrator ->

Server -> services -> log configuration and on the right frame in the toolbar press the 'Installation Default' button. Then restart the instance. Just incase any severe trace generation is slowing down the j2ee, this part will take care of it.

6) You can learn how to read a GC output by going to the following link:

http://java.sun.com/docs/hotspot/gc1.4.2/

Solaris specific commandline options for JDK are found @ :

http://java.sun.com/j2se/1.4.2/docs/tooldocs/solaris/java.html

The JDK Parameters can be found @:

http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

Hope the above inputs help.

Regards,

Snehal

Edited by: Snehal Bhaidasna on Mar 3, 2010 11:36 AM

[Former Member]

Hello my friend

What's the target OS? JDK type and version? Any other major applications running in the same 8G RAM host?

Note 723909 provides a general setting as a rule of thumb for SUN JDK on Win/Linux platform, but sometimes you still need to customize settings in your own case.

Regards,

Effan

[Former Member]

Hi,

I'm on Solaris 10. Nothing is running except Database.

Thanks,

[Former Member]

How much RAM is oracle DB allocated? Is this a AS Java-only instance?

You have 2 server nodes plus dispatcher which is almost 4.5GB allocated. Ideally you need to leave 1GB for OS kernel operation, then oracle should have less than 8-4.5-1=2.5GB. Or just give it a test see if it's memory related, decrease server node to one and then see if it's better.

Regards,

Effan