High number of jlaunch processes after startsap

[Former Member]

We have XI30, based on Ntw 04 with Web As 640, Abap+Java, on Linux.

We discovered after startup of the Sap istance are presents hundreds of jlaunch processes pending, ie 400 or more.

All of these jlauch processes consumes slowly memory resources. Then when the memory available is less than a certain value, the J2EE instance begin to restart by itself.

At this poin the only solution is to close the whole Sap instance, to claen up all of these processes and the shared memory then to restart, till the next

On other systems with the same Os anc the same release of Sap we see just 2 or 3 jlaunch processes in normal operation, and we do not experiment at all this strange behaviour.

We applied already the latest Kernel patch, but nothing is changed.

We try to kill some of these jlaunch processes, but the J2EE get down immediately.

What all of these jlauch processe are ? How can avoid these all these jlauch to start ?

Regards

[KleinM · Accepted solution]

Hello Roberto,

this has something to do with the fact how Java Threads are implemented on Linux. On Linux each Java Thread is a cloned process on Linux OS level. So, this is normal behavior.

You can find more information about it here: Link: [Java Technology on the Linux Platform|http://java.sun.com/developer/technicalArticles/Programming/linux/]

See section: "About Linux Threads" for more details.

Best regards,

Mathias

[Former Member · Accepted solution]

Hi Mathias,

me too was thinking this, but someting it's not working in the proper way.

I get a look to a the jlaunch processes active on a production environment and I found just 3 processes.

From the Sap documentation I read that the normal situation for 1 cluster node as we have: 1 jlauch or the dispatcher, 1 for the server0 and one for the SDM .

Always on this production machine there is only 1 jcontrol process running.

On the machine where we have problems instead for 1 cluster node I see 5 Jcontrol processes and almost 400 , yes four hundred, jlaunch processess.

Probabily almost all of them are ghost processes; the problem is that they causes the Ram to be exausted.

It seems to me during the startup some process enconter problems, then it begin to restart again and again....

Any advise ?

[Former Member · Accepted solution]

Hi Roberto,

If you watch "top" how many jlaunch processes you have ??

Also did you track which jlaunch process is creating an issue ??

You need to check the defaulttrace files and also files in the work directory.

Is this happening immidiately after you start the instance ??

Is there any load the server ??

Any web dispatchers connected to the server ??

There are many such things we need to look to figure out the issue.

Can you please paste the default trace file of dispatcher and server<x> taken at the time of issue.

Cheers.......,

Raghu

[KleinM · Accepted solution]

Hello Roberto,

you say you have 5 JControl processes running on this machine? That's strange!

It should be only one...

How are those JControl processes named? jc.sap<SID>_JC00 ???? All with the same SID?

There is only one Java Instance installed on this server, right?

Can you take a look into the dev_jcontrol file in directory /usr/sap/<SID>/JC00/work?

What happens if you execute "stopsap" are all JControl processes gone after that?

By the way: Which Linux distribution and version are you using (Red Hat? SuSE?)

Command:

cat /etc/*-release

Which Linux kernel version is running on this system?

Command:

uname -a

)

Best regards,

Mathias

Edited by: Mathias Klein on Oct 2, 2009 10:34 AM

[Former Member · Accepted solution]

Hi Ragu

see below the answers:

> If you watch "top" how many jlaunch processes you have ??

with Top I see 4 or 5 Jlaunch processes, while with ps -ef |grep jlaunch I see more than 400 processes.

Top reports:

462 processes: 460 sleeping, 1 running, 1 zombie, 0 stopped

"top -H" does not work.,

> Also did you track which jlaunch process is creating an issue ??

Unfortunately I do not know which is

>You need to check the defaulttrace files and also files in the work directory.

Is this happening immidiately after you start the instance ??

Usually it happens when the developers do the activation of message mapping into the Integration Builder. But not always it happen, probabily it depend from the complexity of the object to be activated.

When the J2ee server0 get down at restart it happen often the server0 start to do ping-pong after the service msp is started.

> Is there any load the server ??

See the previous answer

>Any web dispatchers connected to the server ??

No

> Can you please paste the default trace file of dispatcher and server taken at the time of issue.

See below a piece of std_server0.out when the dump occur:

.......

.......

232K->514262K(1267200K), 0.2103950 secs]

68894.618: [GC 68894.618: [DefNew: 128234K->39672K(130560K), 0.5961440 secs] 601

302K->516758K(1267200K), 0.5962080 secs]

-

---

stdout/stderr redirect

-

---

node name : server0

pid : 4479

system name : ABC

system nr. : 01

started at : Fri Oct 2 12:51:51 2009

CompilerOracle: exclude com/sapportals/portal/pb/layout/taglib/ContainerTag addI

viewResources

CompilerOracle: exclude com/sap/engine/services/keystore/impl/security/CodeBased

SecurityConnector getApplicationDomain

CompilerOracle: exclude com/sap/engine/services/rmi_p4/P4StubSkeletonGenerator g

enerateStub

CompilerOracle: exclude com/sapportals/portal/prt/util/StringUtils escapeToJS

CompilerOracle: exclude com/sapportals/portal/prt/core/broker/PortalServiceItem

startServices

CompilerOracle: exclude com/sap/engine/services/webservices/server/deploy/WSConf

igurationHandler downloadFile

CompilerOracle: exclude com/sapportals/portal/prt/jndisupport/util/AbstractHiera

rchicalContext lookup

CompilerOracle: exclude com/sapportals/portal/navigation/cache/CacheNavigationNo

de getAttributeValue

CompilerOracle: exclude com/sapportals/portal/navigation/TopLevelNavigationiView

PrintNode

CompilerOracle: exclude com/sapportals/wcm/service/ice/wcm/ICEPropertiesCoder en

code

CompilerOracle: exclude com/sap/lcr/pers/delta/importing/ObjectLoader loadObject

s

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/Inst

anceBuilder readElement

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/Inst

anceBuilder readSequence

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/Type

MappingImpl initializeRelations

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/Gene

ratedComplexType _loadInto

SAP J2EE Engine Version 6.40 PatchLevel 108290.313 is starting...

Loading: LogManager ... 816 ms.

Loading: PoolManager ... 6 ms.

Loading: ApplicationThreadManager ... 110 ms.

Loading: ThreadManager ... 418 ms.

Loading: IpVerificationManager ... 12 ms.

Loading: ClassLoaderManager ... 19 ms.

Loading: ClusterManager ... 253 ms.

Loading: LockingManager ... 118 ms.

Loading: ConfigurationManager ... 1938 ms.

Loading: LicensingManager ... 45 ms.

Loading: ServiceManager ...

Loading services.:

Service memory started. (32 ms).

Service userstore started. (12 ms).

Service cross started. (26 ms).

Service file started. (368 ms).

Service timeout started. (181 ms).

Service p4 started. (328 ms).

Service classpath_resolver started. (33 ms).

Service trex.service started. (122 ms).

Service runtimeinfo started. (54 ms).

Service jmx_notification started. (53 ms).

12.695: [GC 12.696: [DefNew: 87039K->6863K(130560K), 0.1437840 secs] 87039K->686

3K(1267200K), 0.1438600 secs]

18.089: [GC 18.089: [DefNew: 93903K->8723K(130560K), 0.1565010 secs] 93903K->872

3K(1267200K), 0.1565610 secs]

20.411: [GC 20.413: [DefNew: 95762K->10212K(130560K), 0.0796130 secs] 95762K->10

212K(1267200K), 0.0796480 secs]

Service log_configurator started. (8340 ms).

Service locking started. (16 ms).

Service naming started. (354 ms).

Service failover started. (73 ms).

Service appclient started. (80 ms).

Service javamail started. (265 ms).

Service ts started. (195 ms).

Service jmsconnector started. (299 ms).

Service licensing started. (7 ms).

Service http started. (798 ms).

Service connector started. (306 ms).

Service webservices started. (687 ms).

Service deploy started. (17943 ms).

Service configuration started. (13 ms).

Service MobileArchiveContainer started. (269 ms).

Service MobileSetupGeneration started. (272 ms).

31.519: [GC 31.521: [DefNew: 97252K->14752K(130560K), 0.1271440 secs] 97252K->14

752K(1267200K), 0.1272050 secs]

Service dbpool started. (2060 ms).

Service com.sap.security.core.ume.service started. (3024 ms).

Service security started. (1257 ms).

Service classload started. (134 ms).

Service applocking started. (137 ms).

Service shell started. (150 ms).

Service tc_eCATTPingservice started. (47 ms).

Service telnet started. (70 ms).

Service webdynpro started. (364 ms).

Service ejb started. (856 ms).

Service servlet_jsp started. (973 ms).

Service keystore started. (833 ms).

Service ssl started. (33 ms).

Service tc_secsecurestorage~service started. (261 ms).

Service dsr started. (673 ms).

Service rfcengine started. (580 ms).

Service jmx started. (1036 ms).

37.850: [GC 37.850: [DefNew: 101785K->23964K(130560K), 0.2141800 secs] 101785K->

23964K(1267200K), 0.2142360 secs]

Service tc_lmctc_culculservice_sda started. (0 ms).

Service tc_secwssec~service started. (1066 ms).

Service tc_lmctc_confsservice_sda started. (796 ms).

Service tc_secdestinations~service started. (1518 ms).

Service sld started. (1922 ms).

Service pmi started. (277 ms).

Service apptracing started. (269 ms).

Service tc_secvsi~service started. (554 ms).

Service basicadmin started. (1848 ms).

Service adminadapter started. (88 ms).

40.400: [GC 40.400: [DefNew: 111004K->29574K(130560K), 0.2340310 secs] 111004K->

29574K(1267200K), 0.2340890 secs]

Service monitor started. (998 ms).

41.653: [GC 41.653: [DefNew: 116608K->30106K(130560K), 0.1720070 secs] 116608K->

30106K(1267200K), 0.1720640 secs]

Service tc.monitoring.logviewer started. (3688 ms).

Service tc_lmctc_culservice_sda started. (5781 ms).

45.277: [GC 45.277: [DefNew: 117146K->37892K(130560K), 0.2398960 secs] 117146K->

37892K(1267200K), 0.2399430 secs]

Service jms_provider started. (7595 ms).

Service com.sap.aii.af.cpa.svc started. (9374 ms).

Service com.sap.aii.af.security.service started. (42 ms).

49.219: [GC 49.219: [DefNew: 124932K->41454K(130560K), 0.2554820 secs] 124932K->

41454K(1267200K), 0.2555410 secs]

Service com.sap.aii.af.svc started. (653 ms).

Service com.sap.aii.af.ms.svc started. (718 ms).

Service com.sap.aii.adapter.marketplace.svc started. (189 ms).

Service com.sap.aii.adapter.bc.svc started. (188 ms).

Service com.sap.aii.adapter.jms.svc started. (247 ms).

53.474: [GC 53.474: [DefNew: 128494K->39612K(130560K), 0.2487170 secs] 128494K->

44448K(1267200K), 0.2487740 secs]

Service com.sap.aii.adapter.xi.svc started. (3247 ms).

Service com.sap.aii.adapter.jdbc.svc started. (181 ms).

Service com.sap.aii.adapter.mail.svc started. (265 ms).

Service com.sap.aii.af.ispeak.svc started. (719 ms).

Service com.sap.aii.adapter.file.svc started. (1583 ms).

Service com.sap.aii.adapter.rfc.svc started. (2098 ms).

ServiceManager started for 50541 ms.

Framework started for 54737 ms.

SAP J2EE Engine Version 6.40 PatchLevel 108290.313 is running!

PatchLevel 108290.313 April 15, 2007 18:55 GMT

After that, after 10 minutes about the file record also the string 'Service mps started'; then often the J2EE get down at that time.

It's strange the msp service appears started after the row "SAP J2EE Engine ... is running!".

[Former Member · Accepted solution]

Hi mathias,

yes there is only one J2EE instance , with one node only.

I see these jcontrol processes with ps -ef |grep jcontrol:

abcadm 4479 4446 0 Sep30 ? 00:00:03 jcontrol pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DSAPSTART=1 -DCONNECT_PORT=65000 -DLISTEN_PORT=64999 -DSAPSYSTEM=01 -DSAPSYSTEMNAME=ABC -DSAPMYNAME=sapserver_ABC_01 -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DFRFC_FALLBACK=ON -DFRFC_FALLBACK_HOST=

abcadm 4494 4479 0 Sep30 ? 00:00:03 jcontrol pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DSAPSTART=1 -DCONNECT_PORT=65000 -DLISTEN_PORT=64999 -DSAPSYSTEM=01 -DSAPSYSTEMNAME=ABC -DSAPMYNAME=sapserver_ABC_01 -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DFRFC_FALLBACK=ON -DFRFC_FALLBACK_HOST=

abcadm 4496 4494 0 Sep30 ? 00:00:03 jcontrol pf=/usr/sap/XS0/SYS/profile/ABC_DVEBMGS01_sapserver -DSAPSTART=1 -DCONNECT_PORT=65000 -DLISTEN_PORT=64999 -DSAPSYSTEM=01 -DSAPSYSTEMNAME=ABC -DSAPMYNAME=sapserver_ABC_01 -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DFRFC_FALLBACK=ON -DFRFC_FALLBACK_HOST=

abcadm 4617 4494 0 Sep30 ? 00:00:00 jcontrol pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DSAPSTART=1 -DCONNECT_PORT=65000 -DLISTEN_PORT=64999 -DSAPSYSTEM=01 -DSAPSYSTEMNAME=ABC -DSAPMYNAME=sapserver_ABC_01 -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DFRFC_FALLBACK=ON -DFRFC_FALLBACK_HOST=

abcadm 4618 4494 0 Sep30 ? 00:00:08 jcontrol pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DSAPSTART=1 -DCONNECT_PORT=65000 -DLISTEN_PORT=64999 -DSAPSYSTEM=01 -DSAPSYSTEMNAME=ABC -DSAPMYNAME=sapserver_ABC_01 -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DFRFC_FALLBACK=ON -DFRFC_FALLBACK_HOST=

I do not know which is the real process and wich the ghosts, if thery are.

[Former Member · Accepted solution]

Below the relative dev_jcontrol (part1) , the j2ee server0 was just restarted by itself:

-

---

trc file: "/usr/sap/ABC/DVEBMGS01/work/dev_jcontrol", trc level: 1, release: "640"

-

---

node name : jcontrol

pid : 4479

system name : ABC

system nr. : 01

started at : Wed Sep 30 18:42:24 2009

arguments :

arg[00] : jcontrol

arg[01] : pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver

arg[02] : -DSAPSTART=1

arg[03] : -DCONNECT_PORT=65000

arg[04] : -DLISTEN_PORT=64999

arg[05] : -DSAPSYSTEM=01

arg[06] : -DSAPSYSTEMNAME=ABC

arg[07] : -DSAPMYNAME=sapserver_ABC_01

arg[08] : -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver

arg[09] : -DFRFC_FALLBACK=ON

arg[10] : -DFRFC_FALLBACK_HOST=localhost

[Thr 16384] Wed Sep 30 18:42:24 2009

[Thr 16384] INFO: Unknown property [box.number=ABCDVEBMGS01sapserver]

[Thr 16384] INFO: Unknown property [ms.host=sapserver]

[Thr 16384] INFO: Unknown property [ms.port=3602]

[Thr 16384] INFO: Unknown property [system.id=1]

**********************************************************************

JStartupReadInstanceProperties: read instance properties [/usr/sap/ABC/DVEBMGS01/j2ee/cluster/instan

ce.properties:/usr/sap/ABC/DVEBMGS01/SDM/program/config/sdm_jstartup.properties]

-> ms host : sapserver

-> ms port : 3602

-> OS libs : /usr/sap/ABC/DVEBMGS01/j2ee/os_libs

-> Admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

Used property files

-> files [00] : /usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

-> files [01] : /usr/sap/ABC/DVEBMGS01/SDM/program/config/sdm_jstartup.properties

**********************************************************************

Instance properties

-> ms host : sapserver

-> ms port : 3602

-> os libs : /usr/sap/ABC/DVEBMGS01/j2ee/os_libs

-> admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

Bootstrap nodes

-> [00] bootstrap : /usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

-> [01] bootstrap_ID14029700 : /usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

-> [02] bootstrap_ID14029750 : /usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

**********************************************************************

Worker nodes

-> [00] ID14029700 : /usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

-> [01] ID14029750 : /usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

-> [02] sdm : /usr/sap/ABC/DVEBMGS01/SDM/program/config/sdm_jstartup.properties

**********************************************************************

[Thr 16384] SigISetDefaultAction : default handling for signal 17

[Thr 16386] JControlDPMessageProxy: Thread 16386 started as handler thread for R/3 dispatcher messag

es.

[Thr 16384] JControlExecuteBootstrap: execute bootstrap process [bootstrap]

[Thr 16384] [Node: bootstrap] java home is set by profile parameter

Java Home: /usr/java/j2sdk1.4.2_16

**********************************************************************

JStartupIReadSection: read node properties [bootstrap]

-> node name : bootstrap

-> node type : bootstrap

-> node execute : yes

-> java path : /usr/java/j2sdk1.4.2_16

-> java parameters : -Djco.jarm=1 -Djco.jarm=1

-> java vm version : 1.4.2_16-b05

-> java vm vendor : Java HotSpot(TM) Server VM (Sun Microsystems Inc.)

-> java vm type : server

-> java vm cpu : i386

-> heap size : 384M

-> root path : /usr/sap/ABC/DVEBMGS01/j2ee/cluster

-> class path : ./bootstrap/launcher.jar

-> OS libs path : /usr/sap/ABC/DVEBMGS01/j2ee/os_libs

-> main class : com.sap.engine.offline.OfflineToolStart

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : /usr/sap/ABC/DVEBMGS01/j2ee/os_libs/jstartup.jar

-> parameters : com.sap.engine.bootstrap.Bootstrap ./bootstrap ID0140297

-> debuggable : yes

-> debug mode : no

-> debug port : 60000

-> shutdown timeout: 120000

**********************************************************************

JStartupStartJLaunch: program = /usr/sap/ABC/DVEBMGS01/j2ee/os_libs/jlaunch

-> arg[00] = /usr/sap/ABC/DVEBMGS01/j2ee/os_libs/jlaunch

-> arg[01] = pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver

-> arg[02] = -DSAPINFO=ABC_01_bootstrap

-> arg[03] = -file=/usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties

-> arg[04] = -nodeName=bootstrap

-> arg[05] = -nodeId=-1

-> arg[06] = -syncSem=22118467

-> arg[07] = -jvmOutFile=/usr/sap/ABC/DVEBMGS01/work/jvm_bootstrap.out

-> arg[08] = -stdOutFile=/usr/sap/ABC/DVEBMGS01/work/std_bootstrap.out

-> arg[09] = -locOutFile=/usr/sap/ABC/DVEBMGS01/work/dev_bootstrap

-> arg[10] = -mode=BOOTSTRAP

-> arg[11] = pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver

-> arg[12] = -DSAPSTART=1

-> arg[13] = -DCONNECT_PORT=35489

-> arg[14] = -DLISTEN_PORT=64999

-> arg[15] = -DSAPSYSTEM=01

-> arg[16] = -DSAPSYSTEMNAME=ABC

-> arg[17] = -DSAPMYNAME=sapserver_ABC_01

-> arg[18] = -DSAPPROFILE=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver

-> arg[19] = -DFRFC_FALLBACK=ON

-> arg[20] = -DFRFC_FALLBACK_HOST=localhost

-> lib path = LD_LIBRARY_PATH=/usr/java/j2sdk1.4.2_16/jre/lib/i386/server:/usr/java/j2sdk1.4.2_16/jr

e/lib/i386:/usr/java/j2sdk1.4.2_16/jre/../lib/i386:/usr/sap/ABC/DVEBMGS01/j2ee/os_libs:/usr/sap/ABC/

SYS/exe/run:/usr/sap/ABC/SYS/exe/runU:/oracle/ABC/920_32/lib:/usr/sap/ABC/DVEBMGS01/j2ee/os_libs:/us

r/sap/ABC/DVEBMGS01/j2ee/os_libs:/usr/sap/ABC/SYS/exe/run:/usr/sap/ABC/SYS/exe/runU:/oracle/ABC/920_

32/lib

-> exe path = PATH=/usr/java/j2sdk1.4.2_16/bin:/usr/sap/ABC/DVEBMGS01/j2ee/os_libs:/oracle/ABC/920_3

2/bin:/usr/java/j2sdk1.4.2_16/bin:.:/home/ABCadm:/usr/sap/ABC/SYS/exe/runU:/usr/sap/ABC/SYS/exe/run:

/usr/kerberos/bin:/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/java/j2sdk1.4.2_07/b

in

[Former Member · Accepted solution]

note the dev_jcontrol attached is just the first piece.

The full file is to bigger to be attched.

Which error or string should I search ?

regards

[Former Member · Accepted solution]

Hi Roberto,

ps -ef |grep jlaunch I see more than 400 processes -> Wow....Very strange............

Dont mind...Few more questions........

Can you post the instance.properties file ??

What is the java version, SP stack level on the server ??

Can u backup and delete *.vmprop files and restart the instance ??

Cheeers....,

Raghu

[Former Member · Accepted solution]

Hi Ragu,

here the instance.property:

ID14029700.ClassPath=./bin/boot/boot.jar:./bin/system/bytecode.jar:.

ID14029700.Debuggable=no

ID14029700.JLaunchParameters=

ID14029700.JavaParameters=-Djava.security.policy=.\java.policy -Djava.security.egd=file:/dev/urandom -Djco.jarm=1

-XX:NewSize=28m -XX:MaxNewSize=28m -Xms170m -verbose:gc -Drdbms.driverLocation=/opt/oracle/ABC/920_32/jdbc/lib/c

lasses12.jar -Dsys.global.dir=/opt/usr/sap/ABC/SYS/global/

ID14029700.LogName=dispatcher

ID14029700.MainClass=com.sap.engine.boot.Start

ID14029700.MaxHeapSize=170

ID14029700.Name=dispatcher

ID14029700.Parameters=

ID14029700.RootPath=/usr/sap/ABC/DVEBMGS01/j2ee/cluster/dispatcher

ID14029700.Type=dispatcher

ID14029750.ClassPath=./bin/boot/boot.jar:./bin/boot/jaas.jar:./bin/system/bytecode.jar:.

ID14029750.DebugMode=no

ID14029750.DebugPort=50121

ID14029750.Debuggable=no

ID14029750.JLaunchParameters=

ID14029750.JavaParameters=-Djava.security.policy=./java.policy -Djava.security.egd=file:/dev/urandom -Dorg.omg.CO

RBA.ORBClass=com.sap.engine.system.ORBProxy -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonP

roxy -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy -Djco.jarm=1 -Ds

un.io.useCanonCaches=false -XX:MaxPermSize=192m -Djava.awt.headless=true -XX:PermSize=192m -XX:SurvivorRatio=2 -X

X:TargetSurvivorRatio=90 -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:PrintGCDetails -XX:PrintGCTimeStamps -Dr

dbms.driverLocation=/oracle/ABC/920_32/jdbc/lib/classes12.jar -Drdbms.driverLocation=/opt/oracle/ABC/920_32/jdbc/

lib/classes12.jar -Dsys.global.dir=/opt/usr/sap/ABC/SYS/global/ -Xms1280m -XX:NewSize=170m -XX:MaxNewSize=170m

ID14029750.LogName=server0

ID14029750.MainClass=com.sap.engine.boot.Start

ID14029750.MaxHeapSize=1280

ID14029750.Name=server0

ID14029750.Parameters=

ID14029750.RootPath=/usr/sap/ABC/DVEBMGS01/j2ee/cluster/server0

ID14029750.Type=server

bootstrap.ClassPath=./bootstrap/launcher.jar

bootstrap.JavaParameters=-Djco.jarm=1 -Djco.jarm=1

bootstrap.MainClass=com.sap.engine.offline.OfflineToolStart

bootstrap.MaxHeapSize=384

bootstrap.Name=bootstrap

bootstrap.Parameters=com.sap.engine.bootstrap.Bootstrap ./bootstrap ID0140297

bootstrap.RootPath=/usr/sap/ABC/DVEBMGS01/j2ee/cluster

bootstrap.Type=bootstrap

bootstrap_ID14029700.ClassPath=./bootstrap/launcher.jar

bootstrap_ID14029700.JLaunchParameters=

bootstrap_ID14029700.JavaParameters=-Djco.jarm=1 -Djco.jarm=1

bootstrap_ID14029700.MainClass=com.sap.engine.offline.OfflineToolStart

bootstrap_ID14029700.MaxHeapSize=170

bootstrap_ID14029700.Name=dispatcher bootstrap

bootstrap_ID14029700.Parameters=com.sap.engine.bootstrap.Bootstrap ./bootstrap ID014029700

bootstrap_ID14029700.RootPath=/usr/sap/ABC/DVEBMGS01/j2ee/cluster

bootstrap_ID14029700.Type=bootstrap

bootstrap_ID14029750.ClassPath=./bootstrap/launcher.jar

bootstrap_ID14029750.JLaunchParameters=

bootstrap_ID14029750.JavaParameters=-Djco.jarm=1 -Djco.jarm=1

bootstrap_ID14029750.MainClass=com.sap.engine.offline.OfflineToolStart

bootstrap_ID14029750.MaxHeapSize=1024

bootstrap_ID14029750.Name=server0 bootstrap

bootstrap_ID14029750.Parameters=com.sap.engine.bootstrap.Bootstrap ./bootstrap ID014029750

bootstrap_ID14029750.RootPath=/usr/sap/ABC/DVEBMGS01/j2ee/cluster

bootstrap_ID14029750.Type=bootstrap

instance.box.number=ABCDVEBMGS01sapserver

instance.en.host=sapserver

instance.en.port=3202

instance.enabled=yes

instance.install.dir=/usr/sap/ABC/DVEBMGS01/j2ee

instance.ms.host=sapserver

instance.ms.port=3602

instance.osLibsPath=/usr/sap/ABC/DVEBMGS01/j2ee/os_libs

instance.runAction=NONE

instance.runMode=NORMAL

instance.system.id=1

I did not try to delete the .vmprop file.

regards

[Former Member · Accepted solution]

Here the number of jlauch processes I see running :

> ps -ef |grep jlaunch | wc -l

335

...definitively there is something wrong.

regards

[Former Member · Accepted solution]

Hi Roberto,

Delete the .vmprop file and restart the instance.

This will create a fresh .vmprop file. You can also take a backup of the file to be safe.

Let me know the result of the same.

Also increase the heap size of the dispatcher 512MB and server heap size to 2048.

Also let me know the RAM size and the no of processes.

Lets see if this helps.

Cheers......,

Raghu

[Former Member · Accepted solution]

Thanks for the feedback Raghu

I cannot extend the heap size, as the system is on 32 bit platform. In the past I already try to do it but the instance was not able to come up for '..unable to allocate heap memory...' error.

Can I delete the *.vmprop file while the system is running ?

regards

[Former Member · Accepted solution]

Hi Roberto,

You have to stop the instance. Delete the .vmprop file and start the instance.

When you start the instance this file gets recreated.

Let us see how it goes after this.

Cheers......,

Raghu

[KleinM · Accepted solution]

Hello Roberto,

there is one thing which confuses me. Your dev_jcontrol contains two different Java Folder in the PATH environment variable.

PATH=/usr/java/j2sdk1.4.2_16/bin:/usr/sap/ABC/DVEBMGS01/j2ee/os_libs:/oracle/ABC/920_3 2/bin:/usr/java/j2sdk1.4.2_16/bin:.:/home/ABCadm:/usr/sap/ABC/SYS/exe/runU:/usr/sap/ABC/SYS/exe/run: /usr/kerberos/bin:/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/java/j2sdk1.4.2_07/bin

This could be a problem.

What output do you get when you execute

java -version

Can you please provide some more information about your system?

NetWeaver Release 6.40? SPxx? (Looks like a Dual Stack ABAP+Java installation)

Linux Distribution and version?

Linux Kernel version?

Best regards,

Mathias

[Former Member · Accepted solution]

Hi Mathias,

here some information.

> sapserver:abcadm 1> java -version

> java version "1.4.2_16"

> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_16-b05)

> Java HotSpot(TM) Client VM (build 1.4.2_16-b05, mixed mode)

and

> sapserver:abcadm 2> uname -a

> Linux sapserver 2.4.21-27.0.4.ELsmp #1 SMP Sat Apr 16 18:43:06 EDT 2005 i686 athlon i386 GNU/Linux

It's Linux RH, 32 bit.

Yes, the system is question is a Dual Stack installation; it's an XI 30, with SAP_ABA and SAP_BASIS 640 at level 20. The J2EE is aligned at level 20.

The Sap kernel is use is :

> SAP R/3 Kernel 640_REL

> Database Client Library OCI_920

> created in Linux GNU SLES-7 i686

> created on Mar 9 2009 20:23:48

> Support Level 0

> Kernel Patch number 274

> Source ID 0.274

> DBSL Patch No. 220

> ICU Version 2.6.1 Unicode Version 4.0

> libsapu16 Version 1.0025 Mar 6 2009 19:33:06

regards

[Former Member · Accepted solution]

I managed the startsap to avoid to start the IGS and the SAPCCMRS Agent, and the instance seems to be more stable.

It does not mean the J2EE server0 never get down, but just it crashes less than before when it's under stress.

As for Sap suggestion we removed also the parameter +ParNewGC.

All these things can attempts to improve the stability.

But nothing explain why at startup almost 300 jlaunch processes are started. I'm in touch also with Oss consultant but no advises from their side on this topic.

Regards

[KleinM · Accepted solution]

Hello Roberto,

you said you are running kernel 2.4.21-27 on Red Hat Linux.

According to this information you must be running Red Hat Enterprise Linux 3 Update 4.

Did you read this SAP Note before you started the installation?

Link: [SAP Note 722273 - Red Hat Enterprise Linux 3.x, 4.x: Installation and Upgrade|https://service.sap.com/sap/support/notes/722273]

Have you set the environment variable LD_ASSUME_KERNEL=2.4.1 ?

What output do you get when you execute:

echo $LD_ASSUME_KERNEL

If this environment variable is set the Java 1.4.2_16 VM would be started in non NPTL (Native Posix Threading Library) mode.

This would explain why you see so many Java processes.

Best regards,

Mathias

[Former Member · Accepted solution]

Hi Mathias

I'm reading the note, anyway I did not the installation on this machine.

here the output of the commads suggested :

> [root@sapserver root]# echo $LD_ASSUME_KERNEL

> [root@sapserver root]# su - abcadm

> sapserver:abcadm 1> echo $LD_ASSUME_KERNEL

> LD_ASSUME_KERNEL: Undefined variable.

> sapserver:abcadm 2>

So for root is not in place, neither for abcadm.

regards

[Former Member]

Hi,

I am having the same issue on Linux.

So what was the solution to this problem as my user root has 2.4.1 set for LD_ASSUME_KERNEL

Thanks,

Shan

[Former Member]

Hi Roberto,

Check "top" and "top -iH". Check which jalunch process is stuck. You will get the PID of the jlaunch process in "top" and compare the same with "ps -ef |grep jlaunch"

See which process is stuck. One jlaunch thread may lock several jlaunch threads.

Accordingly, after finding the process ( dispatcher or server or SDM), check the default trace files.

Please catch the default trace at the time of issue and paste the same.

Cheeers....,

Raghu

[Former Member]

Addendum:

the jlaunch process we see running are of the form :

/usr/sap/ABC/DVEBMGS01/j2ee/os_libs/jlaunch pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_sapserver -DSAPINFO=ABC_01_server -file=/usr/sap/ABC/DVEBMGS01/j2ee/cluster/instance.properties -nodeName=ID14029750 -nodeId=1 -syncSem=19595331 -jvmOutFile=/usr/sap/ABC/DVEBMGS01/work/jvm_

That is what we see using ps. They are about 400 or more.

What they are ? How we can avoid to start all of them ?

regards