Problem with server0 exitcode 666

[Former Member]

Hello,

I have an ABAP+Java system with Oracle 10.2.0.4, windows 2003 server enterprise, 12 gb of ram and 8 cores.

The system starts OK and in a while the server0 stops, and in some time goes back again but sometimes we need to reboot the machine.

in the dev_server0 log i found only this error:

[Thr 3840] Fri Nov 06 08:46:48 2009

[Thr 3840] JLaunchIExitJava: exit hook is called (rc = 666)

[Thr 3840] **********************************************************************

• ERROR => The Java VM terminated with a non-zero exit code.

• Please see SAP Note 943602 , section 'J2EE Engine exit codes'

• for additional information and trouble shooting.

**********************************************************************

[Thr 3840] JLaunchCloseProgram: good bye (exitcode = 666)

-

---

trc file: "F:\usr\sap\FPP\DVEBMGS00\work\dev_server0", trc level: 1, release: "700"

-

---

node name : ID2866450

pid : 7140

system name : FPP

system nr. : 00

started at : Fri Nov 06 08:46:50 2009

arguments :

arg[00] : F:\usr\sap\FPP\DVEBMGS00\exe\jlaunch.exe

arg[01] : pf=F:\usr\sap\FPP\SYS\profile\FPP_DVEBMGS00_qsiscso-fd4

arg[02] : -DSAPINFO=FPP_00_server

arg[03] : pf=F:\usr\sap\FPP\SYS\profile\FPP_DVEBMGS00_qsiscso-fd4

arg[04] : -DSAPSTART=1

arg[05] : -DCONNECT_PORT=64999

arg[06] : -DSAPSYSTEM=00

arg[07] : -DSAPSYSTEMNAME=FPP

arg[08] : -DSAPMYNAME=qsiscso-fd4_FPP_00

arg[09] : -DSAPPROFILE=F:\usr\sap\FPP\SYS\profile\FPP_DVEBMGS00_qsiscso-fd4

arg[10] : -DFRFC_FALLBACK=ON

arg[11] : -DFRFC_FALLBACK_HOST=localhost

[Thr 7136] Fri Nov 06 08:46:50 2009

[Thr 7136] *** WARNING => INFO: Unknown property [instance.box.number=FPPDVEBMGS00qsiscso-fd4] [jstartxx.c 841]

[Thr 7136] *** WARNING => INFO: Unknown property [instance.en.host=qsiscso-fd4] [jstartxx.c 841]

[Thr 7136] *** WARNING => INFO: Unknown property [instance.en.port=3201] [jstartxx.c 841]

[Thr 7136] *** WARNING => INFO: Unknown property [instance.system.id=0] [jstartxx.c 841]

**********************************************************************

JStartupReadInstanceProperties: read instance properties [F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties]

-> ms host : qsiscso-fd4

-> ms port : 3901

-> OS libs : F:\usr\sap\FPP\DVEBMGS00\j2ee\os_libs

-> Admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

**********************************************************************

Used property files

-> files [00] : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties

**********************************************************************

**********************************************************************

Instance properties

-> ms host : qsiscso-fd4

-> ms port : 3901

-> os libs : F:\usr\sap\FPP\DVEBMGS00\j2ee\os_libs

-> admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

**********************************************************************

Bootstrap nodes

-> [00] bootstrap : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties

-> [01] bootstrap_ID2866400 : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties

-> [02] bootstrap_ID2866450 : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties

**********************************************************************

**********************************************************************

Worker nodes

-> [00] ID2866400 : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties

-> [01] ID2866450 : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\instance.properties

**********************************************************************

[Thr 7136] JLaunchRequestQueueInit: create named pipe for ipc

[Thr 7136] JLaunchRequestQueueInit: create pipe listener thread

[Thr 3708] JLaunchRequestFunc: Thread 3708 started as listener thread for np messages.

[Thr 6640] WaitSyncSemThread: Thread 6640 started as semaphore monitor thread.

[Thr 7136] NiInit3: NI already initialized; param 'maxHandles' ignored (1;202)

[Thr 7136] CPIC (version=700.2009.02.11)

[Thr 7136] [Node: server0] java home is set by profile parameter

Java Home: C:\j2sdk1.4.2_17-x64

[Thr 7136] JStartupICheckFrameworkPackage: can't find framework package F:\usr\sap\FPP\DVEBMGS00\exe\jvmx.jar

**********************************************************************

JStartupIReadSection: read node properties [ID2866450]

-> node name : server0

-> node type : server

-> node execute : yes

-> jlaunch parameters :

-> java path : C:\j2sdk1.4.2_17-x64

-> java parameters : -Djco.jarm=1 -XX:MaxPermSize=256M -XX:PermSize=256M -XX:NewSize=171M -XX:MaxNewSize=171M -XX:DisableExplicitGC -verbose:gc -Xloggc:GC.log -XX:PrintGCDetails -XX:+PrintGCTimeStamps -Djava.awt.headless=true -Dsun.io.useCanonCaches=false -XX:SoftRefLRUPolicyMSPerMB=1 -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=90 -Djava.security.policy=./java.policy -Djava.security.egd=file:/dev/urandom -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy

-> java vm version : 1.4.2_17-b06

-> java vm vendor : Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)

-> java vm type : server

-> java vm cpu : amd64

-> heap size : 1024M

-> init heap size : 1024M

-> root path : F:\usr\sap\FPP\DVEBMGS00\j2ee\cluster\server0

-> class path : .\bin\boot\boot.jar;.\bin\boot\jaas.jar;.\bin\system\bytecode.jar;.

-> OS libs path : F:\usr\sap\FPP\DVEBMGS00\j2ee\os_libs

-> main class : com.sap.engine.boot.Start

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : F:\usr\sap\FPP\DVEBMGS00\exe\jstartup.jar;F:\usr\sap\FPP\DVEBMGS00\exe\jvmx.jar

-> shutdown class : com.sap.engine.boot.Start

-> parameters :

-> debuggable : no

-> debug mode : no

-> debug port : 50021

-> shutdown timeout : 120000

**********************************************************************

[Thr 7136] JLaunchISetDebugMode: set debug mode [no]

[Thr 6172] JLaunchIStartFunc: Thread 6172 started as Java VM thread.

[Thr 6172] [JHVM_PrepareVMOptions] use java parameters set by profile parameter

Java Parameters: -Xss2m

Could somebody help me please

Best regards

[Former Member]

Might want to review this Forum entry :[OutOfMemoryError! Node will exit with exit code 666 |;

It sounds like you are running out of memory at some point.

Hope that is helpful.

J. Haynes

[Former Member]

Hello Hernan,

As Joe said exit code 666 means out of memory error.

In your case the reason seems to be low memory configuration for a NetWeaver 7.0 system.

According to note 723909 you should have 2048MB of heap:

For 64 bit platforms (that means not only your OS but also your JDK supports 64 bit) we recommend using 2GB: -Xmx2048m -Xms2048m

It would be best if you apply all parameters recommended in note 723909.

If you still have problems with OutOfMemory with the new parameters refer to note 1004255 for how to create heapdump in order to analyze what is filling up the memory.

Regards,

Ventsi Tsachev

Technology Development Support (J2EE Engine)

SAP Labs, Palo Alto, Ca (USA)

[Former Member]

Apply whole SAP note 723909. it will definately work as it worked for me.

[Former Member]

in dev_w0 i have the following:

N =================================================

N === SSF INITIALIZATION:

N ===...SSF Security Toolkit name SAPSECULIB .

N ===...SSF trace level is 0 .

N ===...SSF library is F:\usr\sap\FPP\DVEBMGS00\exe\sapsecu.dll .

N ===...SSF hash algorithm is SHA1 .

N ===...SSF symmetric encryption algorithm is DES-CBC .

N ===...sucessfully completed.

N =================================================

N MskiInitLogonTicketCacheHandle: Logon Ticket cache pointer retrieved from shared memory.

N MskiInitLogonTicketCacheHandle: Workprocess runs with Logon Ticket cache.

M JrfcVmcRegisterNativesDriver o.k.

W =================================================

W === ipl_Init() called

W ITS Plugin: Path dw_gui

W ITS Plugin: Description ITS Plugin - ITS rendering DLL

W ITS Plugin: sizeof(SAP_UC) 2

W ITS Plugin: Release: 700, [7000.0.229.20050900]

W ITS Plugin: Int.version, [33]

W ITS Plugin: Feature set: [21]

W ===... Calling itsp_Init in external dll ===>

W === ipl_Init() returns 0, ITSPE_OK: OK

W =================================================

E Enqueue Info: rdisp/wp_no_enq=1, rdisp/enqname=<empty>, assume qsiscso-fd4_FPP_00

E Profile-Parameter: enque/deque_wait_answer = FALSE

E Enqueue Info: enque/disable_replication = 2

E Enqueue Info: replication disabled

E EnqCcInitialize: local lock table initialization o.k.

E EnqId_SuppressIpc: local EnqId initialization o.k.

E EnqCcInitialize: local enqueue client init o.k.

M MBUF info for hooks: MS component UP

M ThSetEnqName: set enqname by server list

M ThISetEnqname: enq name = >qsiscso-fd4_FPP_00 <

E

E *************** EnqId_EN_ActionAtMsUpHook ***************

E Hook on upcoming Ms (with EnqSrv), get auth EnqId and check it locally

E

E *************** ObjShMem_CheckAuthoritativeEnqId ***************

E Checking authoritative EnqId from EnqSrv into ObjShMem

E ObjShMem_CheckAuthoritativeEnqId: ObjShMem ...

E EnqId.EnqTabCreaTime = -999

E EnqId.RandomNumber = -999

E ReqOrd.TimeInSecs = -999

E ReqOrd.ReqNumberThisSec = -999

E ObjShMem_CheckAuthoritativeEnqId: ObjShMem ...

E EnqId.EnqTabCreaTime = -999

E EnqId.RandomNumber = -999

E ReqOrd.TimeInSecs = -999

E ReqOrd.ReqNumberThisSec = -999

E ObjShMem_CheckAuthoritativeEnqId: EnqId is initial in ShMem

E ObjShMem_CheckAuthoritativeEnqId: Overwrite incoming auth EnqId, continue

E EnqId inscribed into initial ObjShMem: (ObjShMem_CheckAuthoritativeEnqId)

E -SHMEM--

---

E EnqId: EnqTabCreaTime/RandomNumber = 06.11.2009 02:23:41 1257485021 / 4120

E ReqOrd at Srv: TimeInSecs/ReqNumberThisSec = 06.11.2009 02:23:41 1257485021 / 2

E ReqOrd at Cli: TimeInSecs/ReqNumberThisSec = 06.11.2009 02:23:41 1257485021 / 1

E Status: STATUS_OK

E -

---

M ThActivateServer: state = STARTING

L BtcSysStartRaise: Begin

L Raise event SAP_SYSTEM_START with parameter <qsiscso-fd4_FPP_00 >

L BtcSysStartRaise: End

M SosICreateNewAnchorArray: sos_search_anchor_semantics = 1

A

A Fri Nov 06 02:23:47 2009

A **GENER Trace switched off ***

M

M Fri Nov 06 02:24:36 2009

M rdisp/rb_cleaned_rfc = 0

M Deactivate ASTAT hyper index locking

S

S Fri Nov 06 02:24:56 2009

S found spool memory service RSPO-ACTIONS at 000000002BC0A230

A RFC FRFC> fallback on the central gateway qsiscso-fd4 sapgw00 activ

M

M Fri Nov 06 08:46:48 2009

M ***LOG R49=> ThReceive, CPIC-Error (020223) [thxxhead.c 7559]

M ***LOG R5A=> ThReceive, CPIC-Error (11986091) [thxxhead.c 7564]

M ***LOG R64=> ThReceive, CPIC-Error ( CMSEND(SAP)) [thxxhead.c 7569]

C

C Fri Nov 06 09:53:57 2009

C CbApplInfoGet() failed (ignored 100).