Portal server suddenly went down - logs to be anal...

Former Member · ‎12-02-2009

hi, Today our portal servers EP7 suddenly went down around 9:30 am. We did portal restart which brought the system up. we are doing RCA to find the issue. Kindly let me know the log files to be analysed.

Thanks.

few logs collected :

Log from dev_dispatcher.b00

from

Dec 2 04:50:09 2009

[Unloading class

sun.reflect.GeneratedSerializationConstructorAccessor102]

[Thr 2]

Wed Dec 2 09:31:21 2009

[Thr 2] JLaunchRequestFunc: receive command:17, argument:0 from

pid:12001

[Thr 2] JLaunchIShutdownInvoke: set shutdown interval

(stop:1259746281/end:1259746401/TO:120)

[Thr 2] JLaunchProcessCommand: Invoke VM Shutdown

[Thr 2] JHVM_FrameworkShutdownDirect: invoke direct shutdown

[Thr 196] JLaunchISetState:

change state from [Running (3)] to [Waiting for stop (4)]

[Thr 196] JLaunchISetState: change

state from [Waiting for stop (4)] to [Stopping (5)]

[Thr 82] JLaunchISetHttpPort: set http

port 53200

[Thr 82] JLaunchISetHttpsPort: set https port 53201

[Thr 38] JLaunchISetP4Port:

set p4 port 53204

[Thr 175] JLaunchISetP4Port: set p4 port 53204

[Thr 194]

JLaunchISetHttpPort: set http port 53200

[Thr 194] JLaunchISetHttpsPort: set https port

53201

[Thr 176] JLaunchISetP4Port: set p4 port 53204

[Thr 106] Wed Dec 2 09:31:22 2009

[Thr 106] JLaunchISetHttpsPort: set https port 53201

[Thr 106] JLaunchISetHttpPort: set http

port 53200

[Thr 190] Wed Dec 2 09:31:28 2009

[Thr 190] JLaunchISetTelnetPort: set telnet

port 53208

[Thr 187] Wed Dec 2 09:31:29 2009

[Thr 187] JLaunchISetHttpPort: set http port

53200

[Thr 196] Wed Dec 2 09:31:32 2009

[Thr 196] JLaunchISetState: change state from

[Stopping (5)] to [Stopped (6)]

[Thr 5] JLaunchIExitJava: exit hook is called (rc = 0)

[Thr

5] SigISetIgnoreAction : SIG_IGN for signal 18

[Thr 5] JLaunchCloseProgram: good bye

(exitcode = 0)

*************

jvm_sdm.o00

Starting SDM - Software Deployment Manager...

tc/SL/SDM/SDM/sap.com/SAP

AG/7.0019.20090401124145.0000

JStartupFramework is requesting to start the SDM Server.

Start

the SDM Server integrated in the JStartupFramework.

Initializing Network Manager (53217)

Starting SDM Server listening on port 53218

SDM started successfully.

JControl is shutting

SDM Server down

Asynchroneously processing shutdown request.

Server finished sending back

the answer of the ShutdownRequest.

Shutting dispatcher down

Bye, bye!

**************

dev_sdm.b00

Wed Dec 2 09:31:21 2009

[Thr 2] JLaunchRequestFunc: receive command:17, argument:0 from pid:12001

[Thr 2] JLaunchIShutdownInvoke: set shutdown interval (stop:1259746281/end:1259746401/TO:120)

[Thr 2] JLaunchProcessCommand: Invoke VM Shutdown

[Thr 2] JHVM_FrameworkShutdownDirect: invoke direct shutdown

[Thr 2] JLaunchISetState: change state from [Running (3)] to [Stopping (5)]

[Thr 5] JLaunchIExitJava: exit hook is called (rc = 0)

[Thr 5] SigISetIgnoreAction : SIG_IGN for signal 18

[Thr 5] JLaunchCloseProgram: good bye (exitcode = 0)

Former Member · ‎07-23-2010

Hi

we had same problem ....

unfort. the note mentioned for problem solving seems to be wrong .

is it miss_spelled ?

thx

Former Member · ‎12-08-2009

Hi,

We got the issue resolved with the help of sap. Temp solution is - restarting HTTP service in VA.

Permanenet solution - applying patch as per note - 1347861

Thanks

Former Member · ‎12-03-2009

Hi Prabhu,

It looks to me that your SDM doing all the mess here...Please ensure SDM started in 'integrated' mode

jvm_sdm.o00 Starting SDM - Software Deployment Manager... tc/SL/SDM/SDM/sap.com/SAP AG/7.0019.20090401124145.0000 JStartupFramework is requesting to start the SDM Server. Start the SDM Server integrated in the JStartupFramework. Initializing Network Manager (53217) Starting SDM Server listening on port 53218 SDM started successfully. JControl is shutting SDM Server down Asynchroneously processing shutdown request. Server finished sending back the answer of the ShutdownRequest. Shutting dispatcher down Bye, bye! ************** dev_sdm.b00 Wed Dec 2 09:31:21 2009 [Thr 2] JLaunchRequestFunc: receive command:17, argument:0 from pid:12001 [Thr 2] JLaunchIShutdownInvoke: set shutdown interval

Generally if SDM is detaching from the JStartupFramework, you must set SDM in 'integrated' mode.

In order to start it via JStartupFramework you should set it to "integrated" mode. Open command prompt in directory /usr/sap///SDM/program and execute:

sdm jstartup mode=integrated

I think, while applying the patch the SDM switches to standalone mode.

Please refer to below URL if you want more info on SDM startup/Shutdown

http://help.sap.com/saphelp_erp2004/helpdata/en/41/a30f31e2cf45d09ad5aac66b71f896/frameset.htm

Secondly, Instead of useing the command you can also edit the config file on your server.

\usr\J2E\JC00\SDM\program\config\sdmrepository.sdc

<sdm-opertaion-mode>

integrated

</sdm-operation-mode>

Thanks

Sekhar

Former Member · ‎12-02-2009

dev_server0.boo --->

[Thr 2] Wed Dec 2 09:31:21 2009

[Thr 2] JLaunchRequestFunc: receive command:17,

argument:0 from pid:12001

[Thr 2] JLaunchIShutdownInvoke: set shutdown interval

(stop:1259746281/end:1259746401/TO:120)

[Thr 2] JLaunchProcessCommand: Invoke VM Shutdown

[Thr 2] JHVM_FrameworkShutdownDirect: invoke direct shutdown

[Thr 7276] JLaunchISetState:

change state from [Running (3)] to [Waiting for stop (4)]

[Thr 7276] JLaunchISetState:

change state from [Waiting for stop (4)] to [Stopping (5)]

Wed Dec 2 09:31:27 2009

394979.347: [GC 394979.349: [ParNew: 105402K->17693K(131328K), 0.1623975 secs]

743527K->656016K(1293064K), 0.1628020 secs]

Wed Dec 2 09:31:33 2009

394985.879: [GC

394985.879: [ParNew: 105245K->14666K(131328K), 0.1096688 secs] 743568K->653157K(1293064K),

0.1099241 secs]

Wed Dec 2 09:31:37 2009

394989.881: [GC 394989.882: [ParNew:

102218K->14251K(131328K), 0.1009569 secs] 740709K->652911K(1293064K), 0.1011890 secs]

Wed

Dec 2 09:31:40 2009

394992.799: [GC 394992.799: [ParNew: 101803K->14119K(131328K),

0.0979001 secs] 740463K->652914K(1293064K), 0.0981073 secs]

Wed Dec 2 09:31:43 2009

394995.769: [GC 394995.769: [ParNew: 101671K->13972K(131328K), 0.0978219 secs]

740466K->652991K(1293064K), 0.0980658 secs]

Wed Dec 2 09:31:47 2009

394999.838: [GC

394999.838: [ParNew: 101524K->13323K(131328K), 0.0988114 secs] 740543K->652948K(1293064K),

0.0990286 secs]

Wed Dec 2 09:31:50 2009

395002.649: [GC 395002.649: [ParNew:

100875K->12882K(131328K), 0.0938378 secs] 740500K->652980K(1293064K), 0.0940466 secs]

Wed

Dec 2 09:31:53 2009

395005.452: [GC 395005.452: [ParNew: 100434K->12623K(131328K),

0.0928746 secs] 740532K->653009K(1293064K), 0.0930924 secs]

Wed Dec 2 09:31:56 2009

395008.826: [GC 395008.826: [ParNew: 100175K->12123K(131328K), 0.0910206 secs]

740561K->652891K(1293064K), 0.0912409 secs]

Wed Dec 2 09:32:00 2009

395012.321: [GC

395012.321: [ParNew: 99675K->11303K(131328K), 0.0967564 secs] 740443K->652985K(1293064K),

0.0969776 secs]

Wed Dec 2 09:32:03 2009

395015.135: [GC 395015.136: [ParNew:

98855K->10834K(131328K), 0.0901516 secs] 740537K->652939K(1293064K), 0.0903539 secs]

Wed

Dec 2 09:32:06 2009

395018.817: [GC 395018.818: [ParNew: 98386K->10663K(131328K), 0.0866679

secs] 740491K->652961K(1293064K), 0.0868988 secs]

Wed Dec 2 09:32:10 2009

395022.172: [GC

395022.172: [ParNew: 98215K->10362K(131328K), 0.0876151 secs] 740513K->652938K(1293064K),

0.0878429 secs]

Wed Dec 2 09:32:13 2009

395025.190: [GC 395025.190: [ParNew:

97914K->10233K(131328K), 0.0858441 secs] 740490K->652968K(1293064K), 0.0860755 secs]

Wed

Dec 2 09:32:16 2009

395028.887: [GC 395028.887: [ParNew: 97737K->9950K(131328K), 0.0873147

secs] 740473K->652922K(1293064K), 0.0875171 secs]

Wed Dec 2 09:32:19 2009

395031.795: [GC

395031.796: [ParNew: 97502K->9610K(131328K), 0.0827926 secs] 740474K->652935K(1293064K),

0.0830060 secs]

Former Member · ‎12-02-2009

Hi, also check some more logs

dev_dispatcher.b00

[Thr 2] Wed Dec 2 09:31:21 2009

[Thr 2] JLaunchRequestFunc: receive command:17,

argument:0 from pid:12001

[Thr 2] JLaunchIShutdownInvoke: set shutdown interval

(stop:1259746281/end:1259746401/TO:120)

[Thr 2] JLaunchProcessCommand: Invoke VM Shutdown

[Thr 2] JHVM_FrameworkShutdownDirect: invoke direct shutdown

[Thr 196] JLaunchISetState:

change state from [Running (3)] to [Waiting for stop (4)]

[Thr 196] JLaunchISetState: change

state from [Waiting for stop (4)] to [Stopping (5)]

[Thr 82] JLaunchISetHttpPort: set http

port 53200

[Thr 82] JLaunchISetHttpsPort: set https port 53201

[Thr 38] JLaunchISetP4Port:

set p4 port 53204

[Thr 175] JLaunchISetP4Port: set p4 port 53204

[Thr 194]

JLaunchISetHttpPort: set http port 53200

[Thr 194] JLaunchISetHttpsPort: set https port

53201

[Thr 176] JLaunchISetP4Port: set p4 port 53204

[Thr 106] Wed Dec 2 09:31:22 2009

[Thr 106] JLaunchISetHttpsPort: set https port 53201

[Thr 106] JLaunchISetHttpPort: set http

port 53200

[Thr 190] Wed Dec 2 09:31:28 2009

[Thr 190] JLaunchISetTelnetPort: set telnet

port 53208

[Thr 187] Wed Dec 2 09:31:29 2009

[Thr 187] JLaunchISetHttpPort: set http port

53200

[Thr 196] Wed Dec 2 09:31:32 2009

[Thr 196] JLaunchISetState: change state from

[Stopping (5)] to [Stopped (6)]

[Thr 5] JLaunchIExitJava: exit hook is called (rc = 0)

[Thr

5] SigISetIgnoreAction : SIG_IGN for signal 18

[Thr 5] JLaunchCloseProgram: good bye

(exitcode = 0)

**************

std_dispatcher.o00

at java.lang.Thread.run(Thread.java:534)

"Signal Dispatcher" daemon prio=10

tid=0x0000000101555190 nid=0x9 waiting on condition

[0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=8 tid=0x000000010154f890

nid=0x7 in Object.wait() [0xffffffff654ff000..0xffffffff654ff8b0]

at

java.lang.Object.wait(Native Method)

at

java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)

- locked

<0xffffffff69356840> (a java.lang.ref.ReferenceQueue$Lock)

at

java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)

at

java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon

prio=10 tid=0x000000010154dfa0 nid=0x6 in Object.wait()

[0xffffffff656ff000..0xffffffff656ff8b0]

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:429)

at

java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)

- locked

<0xffffffff69355ad0> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x000000010146bc90

nid=0x4 in Object.wait() [0xffffffff7bcff000..0xffffffff7bcffab0]

at

java.lang.Object.wait(Native Method)

- waiting on <0xffffffff693568e8> (a

com.sap.engine.boot.StartFrame)

at java.lang.Object.wait(Object.java:429)

at

com.sap.engine.boot.StartFrame.work(StartFrame.java:121)

- locked

<0xffffffff693568e8> (a com.sap.engine.boot.StartFrame)

at

com.sap.engine.boot.Start.main(Start.java:34)

"VM Thread" prio=5 tid=0x000000010154d650

nid=0x5 runnable

"VM Periodic Task Thread" prio=10 tid=0x000000010155c5e0 nid=0xd waiting

on condition

"Suspend Checker Thread" prio=10 tid=0x00000001015547c0 nid=0x8 runnable

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor102]

SAP J2EE Engine

Version 7.00 PatchLevel 56839.450 is shutting down! PatchLevel 56839.450 April 25, 2009

07:00 GMT

Stopping services.

Service classload stopped. (152 ms)

Service

tc.monitoring.logviewer stopped. (125 ms)

Service p4 stopped. (38 ms)

Service ssl

stopped. (1 ms)

Service keystore stopped. (9 ms)

Service iiop stopped. (3025 ms)

Service adminadapter stopped. (5 ms)

Service monitor stopped. (198 ms)

Service security

stopped. (0 ms)

Service jms_provider stopped. (1010 ms)

Service basicadmin stopped. (83

ms)

Service jmx stopped. (1 ms)

Service webservices stopped. (1 ms)

Service sld

stopped. (0 ms)

Service telnet stopped. (1012 ms)

Service http stopped. (1007 ms)

Service timeout stopped. (1 ms)

Service configuration stopped. (0 ms)

Service

log_configurator stopped. (1 ms)

Service memory stopped. (0 ms)

Service licensing

stopped. (0 ms)

Service httptunneling stopped. (0 ms)

Service shell stopped. (53 ms)

Service jmx_notification stopped. (1 ms)

Former Member · ‎12-02-2009

Hi,

Check dev_server0 and std_server0.out files under work directory.

Cheers....,

Raghu

Portal server suddenly went down - logs to be analysed

Accepted Solutions (0)

Answers (6)

Answers (6)

Re: HTTP Receiver adapter with Dynamic value in UR...

How to create a rolling forecast in SAC? I have an...

In SAP Hana Cloud, do packages exist as database o...

Re: Mass Update a Zfield in Std Table with Split &...

Re: Change BW Query Initial Variable Input at runt...