J2EE engine failing to start on BI Production Server

[Former Member]

Hi guys

I am trying to start up BI Production after what was meant to be a routine restart to take on a new abap/buffersize setting. All BI App servers have come up except for one left (Cluster Server 1).

That cluster server 1 has multiple SAP instances on it but these app server parts (not enqueue or dispatcher) are all shut down to ensure there should be no port problems

I have ensured that the cluster resources are pointing at the correct servers to ensure that DNS is not an issue

below is the tail end of the trace file dev_server0

[Thr 12000] *  ERROR       connection to partner '192.168.1.196:sapgw44' broken
[Thr 12000] *
*  TIME        Wed May 09 17:06:27 2012
[Thr 12000] *  RELEASE     700
[Thr 12000] *  COMPONENT   NI (network interface)
[Thr 12000] *  VERSION     38
[Thr 12000] *  RC          -6
[Thr 12000] *  MODULE      nixxi.cpp
[Thr 12000] *  LINE        3958
[Thr 12000] *  DETAIL      NiIWrite
[Thr 12000] *  SYSTEM CALL WSASend
[Thr 12000] *  ERRNO       10054
[Thr 12000] *  ERRNO TEXT  WSAECONNRESET: Connection reset by peer
[Thr 12000] *  COUNTER     1
[Thr 12000] *
[Thr 12000] *****************************************************************************
[Thr 12000] *** ERROR => STIAsSendToGw: STINiWrite failed (NIECONN_BROKEN) [r3cpic.c     5384]
[Thr 12000] ***LOG Q0I=> NiIRead: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 4434]
[Thr 12000] *** ERROR => NiIRead: SiRecv failed for hdl 28 / sock 3874568
    (SI_ECONN_BROKEN/10054; I4; ST; 192.168.1.196:3344) [nixxi.cpp    4434]
[Thr 11780] JLaunchISetState: change state from [Stopping (5)] to [Stopped (6)]
[Thr 10680] Wed May 09 17:06:29 2012
[Thr 10680] JLaunchIExitJava: exit hook is called (rc = 0)
[Thr 10680] JLaunchCloseProgram: good bye (exitcode = 0)

And for the developers trace file from the sap mmc session

Wed May 09 19:54:33 2012
J2EE server info
  start = TRUE
  state = STARTED
  pid = 6980
  argv[0] = D:\usr\sap\BIP\D44\exe\jcontrol.EXE
  argv[1] = D:\usr\sap\BIP\D44\exe\jcontrol.EXE
  argv[2] = pf=\\sapprdappbi\sapmnt\BIP\SYS\profile\BIP_D44_sapprdappcl1
  argv[3] = -DSAPSTART=1
  argv[4] = -DCONNECT_PORT=64994
  argv[5] = -DSAPSYSTEM=44
  argv[6] = -DSAPSYSTEMNAME=BIP
  argv[7] = -DSAPMYNAME=sapprdappcl1_BIP_44
  argv[8] = -DSAPPROFILE=\\sapprdappbi\sapmnt\BIP\SYS\profile\BIP_D44_sapprdappcl1
  argv[9] = -DFRFC_FALLBACK=ON
  argv[10] = -DFRFC_FALLBACK_HOST=localhost
  start_lazy = 0
  start_control = SAP J2EE startup framework
DpJ2eeStart: j2ee state = STARTED
DpJ2eeLogin: j2ee state = CONNECTED
***LOG Q0I=> NiIRead: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 4434]
*** ERROR => NiIRead: SiRecv failed for hdl 7 / sock 128
    (SI_ECONN_BROKEN/10054; I4; ST; 127.0.0.1:6933) [nixxi.cpp    4434]
*** ERROR => DpJ2eeMsgProcess: NiRead failed (NIECONN_BROKEN) [dpxxj2ee.c   1212]
DpJ2eeMsgProcess: j2ee state = CONNECTED (NIECONN_BROKEN)
Wed May 09 19:54:53 2012
DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN
Wed May 09 19:55:13 2012
DpEnvCheckJ2ee: switch off j2ee start flag

Any assistance would be great thanks

[Former Member · Accepted solution]

Ok, I don't want to jump the gun as Java is not yet fully started but It has progressed to starting so that is looking very hopeful.

Based on the trace information above, I then went into the instance parameters file located in:

\\sapprdappcl2\d$\usr\sap\BIP\SYS\profile

I then edited the instance profile file (BIP_D44_sapprdappcl1 ) for cluster 1 and found that it was indeed pointing to the 15 directory (the wrong one) so I backed up the file and changed it to point at the 19 directory and what do you know Bobs your uncle (or aunty depending on how he rolls) and it started up Java.

What a mish - 5 hours of pain for something that got weirdly changed that should never have changed.

[Former Member · Accepted solution]

Yup all sorted as per above!!!!!!!!! Thanks to everyone for being so helpful

[Former Member · Accepted solution]

Great!! but the JDK changing mysteriously on prod system seems alarming

[Former Member · Accepted solution]

Thats the next mystery to unravel in this convoluted puzzle

[Former Member]

I have run an analysis on the trace information found within the SAP startup in MMC

Below is the pic containing the results of this:

- note the cannot find java compiler error (that cant be good)

[Former Member]

Hi,

Strange! Are the java binary files existing in the referred path? I would assume yes.

Was the JDK version changed recently?

Regards,

Srikishan

[Former Member]

Very strange, and even weirder is that that directory does not exist on the server

C:\j2sdk1.4.2_15-x64\bin\javac.exe is the wrong directory it should be C:\j2sdk1.4.2_19-x64\bin>java.exe

- note the 19 instead of the 15

[Former Member]

Hi,

Then perhaps the wrong path for old JDK is still existing in maybe one of the profile files or in the Config Tool.

Regards,

Srikishan

[Former Member]

Thanks mate, I just got it sorted before you posted

[Former Member]

Hi,

From the log:

ERROR       connection to partner '192.168.1.196:sapgw44' broken

[Thr 12000]

Is this partner/service up? I guess this should be the BI ABAP service being checked for.

Regards,

Srikishan

[Former Member]

Hi, looks like it is as I can see all SAP Applications up on that server in the mgmt console

[nicholas_chang]

Hi,

check the logs on /j2ee/cluster/server0/log.

Also, what's your sequence of bringing up BI abap and java? If your UME is on BI ABAP, please start the ABAP first, wait until abap startup sucessfully, start the java engine.

Let us know how it goes.

Thanks.

Nicholas Chang

[Former Member]

Hi Nicholas

The J2EE logs don't contain any data since 5pm today (its now 9pm) as the J2EE service has not been running (so not logging on the server )

I don't as far as I can see get an option to start ABAP first - I've attached a screenshot of the mgmt console section that shows the disp+work section.

It has previously always come up without a hitch.

[Former Member]

sorry heres the pic

[Former Member]

Hi Mapp!

abap/bufferssize can only be set on an ABAP instance. The log file you attached is from a Java instance.

You may have to increase shared memory buffers

Please run "sappfpar pf=<full path to your instance profile file> check" as <sid>adm.

BR,

Samuli

[Former Member]

Hi Samuli

Yes that is correct, I set the abap/buffersize on the ABAP instance - I can log into the ABAP Client not a problem - its just dodgy old java that is causing the issue.