SAP NetWeaver 7.30 on RHEL 6 (6.u2 in fact) - kernel issues

[Former Member]

Hello SAP community,

I'm hoping that I can benefit from the wealth of experience in this community and maybe make this a useful thread for others too.

I'm work to get a SAP NetWeaver 7.30 AS Java system working after an OS upgrade from Red Hat Enterprise Linux (RHEL) 5.u7 to 6.u2.

So system specs:

OS: RHEL (x86_64) 6.u2 (previously 5.u7)

DB: ORACLE 11g (11.2.0.3)

SAP: SAP NetWeaver 7.30 SP7 (no EHP) - AS Java (Unicode)

The system previously ran without issue on RHEL 5.u7 with kernel 720_REL (for Linux x86_64 Unicode).

Following the upgrade we planned to change to 720_EXT_REL as this is compatible with RHEL 6 according to many SAP notes - also we have a NW 7.31 (note 7.31, not 7.30) which runs this kernel without issue. However at startup the Java server node (a single one) fails to startup. They key error messages in the <instance>/work directory being:

[std_server0.out]

Loading [LockingManager]...

FATAL: Server process failed to start; manager [LockingManager] could not be started

[dev_server0]

F [Thr 140163195279104] Tue Sep 11 07:16:00 2012

F  [Thr 140163195279104] *** LOG => SfCJavaVm: exit hook is called. (rc = -1501)

F

F  ********************************************************************************

F  *** ERROR => Java node 'server0' terminated with exit code -1501.

F  ***

F  *** Please see section 'Java program exit codes'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

[dev_jstart]

M [Thr 140065878832992] Tue Sep 11 07:30:27 2012

M  [Thr 140065878832992] ***LOG Q0I=> NiPConnect2: 143.22.104.71:3201: connect (111: Connection refused) [nixxi.cpp 3286]

M  [Thr 140065878832992] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 993/sock 130

    (SI_ECONN_REFUSE/111; I4; ST; 143.22.104.71:3201) [nixxi.cpp    3286]

E  [Thr 140065878832992] *** ERROR => EnqTCDisconnect: EnqTCIGetConn failed; invalid TChdl 993 [enxxtc.c     346]

I re-downloaded this kernel from OSS to check the right version and yet it seemed fine - yet there is obviously some issue.

Here is the kernel version we're now trying confirmed:

--------------------

disp+work information

--------------------

kernel release                720

kernel make variant           720_EXT_REL

DBMS client library           OCI_112,

DBSL shared library version   720.00

compiled on                   Linux GNU SLES-11 x86_64 cc4.3.4 use-pr120614 for linuxx86_64

compiled for                  64 BIT

compilation mode              UNICODE

compile time                  Jul  8 2012 19:33:31

update level                  0

patch number                  300

source id                     0.300

---------------------

supported environment

---------------------

database (SAP, table SVERS)   700

                              710

                              701

                              702

                              703

                              711

                              720

                              730

                              731

                              732

DBMS server

   ORACLE 10.1.0.*.*

   ORACLE 10.2.0.*.*

   ORACLE 11.2.*.*.*

operating system

Linux 2.6

Linux 3

Any help or experience from anyone running 7.30 on RHEL 6 would be greatly appreciated!

[former_member189725 · Accepted solution]

Please check if the SCS instance is up and running or not . It seems the JAVA CI is not able to connect to SCS enque process  host 143.22.104.71 on 3201 port.

I suggest you stop the SCS instance , clean up the memory segments for IPC (cleanipc ) and then start the SCS instance .

Then check using ensmon if the connection to the enque process is fine

# ensmon pf=<SCS_instance_profile>

Regards

Ratnajit

[Former Member · Accepted solution]

3201 sounds like a dispatcher port to me !! O

143.22.104.71 - what is this IP pointing to ?? Is it hosting an ABAP system with CI/PAS having instance number 01 ??

Is it UME backend of ABAP system which is down ?

Or is enqueue service port specified as 3201 ???

Regards.

Ruchit.

[Former Member · Accepted solution]

I don't think so - only change is the OS upgrade from 5.u7 to 6.u2 - unless that locked down ports somehow... I will check /etc/services is still AOK

[Former Member · Accepted solution]

Although it is perhaps that the OS team may not have prepetuated entries in /etc/services....

[former_member189725 · Accepted solution]

Please restart the SCS instance , its just that the enqueue process is not reachable. Everything looks ok in the system and is normal. The port 3201 should be the one to which the standalone java enqueue process should be listening to.

Between the SCS instance stop and start , execute

# cleanipc <SCS_instance_no> remove

Regards

Ratnajit

[Former Member · Accepted solution]

What does /etc/services have to do with it ? You got check it in SCS profile.

In any case does 143.22.104.71 and 3201 combination make any sense to you ?

Regards.

Ruchit

[Former Member · Accepted solution]

You have mentioned XXX_SCS01_server.

Please do an nslookup on <server> or even a simple ping to get the IP ? Is it same as

143.22.104.71?

Regards.

Ruchit.

[Former Member · Accepted solution]

Okay looks like enque/encni/port might be in play here.

Regards.

Ruchit.

[Former Member]

please paste following log file in enqueue work directory:

dev_enqsrv

dev_enqlisten

check the enqueue is running or not.

if it is possible ( in case the SCS & Apps installed separately), can you turn off iptables/firewall ( if in user) for a moment (#service iptables stop)?

[Asha_Pillai]

Hi Jamie,

It looks like there might be some issues with locks on the Enqueue Server.Please check SCS work

directory at /usr/sap/<SID>/SCSxx/work for clues.

You can also try to restart Java cluster and see if it helps.

Regards,

Asha

[Former Member]

hmm yes the sapstart.log file in SCS has these types of errors recurring:

(30539) Waiting for Child Processes to terminate.

(30539) **** 2012/09/12 04:05:51 Child 30557 terminated with Status 8 . ****

(30539) **** 2012/09/12 04:05:51 Execute Restart Command local en.sapXXX_SCS01 pf=/usr/sap/XXX/SYS/profile/XXX_SCS01_server ****

(30566) Starting: local en.sapXXX_SCS01 pf=/usr/sap/XXX/SYS/profile/XXX_SCS01_server

[JPReyes]

What oracle client are you using?

Regards

Juan

[Former Member]

server:xxxadm 55> ldd dboraslib.so

        linux-vdso.so.1 =>  (0x00007fffe81ff000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5076a98000)

        librt.so.1 => /lib64/librt.so.1 (0x00007f507688f000)

        libclntsh.so.11.1 => /oracle/client/11x_64/instantclient/libclntsh.so.11.1 (0x00007f5073ed9000)

        libc.so.6 => /lib64/libc.so.6 (0x00007f5073b38000)

        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f5073831000)

        libm.so.6 => /lib64/libm.so.6 (0x00007f50735ad000)

        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5073397000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f507317a000)

        /lib64/ld-linux-x86-64.so.2 (0x0000003659000000)

        libnnz11.so => /oracle/client/11x_64/instantclient/libnnz11.so (0x00007f5072dae000)

        libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f5072b95000)

        libaio.so.1 => /lib64/libaio.so.1 (0x00007f5072993000)