503 Service not available, ICM running but no server with HTTP connected

[Former Member]

Hi,

i am getting some issue in java system after restarting the system.

please find here with attched error and dev_icm log file... please suggest me

error when accessing java

503 Service not available

--------------------------------------------------------------------------------
Error: -6
Version: 7200
Component: ICM Java
Date/Time: Sun Jun 29 21:03:01 2014 
Module: http_j2ee2_mt.c
Line: 885
Server: xxxxxxxxx_00
Error Tag: {-}
Detail: ICM running but no server with HTTP connected

dev_icm log file

---------------------------------------------------
trc file: "dev_icm", trc level: 1, release: "720"
---------------------------------------------------
sysno      00
sid        JSD
systemid   324 (IBM RS/6000 with AIX)
relno      7200
patchlevel 0
patchno    324
intno      20020600
make       multithreaded, Unicode, 64 bit, optimized
profile    /usr/sap/JSD/SYS/profile/JSD_J00_soldbci12
pid        29950142

[Thr 01] Sun Jun 29 19:49:37 2014
[Thr 01] Parameter system/type has value: "J2EE"
[Thr 01] started security log to file ./dev_icm_sec_1
[Thr 01] ICM running on: soldbci12.scj.com
[Thr 01] MtxInit: 30001 0 2
[Thr 01] ***LOG IM1=> IcmInit, Startup (ICM&soldbci12.scj.com&29950142&) [icxxman_mt.c 1931]
[Thr 01] IcmInit: listening to admin port: 65000
[Thr 01] MPI: dynamic quotas disabled.
[Thr 01] MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%
[Thr 01] CCMS: Initialized monitoring segment of size 60000000.
[Thr 01] CCMS: Initialized CCMS Headers in the shared monitoring segment.
[Thr 01] CCMS: Checking Downtime Configuration of Monitoring Segment.
[Thr 01] IcmInit: icm running without ABAP stack
[Thr 515] IcmProxyWatchDog: proxy watchdog started
[Thr 772] IcmJSessCommThread started
[Thr 1029] IcmMplxThread started
[Thr 01] IcmJsfAttach: attached to Java Startup Framework
[Thr 01] WLM Tag 'JSD/ICM' successfully set for this process
[Thr 01] IcmCreateWorkerThreads: created worker thread 0
[Thr 01] IcmCreateWorkerThreads: created worker thread 1
[Thr 01] IcmCreateWorkerThreads: created worker thread 2
[Thr 01] IcmCreateWorkerThreads: created worker thread 3
[Thr 01] IcmCreateWorkerThreads: created worker thread 4
[Thr 01] IcmCreateWorkerThreads: created worker thread 5
[Thr 01] IcmCreateWorkerThreads: created worker thread 6
[Thr 01] IcmCreateWorkerThreads: created worker thread 7
[Thr 01] IcmCreateWorkerThreads: created worker thread 8
[Thr 01] IcmCreateWorkerThreads: created worker thread 9
[Thr 3856] IcmMpiWatchDogThread: MPI watchdog started
[Thr 4113] IcmWatchDogThread: watchdog started
[Thr 4370] *** ERROR => can't open directory '/usr/sap/JSD/J00/data/cache' [ictxxcache_r 4085]
[Thr 4370] *** ERROR => IctCmCreateCache() failed rc=3 [http_cache_m 2157]
[Thr 4370] WARNING: could not create disk cache. running with memory cache only
[Thr 4370] ISC: created 50 MB memory cache.
[Thr 4370] HttpISubHandlerAdd: Added handler HttpCacheHandler(111ab28b0), slot=0, flags=12293) for /, active: 1, table 111ab22f0
[Thr 4370] HttpExtractArchive: files from archive /usr/sap/JSD/J00/exe/icmadmin.SAR in directory /usr/sap/JSD/J00/data/icmandir are up to date
[Thr 4370] HttpISubHandlerAdd: Added handler HttpAdminHandler(115d2d230), slot=1, flags=36869) for /sap/admin, active: 1, table 111ab22f0
[Thr 4370] HttpISubHandlerAdd: Added handler HttpLogHandler(115d2d350), slot=2, flags=12293) for /, active: 1, table 111ab22f0
[Thr 4370] HttpISubHandlerAdd: Added handler HttpModHandler(115d2db50), slot=3, flags=12293) for /, active: 1, table 111ab22f0
[Thr 4370] CsiInit(): Initializing the Content Scan Interface
[Thr 4370]            IBM RS/6000 with AIX (mt,unicode,SAP_CHAR/size_t/void* = 16/64/64)
[Thr 4370] CsiInit(): CSA_LIB = "/usr/sap/JSD/J00/exe/libsapcsa.o"
[Thr 4370] HttpISubHandlerAdd: Added handler HttpAuthHandler(115d2dcb0), slot=4, flags=12293) for /, active: 1, table 111ab22f0
[Thr 4370] ICM running without ABAP
[Thr 4370] HttpISubHandlerAdd: Added handler HttpJ2EE2Handler(115d452d0), slot=5, flags=1060869) for /, active: 1, table 111ab22f0
[Thr 4370] Started service PORT=50000,PROT=HTTP,TIMEOUT=60,PROCTIMEOUT=300
[Thr 4370] Started service PORT=50004,PROT=P4,TIMEOUT=60,PROCTIMEOUT=300
[Thr 4370] =================================================
[Thr 4370] = SSL Initialization    platform tag=(rs6000_5.2_64)
[Thr 4370] =   (720_REL,Jan 15 2013,mt,ascii-uc,SAP_UC/size_t/void* = 16/64/64)
[Thr 1029] JNCMIHttpMsPutLogon: http port information not available
[Thr 1029] JNCMIP4MsPutLogon: p4 port information not available
[Thr 1029] JNCMIIIOPMsPutLogon: iiop port information not available
[Thr 4370] =   found SAPCRYPTOLIB  5.5.5C pl34  (Mar  1 2012) MT,NB
[Thr 4370] =   current UserID: "JSDadm",  env-var USER="JSDadm"
[Thr 4370] =   using SECUDIR=/usr/sap/JSD/J00/sec
[Thr 4370] =  secudessl_Create_SSL_CTX():  PSE "/usr/sap/JSD/J00/sec/SAPSSLC.pse" not found,
[Thr 4370] =      using PSE "/usr/sap/JSD/J00/sec/SAPSSLS.pse" as fallback
[Thr 4370] =  secudessl_Create_SSL_CTX():  PSE "/usr/sap/JSD/J00/sec/SAPSSLA.pse" not found,
[Thr 4370] =      using PSE "/usr/sap/JSD/J00/sec/SAPSSLS.pse" as fallback
[Thr 4370] ******** Warning ********
[Thr 4370] *** No SSL-client PSE "SAPSSLC.pse" available
[Thr 4370] *** -- this will probably limit SSL-client side connectivity
[Thr 4370] ********
[Thr 4370] = Success -- SapCryptoLib SSL ready!
[Thr 4370] =================================================
[Thr 4370]
[Thr 4370] Started service PORT=50006,PROT=P4SEC,TIMEOUT=60,PROCTIMEOUT=60,VCLIENT=1
[Thr 4370] SSL settings: verify_client: 1, cache_size: -1, cache_lifetime: -1, credfile: SAPSSLS.pse, ciphers: default
[Thr 4370] Started service PORT=50007,PROT=IIOP,TIMEOUT=60,PROCTIMEOUT=60
[Thr 4370] Started service PORT=50003,PROT=IIOPSEC,TIMEOUT=60,PROCTIMEOUT=60,VCLIENT=1
[Thr 4370] SSL settings: verify_client: 1, cache_size: -1, cache_lifetime: -1, credfile: SAPSSLS.pse, ciphers: default
[Thr 4370] Started service HOST=localhost,PORT=50008,PROT=TELNET,TIMEOUT=60,PROCTIMEOUT=60
[Thr 1029] JNCMIHttpMsPutLogon: delete http logon information
[Thr 1029] JNCMIP4MsPutLogon: reset p4 logon information
[Thr 1029] JNCMIP4MsPutLogon: reset p4s logon information
[Thr 1029] JNCMIIIOPMsPutLogon: reset iiop logon information
[Thr 1029] JNCMIIIOPMsPutLogon: reset iiops logon information
[Thr 1029] JNCMITelnetMsPutLogon: reset telnet logon information
[Thr 4370] IcmNetCheck: network check passed without detecting problems
[Thr 1029] JNCMIHttpMsPutLogon: delete http logon information
[Thr 1029] JNCMIP4MsPutLogon: reset p4 logon information
[Thr 1029] JNCMIP4MsPutLogon: reset p4s logon information
[Thr 1029] JNCMIIIOPMsPutLogon: reset iiop logon information
[Thr 1029] JNCMIIIOPMsPutLogon: reset iiops logon information
[Thr 1029] JNCMITelnetMsPutLogon: reset telnet logon information
[Thr 1029] JNCMIHttpMsPutLogon: delete http logon information
[Thr 1029] JNCMIP4MsPutLogon: reset p4 logon information
[Thr 1029] JNCMIP4MsPutLogon: reset p4s logon information
[Thr 1029] JNCMIIIOPMsPutLogon: reset iiop logon information
[Thr 1029] JNCMIIIOPMsPutLogon: reset iiops logon information
[Thr 1029] JNCMITelnetMsPutLogon: reset telnet logon information

[Thr 4113] Sun Jun 29 19:51:27 2014
[Thr 4113] {0000000b} Traffic Control: Nettimeout (30) exceeded by peer: 110.56.36.96:65144 [icxxthr_mt.c 4285]
[Thr 4113] CONNECTION (id=0/11):
    used: 1, type: default, role: Server(1), stateful: 0
    NI_HDL: 132, protocol: HTTP(1)
    local host:  152.72.244.17:50000 ()
    remote host: 110.56.36.96:65144 ()
    status: READ_REQUEST
    connect time: 29.06.2014 19:50:55
    MPI request:        <13>     MPI response:        <16>
    request_buf_size:   0        response_buf_size:   0
    request_buf_used:   0        response_buf_used:   0
    request_buf_offset: 0        response_buf_offset: 0

[Thr 4113] Sun Jun 29 19:52:07 2014
[Thr 4113] {00000011} Traffic Control: Nettimeout (30) exceeded by peer: 110.56.36.96:65149 [icxxthr_mt.c 4285]
[Thr 4113] CONNECTION (id=0/17):
    used: 1, type: default, role: Server(1), stateful: 0
    NI_HDL: 133, protocol: HTTP(1)
    local host:  152.72.244.17:50000 ()
    remote host: 110.56.36.96:65149 ()
    status: READ_REQUEST
    connect time: 29.06.2014 19:51:30
    MPI request:        <1d>     MPI response:        <1e>
    request_buf_size:   0        response_buf_size:   0
    request_buf_used:   0        response_buf_used:   0
    request_buf_offset: 0        response_buf_offset: 0

[Thr 4113] Sun Jun 29 19:54:57 2014
[Thr 4113] {00000024} Traffic Control: Nettimeout (30) exceeded by peer: 110.56.36.96:65178 [icxxthr_mt.c 4285]
[Thr 4113] CONNECTION (id=0/36):
    used: 1, type: default, role: Server(1), stateful: 0
    NI_HDL: 130, protocol: HTTP(1)
    local host:  152.72.244.17:50000 ()
    remote host: 110.56.36.96:65178 ()
    status: READ_REQUEST
    connect time: 29.06.2014 19:54:25
    MPI request:        <37>     MPI response:        <38>
    request_buf_size:   0        response_buf_size:   0
    request_buf_used:   0        response_buf_used:   0
    request_buf_offset: 0        response_buf_offset: 0

[Thr 4113] Sun Jun 29 19:59:27 2014
[Thr 4113] {00000043} Traffic Control: Nettimeout (30) exceeded by peer: 110.56.36.96:65231 [icxxthr_mt.c 4285]
[Thr 4113] CONNECTION (id=0/67):
    used: 1, type: default, role: Server(1), stateful: 0
    NI_HDL: 128, protocol: HTTP(1)
    local host:  152.72.244.17:50000 ()
    remote host: 110.56.36.96:65231 ()
    status: READ_REQUEST
    connect time: 29.06.2014 19:58:48
    MPI request:        <63>     MPI response:        <64>
    request_buf_size:   0        response_buf_size:   0
    request_buf_used:   0        response_buf_used:   0
    request_buf_offset: 0        response_buf_offset: 0

[Thr 4113] Sun Jun 29 20:30:17 2014
[Thr 4113] {00010136} Traffic Control: Nettimeout (30) exceeded by peer: 110.56.36.96:65514 [icxxthr_mt.c 4285]
[Thr 4113] CONNECTION (id=1/310):
    used: 1, type: default, role: Server(1), stateful: 0
    NI_HDL: 136, protocol: HTTP(1)
    local host:  152.72.244.17:50000 ()
    remote host: 110.56.36.96:65514 ()
    status: READ_REQUEST
    connect time: 29.06.2014 20:29:42
    MPI request:        <1cf>    MPI response:        <1d0>
    request_buf_size:   0        response_buf_size:   0
    request_buf_used:   0        response_buf_used:   0
    request_buf_offset: 0        response_buf_offset: 0

[Thr 1029] Sun Jun 29 20:46:14 2014
[Thr 1029] *** ERROR => IcmJSessSendMsg: no server with protocol P4 connected [icxxjcom_mt. 5313]
[Thr 1029] *** WARNING => P4PlugInReadHandler(id=0/554): P4ProcessRequest failed: Dispatching error(-26) [p4_plg_mt.c  1378]
[Thr 1029] *** WARNING => IcmJSessSendConnClosedMsg: server 0 with protocol P4 not connected [icxxjcom_mt. 6142]
[Thr 1029] *** ERROR => IcmJSessSendMsg: no server with protocol P4 connected [icxxjcom_mt. 5313]
[Thr 1029] *** WARNING => P4PlugInReadHandler(id=0/555): P4ProcessRequest failed: Dispatching error(-26) [p4_plg_mt.c  1378]
[Thr 1029] *** WARNING => IcmJSessSendConnClosedMsg: server 0 with protocol P4 not connected [icxxjcom_mt. 6142]
[Thr 1029] *** ERROR => IcmJSessSendMsg: no server with protocol P4 connected [icxxjcom_mt. 5313]
[Thr 1029] *** WARNING => P4PlugInReadHandler(id=0/556): P4ProcessRequest failed: Dispatching error(-26) [p4_plg_mt.c  1378]
[Thr 1029] *** WARNING => IcmJSessSendConnClosedMsg: server 0 with protocol P4 not connected [icxxjcom_mt. 6142]

[Former Member]

Please check if the issue is related to -  1981961 - ICM doesn't reconnect after the FCA connection to Java is broken This notes suggest to upgrade the patch of kernel/ICM. Regards, Mijin

[divyanshu_srivastava3]

Hi Shiv,

Can you give dev_dispatcher and std_server0 logs ?

Also, please check /usr/sap/<SID> file system space.

Regards,

Divyanshu

[Former Member]

Hi,

503 error comes when the client finds that the target server is not running or accessible.

Is the java system up and running? Check if the server nodes are up ?

check for the logs in work dir and if only server node is not coming up then check for default traces.

Amit.