SAP Portal SERVER0 folder over 500GB

[former_member196664]

Hi All,

I am keep getting out of space issue in Portal. When I checked E:\usr\sap\EPQ\JC00\j2ee\cluster\server0 is too big. Lot of files being generated and its size is over 500GB.
Can I delete E:\usr\sap\EPQ\JC00\j2ee\cluster\server0_old2 and old1 folders?

E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\log is over 22 GB. There is lot of dev_jrfc.20150225_200323.trc trace files under E:\usr\sap\EPQ\JC00\j2ee\cluster\server0 itself. Please advise how to reduce space. What is making this folder fill-up so quickly. Please help to understand this trace files and how to reduce them. Thanks for the help. Portal 7.0 on windows/sql platform.

Thanks,

Kavitha Rajan.

[Former Member · Accepted solution]

Hi Kavitha,

Apart from deleting the old logs just do verify if the trace level is "error" or Warning or information.

If its debug/info make it either "Error" which will reduce the size of the trace significantly.

Also you can set the archiving of the old trace files which will zip the existing traces older than certain date or trace number like defaultrace.1,..10 and then zip the old onces. Refer below thread for more information.

Log and Trace configuration on NW 7.1 - SAP Netweaver Application Server Java - SCN Wiki

Let us know if you need more help on this.

Regards,

Ram

[former_member196664 · Accepted solution]

Ram,

This is portal 7.0 wiki is showing for 7.1

Thanks,

Kavitha Rajan

[Former Member · Accepted solution]

Hi Kavitha,

The wiki was shared to give you an idea to how to check and change the trace level settings and archiving for the logs.

And Do verify in your system if any of the services are in Debug mode which may cause  the heavy trace detailing.

Regards,

Ram

[hemanth2]

Dear Kavitha,

Hope you are doing good.

Nice to hear from you again.

In such cases, the first step of investigation is to manually set the parameters back to the state they were after installation (if the logs are filling up with errors). This will synchronise the configurations with the default SAP settings and restart the cluster. Please have a look at step 3 of sap note 1589548 and apply the settings.

Restart the server and check if the issue persists.

Hope this helps.

_ _ _ _ _ _ _ _ _

Kind Regards,

Hemanth
SAP Active Global Support

_ _ _ _ _ _ _ _ _
 

[former_member196664]

Hi All,

I followed the procedure to installation defaults in VA. I tried to restart Portal system. Message server is in stopped state. Please help ASAP to start the message server. Jcontrol.exe also stopped.

---------------------------------------------------

trc file: "dev_ms", trc level: 1, release: "720"

---------------------------------------------------

[Thr 3720] Tue Aug 11 15:04:30 2015

[Thr 3720] ms/http_max_clients = 500 -> 500

[Thr 3720] MsSSetTrcLog: trc logging active, max size = 52428800 bytes

systemid   562 (PC with Windows NT)

relno      7200

patchlevel 0

patchno    620

intno      20020600

make       multithreaded, Unicode, 64 bit, optimized

pid        1988

[Thr 3720] ***LOG Q01=> MsSInit, MSStart (Msg Server 1 1988) [msxxserv.c   2299]

[Thr 3720] ***LOG Q0I=> NiIBindSocket: bind (10013: WSAEACCES: Permission denied) [nixxi.cpp 3735]

[Thr 3720] *** ERROR => NiIBindSocket: SiBind failed for hdl 17/sock 1664

    (SI_EPORT_INUSE/10013; I4; ST; 0.0.0.0:3901) [nixxi.cpp    3735]

[Thr 3720] *** ERROR => MsSCommInit: NiBufListen(3901) (rc=NIESERV_USED) [msxxserv.c   12343]

[Thr 3720] *** ERROR => MsSInit: MsSCommInit (internal) [msxxserv.c   2379]

[Thr 3720] *** ERROR => MsSInit failed, see dev_ms for details

[msxxserv.c   7670]

[Thr 3720] ***LOG Q02=> MsSHalt, MSStop (Msg Server 1988) [msxxserv.c   7729]

--------------------------------------------------

Bootstrap MODE:

<SINGLE INSTANCE>

determined by parameter [ID005821950].

--------------------------------------------------

+----------------[ DB info ]----------------

| Cluster ID in DB        : ID5821950

| Type                    : server

| Installation Local Home : .\server0

| OS                      : ntamd64

| Unicode support         : nuc

| Bit-length              : 64

| Binaries Synch Status   : 2

| Database bin version    : 1562

| Local bin version       : 1562

+--------------------------------------------

* Synchronizing configuration...

No local properties available for [ApplicationThreadManager].

No local properties available for [LicensingManager].

No local properties available for [ClassLoaderManager].

No local properties available for [PoolManager].

No local properties available for [LogManager].

No local properties available for [CacheManager].

No local properties available for [IpVerificationManager].

No local properties available for [kernel_shell_commands].

No local properties available for [ThreadManager].

No local properties available for [LockingManager].

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\cfg\kernel\log-configuration.xml].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\cfg\kernel\enqt.pf].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\cfg\kernel\kernel.properties].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\cfg\kernel\hosts.txt].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\cfg\kernel\global-custom-log-configuration.xml].

...Synched ok!

Ok! (94 ms)

* Synchronizing binaries...

Ok! synched: 0 file(s); deleted: 0 file(s) (0 ms)

* Synchronizing additional files...

Synchronizing file [.\server0\version.txt].

...Synched ok!

Synchronizing file [.\server0\.hotspot_compiler].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\META-INF\mailcap].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\META-INF\javamail.default.providers].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\remote.policy].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\java.policy].

...Synched ok!

Ok! (31 ms)

* Synchronizing DTD files...

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\library.provider.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\interface.provider.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\jms-destinations.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\jms-factories.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\data-source-aliases.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\appclient-j2ee-engine.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\data-sources.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\log-configuration.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\connector-j2ee-engine.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\persistent.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\web-j2ee-engine.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\service.provider.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\ejb-j2ee-engine.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\monitor-configuration.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\application-j2ee-engine.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\application-client-additional.dtd].

...Synched ok!

Synchronizing file [E:\usr\sap\EPQ\JC00\j2ee\cluster\server0\dtd\db-init.dtd].

...Synched ok!

Ok! (78 ms)

Done!

Synch time: 6578 ms

dev_jcontrol

[Thr 796] Tue Aug 11 15:04:52 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 25/sock 1492
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:04:58 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 26/sock 1492
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:04 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 27/sock 1472
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:10 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 28/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:16 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 29/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:22 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 30/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:28 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 31/sock 1492
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:34 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 24/sock 1472
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:40 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 25/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:46 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 26/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:51 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 27/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** WARNING => Can't attach to message server (sapepqas/3901) [rc = -100]-> reconnect [jcntrms.c    371]

[Thr 796] Tue Aug 11 15:05:57 2015
[Thr 796] ***LOG Q0I=> NiPConnect2: 192.168.200.191:3901: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3283]
[Thr 796] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 28/sock 1496
    (SI_ECONN_REFUSE/10061; I4; ST; 192.168.200.191:3901) [nixxi.cpp    3283]
[Thr 796] *** ERROR => MsIAttachEx: NiBufConnect to sapepqas/3901 failed (rc=NIECONN_REFUSED) [msxxi.c      718]
[Thr 796] *** ERROR => Can't attach to message server (sapepqas/3901) [rc = -100]-> terminate [jcntrms.c    377]
[Thr 796] JControlCloseProgram: started (exitcode = -100)
[Thr 796] *** ERROR => MsIModState: not_attached [msxxi.c      4233]
[Thr 796] *** ERROR => Can't modify message server state (state = 7, rc = -3) [jcntrms.c    842]
[Thr 796] JControlCloseProgram: good bye... (exitcode = -100)

[former_member196664]

Portal started after I rebooted the server. I have 180 GB free space after deleting old trace files. Now will monitor space if it fills up. I have already defaulted the settings to installation  defaults as per Hemanth. Will check after sometime and update if huge trace files getting triggered.

Thanks,

Kavitha Rajan.

[hemanth2]

Dear Kavitha,

Hope you are doing good.

This looks like a message server issue alright.

First of all, as the KBA 1910223 explains, have a look at the below notes and make sure that

the network connection is fine. We have seen similar issues due to bad/ slow network connection. Please check the below notes with your network to ascertain the network health:

#124562     Hostname resolution problems (DNS timeouts)

#500235     Network Diagnosis with NIPING

#413330     Network timeout

Another important point is the kernel being used. Many new bug fixes are part of the latest kernel in this regard (the  J2EE engine kernel consists of executables that are directly used by the SCS processes ); please make sure that you are on the latest level.

The below links will help you if there are any queries with the kernel update procedure:

SAP Note No. 19466 - Downloading SAP kernel patches

Also check:

<http://scn.sap.com/docs/DOC-1206>

Hope this helps.

_ _ _ _ _ _ _ _ _

Kind Regards,

Hemanth

[former_member196664]

All,

Please help to stop generating these trace files under Server0 folder ASAP.

Thanks,

Kavitha Rajan.

[TomCenens]

Hi Kavitha

What's in folder server0_old1 and server0_old2 ? I assume someone created those because those are non standard. If it's some kind of backup, you could move it elsewhere or delete it if no longer in use.

As far other trace files, you can delete old traces that are not in use, furthermore, if you want less trace files to be written / they are way to detailed you would have to look at the log configuration - do you have logging / tracing on in debug mode for example - that would generate a lot of files for sure.

Kind regards

Tom