CCMS Agents availability

[Former Member]

Hi All,

ccms agent availability is showing 0% after rebooting the remote system for winter time change.

agent connection test is successful and work directory is accessible in rz21. looks like agent is not able to talk to central system. any troubleshooting steps?

the error in "sapstartsrv.log" is

CCMS agent initialization for instance type ABAP: return code 0.

CCMS agent start: return code 411.

Thanks,

Ranjith

[Former Member]

Hi ,

We have faced the same issue , in which agent service was not restarting.

try with this

1. Rename agent.lock file under CCMS directory .

2. Clear shared memory thorugh -initshm command.

3. Start the service.

Issue will get resolved.

Br, Vaibhav

[bin_sun]

2248316 - Integrated CCMS agent stops pushing data to CEN

This KBA may be helpful.

[alwina_enns]

Hello Ranjith,

could you please check the agent trace sapstartsrv_ccms.log in DIR_LOGGING/sapccm4x directory of the affected instance, if any errors are reported for the CSMREG user and if this user could connect to CEN?

Regards,
Alwina

[Former Member]

Hello Alwina,

I reregistered the agents and availability became 100% and then dropped to 0%. then again it became 33% and drops to 0%.

sapstartsrv_ccms.log has below errors.

ERROR: OS monitoring: Error-253 Can not attach same MTE [Utilization] with different MTE class [CPU_UsageUtil]. Existing MTE class: [CPU_Utilization]

ERROR: OS monitoring: Error-253 Can not attach same MTE [I/O Wait] with different MTE class [CPU_UsageIOWait]. Existing MTE class: [CPU_IO_Wait]

ERROR: OS monitoring: Error-253 Can not create MTE [Solution Version]. Same name but different types MESSAGE_CONTAINER vs. SHORTTEXT

sapstartsrv.log has error for ccms

CCMS agent initialization for instance type ABAP: return code 0.

CCMS agent start: return code 411.

Thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

in which client you was logged on during the agent registration? The registration of the integrated agents is client dependent.

What is the basis support package level of CEN? As of certain basis support package level a new component registration was delivered in RZ21.

What is the kernel pach level of the monitored system?

In the agent trace please search for CSMREG user. The errors above error-253 indicates some inconsistency in the OS monitoring, but these errors are not the cause of the issue. If the CSMREG user could connect to CEN without problems, you should see an entry like this:

[Former Member]

Hi Alwina,

i registered in client 000 and basis is sp15      SAPKB70215. kernel patch of monitored system is 721_ext patch 500.

i tried searching csmreg in trace file but nothing came up

hbpadm> grep CSMREG sapstartsrv_ccms.log

hbpadm>

thanks,

Ranjith

[alwina_enns]

Hello Ranjith,

thank you, SAPKB70215 contains the old component registration in RZ21. If sapstartsrv_ccms.log does not contain any entries about CSMREG user, then it is probably too big because of too many entries with the Error-253 and you should check, if sapstartsrv_ccms.log was already moved to sapstartsrv_ccms_1st.log. If such logs exist, please search in all sapstartsrv_ccms* files for CSMREG user. You should find either an error for CSMREG or the info, that CSMREG is connected to CEN.

Did you get any errors or messages during the system registration in RZ21?

Regards,
Alwina

[Former Member]

Hi Alwina,

there are no messages related to csmreg in any of the files. i did not get any error during reregistration. wonder why the error is coming after restart on Nov 1. csmreg initial password expired and i had to reset it before re-registering the agents.

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

if you have changed the password for the CSMREG user, then this is possible, that the old password is still stored somewhere, either in CEN or in the working directory of the agent, and the reregistration should help. Could you please:

1. Unregister the agent again by deleting it in "RZ21 --> Topology --> Agents for Remote Systems --> Display Overview"

2. On the monitored system please execute one time manually:

sapccm4x -initshm pf=<full path to the profile used by the integrated agent>

(please look in the first trace sapstartsrv_ccms_1st.log for the first entries, which profile is used)

This will reinitilize the entries of the agent in the CCMS shared memory

3. Please move the whole content of the agent working dircetory DIR_LOGGING/sapccm4x to another place

4. Restart sapstartsrv of the affected monitored instance with:

sapcontrol -nr NN -funciton RestartService

("NN" - instance number)

5. Logon in client 000 in CEN and register the system in RZ21 again

6. If the CCMS in CEN is configured in a different client than 000, you will need to repeat the system registration also in this client

Regards,
Alwina

[Former Member]

Hi Alwina,

i will perform these steps and update you. I cannot restart sapstartsrv as it is PRD system. i am also seeing this error in sapstartsrv. normally we should have return code 0 . isnt it?

the error in "sapstartsrv.log" is

CCMS agent initialization for instance type ABAP: return code 0.

CCMS agent start: return code 411.

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

yes, the return code for the agent start should be 0, I do not know, what the return code 411 means, but you should see more details in the agent trace, what was the problem during the start.

The sapstartsrv can be restarted on OS level with sapcontrol as above  at any time, it does not restart the whole system, you can try it out on a test system first. It could be only a problem, if the affected instance is running in a cluster, you should clarify first, how cluster software will react, if sapstartsrv is not available for a short time.

Regards,
Alwina

[Former Member]

Hi Alwina,

I performed all the steps you gave and now the error during reregistration is

Error when sending the configuration data to the agent

SoapFaultCode:5 CCMS__RegisterCentralSystem failed: 413: the agent is restarting now, please resend your request latter.

Thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

please check the agent trace sapstartsrv_ccms.log, what is the problem with the agent?

Regards,
Alwina

[Former Member]

Hello Alwina,

this is the log in one of the server where sapcontrol restart gave me error

sapcontrol -nr 02 -function RestartService

12.11.2015 07:43:49

RestartService

FAIL: HTTP error, HTTP/1.1 401 Unauthorized

> more sapstartsrv_ccms.log

[Thr 139760988591968]

Thu Nov 12 07:08:55 2015

      Initialize CCMS agent sapccm4x

INFO:

      CCMS version  20060508, 64 bit, multithreaded, Unicode 7210

      compiled at   Apr 27 2015

      systemid      390 (AMD/Intel x86_64 with Linux)

      running on     Linux 2.6.32-573.1.1.el6.x86_64 #1 SMP Tue Jul 14 02:46:51 EDT 2015 x86_64

      patch text    patch collection 2014/1, OSS note 1453112, PL 500

      profile       /usr/sap//SYS/profile/

INFO: cannot open config file /usr/sap//D02/log/sapccm4x/csmconf_sapstartsrv.

INFO: CCMS agent is not registered at central CCMS system.

INFO: No central CCMS system configured.

INFO: CCMS agent initialization finished.

INFO: Starting CCMS agent threads...

INFO: Checking CCMS monitoring segment ...

INFO: Attached to a monitoring segment type APPLSERVER (1), running in shared memory.

INFO: Agent status in the attached monitoring segment is ACTIVE (20).

ERROR: Starting integrated CCMS agent SAPCCM4X encountered troubles when attaching CCMS monitoring segment of type APPLSERVER:

       There is an agent, pid 29069, with the same profile already active. Cannot start twice!

       If there is no other standalone or integrated CCMS agent SAPCCM4X running with a profile containing

       parameters "SAPSYSTEMNAME = " and "SAPSYSTEM = 2", start standalone agent sapccm4x with option

       -initshm and profile to perform cleanup of CCMS Monitoring Segment in shared memory.

INFO: Checking CCMS monitoring segment finished.

****************************

* CCMS AGENT START ABORTED *

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

thnak you for the info. Under which user you are logged on to restart the sapstartsrv? The <sid>adm user should not get an authorization error.

It looks like there is an agent.lock file in the working dircetory of the agent? The "initshm" option should delete this file. Please execute the agent with the "initshm" option again under the <sid>adm user:

sapccm4x -initshm pf=<path to the profile used by the integrated agent>

and restart sapstartsrv (under <sid>adm user) and check the sapstartsrv_ccms.log trace, if the agent could start.

Regards,
Alwina

[Former Member]

Hello Alwina,

i did try the steps again but it didnt help. .sapstream5xx13 is not getting created in /tmp directory . i might have to restart the entire instance.

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

is this still the same authorization error also for the <sid>adm user? Or which error did you get?

Regards,
Alwina

[Former Member]

Hello Alwina,

it is same error for sidadm

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

did you check already the note "1565645 - SAP composite note:sapcontrol" for the reasons of the "Unauthorized" error?

Could you please execute sapcontrol with the <sid>adm user and password:

sapcontrol -nr NN -user <sid>adm <password> -function RestartService

Does it work?

You have mentioned, that the file .sapstream5xx13 is not getting created. What/who has removed this file?

Regards,
Alwina

[Former Member]

Hi Alwina,

I tried sapcontrol -nr NN -user <sid>adm <password> -function RestartService but syntax seems wrong. please share help link for sapcontrol commands syntax?

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

on our test system it works:

[alwina_enns]

you can display the valid functions for sapcontrol with "sapcontrol -help"

[Former Member]

Hi Alwina,

for some reason it is not working.

sapcontrol -nr 02 -user sidadm $pring9SID -function RestartService

I am getting sapcontrol help message when I execute above command

Thanks,

Ranjith

[alwina_enns]

Hello Ranjith,

for "sidadm" you have choosen the real sidadm user like "nwcadm"?

Regards,
Alwina

[alwina_enns]

probably there is a problem with the special character in the password, for your password I get also only the help message instead of "invalid credentials"

Could you please use aphostroph for the password like '$pring9SID', will it work?

[Former Member]

Hi Alwina,

I changed the password without special characters and it helped but still facing the error agent is restarting. a clean restart might be needed i believe.

thanks,
Ranjith

[alwina_enns]

Hello Ranjith,

it could be the error as described in the note "2165518 - CCMS:sapstartsrv process hangs after stop action". The fix is in PL512 and you have mentioned the affected system is on PL500.

Regards, Alwina