Diagnostic Agents on the fly consuming all the CPU

[Farid]

Hello,

We are running a HP-UX Superdome server, with 10 SAP systems running on this

same hardware platform, so as you may imagine, it would make a lot of

sense to use  the Diagnotic Agents on the fly feature as described in

document linked to sap note 1365123.

So I have followed the instructions of the sap note 1365123, and diagnostics agents are running,

the only problems is that more than 30 diagnostic agents are running and comuning all the CPU,

whenever I start the diagostic agents, the CPU Idle % on the server drops from 50% to

5%

The reason being that we have a lot of logical names defnied on

that server : third party tools, two LAN layers, and we can not

remove those logical server name from the DNS obviously.

So most of theses diagnotics agents are running but not related to any

sap system

So, At first I tried to manually kill some of unecessary the diagostic

agents using the "kill -9" command it did not work, the process

automatically reappears

Then I tried to stop them from the solution manager console,

"stopdiagnostic agent", or "reset server name" did not stop the process

atthe OS level.

Then I tired to adjust the smdadgent.property file, and

comment all theunecessary DIA it did work for 2 minutes, afterwards, the

file is automatically updated with unecessary DIA.

Anyone facing the same issue ?

Thank you for your help

Bes tregards

[benjamin_wegener · Accepted solution]

You can limit the number of Diagnostic Agents installed by configuring the main physical host properly in the solution manager to which the agents connect.

1. Go to transaction SOLMAN_SETUP > Managed Systems Configuration > Hosts

2. Select your physical host and click on configure system

3. In step 2, you can not only de / activate the on-the-fly feature but also create a exclude list for the hosts with regular expressions (Perl compliant)

[Henk]

Hi Raoul,

I assume that you have an SAP Host Agent running on the Superdome. Do you only have the SAP application Servers running on the server or also their databases?

Henk.

[Farid]

Hi Hemk,

Yes, the SAP Host Agent (latest verion) is also running on the superdome server.The sap systems are running on the server (both the database and the applications servers) and also dozens of third party tools , and each one of thoses tools has its own logical hostname and therefore its own DIA on the fly, even if we absolutely do not need one for them.

Thank you for your attention

BestRegards

[Henk]

Hi Raoul,

The problem is probably caused by the number of databases combined with the number of Diagnostics Agents. The SAP Host Agent keeps track on the databases that reside on your Superdome. Each and every Diagnostics Agent will contact the SAP Host Agent at a certain time interval and gets the list of databases. In stead of performing outside discovery for just one database, every Diagnostics Agent will perform outside discovery for all databases on your server.

If you have, for instance, 20 Diagnostics Agents and 30 databases running on the Superdome, all 20 Agents will query all 30 databases every 5 minutes (default).

You can tweak this a little bit by going to the Agent Administration and choose the tab strip "Application Configuration". In the left column ("Agent Applications"), select configuration 'com.sap.smd.agent.application.e2edcc'. In the right half of the screen, the properties of this configuration will be displayed. You can adjust some values here, to see if this will take some load off your server.

Setting the parameter 'job.dbinfo.schedule.period.m' to value 60, for instance, will reduce the number of times the Agents query the databases to once every hour.

However, there will still be 20 Agents (in our example) querying 30 databases every hour.

There are of course some more parameters you can tweak here.

Please also be advised that SAP states that every Diagnostics Agent will take 5% from your amount of CPU. (This is in a guide somewhere, but I don't have access to it at the moment.) I always thought it was a pretty strange formula, since you can have more or less CPU's in a single server, and this rule of thumb does not take this into account. But this would already mean that 20 Diagnostics Agents will consume every % of CPU your sever can possibly have.

I'm not familiar with the on-the-fly concept. However, on our Superdome with 30+ databases, I shut all SMDA's down when I don't need them; and this is actually almost always.

Hope this clarifies a bit.

Henk.

Update:

When you navigate to http://service.sap.com/diagnostics --> Media Library, you can download the "SAP Solution Manager 7.1 Sizing Toolkit". If you extract the resulting archive, you'll find a document inside, called "SAPSolutionManager7.1_SizingGuide.pdf". The rule of thumb I was talking about, is on page 27 of this document.

[Farid]

Hello Henk,

Actually we "only" have 10 Databases running on our Superdome, but we have 32 Diagnostic Agents, each one of them separatly consuming a "small" (yet to be determined) amount of CPU, but all thoses DIA agents combined using more than 50% of the whole CPU !

The idea behind DIA OTF (Diagnostic Agents on the fly) is that the number of DIA in independant

of the number of SAP systems or database, but it depends on the number of vitual/physical hostname defined (ib you Superdome for example)

http://wiki.sdn.sap.com/wiki/display/smsetup/diagnostics+agent+and+ha+support

"the number of Diagnostics Agent instances (allocated Instance Number) on a Physical or Virtual, is also independant of the number of Managed Systems/Logical Hosts"

In case you have several SAP systems running on the same hardware platfrom, instead of installing a dedicated diagnostic agent for each one of tose sap managed systems, you only install one DIAOTF, and it will automatically create the others diagnostic agents for all your sap systems but not only (unfortunatly).

As it is certianly the case in your Superdome, you have several virtual/physical hostnames ( for your LAN layers, non SAP products, third party tools), you end up with much more DIA agents than SAP systems.

Thank and Regards