Max concurrent RT engines is ignored

[lambert-giese]

Dear experts,

my use case is to execute initial load jobs for a large number of backend systems in an automated fashion without any resource contention.

I've configured an IdM 7.2 SP7 system with one dispatcher to start at most two concurrent runtime engines. This is configured both as a global setting and on the single dispatcher (see screenshots).

Then I execute a normal job with one toGeneric pass whose entry script uses uRunJobNow() to start the ~50 initial load jobs. My expectation was that these jobs would now be executed in sequence, with at most two jobs running in parallel.

Unfortunately, that's not the case. In fact, up to 16 runtimes are started in parallel, so resource contention occurs with the usual problems (OutOfMemory, timeout etc.). The dispatcher log emits warning messages:

Not allowed to start ANY runtimes (right now running runtimes = 16, allowed total runtimes = 2

Question: is there anything I need to change in the dispatcher configuration to achieve the desired behavior?

Best regards,

Lambert

[Former Member · Accepted solution]

I think that in SP7 it simply counts running jobs (mc_jobs.state=2), so if a job takes a long time to start and the check frequency is low (1 or 2 seconds for instance) then the dispatcher has time to start more runtimes before the state of 2 jobs are changed to 2. I believe that later versions of the dispatcher checks the pids that were started to see how many are still active.

To test, change the max engines to start to 1, increase the dispatchers policy/check interval to 30 seconds. This should result in 1 runtime being started every 30 seconds and make sure that the jobs have had time to start properly and set their state to running. Setting the state to 2 should be almost instantaneous though so that 16 have time to start almost sounds like some setting is being ignored. You might also want to check that there isn't an excessive amount of javaclasses getting loaded that bloats the startup times.

Br,

Per Christian

Message was edited by: Per Krabsetsve

[lambert-giese · Accepted solution]

Thanks Chris,

that's very helpful feedback. I changed "Max RT engines to start" to 1 as per your suggestion. Check interval was already 5 sec - the SAP recommendation for environments with one dispatcher - so I left that alone for now, assuming it is OK.

Changing "Max RT engines to start" alone did not result in any improvement yet, however. What fixed the problem was to add an artificial wait time using uSleep(10000) after each invocation of uRunJobNow(). So it seems the key part of your tip was:

make sure that the jobs have had time to start properly and set their state to running

With these two changes in place, job execution proceeds with at most two jobs in parallel as expected.

Best regards,

Lambert