How to limit provisioning task to a batch of entries from the queue

[siarhei_pisarenka3]

Hi Folks,

In our IDM system we have 60000 entries sitting in the provisioning queue for a single repository for a single task. The provisioning task handles such amount of entries very slow working for days.

I'm wondering if it's possible to limit somehow the provisioning task to a certain batch of entries from the queue. Let say, to force somehow the provisioning task to handle TOP 1000 entries at once and then finish. In such a way it will start multiple times to process the 60000 entries, but each iteration of the task will not longer than 1000 entries.

I have the idea to overwrite the default view MXPROV_ENTRIES for the task to return top 1000 entries only. Do you think this will work ?

Any other ideas please welcome.

Thank you a lot!

Siarhei

[Former Member · Accepted solution]

Hi,

You don't mention what kind of task it is queued for, or perhaps it's even an action? Or the IdM version for that matter. I'm not sure how changing the views would change anything though.

In case it's a task (ordered, condt., switch etc) the dispatcher has for the latest versions been using the views mxpv_grouptasks_ordered/switch/conditional which already have 1000 row limiters in them.

If it's an action then the runtime should not be much affected by the size or really care how many are in the mxprov_entries view. It might take a few seconds to get the entire list og 60.000 rows if it doesn't add it's own limitation in addition to the action/repid filter. If they're all for the same actionid/repository combination it will just works through it one entry at a time untill it's done and each entry is enclosed in its own transaction so at this point there would be no effect of having a top 1000 or not It's also not possible to split up the queue action/repository combination between multiple runtimes either.

Br,

Per Christian

[siarhei_pisarenka3 · Accepted solution]

Hi Per

Originally I thougth it would be possible to boost processing of the queue if we split the 60000 actions tasks onto batches. But now I see (and also based on your reply) that the provisioning job handles the entries one by one anyway commiting transaction after a single entry is processed. It means that the speed of queue processing does not really matter if the provisioning task is done or not yet - it should be the same.

But today we identified that our queue was processed very slow in general, due to one unused DB index on the MXP_PROVISION table. The index included the exectime attribute. When I disabled the index in the table it reduced the time of processing of a single action task dramatically in more than 100 times !!

I am not really sure why IDM developers created the index if there is no visible usage of it in database.

I will monitor the system with the index disabled for some time in order to be completely sure that everything is OK.

Siarhei

[siarhei_pisarenka3]

Finally the issue was resolved with improving a generic performance in database. So all 60000 events have been quickly processed.

[former_member2987]

Sirahei,

I don't think it will be possible to overwrite the view. 

In the future the best bet is to slowly load data via the initial load job.  It's easier to break up the data into chunks that your IDM infrastructure can handle. In SQL Server you can control this as you say by Top, or in Oracle and DB2 you can use ROWNUM.

I don't recall if there is an official way to clear the provisioning queue.  You'll need to check this via an OSS note.

Thanks,

Matt