Any beginner doc on IDM delta management?

[Former Member]

Hi, all.

I have been searching the whole night and come up with nothing on some basic doc on the IDM delta management. By the way, I have the course content for IDM 920 and the material for delta is minimal. Can you point me to the right direction and that would be much appreciated? For example, where is the delta table and what is the table name? How to make use of the delta table?

Most of the blogs are too "advanced" and too brief for me on the SDN.

Thanks a million,

Jonathan.

[Former Member · Accepted solution]

Peters summary is good and as he mentions it has its challenges. Often it might be easier to check if the datasource provides a type of changelog or changenumbers & tombstones in the case of AD.

One big challenge is to control error situations and this is where the difference between using delta on To or From passes become important. If the source repository returns a partial resultset due to a network error or something else, will your next pass process and delete every user in IdM because of that? A To Pass with delta and autodelete with max limit" will abort if x% of the entries are marked for deletion. If you use a delete job with a from pass that selects from the delta table "where operation=5" you're on your own to implement the same safety mechanism.

The delta mechanism has been around since the first versions of the product in 1996/7(!) so it should be well documented and tested.

Br,

Chris

[Former Member]

Thank you Peter and Tero for your feedback. I guess I will need to play around more to get some understanding of the subject.

So do you normal put the delta in the "from pass" or the "to pass"? If I want to get the delta to only compare a couple of fields in the source data, how do you do that as the delta key is greyed out? What does source key mean?

Thanks,

Jonathan.

[Former Member]

Whether its from or to depends on what you need.  In most cases and usually for things like AD, you put it in the from pass.  You want to know whats changed in AD.

You can exclude from delta with a prefix on the row (can't remember which one but its in the dropdown list).  Just exclude everything you don't want.  Note you can have multiple prefixes but only one from the list.  The others go on the attribute (see the online help).

The source key is what its called in the delta_defs table iirc.  You can then use your sql to get the changes for this job specifically.  Select * from Logentries WHERE jobid = (select id from delta_defs where sourcename = ...  (NOTE column names will be wrong! )

You can then do a delete job which has the source of:

SELECT * FROM Logentries WHERE JobID = (select id from delta_defs where sourcename = myjob) AND Operation = 5

This will give you a list of all the objects that were in your table that are no longer in the source environment.  You can then process them.  This should be immediately followed by:

Set Operation = 6 to indicate that its been processed.

SPECIAL NOTE:  Groups require a multi-key (the key for each file is the first line of the TO-PASS ie the MSKEYVALUE).  This is because the group name will be updated multiple times for each member.  You can add a second component to the MSKEYVALUE which just gets written to the Logentries table as the key.  I THINK the format is:

MSKEYVALUE  %groupname%!!%membername%

Its somewhere in the online help.  When it writes the delta entry, the entire thing gets put in as the key but when it gets written to the IDM datastore, the second component is removed.

Hope that helps

Peter

[Former Member]

I agree - the documentation on this is poor.  The online help has some useful tips for advanced stuff but doesn't help with the basics.  The best source I found for information was the Jobs that come with the provisioning framework and just deconstructing them.

Essentially there are four tables but these two are the important ones (names may be incorrect - its off the top of my head)

Delta Defs

Logentries

Delta Defs contains the job definition

Logentries contains ALL the deltas for all the jobs, with a link to the delta defs table.

In logentries, the delta_operation column has a numeric value which tells you what the operation was that updated it.  You can use this to get deleted entries by searching for delta_operation = 4 and then doing processing against the identity in IDM.  Afterwards you set the delta_op to 5 to mark it as 'processed'.

That's the absolute basics.

Peter

[former_member2987]

Jonathan,

Let me see if I have anything from back in the MaXware days.  I think we had something. I'm sure I can whip up a Blog Entry or something.  Anything in particular you're looking to understand?

Thanks,

Matt

[Former Member]

Hi, Matt

I really appreciate your suggestion. What I was hoping to find in google yesterday were the following information:

1) An step by step delta scenario. For example, one scenario we probably need is to synchronize the data between LDAP and IDS and then to the ECC system. There can be new records, deleted records or updates. How does the IDM delta handle the 3 main situations? We can use flat file to simulate the LDAP data.

2) Delta can be created in both "From Pass" and "To Pass". In what situation do we put it in the from or to pass? Why?

3) What are the different delta options used for in either the from or to pass? What do they really mean?

I only found bits and pieces of information in the IDM help page. I also saw the blog of Ian and Murali but they are too brief for me to understand. Probably the IDM veterans in this board already know the subject very well. But for newbie like myself, it would be awfully helpful to get some basic understanding as I can see that it would be used very often in the project.

Thanks a lot,

Jonathan.

[former_member2987]

Jonathan,

I'm on the road at the moment, but should be able to find something over the weekend.

What I will say is this, for the most part Delta is only useful in mass jobs and not really at all useful in single shot IDM Provisioning tasks.  I find it is most useful in scenarios where there is no defined "marker" for determining how data fits in. Attributes like these would be a timestamp or changenumber.

As I recall, Delta is most helpful in FROM pass scenarios.

Matt

[terovirta]

The online help has something:

SAP NetWeaver Identity Management

Search with terms "delta" or "logentries".

Is there anything specific you're doing with delta? For simpler cases like "does the record exist already or has it been deleted from the source" I have found easier (just personal preference) to store the last import to a temp table and compare that with the new batch from the source. If it's more complex case like "has the record changed" and the record has 15 fields to compare then the delta is better.

regards, Tero