Provisioning failing when updating users Business Role

[Former Member]

All,

We had a few different consultants come in and install IDM for our organization and after multiple restarts finally got it running to a point.    Nothing was really documented on how specific tasks are triggered and where that is configured and honestly I am having the time of it trying to find a document that discusses the entire workflow task hierarchy for numerous out of box provisioning tasks.  

That said, I also have a few Warning messages that show up everyday in our system that I guess were ignored by the consultant that set up the system and I think are causing a majority of my provisioning hold-ups and dispatcher failures every day.

Any help with identifying a solution for the below would be helpful.

Dispatcher2

Error in Sys Log:  Exception from Add operation:ToCustom.addEntry failed for entry E0090986 - Exception:com.sap.idm.ic.ToPassException: User E0090986 exists

Situation:

User was hired in last year, was already in a role and provisioned successfully to the backend ABAP system.    The users shows in the IDM web UI that they have an account on our Training box:

User changed jobs and an HCM event triggered our modify task which worked perfectly up untill the "create ABAP identify" task.  On this task, even though IDM shows (see above) that it knows a user accout exists in our Training (consultant labeled it test) environment, it fails with a stating that account already exists.

Below are the details of the task.

My personal thoughts are that IF this was a user move then shouldn't this have triggered the "Modify" ABAP provisioning task and not the "Create" task?      If so then how do you tweek the system to recognize a move in the organization to modify a users backend SAP access instead of trying to create it new which causes the error?

Secondly to this, it is NOT failing on the PRD SAP or GRC access which strikes me wierd..

So far the only way to get the user to provision to the backend system is for me to delete the user account and then retry the job.

Side note****   This happens frequently with different user accounts across our PRD, TRN and GRC backend systems, in that Some provisioning will work correctly while others fail with the above error message even though the identify store sees them.    In all cases if I delete the ID in the backend and retry in any system it seems to work after that. 

Below is the log details from the Job.

<Log details>

<?xml version="1.0" encoding="UTF-8"?>

<mx:EMSLOG xmlns:mx="http://www.maxware.com/EMS">

<mx:GENERAL>

<mx:DATE>03.09.2014 13:39:00</mx:DATE>

<mx:VERSION>DSE.JAR version: 7.2.8.0 Built: 15.05.2013 18:37:10 - 176434 (c) Copyright 2012 SAP AG. All rights reserved.</mx:VERSION>

<mx:JOB>Identity Center JobID:1</mx:JOB>

<mx:JOBName>CreateABAPIdentity</mx:JOBName>

<mx:MCVersion>7.20.8.0-SQL-2013-06-21 Schema update:1100</mx:MCVersion>

<mx:JDBCInfo>jdbc:sqlserver://****************.*****************.com:1433;xopenStates=false;sendTimeAsDatetime=true;trustServerCertificate=false;sendStringParametersAsUnicode=true;selectMethod=direct;responseBuffering=adaptive;packetSize=8000;loginTimeout=15;lockTimeout=-1;lastUpdateCount=true;encrypt=false;disableStatementPooling=true;databaseName=PN2_db;applicationName=Microsoft SQL Server JDBC Driver;</mx:JDBCInfo>

<mx:MACHINE>idmdispatcher2</mx:MACHINE>

<mx:JOBID>976EB10B-0609-432B-B4E0-924E7C824E95</mx:JOBID>

<mx:WORKAREA>E:/usr/sap/IdM/Identity Center/Jobs/976EB10B-0609-432B-B4E0-924E7C824E95</mx:WORKAREA>

<mx:PRODUCT>Provisioning</mx:PRODUCT>

<mx:CUSTOMER>SAP customer : f9c1c5cd66189d133765ac44ea6c127a</mx:CUSTOMER>

<mx:TIMEUSED>2</mx:TIMEUSED>

<mx:NERRORS>1</mx:NERRORS>

<mx:NWARNINGS>0</mx:NWARNINGS>

<mx:NENTRIES adds="0" dels="0" markdels="0" mods="0" noops="0">0</mx:NENTRIES>

</mx:GENERAL>

<mx:PASSES>

<mx:PASS name="06B5C9F3-3F04-4FB6-ABDB-D606847D761F" seq="1" title="CreateABAPIdentity" type="To Custom">

<mx:REPOSITORYNAME>ECC_TEST</mx:REPOSITORYNAME>

<mx:MESSAGES>

<mx:ERROR seq="1">

<mx:TEXT>putNextEntry failed storingE0090986</mx:TEXT>

<mx:TEXT>Exception from Add operation:com.sap.idm.ic.ToPassException: User E0090986 exists</mx:TEXT>

<mx:ENTRY>

<mx:ATTR name="MX_DATEFORMAT">2</mx:ATTR>

<mx:ATTR name="MX_MAIL_PRIMARY">email.address@email.com</mx:ATTR>

<mx:ATTR name="MX_NUMBERFORMAT">X</mx:ATTR>

<mx:ATTR name="MXREF_MX_COMPANY_ADDRESS"/>

<mx:ATTR name="MX_PRINTERSETTINGS_SPDB">G</mx:ATTR>

<mx:ATTR name="MX_FIRSTNAME">*************</mx:ATTR>

<mx:ATTR name="MX_COMMUNICATION_METHOD">INT</mx:ATTR>

<mx:ATTR name="MX_PRINTERSETTINGS_SPLD">AGLOCAL1</mx:ATTR>

<mx:ATTR name="MX_ENCRYPTED_PASSWORD">************************************************************</mx:ATTR>

<mx:ATTR name="MX_VALIDFROM">2012-11-12</mx:ATTR>

<mx:ATTR name="MX_VALIDTO">9999-12-31</mx:ATTR>

<mx:ATTR name="MX_ADMIN_UNIT">NONSUPER</mx:ATTR>

<mx:ATTR name="MX_REFERENCE_USER"/>

<mx:ATTR name="MX_ENTRYTYPE">MX_PERSON</mx:ATTR>

<mx:ATTR name="DISPLAYNAME">***********************</mx:ATTR>

<mx:ATTR name="MX_LASTNAME">***************</mx:ATTR>

<mx:ATTR name="MX_ENTRY_REFERENCE"/>

<mx:ATTR name="MSKEY">150232</mx:ATTR>

<mx:ATTR name="MSKEYVALUE">e0090986</mx:ATTR>

</mx:ENTRY>

</mx:ERROR>

</mx:MESSAGES>

<mx:DELTA>0</mx:DELTA>

<mx:TIMEUSED>2</mx:TIMEUSED>

<mx:NENTRIES adds="0" dels="0" markdels="0" mods="0" noops="0">0</mx:NENTRIES>

<mx:NERRORS>1</mx:NERRORS>

<mx:NWARNINGS>0</mx:NWARNINGS>

</mx:PASS>

</mx:PASSES>

<mx:PROVISIONING fail="1" num="1" ok="0">

<mx:PROVISION auditid="924904" mskey="150232" seq="1" status="FAIL">

<mx:TEXT>ToCustom.addEntry failed for entry E0090986</mx:TEXT>

</mx:PROVISION>

</mx:PROVISIONING>

</mx:EMSLOG>

[Former Member · Accepted solution]

The failure is on the assignment of the "ecc_test-only" privalege assignment in the Business role.   I was told that, that must be assigned to the user as part of the account creation before any other privilages are assigned.  If you don't assign that but then try and assign other previlages (or sap roles) then those will fail.     again, some user ID's work fine others don't....and if I delete the ID itself on the backend ABAP system it provisions fine on a retry with that ECC TEST only privilage.

[Steffi_Warnecke · Accepted solution]

Yeah. Then I mean the system-privilege, that might be missing. The perils of not having the system in front of me to check the names of the privileges. ^^

The account-privilege of the repository must be assigned to create an account there and to get the provisioning working for that user to that system, that is correct.

But your problem here is, that the account alreay exists, but IDM is still trying to create the user again and therefor fails, because it already exists. The modify-task is the one, that should be triggered, as you found out nicely.

So the reason it's not triggering is, that in the first check in IDM (where it goes either to "create" or to "modify" something is missing, that shoves the entry to modify. And my guess is, that the second repository-privilege is missing. The one, that you can't see in the UI, but only in the database. IDM should assign this privilege itself, when you assign the other one. So it's not something you control directly, normally.

I wish I had my system now, because I had created an SQL-query for this like one or two weeks ago, because I had the same issue as you have. Only in reverse. The privilege in question was still there and so instead of going to "create" my workflow went to "modify" for some users.

I don't remember the name of the view you have to check right now. GNAR! Was it one of the ext-views?

[Former Member · Accepted solution]

OK, so if I do find that an SQL entry is missing for the user, how do I get IDM to update that automatically if it is not correct on the backend?      is there as a synch command i can create a job in IDM for to perform this?

deleting ID's in the backend does not seem like the optimal solution

[Steffi_Warnecke · Accepted solution]

I would create a simple job in the MMC, where the source is an sql-query to find all users, that have the only-privilege of a system, but not the system-privilege and in the destination-tab you could just add the missing privilege to those users.

As I needed the reverse-thingy, I created a query to find all users that still had the system-priv assigned, but not the only-priv and delete the first privilege from them.

But that's all from memory now (and it's late where I am). ^^ So I'm a little shaky on the details. I can get back to you tomorrow with some more infos if you like and one of the other IDMlers here hasn't chimed in yet. Because I don't know, if there is a "best practise" for this kind of account-trouble.

[Former Member · Accepted solution]

Thank you, I will look for a response tomorrow!

[Steffi_Warnecke · Accepted solution]

Okay, here we go. ^^

For each repository there are two "main"-privileges created with the initial load-job:

PRIV:%$rep.$NAME%:ONLY

which you can see in the UI and assign to a user to create an account in that system

PRIV:SYSTEM:%$rep.$NAME%

which you can NOT see in the UI, but find through a SQL-query in the views "idmv_link_ext" and "idmv_link_ext2"

The last one should be assigned by IDM automaticly, if the first one is assigned either through the UI or a task or job.

For my purpose I have a job, where the source-query looks like this:

select distinct mcthismskeyvalue from idmv_link_ext2
where mcothermskeyvalue='PRIV:SYSTEM:%$rep.$NAME%'
minus
select distinct mcthismskeyvalue from idmv_link_ext2
where mcothermskeyvalue='PRIV:%$rep.$NAME%:ONLY'

So it should find all mcthismskeyvalues (which are is the MSKEYVALUEs of the users in my use case) that have the system-privilege assigned MINUS those, that have the only-privilege assigned, too. Which leaves those users, that have only the system-privilege assigned.

That's because the account in the backend system was deleted from IDM, but something went a bit wonky and only the account-privilege (ONLY-priv) was deleted from the user in IDM and the system-priv stayed. And THAT starts trouble, once you try to create a new account in one of those systems. ^^

Yeah, that's the base and the pool of users to work with. Now in destination-tab I just have a simple delete of the system-privilege for the found users looking like this:

This way I clean up the system-privileges, that are not supposed to be assigned anymore.

In my understanding, you just need to reverse this (source and destination). Look for the users, that have only the only-privilege assigned and add the system-priv to them.

Though it can't hurt to look for those users, too, that have only the system-priv to clean that up, too. ^^

Since you can easily test that with just one user, too, there shouldn't be any big issue.

So that's it. Like I said before, maybe I'm doing this too complicated or plainly wrong and there is a nice, simple best practise I haven't found and heard of yet. But this works for me so far.

Regards,

Steffi.

[former_member2987 · Accepted solution]

Nice description, Steffi.

Michael, sounds like you need to reconcile between IDM and your back end systems as well.

It might not be a bad idea to throw a script into that job that would run uPrivReconcile as well.

[Former Member · Accepted solution]

Thanks for the reply Steffi, I was pulled into a nother project and havn't had a chance to run this yet.   I'll try it in the next couple of days and get back to give correct Answer high light!!

[Former Member · Accepted solution]

Steffi,

Again sorry for the late reply and also in advance for any questions that may be obvious, but I am truely an IDM novice.

I followed what I thought to be your advice (swapping the priv and system) and have the following results:

1.  I had to update the "Minus" to be "except" because of the version of SQL we are on but the scirpt seemed to work except that I got some wierd results when i tested directly in SQL.   It seems that it pulled all of my profiles and IDM business roles as being the ID's that needed to be updated, and not any "User ID's".  (this happened in both QAS and PRD)

Sould this only pull the differences User MSKeys that don't have the system priv?

I have also created the following job in our QAS environment and am wondering if you could look at it and give me a thumbs up or down?

IF this is right and what I am seeing in my first question is acceptable then how would I just test this with one users ID that I know is not correct?

@Matt  -    Do you have a link to a script that would perform that?     I would also have the same question as to whether I can valdiate it with a single user ID first before I go and run it against the whole system.

[former_member2987 · Accepted solution]

Michael,

The script would be custom made, yes you can restrict to test with a single user by changing your source query.

Matt

[Former Member]

It worked like a charm!!!

I went through a couple of users in Test mode to test the script and job, and was successful, and then ran them out of test mode and they worked perfectly Steffi.      I am now going to tweak this a little to do the opposite as you suggested also.

Thanks alot Steffi and Matt for your excellent input!

[Steffi_Warnecke]

Yeah, great! Stuff likes this is kind of nerve-wrecking, but you can learn a lot by working through it. At least that is my experience.

You're very welcome, Michael! Team work is awesome.  

[former_member2987]

Michael,

Taking a look at the original post I see that you reference that the Modify Job is not running.  I'm assuming that one has been assigned to the system and that it runs (enabled, dispatcher assigned)

I'm also wondering if it happens for all users or just some / none?

It's starting to sound like Steffi's idea of assigning the roles back to the user make the most sense.  If you try one of the affected users and re add the only and system roles, does it work? If only one is assigned you might need to drop it before readding both.

Just a couple of thoughts.

Matt

[Former Member]

Matt,

Yes the modify job is running and is assigned to a dispatcher, and it is only happening to a few users who I suspect were some of our pre-loads prior to IDM being live.  (I loaded them via LSMW script at go-live)

If I remove the users from the role and then go back and delete the users accounts in the backend system, then they provision correctly after I re-add the role...I was just hoping to get ahead of the curve by finding out all of the users that might be in this "unknown" state to our Production ECC system.

If I can get a working script that truely identifies those users that are missing the PRIV:SYSTEM:ECC role in the IDStore and the job that Steffi showed me corrects that, then I think my problems on THIS issue will be resolved.    

I have about 5 others I need to work through after this that I will probably post later, and most I think I know why they are failing and what changes I have to make, but want to confirm my assumptions with admins more knowledgable than me before I make a change and accidently break something.

[Steffi_Warnecke]

We're here to help and brainstorm. ^^

Since I happen to be on a IDM training this week (that's the reason I'm so slow at answering, sorry about that), I asked the trainer today if the ONLY- and SYSTEM-privilege can be assigned to roles and privileges, too. To this day I thought, they are only assigned to users, but alas... no. Since the IDM needs to know to which repository they belong to and need to be provisioned to if there are changes to them in IDM, they can have those main privileges, too. At least that is what I was told.

I'd love to here from , if this is indeed the case.

Either way, I think adding to the SQL statement another line to filter just the identities, that have the entry type "MX_PERSON" should help here.

@Michael: You could test the job by just filling the MSKEYVALUE of one of the failing users (like the one in your start post) directly in the destination tab and leaving the source tab empty for now, since you have the complete SQL statement to find all the users that are missing the system privilege.

[Former Member]

ok so I updated your script to the following Steffi and I have gotten what I needed.   I will not try the job with one of the user ID's in PRD and see if it updates as needed and get back to this post.

distinct mcThisMSKEYVALUE FROM idmv_link_ext2 where mcOtherMSKEYVALUE='PRIV:ECC:ONLY' and mcThisOcName='MX_PERSON'

excecpt

distinct mcThisMSKEYVALUE FROM idmv_link_ext2 where mcOtherMSKEYVALUE='PRIV:SYSTEM:ECC' AND mcThisOcName='MX_PERSON'

[former_member2987]

HI

Who is doing the training?

Matt

[Steffi_Warnecke]

It was one of SAP's own IDM consultants, a Mister Benjamin Schmidt.

[former_member2987]

Steffi,

Cool.  How was it?  Did you take the general IDM training or the version 8 Delta training?

Matt

[Former Member]

Hi Mike ,

Can you please clarify how Master Privilege concept was designed in your Organization . SQL Query mentioned above provided a list of PROFILES or ROLES as OUTPUT , while output should be USER ID or User MSKEYVALUE .

Can you please try below mentioned SQL to extract user with the difference :

SELECT A.mcthismskeyvalue  FROM IDMV_LINK_EXT_ACTIVE A, IDMV_LINK_EXT_ACTIVE B

where

    A.mcothermskeyvalue='PRIV:Replace with SystemID:ONLY'

    AND B.MCOTHERMSKEYVALUE='PRIV:SYSTEM:Replace with SystemID'

    AND A.MCTHISOCNAME='MX_PERSON'

    AND B.MCTHISOCNAME='MX_PERSON' 

    AND A.MCTHISMSKEY=B.MCTHISMSKEY

    AND A.MCTHISMSKEY IS NOT NULL;

Is the design is something like , PRIV:<SYSTEM>:ONLY is added to all Generic ROLES OR PROFILES ? .Above Query will provide Users which has ONLY Privilege assigned & System Privilege missing . As explained by Steffi above , you can write a simple job to do a reconcilation .

Hope it helps ,

Thanks ,

Jerry George

[Former Member]

For our organization we are doing the following:

1. HCM feed comes in and creates a user account (if new hire) and assignes a business role called "everybody".   That everybody role has an privalege that updates the 105 record of the user with system UME ID and email which is required by other automated processes.   The role also contains the PRIV:ECC:ONLY privilage because all users loaded into our environment (whether using SAP or not) must have a valid UME account to populate the 105.

so by default every user on hire gets everybody which provides them ecc ume account.

2. depending on if the user requires access in our SAP environment they are assigned to a positon in HCM which is associated with an IDM business role.    Each buisness role contains the required ECC priv to grant them their required authorizations, as well as any other system only privs if that role requires access to others beyond our primary ECC client.

exm:    Role A is a Finance user so their Business Role contains

Priv:CompRoleNameFinanceUserA

Priv:GRC:Only  (as all finance users may need to request FFID or get approvals via GRC for other FI access not provided in their primary composite role granted

Priv:GRCENDUSERROLES

Priv:ECC:TRN  (we have a training client that users can log onto to with managment to train on which will have the exact same access they have in PRD but fake data)

PRIV:CompRoleNameFinanceUserA

As you can see we don't have the PRIV:ECC:ONLY in the Business role because it was already in the Everybody Role which was assigned at Hire, and Since a users association to a position in our ORG is a manual process completely by HR, there was never a problem with a positional Business Role triggering before the new hire everybody role assignment which runs in the middle of the night.

The problem for our org came about, because the project team for our ECC FI/PLM go live completed before the IDM HCM implementation and we had to load the users first via a combination of BODS and LSMW script.      Those user loads were not done all at the same time and not by the same person so when IDM did come on line and we did a synch of our single ECC environment to the IDStore and triggered our initial HCM job....it created a mess...not to mention I think some of the IDM jobs created by the 2 or 3 contractors that came through trying to stand the system up all tried to accomplish what we wanted but via different ways...(some more successful then others)

So now I am having mutliple issues in my provisioning process that I am slowly learning are all related to data inconsistancies in what the ID Store sees and what is truely in the backend system.      (which is why I am trying to following the SCN article that walks us through the RDS repository synch data file jobs so I can see how bad it is and then hopefully find enough information on SCN to fix all of my different data problems using jobs).

For now though this issue is a blatent one that I hope is an easy fix.

@Jerry    I tried you script directly in SQL against our QAS environment so that I could validate it did find only users that were missing the "System:only" priv and yours seemed to return all users with both privs  (I spot checked 5 users and all seemed good)

here are my screen shots:

your script with updated environment variables added:

Check of one of the 341 returned users from above script using following command:

select * from [QN2_db].[dbo].[idmv_link_ext2] where mcThisMSKEYVALUE='e0081004'

As you can see he contains both which means he is a working user...where as I need to only find users who are missing one or the other, and remediate.

Sorry in advance for long winded email, I appreciate the effort in helping me work through this, as I am learning more and more everyday.   (didn't know a single SQL command before someone thorugh me into IDM) 

[Steffi_Warnecke]

Hello Michael,

in your jobs in the destination-tab you need to set "Entry type" to "MX_PERSON". The dropdown, that's empty right now.

About your query-result:. Maybe it's the late hour here, but I'm a bit confused how this result came up. Maybe it helps to add another line that checks, if the mcthismskeyvalue belongs to an entry that has the type "MX_PERSON", so it just gives back persons.

I'll check it tomorrow in my system to see, what's what.

[Steffi_Warnecke]

Hello Michael,

welcome in the scavenger hunt that is a already set up IDM system.

You could check, if the users with this wrong trigger have the system-privilege and only-privilege of the repository, that is throwing the error. I think it was the "only"-privilege, that you can only (no pun intended) find in the database through an sql-query. It won't show in the UI.