Missing Master and Derived Roles

[Former Member]

Hello All,

I have got an odd scenario and I am hoping some of you might have run into the same issue or might point me to the right direction.

Back ground

We are on ECC 5.0 and have Master Derived Concept, and then Derived Roles are grouped in Composites

We recently( Last week ) created some ( say 34 ) Derived roles and some (10) composites using a combinition of the newly created derived and some Old derived roles.

Transported The derived seperatly and Composites seperately. Transports went successfully into QA and PRD.

This week we noticed that all of the 34 derived roles are missing in DEV ONLY along with 28 Master of the 34 Child Roles. All the Childs and master still exist in QA and PRD.

We have tried to look up the change Doc of the missing roles or the profiles or the authorizations of the missing roles and there is no change log under suim. Change Log shows when the role was created but nothing after that. According to Basis transports does not have any unusual log

Since its a DEV system so no delete transports have come into DEV, therefore delete transport could not be an option.

I have also uploaded one of the missing master roles from the PRD to DEV and it is succfully established the relation with the childs. I was hoping it might shake up the Change History regarding missing role but it did not, It now shows when the role was created earlier( 2006 ) and This week agian but no Delete History

Any Ideas on how to explain this behavior

[Former Member]

SUIM sometimes will not show all the change history on the roles. Try this instead PFCG->Environment->Display Changes "enter your missing derive roles"->From Date->under Change Documents select "Create and delete roles" then execute. This will display when the roles were deleted and who deleted them. This will be a good start to investigating your issue.

Good Luck!

[Former Member]

John,

I have tried that and still no Luck

[Former Member]

I'm curious to see the roles included on your initial transport to QA and PROD. In SE10 look up the transport number and under ROLE does it show the missing roles.

[Former Member]

The Transport and all its Roles are intact.... and I have already checked that if any of the roles was transported after mine and the answer is no, last time it was in any transport it was mine which went into Prd

If I also look into AGR_Define for the Child Roles for which master was deleted, The entries for the master are there... I can not draw any conclusiong from it but this is an FYI

Edited by: Afzaal Hassan on Oct 12, 2010 8:23 PM

[Former Member]

Hi Afzaal

It sounds like everything you did was standard process, creating derived singles and adding to composites etc.

Is there any way that the DEV client has been copied back/refreshed from a backup which might explain why you don't have any history apart from role creation (say, before the date of the backup?) or were the roles created in a different DEV client 001 instead of 010 or similar?

It's a long shot - I have to admit that I know bugger all about basis so this is just a suggestion.

Good luck

David

Edited by: David Berry on Oct 12, 2010 8:21 PM

[Former Member]

No there is no Dev Refersh.

Even if there was a Dev refresh from QA or PRD then it should not make these roles missing like I mentioned that roles are in QA and PRD, especially Master ones and they have been in the system for quite a while. In addition why only these specific roles. I could be wrong but I have not seen any Dev refreshes for the few clients I have worked with

The roles were only created in abc client and transported from there, our other dev clients dont have any business roles, niether are in the transport route of the transport from abc

Edited by: Afzaal Hassan on Oct 12, 2010 9:29 PM

[Former Member]

Hi

Then let's treat this a one off gremlin

Download from qas upload to dev and keep an eye on the roles

[Former Member]

You are right, thats is the only way to fix it( Download from Q and Upload to D) even though I have some seen some comments from few people to be cautious of this approach,

I will wait for couple of days to see if anybody else has other thoughts

[Former Member]

Make sure that you match the profile names up during the upload or you will have issues down the road when you transport the roles again.

[Former Member]

Hi Melissa

Looking at the techniques applied so far I don't expect any issues on profiles - can't see Afzaal creating in a non DEV system but good advice.

Cheers

David

[Former Member]

Thanks David/Melissa/Julius/Alex.. really appreciate your thought full analysis and comments

We still have the OSS message open with SAP, Lets c if they can conlcude something. We have turned on the audit logs for some agr tables, that way we will have more informartion to look at in case it happens again. If anything interesting come up in the next few days which will lead us to the root cause I will share it with you.

[Former Member]

Another possible and imaginable human error worth looking into is that at some stage in the past a transport request was created for the master and child roles -- okay.

Then the child roles were "broken" by changing org. levels and other fields in the authorization maintenance, so the roles themselves were deleted with the intention of creating them again from one of the "template" child-roles --> okay, seems reasonable to have happened.

Then (here is the problem!) someone released the transport before the new child roles were created. This is interpreted by the system to be a deletion transport of roles.

Additionally the sequence of the transports might have added additional obscurity to the issue and now, much later on, someone imported the transport into production which deleted the roles.

<conspiracy_theory>

The person then deleted the transport request from the queues and archived the change documents in SU83.

</conspiracy_theory>

Cheers,

Julius

[Former Member]

Julius,

Its a poosibility in general but not applicable since the child roles were created the very FIRST Time last week and after generating the profiles we put them in the transport.

I like the theory but I am not going to share with others they might get me in trouble if I keep alluding to them that its a thoughtfull human error I think its Aliens...they are coming.................

[Former Member]

the roles were created under which language, please check

just a guess, that can sometime cause problem

[Former Member]

English All the way

[Former Member]

Hmmm... go to SE11 and display the table AGR_DEFINE. Then right-click the table's name and choose Where-Used-List.

Are there any Z-programs there? If so, does any of them mention the work "delete" in the program name or scan then with report RS_ABAP_SOURCE_SCAN for the keyword "delete".

Cheers,

Julius

[Former Member]

Yeh, We have got couple of Z programs but none of them have delete statement to delete entries from AGR_DEFINE... Since we have GRC, there are lot of GRC program shows up in the list and so far SAP is suspecting that GRC program might have deleted the roles but no reason why, I guess its our Job to figure it out

[Former Member]

Hi,

Your Basis guys should be able to identify the deletes at the table level and the ID that did them (this is independent of table logging and has much less detail). It's a bit of a blunt tool but should be able to give you a bit more info than you have.

[Former Member]

Alex,

I think I am not quite clear what u exactly mean by identifying the deletes at the table level.

1) Like most of us know there are SUIM Logs ( as mentioned before no logs there for this behavior )

2) We could also turn on the logs on the table level in SE11 and that is not turned on for AGR_DEFINE or AGR_PROF ( have not looked others ) so can not use that either

Is there a third way that you r refering to here?

[Former Member]

The PFCG functions called by GRC ERM are not released (externally) and do not respect the enqueue / dequeue process ( no locks are set or checked in SM12) so all sorts of strange things can happen if you use both at the same time.

Cheers,

Julius

[Former Member]

Julius,

In GRC tool we are only using RAR, SPM and HR trigger for CUP. We are not using ERM

Afzaal

[Former Member]

Hi Afzaal

I didn't allow for GRC (possibly ERM), I know you don't think ERM is being used but is there a possibility that there is some experimentation going on that you aren't aware of? The GRC implementation can start off just with RAR then CUP and, once that is looking stable, ERM.

Maybe you need to just double check what Alex and Julius have mentioned just in case this happens again?

Changes in SUIM do reflect user changes/batch jobs etc but things like transports being re-transported (and pixie dust GRC?) may not show up. The changes that Basis can get reports on go way below the SAP reports - they are, as advised, your next best option before restoring from Q.

Ask them - they won't mind and you've ticked off another check for future audit.

Cheers

David

Edited by: David Berry on Oct 13, 2010 8:39 PM

[Former Member]

David,

I have asked our GRC team and the rest of the team and nobody has lately or before experimented with ERM and neither there are any connectors created in ERM.

I have also asked our Basis team and they are only aware of the two logging menthods that I mentioned earlier so not sure if Alex was refering to any third way. They are aware of the issue and are looking into it but so far no luck.

We think we might have narrowed down the incident date. Since we have implemented RAR (Risk Analysis Remediation). We run DAILY jobs to sync any role changes in Dev. When the Job run on abc time, log shows that RAR could not find the Deleted Roles and hence Update its tables to reflect the changes, while the Job run fine the day before

[Former Member]

I'm not a Basis expert and I forget now how they did it but I have got this info from Basis in the past where they examine the DB records for the deletions. All I can tell you is that it's done through SAP and is independent of change records or logging.

We had a situation where a large number of records were deleted from a specific table. They could give us the ID and the name of the program that performed the action. I wish I had paid more attention to what they did but as you can appreciate, I had bigger things on my plate at that time. I am 100% certain it is standard SAP though, possibly through ST03.

[Former Member]

Try ST11 (developer trace) or ST06 (invalidations).

Cheers,

Julius

[Former Member]

Afzal,

Can you please try to confirm with your team if they had tried to delete any connector in RAR recently.

Or had they actually deleted a connector after running sql query.

regards,

Surpreet

[Former Member]

Surpreet,

If your recent means since last week when the incident happended, the answer is no

Assume if they did, how does it relate to our missing roles