Data migration between systems

[Former Member]

Hi All,

Can anybody tell the methods for uploading of data for the first time

from remote systems to MDM?

Please also circulate some links or documents if any.

Regards,

Pramod

[Former Member]

Hi Pramod,

A general rule of thumb for Migrating data into MDM is that while you upload data into MDM the validation part should be Minimum so that you have the traceability.

Let me expalin with a scenario.

Legacy -> Excel -> MDM.

After you extract the data from a legacy system do the conversion/validation logic in the Middle layer (here its excel).Then Upload mass data ,this is the most effective way.This ensures that the data which resides in MDM is of good quality.You are not adding junk into your Master Repository.

Reg your question on the Landscape of Server, I would suggest you onething.

While you do the Upload in Dev Box first, you will be saving the import maps .donot use these import maps in Quality or prod box.Do a fresh upload in Quality and Production because there will be changes in the Repository often, so the structure of repository wont be the same as your import maps.

Thaks,

Jerome Felix

[Former Member]

Hi Pramod,

Please follow the steps:

1. Create a relevant repository in MDM which will accomodate the fields for the source systems whose value you want to get in MDM..

2. Identify the source systems and create the Remote systems Under Admin node in Console

3. Create Ports( as type Inbound) for this case as you want to get the data into MDM for each Remote system

4. This will aumatically create a folder in the MDM Server folder which will have a Inbound Ready folder for each .

5.. Now as Simona has menitoned that you will require the source file from the various source system.which you will have to place in the Inbound ready folder to import it.

Now as per your question, it would be always better if you clean the data in the Excel sheet itself. Excel itself has many feature incorporated in it which can be used for cleaning data to some extent.

However, if not possible you can always make use of the MDM features such as Validations, Matching Merging etc. to clean and de-duplicate the data after imported it into MDM.

Hope it helps.

Thanks and Regards

Nitin Jain

[Former Member]

Hi Pramod,

Uploading of data into MDM depends on the remote system in use.

- For eg you are importing data from R/3 or ECC the you will have to use MDMGX transcation to upload the lookup data in to MDM from Ecc and the MDM_Clnt_Extr for the Main Table Extraction

- You can foolw these links for the same

MDMGX

https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/f0740b31-a934-2b10-2e95-af225299...

MDM_Clnt_Extr

https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/50260ae5-5589-2910-4791-fc091d9b...

Hope It Helped,

Thanks & Regards

Simona Pinto

[Former Member]

Hi Simona,

Thanks for your quick reply.

I am uploading data for the first time in the form of Excel sheet.

What would you suggest me:

1. Upload the data as is & cleanse it in regular intervals.

2. Upload the data & run validations on the uploaded data.

It is a three system landscape.

Regards,

Pramod

[Former Member]

Hi Pramod,

Source data will be having invalid and inconsistent master data .It is the functionality of MDM to cleanse the Data.

I would suggest you that

- Take the one time data into MDM by using the extraction mechanism.

- And then cleanse it in MDM using the matching merging feature

- MDM is not destined to just perform the duplication identification but also enrichment and governance of data

- After the duplicates are eliminated in MDM you can always avoid bad data from creation in the future in the source sytsem itself

- Then you must go for a CMDM situation where hence forth cleanse data is created within MDM itself and MDM acts as the data hub

- You can use the MDM workflows for data governace

Hope It Helped,

Thanks & Regards

Simona Pinto

[Former Member]

Hi Simona,

I have huge amount of data ( might be 50k records).

Should i use extensive validations or it should be minimal during upload?

Please share any documents or links on these.

Please let me know if there are any other methods for initial upload.

Regards,

Pramod

[Former Member]

Hi Pramod,

If the data is huge, then you must break down the file into smaller files and then import those smaller files into MDM one by one. This will definitely improve performance of your system.

Now coming to using Validations. You do not have to run any validation while importing. After importing all the records into MDM one by one, you can run Validation on them and make your data consistent accordingly

I have done this scenario myself by breaking the file into smaller sizes of 10K each to get better performance.

You can go though the linke below to get more ideas on other methods of importing.

Hope it helps.

Thanks and Regards

Nitin Jain

[Former Member]

Hi Pramod,

As I mentioned in my earlier post All the data Cleansing and governing can be done effectively in MDM using the MDM workflows

- In this wf we can use the validations as well which will validate the records on different conditions

- Data validation will run only after upload and not during upload.(unless you are using a third party tool for validations)

- So even if you have a million records in MDM you can run the mdm validations on them with ease.

- With workflows Validation can be run automatically on records and we can use Roles and users who will be authorised to approve these recrds or dissapprove them

- From the performance aspect importing large data can be a load so you can break the records into sets as mentioned by nitin .

You can follow the below links on how the MDM workflows works

https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/60f28084-b90e-2b10-3eb6-d6565367...

https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/001d22fb-622d-2b10-22ac-998acecf...

MDM Validations

(mdm validations)

(mdm validations)

[Former Member]

Hi Simona,

I have Development, QA & production systems in place & the time gap between to start work on

each systems is a month.

How to proceed?

Regards,

Pramod

[Former Member]

Hi Pramod,

This is generally the way the project develops in stages and it is no different for master data team.

So these stages will be there.

The Recommented way to proceed is:

- Start working with the development server.

- The data maodel may change from time to time depending on the busines requirement

- So you must have th back up of your existing data model always

- For this You must archive the dev rep and keep for future ref

- You can also use the master slave rep when you migrate to the quality system where your master rep will be the dev rep and the slave the quality rep you can make changes to the dev system and then synchronize the slave to it.

- You can also use the export import schema at any time when required to replicate the data structure of a rep at any level.

- Always client will work with test data upto the dev sytem even sometime still the quality level as we are dealing with the master data

- So client will not migrate the real master data into MDM untill the data modelling is up to mark and bug free.

- You must contduc rigourous testing till the quality level and only after approval move to the production system

- As changes are not expected on reaching data state but only the data flow must happen

Hope It Helped,

Thanks & Regards

Simona Pinto

[Former Member]

Hello Pramod,

You can refer the link as mentioned below:

Transportation of MDM objects within MDM5.5 Sytems:

https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/20751e78-f81f-2a10-228f-eb3a1642...

I hope this will solve your req.

TNR,

Saurabh...

[Former Member]

Hi Pramod,

I have Development, QA & production systems in place & the time gap between to start work on

each systems is a month.

How to proceed?

I had a similar requirement in a project.

1) You will have to start working upon the deveopment server. You will have to model the repository based on the data model.In the course of this, you will have to import the data into the Lookup tables.

2) You will then import the MainTable data into MDM.

3)There is a high probability that data model keeps on changing with time. Hence you need to Take care of the IMPACT of all the changes in the data model in other processes before actually implementing it.

3) Once the deveopment is ready, you will have to desgin Test scripts and test the devepments done and check whether the deveopment is sutiing your requirement.

4) Once the repository is ready, you can take an archive of the repository and load it on the QA server. Ideally, you should load the data again and check whether it is runningg fine.

It will be done easily within 1 month if the prior tasks are done properly.

5) You also need to take care of the performance related from the begining itself.

6) Similar steps have to be carried while shifting the repoisotry from Quality to production.

Please note that the repository model should be synchronous in all the 3 repositories.

Hope it helps.

Thanks and Regards

Nitin jain