two million records!

[Erdal_Şimşek]

Hi@all,

I am doing a migation where we are dealing with millions of records!!!

How is MDM able to handle this huge amount? The MDM server is really allocating up to 20 GB during import (which is excel and 65535 records each file)! right now I have already imported around 2,000,000 records.

The MDM server would collapse and be unstable. Any idea how to adjust system settings so that the performance turns into normal?

Thanks a lot!!!

BR,

Erdal

[Former Member]

Hi Erdal,

IN addition to the suggestions given in the above posts, I would like to give you a workaround.

Instaed of loading the IMport Manager with 2 millions records at a time, I would suggest you to break your import fiel into smaller portions having less number of records.

It would be much easier and faster to import and would not load the server.

You can use a saved map and then import the files one after the other to get all the 2 million records imported into MDM. We had done the same for better efficiency in our project.

Hope it helps.

• Kindly reward points if helpful.

Thanks and Regards

Nitin jain

[Former Member]

Hi Erdal,

I do not know the exact specifications needed so as to make the sytem performance good.

But howevr I can guide you on few steps if taken carefully will definitely impact MDm performance.

- When you deals with millions of records.It is essential that your data model design is correct and effective if not it can lower your performance terribly.

- By system performance I mean The Number of Fields in your repository,Number of fields as Display Fields,Number of feilds marked as Keyword search,Number of feilds marked As Sort index.

- As i have noticed that even for thousands of records if There are many display fields it affects the performance to a level where ene loading and unloading your repository will atke several minutes.

- So I suggest you minimize the number of fields for the baove mentioned properties and will help you give a better performance

- Besides this you can also increase the chunksize of importing in the mdis .ini file if you are perfoming automatic importing.

Follow the below links which will guide you on how the improve MDM performance futher

Hope It Helped,

Kindly Reward Points if found useful

Thanks & Regards

Simona Pinto

[Former Member]

Erdal, How much of your data is intended for the Main table, or as Qualified links?

You'd have to generate a ton of lookup records (flat or qualified) to get up to a 20GB cache. Are your Display Fields really deep?

We've found that for one-time bulk migrations via the Import Manager, MS-Access is the most efficient data source - everything just works faster (including the record matching step)! Not the most convenient (XML, Flat files, Oracle etc would be nicer), but it sure helps a large migration.

2 million records in the Main table, 17 million qualified links, on about 16GB of server side memory.

YMMV,

...Dee

[Former Member]

I am not sure what you mean by the server is allocating 20GB during import.

What is the spec of your MDM server? Are you loading via MDIS or Import manager?

We recently did a benchmark test for the following:

1) 2 Server setup

2) MDM Server: CPU 3.16 GHZ, RAM 16.0 GB, 500 GB Disk space

3) MDIS&DBMS: CPU 3.16 GHZ, RAM 8.0 GB, 500 GB Disk space

4) SQL Server DBMS, Window OS.

5) Standard Content Model for Materials

6) Loading Data for Global Master Data

7) Flat text file

😎 Port congif for MDIS auto load.

The results in short:

3 million records loaded, Avg. 1 million records in 1 hour. Max 15% CPU usage during loading.

For volume loading I would recommend trying to standardised a text files generation process and use the MDIS for an Automated or semi-Automated loading/reloading process to MDM

Rgs

Con