Import of 500000 items with 4000 attributes!

[Former Member]

Hi,

we are challenged by the problem what is the best ad most performant way to import 500000 items with 4000 different attributes.

The approach was to go with the Import Manager and Excel Flat files. Here comes the Limitation of approx. 50000 lines and 100 columns. So this would mean in the end 400 excel files that need to be imported manually.

- Any recommandations how to handle this the best way?

- Is the automap functionality in the import Manager 100% reliable or do I have to map/check every atribute manually?

Answers will be rewarded.

BR

Nicolas

[Former Member · Accepted solution]

I did something similar with Item, only I used a SQL DB as my source. Import Manager was extremely slow. I had to process my files in chunks of 5,000 or less, because the more record you send, the longer each batch takes when they load in memory.

If you were to use Excel, you would be limited by the number of rows and columns you get.

You could use XI to get the data from the source system and pass to MDM's Import Server (not Import Manager). This automates the process so you don't have to manually process the data in small chunks.

Since I need my source system data in SQL for reconciliation after the load, I wrote a .NET application that reads the data in SQL, translates to XML and dumps the XML files on the XI share so MDM Import Server will pick them up. There is still the issue of file size, so I send 10,000 Item records per XML file. If you have 4,000 attributes, I would not attempt a chunk size that large to begin with. This is for initial bulk data loads only.

[Former Member · Accepted solution]

Hi Andrew,

we use now SQL DB as well because of the column limitation of Excel.

The problem is the performance. Just connection with the Import Manager to a SQL instance with one table takes very long (30 minutes).

But in the end this looks like the best solution.

Thanks for the tip anyway.

BR

Nicolas

[Former Member · Accepted solution]

Hi,

I have another question regarding this topic. I used now the SQL DB connection option. Automap the fields (attributes) worked for several 100 fields. But the value mapping stays blank and I have to manually map/add also the value mappings.

This is should not e since with the "automap" all fields I want to avoid this manuall work.

What is the setting for automatic adding of values when not existing? Not for the batch, I've found that setting. I need to know for the Import Manager. What setting makes automatic value mapping if a field was mapped.

Thanks

Nicolas

[Former Member]

May like to try with Access.

In this case you can accomodate all rows but I am not sure about 4000 attributes.

Ms Access loading data being simple , is worth trying

-reo

[Former Member]

One suggestion is to create an Oracle Table with the required fields and then use the import manager to load the data.