problem - Bulk Loading from sql server using import manager

[Former Member]

Hi MDM Gurus

I am trying to import 260,000 records from MS Sql Server DB to my MDM Server Repository. The MDM Server is running on a 1GB RAM Machine.

While importing the records it has thrown a n out of memory Error.

How do i get it solved?

Is there any other best practice to mass upload Data in MDM Server.

Should i import it in batches?

Vijayasararthy

[Former Member · Accepted solution]

Hi Vijayasararthy,

Some of the options would be:

1. Increase RAM to 2GB or more (may not fully solve the problem)

2. Divide 260 K records into small batches, may be 50 K or so and import

HTH

--

Venkat

[Former Member]

Hi all,

I tried importing 150,000 records using the import manager and it successfully imported.

Thanks to all

[Former Member]

closing the thraed

[Former Member]

Hi Vijayasararthy,

I am facing the same problem, could you please tell me

How many hours does the loading of 150,000 records cost?

How powerful is the compter you use for that?

How complicated is your data structure? e.g. if there any lookup qualified table and so you need to do create combined field by creating mapping?

thanks & regards,

Bin

[Former Member]

Hi,

I think using the Import Manager Batch is advisable as it is designed for mass uploads and can be run in the background. However, I am sceptical whether or not the IM Batch will give the same problem. Please check and do reply.

I suggest another approach - why dont u create a workflow with foll properties:

Trigger Action: Record Import

Max records: >260,000

I am not sure aboout the success of this approach. It may again give the same error. But it just might work. Do let me the result.

Regards,

Siddharth

[Former Member]

Hi I tried it with IMBatch still the same result.

Not yet tried the workflow approach.

Let me see.

[Former Member]

Hi Vijay,

Let me know the following constraint is set for ur case:

" MDM Import Manager does not check any security constraints set in the MDM Console. Specifically, with the following exceptions:

(1) the import will fail if any record constraints are set for the import user; and

(2) you must have write access to the Family Hierarchy if the Synchronize Family Hierarchy After Import configuration option is set to Yes. "

hope this might solve ur problem.

thanks

Alexander

p.s. Kindly reward the points if found problem solved.

[Former Member]

Hi,

Importing of records also depends on the machine on which Import Manager is installed. Import Manager loads all the data in the input file into memore along with the data available in repository in order to do a comparision of data before saying so many number of records are already available in the repository.

Hence, check the system from which you are trying to load the data and try to follow the options mentioned by others.... I would say, split the data and import multiple times with the saved map.

Regards,

Rajani

[Former Member]

Hi Vijayasararthy !

Please keep in mind that Import manager is (also) allocating memory (huge amount) in order to do record matching and to get unique attribute values.

1. Not sure where you run import manager but consider to execute it on machine with more memory (maybe machine where mdm server is located will be nice idea).

2. you could also split import data horizontally or vertically - to load data in batches or load few times with different set of attributes (keeping same matching key)

3. try to use import batch or import server instead of using import manager (interactive gui)

4. (this one I prefer) load data using custom program that connects to repository with java-api.

Regards, marcin

ps. please also check the amount of memory that import manager / server / batch is allocating during import operation and consider using 64-bit version of system and mdm components.