timing stats for Import Manager / Import server

[Former Member]

For Customer master data, say an XML has 5K records (size is roughly 100-200 MB) we are logging atleast 20 mins to complete all the steps via Import manager. I wonder if anyone of you have the timing stats handy to tell if this is norm? I know it depends a lot on sizing an other stuff but wonder what the rest of the world is looking at? And how good is import server at this job? the logs are too heavy just to open up in IE or console(70 MB atleast)

Any shots at this ...highly apreciated!

-Sudhir

[Former Member · Accepted solution]

Hi Sudhir,

The timings your Importing is taking for a set of 5K records is normal.

Even I tried with a set of around 6K and it took me around half an hour on an average.

So i think its behaviour is normal.

Hope It Helped,

Thanks & Regards

Simona Pinto

[Former Member · Accepted solution]

Hi Simona,

Did you try the 6K via Import server? was that way fast than IM. what was the chunk size you have used, is it a good idea to make chunk size 3000 and have tqo chunks to run in parallel?

-Sudhir,

I tried chunk size 50000 with no much difference in time!

[Former Member · Accepted solution]

Hi Sudhir,

I did the importing through the Import manager,it was basically a test carried out to measure the performance in terms of Speed of processing.

Using Import server will definitely have a faster response time as compared using the Import manager for manual importing.

Regarding the chunk size, do amke a note that more records you include in the chunk the less will be the overhead/record however thsi will increase the wait time.Because MDIS process the MDM import chunks in queued manner so each chunk passes through one process on teh completion of which the next chunk follows up.

Time interval in the MDIS just marks the time after which the import server starts polling the ports reday folder for any source file to import,It should ideally not cause any type of exception.

If you are facing a structural exception then check for the following:

- Open the Source XML and the destination XML schema and verfy every line to check for any mismatch in terms of Field Names used in source and in the MDM rep.

- Order of the fields in both the structures

- Also check to see if any value mapping or field mapping is missing by performing a manual import.

Hope It Helped,

Thanks & Regards

Simona Pinto

[Former Member]

Hi Sudhir,

Jitesh is right, its better to use import server to import such huge records but in that you have to be carefull .i.e. time interval should be proper and large enough so it will complete importing in exact time mentioned in MDIS, other wise it will throw structural exception.

Regards,

Neethu Joy.

[Former Member]

Why will Time Interval throw a structural Exception, I am assuming Time INterval is like polling interval, and it should result in much load but not any exception right?

-Sudhir

[Former Member]

Hi Sudhir,

Its better to use Import Server if you want to import huge number of records. Also there is restrcition on Import Manager that it cannot handle file containing more than 50000 records and on the other hand Import Server is capable to handle files with huge records since it internally splits the file and then import Chunk by Chunk.

In your case its better to use Import Server because it works in the background so no user intervention is required.

Regards,

Jitesh Talreja