Import Server (MDIS) parallel processing
We are running import server and each night it processes MATMAS XML files from R/3 into our main MDM repository. I have been working to tune this process with some good results, but have some questions regarding processing file chunks in parallel. I have lowered the chunk size (in MDIS.ini) to 12000 and set the number of chunks that can be processed in parallel to 7. Here is the snippet from mdis.ini :
No. Of Chunks Proccessed In Parallel=7
I know the "proccessed" isn't spelled right, but MDM put it there and I know MDIS is finding it because at the top of the import server log is the following (and I tried correcting the spelling without any change!) :
[ISKU_PROD\] Import Task Started. Chunk size \[12000\], No. parallel chunks\[7\]
Now I know that most of our files have multiple chunks since these MATMAS files can be quite large. Here is an example of a section of import log.
1756 2008/09/15 05:44:32.494 Timer: name: Import Records - Stage 1 - Prepare Import Records total ms: 18892.865415 6
1756 2008/09/15 05:44:37.962 Timer: name: Import Records - Stage 2 - Filter/Merge total ms: 5406.211170 6
1756 2008/09/15 05:44:38.697 Timer: name: Import Records - Stage 3 [Time spent on MDS ] total ms: 729.913273 6
1756 2008/09/15 05:44:38.697 Timer: name: Import Records - Stage 4 - exception generation total ms: 0.004424 6
1756 2008/09/15 05:44:38.697 Import action: Skip: 0 Create: 0 Updated (NULL fields only): 0 Updated (all mapped fields): 12000 Replace: 0 Delete [destination]: 0
1756 2008/09/15 05:44:38.728 Timer: name: Import Chunk total ms: 81269.620471
The '6' at the end of each line means that this refers to chunk number 6, and you can see this chunk contained 12000 records. Now for the question. Despite being told to process 7 chunks in parallel, each chunk is processed one after the other, ie sequentially and using the same thread. The import log shows quite clearing 1 chunk being processed after another.
I've been digging around in the documenation and the OSS notes but can't find anything to indicate why. I read the notes on streaming, but I don't think they apply since I'm only importing one file at a time. And it clearly knows the file has multiple chunks.
For reference, import server is running on a 4 CPU machine with 4GB of memory running Windows 2003 server (32-bit).
All suggestions gratefully received!