Import Server Performance

[Former Member]

Hi,

I need to find out about MDM Import Server performance and how the settings to the port and import server configuration affect it.

In the Console for port specification there is the one attribute that can affect performance:

- File Threshold Count

In the Import Server I can find the following attributes:

- Chunk Size

- Interval

- No. Of Chunks Proccessed In Parallel

I would be glad if somebody with practical experience would share his knowledge regarding this issue.

Thanks & Best Regards,

Ingo

[Former Member]

Does File Threshold Count influence the performance?

It is used for batch processing of multiple files in the inbound port.

Thanks,

Best Regards,

Ingo

[Former Member]

Hi Taraske,

Yes you are right, its batch processing.

and It will decrease your system performance while importing data using import server as the no of threshold increases.

Although we dont have any performance monitoring tool in MDM SP05, soon we will see it in MDM SP06.

Rewards if found helpful.

BR,

Alok Sharma

[Former Member]

Hi,

I'm asking for performance tweaks because the import performance is not quite what we expected it would be:

MDM Import Server 5.5.42.92

MDM Server 5.5.42.92

Computer:

Microsoft Windows Server 2003 R2

Enterprise X64 Edition

Service Pack 2

IBM XSeries 3950M2

Intel Xeon XPU 3,16 Ghz

16 GB RAM

Microsoft SQL 2005

The interface we use is MATMAS05. We are importing one record per file. Repository has about 27 fields in main table, 3 Qualified Tables, one 15 fields, the other 2 about 8 fields.

Average import time per record: 2 Minutes

=> 30 Records per hour

Best Regards,

Ingo

[Former Member]

Hi Ingo,

Yes Ingo, it is batch processing and will decrease the performance when you increase the no of threshold.

you go through these---->

In certain scenarios, it may be more efficient for MDIS to import files from a port in batches rather than as individual files. For example, if you are generating large numbers of small import files (containing one or two records each), MDIS can import these files faster by processing them collectively than it could by processing each file separately.

Files which contain large numbers of records are most efficiently processed as individual files. Ports likely to receive such files should not be set up for aggregation.

The File Threshold Count property, located on the MDM Console's Port Detail pane, instructs MDIS to import files in batches of the number specified by the property. If the number of files in a folder is ever less than the specified count, MDIS simply imports the files as their own batch without waiting for more files to be added.

Optimal setting depends on the available memory and processing power of the machine running MDIS.

Other things for increases the MDIS performance:

• Setting port type as XML Schema instead of XML.

• Using file aggregation when importing large numbers of small files.

• Installing MDIS on a separate machine from the Import Manager or MDM Server.

• Not running any other import, syndication, archive, or data editing operations while a mass import is occurring, as each of these operations locks use of the repository from all other operations.

Imp:Using file aggregation to import files containing more than just a few records each can actually decrease overall performance due to the memory required to process large numbers of records.

I hope it will help you a lot.

Regards,

Saurabh

Reward with points if found helpful.

Edited by: Saurabh Kumar Sahu on Jan 24, 2008 9:39 AM

[Former Member]

Hi Saurabh,

• port type is XML Schema; interface is MATMAS05

• used file aggregation threshold = 20, Import Server crashed after importing the first batch. We are using a small workflow on import assigning values for two fields, it's stable when not using batch processing. Now I got back to threshold = 1 and no crashes occur.

• MDIS runs on the same machine

• Currently in development we don't touch MDM during the import. In a Multi-User environment you can't exclude this though. Just imagine users working with data in MDM while every morning a delta of 200 new records comes in.

With the setting described in my last post we get one record imported every 2 Minutes.

Best Regards,

Ingo

Looking at the MDIS log file I can see that the xSaxParser and xStructuralTransformer need a long time to process:

[MDS=SAPMDMD Repos=MATERIAL ClientSystem=SUPPLIER Port=SUPPLIER_ZMDM_MATMAS_IN]: xSaxParser: Thread finished; Start -> End: 40.328125000 seconds.

3640 2008/01/24 06:51:01.445 GMT [MDS=SAPMDMD Repos=MATERIAL ClientSystem=SUPPLIER Port=SUPPLIER_ZMDM_MATMAS_IN]: xStructuralTransformer: Thread finished; Start -> End: 48.562500000 seconds.

mdis.ini:

[GLOBAL]

String Resource Dir=C:\PROGRA_1\SAPMDM1.5\IMPORT~1\LangStrings\

Log Dir=C:\PROGRA_1\SAPMDM1.5\IMPORT~1\Logs

Server=SAPMDMD

SLD Registration=False

Interval=10

MapScanTopToBottom=False

Verbose=0

[MATERIALSAPMDMDMSQL_8_7_4_3]

Chunk Size=50000

Interval=1

No. Of Chunks Proccessed In Parallel=10

Login=Admin

PasswordE=xx

Log Protocol Transactions=False

Edited by: Ingo Taraske on Jan 24, 2008 10:04 AM

[Former Member]

Hi Ingo,

Check the Import Server Logs. You find the log folders next to the port folder. Check if there is a error description visible. Additionaly you can increase the "Verbose" mode in MDIS.ini. Set it to FFFF to get a full logging and error reporting.

try to see in the log file, if there is any log regarding the Importing of data.

or just try with the configuration mentioned below:

MATERIALSAPMDMDMSQL_8_7_4_3

Chunk Size=50000(Just reduce this size)

Interval=1(Increase the interval as 30 as SAP recomamded)

No. Of Chunks Proccessed In Parallel=10(Reduce this upto 5)

and delete all the log files of MDIS from your MDM server directory.as these consumes the memory.

else you consult with the SAP regarding the more performance improvement.

Hope this will help you.

Regards,

Saurabh....

Edited by: Saurabh Kumar Sahu on Jan 24, 2008 11:18 AM

[Former Member]

Hi Ingo,

As per you query---

1. Chunk Size and No. Of Chunks Proccessed In Parallel : The Chunk Size and No. of Chunks Processed In Parallel parameters in mdis.ini help optimize MDIS performance and reduce overall import times.

Default Value for chunk size as 50000 is Recommended Settings by SAP For Optimal MDIS Performance.

and for No. of Chunks Processed In Parallel--> 5-10

Minimum requirements = 4 GB available memory and dual-core processor. Increase

value from 5 to 10 in proportion to actual system specifications.

The larger the chunk size, the more memory is consumed by MDIS. For this reason, is best to start with the default value.

2. Interval: MDIS processes all files in a port's Ready folder before scanning the

next port in the sequence. When more than one import file is present in

the folder, MDIS processes the files in a FIFO (first in, first out) order,

meaning the oldest file in the port is processed first, then the next

oldest.

Once all ports on the MDM Server have been scanned, MDIS waits the

number of seconds specified in the Interval property of the mdis.ini

file before restarting the sequence.

I hope this will help you.

Regards,

Saurabh..

Reward with points if found helpful.

Edited by: Saurabh Kumar Sahu on Jan 23, 2008 5:02 PM

[Former Member]

I noticed better performance when I increased the Chunk Size and reduced the No. of Chunks Processed in Parallel. I am working on a Sun OS server.

BR