Performance impact of syndicating large data files from MDM

[Former Member]

Hi,

We need a daily syndication of CSV file from MDM. The file might contain around 14 K records and the file size will be around 3 MB. We are using PI as middleware to transform the syndicated data(XML message) to the target CSV file.

Questions:

1. What will be performance impact of syndicating huge data files?

2. Is there any metrics or statistics to indicate the response time for syndicating large data files from your past experience?

Please give your valuable suggestions.

Thanks,

Surendra

[Former Member]

Hi Surendra,

We need a daily syndication of CSV file from MDM. The file might contain around 14 K records and the file size will be around 3 MB. We are using PI as middleware to transform the syndicated data(XML message) to the target CSV file.

Questions:

1. What will be performance impact of syndicating huge data files?

2. Is there any metrics or statistics to indicate the response time for syndicating large data files from your past experience?

I would like to give few suggestion from my past experience :

1) Use the feature "Suppress Unchanges Records" - YES, in the Map properties

This would suppress all those records which have not been changed at all from the previous time of syndication. And hence the records to be syndicated will come down to certain Level.

2) Also the performance also depends upon the number of fields getting syndicated.If the Number of fields is less, then even 14K records will not drag down the performance so much.

Otherwise, the performance will be definitely impacted if the number of fields and records are large. It will take quite a time to perform this activity.

Hence I would advice you to keep all these little things in mind and carefully create the syndication map.

Hope it helps.

Thanks and Regards

Nitin jain

[Former Member]

Hi Surendra,

- The syndication of Huge data files using automatic syndication will take a lot of time if you are using a set of 14000 records

- A performance effective step would be if you can break down your records set into a set of 5k each by using some search mechanism in syndicator which will pick up a smaller set of rec everytime .this will increase the speed of the operation of syndication

- Also MDM is cabable enough the syndicate in .csv file it self so you can elminate teh use of PI is the scenario and thus reduce the complexity and overhead on the system.

- Also make sure you have set the option in ports for block on exception in console.so that inconsistent records may not proceed syndication and be alerted at time.

Hope It Helped,

Thanks & Regards

Simona Pinto

[Former Member]

Hi Simona,

You had mentioned about breaking the records into chunks of 5K in the response.

I would like to know, if the search on MDM repository takes time or the syndication of the data?

Thanks

[Former Member]

Hi Surendra,

Seraching capability in MDM is faster,So it will not take you much time to filter your records uisng a search criteria.

For example if you are maintaining material records say and there are some 5k records that can be filtered out using a search on say a particular material type or material group.

Then you can use this saerch in MDM syndicator and even save it if want to be reused later.This process should not take more than 5 to 8 min max.

then you can syndicate the subset you have obtained.

This could be one of the way around to improve the performance time.

Hope It Helped,

Thanks & Regards

Simona Pinto