Auto syndication for 5M records - MDSS fails

[Former Member]

Hi All,

As we need to extract our complete repository data into flat structure, we have created flat syndication map and scheduled for auto syndication, but the process didn't complete and threw an error log " Could not export repository data. RC = 0x8002001a, RepTag = X, MDS host = X, Agency = MDM, ID = 15, type = PER "

Total Number of records: 25K

Total number of fields: 322

With Tuple (one-to-many): ~200 rows for each record / so the output expected 5M records

• Would XML be the right option? instead of flat?

• or anything else?

your thoughts will be greatly appreciated.

thanks,

Alexander.

[Former Member · Accepted solution]

Hi,

For one time syndication of large number of records, try using SAP MDM Syndicator manually instead of the automated way using SAP MDM Syndication Server.

Once the large number of records are syndicated out then you can use automated syndication for the delta records using "Suppress Unchanged Records" putting less load on the syndication server.

Read this point from the standard SAP Syndicator Reference Guide - http://help.sap.com/saphelp_mdm550/helpdata/en/ff/37e4c6857449388ddb6c77f160ac4f/frameset.htm

"A syndication file may become too big for MDSS to send over a network to its proper port folder. Or the file may simply take a long time to process, preventing MDSS from syndicating data to other ports on a timely basis. Both problems can be particularly severe when you are syndicating data from a repository for the first time.

The remedy for this problem is to run the large syndication job once manually from the Syndicator. Then, enable the Suppress Unchanged Records option on the mapu2019s Map Properties tab. This should result in much smaller syndication files being generated in the future, which MDSS is well-suited to process."

Thanks,

Shambhu.

[Former Member]

HI Alex,

Please follow the SAP NOTE - 1608472.This should solve your problem.

Apart from this I think Shambhu's inputs really makes sense.

Since your data load is high on syndication server it makes sense to optimize syndication parameters too.

Please go through this link for further details:

http://help.sap.com/saphelp_mdm550/helpdata/en/93/e0cc3ac77c4319ad70ef1da0f969b9/frameset.htm

Thanks,

Ravi

[Former Member]

Experts, thanks a lot for all your inputs. We kind of did the stress test and MDSS cannot handle if the record output is huge. ( in our case 25K records expands into 5M because of one to many relations). We will consider either chunk the outputs by suitable parameter or send the delta instead of everything.

thanks,

Alexander.

[Former Member]

Hi Alex,

Yes, I feel that XML would be the right option for this scenario. See, I am not sure about the maximum no. of records you can syndicate in one go without any error but with using XML option you can set the maximum number of records to include in an XML syndication file, after which a new file is created. By this I mean you distribute the data into small form of chunks in case of 5 M records. So it would be quite handy if you set Property Max Records Per Output File = 10,000 under The Map Properties Tab of syndicator. So your 5 M records exported through MDM Syndicator with each new file created with 10,000. Thus, MDM Syndicator appends incrementing numbers to the name of each new file created (for example, Filename[1].xml, Filename[2].xml and so-on). So in this way I surely feel your auto syndication for 5 M records will not fail and you keep getting no. of records with 10,000 in each file till all the records not get syndicated.

Note: Please try the same property Max Records Per Output File = 10,000 under The Map Properties Tab for flat file, may be it works for flat file too. So in this way if it works for you, you can continue working with flat files

Kindly revert with result.

Regards,

Mandeep Saini