MDM Repository gets hanged while executing Matching Strategy

[Former Member]

Hi,

we are using MDM API 2.0 and trying to execute the MDM Matching Strategy. There are 6500 records and 50000 operations in the matching strategy.

when we try to execute the strategy from MDM API , the repository gets hanged and some time we get "JAVA : Out of memory" exception.

Help us in resolving the issues !!!

Please let me know if you need further details.

Thanks

Prabhu.M...

[Former Member · Accepted solution]

I would suggest checking the match results retrieval process in the API program.

Java Out of memory exception might happen if large volume of records is retrieved at that process.

Regards

Orit

[Former Member · Accepted solution]

Thanks for the response.

We are trying to retrieve results of 20 coulmns from 6500 records. When we try with 15 columns , it works fine, but when we increase to 20 column, it hangs. We tried increasing our JVM memory, but we face the same problem.

Please help us in resolving the issue.

Regards

Prabhu.M...

[Former Member · Accepted solution]

Please check the number of records, and not the number of columns.

There are 2 nested iterations for retrieving matching results:

1)Uuter loop: RetrieveMatchingResultCommand is used to get the source record Ids

2)Inner loop: For each source record at the outer loop, RetrieveMatchedRecordsCommand is used to get the target records and their matching scores.

Please check what is the number of target records retrieved for each source record at the inner loop.

Only records with a score which is above the strategy low threshold should be retrieved with the API command.

Regards

Orit

[Former Member · Accepted solution]

Now the scenario is out of matching strategy.

When we try to retrieve using RetrieveLimitedRecordsCommand , we get the error. Any other suggestions?.

Thanks

[Former Member · Accepted solution]

Use the search criteria which is used at that command to narrow the number of retrieved record.

Regards

Orit

[Former Member]

Issue resolved thanks

[Former Member]

Also we use

NWDS 7.0.14 & MDM 5.5 SP 06.0.67

In one of my collegues project where they use the same matching logic they use

NWDS 7.0.14 & MDM 5.5 SP 06.3.57

Does this difference makes these kind of issues?

Also

u2022 When we try to execute matching strategy on less number of records, it works fine. However when the same is executed on huge number of records, both client system and server system stops responding. Below may be the root cause of this issue

Record Count Strategy type Number of matches performed Time to execute from MDM Data Manager Output

2029 Selected Vs Selected 2029*2029 = 4116841 3 to 4 Mins Perfect

6250 Selected Vs Selected 6250*6250 = 39062500 35 to 40 Mins Error

u2022 The code seems to be working fine, as the program works perfectly fine for 2029 records and when the number of records are increased to 6025, both client and server stops responding; when the matching strategy is executed.

Thanks

Prabhu.M...

[Former Member]

Hi Prabhu

This may be the reasons for server going down or repository not responding. please follow the link for optimizing the Matching strategy:

https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/408b8031-6fc4-2b10-c18b-a77abefb...

Also refer to the release notes of MDM version for next available patches. We can find lots of improvement on the matching performance.

best regards

Ravi Kumar

[Former Member]

Hi, I tried with the sap note you have mentioned.

In that I found a step as

RecordResultSet rrs = retrieveCommand.getRecords();

To get records we need to use execute command else it will return null. So when we try to execute again we get into loop that we need to set the page size. The same problem continues.

Any workaround ?

Thanks

Prabhu.M...

[Former Member]

Hi Ravi,

I feel there is a miscommunication from my end. Actually the total records are 6500 and the total duplicates count around 60.

[Former Member]

Sorry Prabhu for understanding wrongly.

In that case, I did some digging in SAP notes and could find note 1309363.

Please see if this helps.

regards

Ravi

[Former Member]

As a result of logic to retrieve only those records which are having score morre than a cutoff - records we received are 6500.

[Former Member]

Hi Prabhumsundar,

Kindly please paste the relevant code lines which were built to retrieve matching results + code explanations + specific MDM version.

Regards

Orit

[Former Member]

Dear Prabhu

This means that the logic may not be correct. Try changing the low threshold score for the matching strategy in MDM to a higher value. Showing 6500 potential duplicates to User on Portaldoesnot make any sense.

best regards

Ravi Kumar

[Former Member]

Hi , we have used the search criteria and problem still continues

[Former Member]

How many records are retrieved with RetrieveLimitedRecordsCommand ?

[Former Member]

6500 records with 20 columns

[Former Member]

OK, than consider changing the search criteria to have less records at the result.

[Former Member]

Hi Prabhu

why do you want to see 6500 records retrieved from matching? Put a logic to retrieve only those records which are having score morre than a cutoff.

regards

ravi

[Former Member]

Hi Prabhu,

If the matching stratergy is not defined properly or if the matching stratergy is defined such that all 6500 records get retrieved then there is a possibility of gettings such errors.

[Former Member]

Hi Prabhumsundar,

If your defined strategy is consisiting of many rules and transofmations (Token equaled), it is recommended to break your strategy into no. of accurate strategies.

Refer https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/408b8031-6fc4-2b10-c18b-a77abefb...

Hope it helps.

Thanks,

Minaz

[Former Member]

Hi Prabhu,

For executing matching startegy, you must ensure that the fields are indexed on which you are putting some rule. Also ensure that you have logically rules should fit into the Matching starategy. As you said it should not be a problem executing a matching strategy on just 6500 records, it should work without any performance issues. Check the Matching strategy and optimize it for better performance.

Regards,

Arafat

[Former Member]

Hi Prabhu

Matching strategy performance depends heavily on number of rules and transformations used.

Are you also using token equals for the rules? Try to reduce the number of token equals or transformations.

Otherwise when executing the matching strategy use clustering of records, that is run the matching for selected Vs results. You can firstdo a search based on some parameter like Country/region and then run the matching strategy.

regards

Ravi

[Former Member]

Hi,

When we try to execute the matching strategy from MDM Console, it goes fine without any problem. But when we execute through portal and when we terminate the process it hangs.

Any Suggestions?

[Former Member]

Hi Prabhu,

Please specify the strategy and matching rules you run with the API.

Do you use matching transformations for the rules ?

At which API command does the "out of memory" error appear ?

What is MDM version ?

Regards

Orit