MDM Data Migration

[Former Member]

I have a current system where I load data from one R3 environement to another by extracting the data with an R3 DCOM connector via ASP pages, dump the data to SQL for manipulation, then submit back to SAP via the DCOM.

We are upgrading and implimenting MDM. I will need to move data into MDM and was wondering if I could migrate from SQL to MDM using the same or similar method and avoid XI and the Import manager. Is there a COM connector or API availeable to gain access into MDM?

Thanks in advance.

-Motley Drew

[Former Member · Accepted solution]

Andrew,

MDM comes with COM API also along with Java API.

ABAP API with a very limited functionality is also available to a very limited customers as of now.

Regards,

Rajani Kumar

[Former Member]

Hi Andrew,

In general, your engineering team is correct in that XI may not be suitable for mass loading of data into MDM. Of course, this also depends on the hardware (mainly cpu and memory) and the amount of data to load.

Aside from the hardware, there are 4 main factors in mass loading that can effect the appropriateness of using XI:

1. Timeframe required for the load.

2. Number of messages (or IDOCs in this case) to be loaded.

3. Size of each message.

4. Complexity of mapping requirement.

The size of the message (in XML format) and the mapping complexity will determine the J2EE memory requirement for each message. Especially, when the message is very large and the mapping is poorly developed, the memory requirement for each message can increase in size several times.

The number of messages and the time required to load will also determine memory requirement. If a limited time is required to mass load the messages, then data will have to be loaded concurrently. The amount of concurrency will determine memory requirement. Also, keep in mind there might be other messages being processed on the system.

Hundreds of thousands of message is definitely manageable by XI. In many mass loadings, the messages are usually in the hundreds of millions. In your case, I think it would be worth a look. The mass load processing can also be used for future incremental loading through XI. So, your work will not be in vain.

If timeframe is not critical, then it is possible the messages can be processed in sequence. This can avoid the memory requirement of processing multiple messages concurrently. Provided, of course, that there is enough time.

I would propose the following:

1. Design and configure the end-to-end processing of the message for the load.

IDOC -> XI (w/mapping) -> RFC/IDOC/ABAP Proxy

2. Measure the time required to execute 1 message. Do this several times; the 1st message will take longer due to initialization.

3. Calculate if this will meet your timeframe if processed sequentially. If so, then the problem is solved, and you can set up the message processing as EOIO (exactly once in order).

4. If not, then process the messages concurrently by sending as many as you can at one time. Make sure the J2EE, ABAP work dialog processes, XI queues, etc. are configured properly. Improperly tuned XI system can negatively affect performance. The XI basis person should be able to do this.

More likely, concurrent processing will be required. If a timeframe can be arranged to allocate XI exclusively for mass loading and the XI system is properly tuned, I think loading hundreds of thousands of messages will not be a problem.

The QuickSizer can help to give an "estimate" of the CPU and memory requirement of the hardware.

Regards,

Bill

[Former Member]

Thank you very much for the replies. I have a few more questions though.

I realize the DCOM connector is older technology, but we have an SAP.NET connector available to replace that piece if needed. The reason we needed to manipulate the data in SQL in the past when moving data from one pipe to another is because of configuration issues. Example, date formats needed to be submitted one way due to the config, but they were pulled another way, so we had to run scripts on the data to fix issues like that.

I'm getting the impression that XI is the way to go in terms data integrity and simplicity, but our engineering team will not allow us to process bulk conversions through XI because they say it won't handle large transfers. Also, they are saying that SAP clearly states in their documentation that “XI is not a replacement for MASS load utilities available within SAP”. Do we know of a mass load utility that works with R3 and MDM? We may potentially have 100's of 1,000's of records to migrate. For instance, Material Master data. We are using a R3 46B system and were thinking about using BD10 to send IDOCs via XI to MDM for initial load. We don’t know how XI will handle this initial load. Has anyone had any experience processing large amounts of data through XI? Is the right approach to use XI for this initial load, or should I use SAP .NET connector and a SQL database?

[Former Member]

Hello Andrew:

The DCOM conector is rather old technology and is not available for use with MDM.

MDM has a JAVA API which is ready available for development.

However, unless you have performed some important manipulation to the data in the SQL database, I would recomend you to use the XI middleware and to extract everything directly from R/3.

The reason for this is that this way you can use standard business content for importing maps in MDM as well as conversion maps in XI, so you don't need to develop anything from scratch, just set up everything in the standard way and do no programing job.

Hope this helps.

Regards,

Jorge.