Remove duplicate records from two different files

[Former Member]

Hi Experts,

I have to send two files from one FTP server to another, but could be the same data record included in both files, I am just wondering can PI remove the duplicates from TWO different files.

Thanks a lot

Blue

[justin_santhanam · Accepted solution]

You can do it with PO (PI+ BPM)... Collect the files in BPM and do the mapping to remove the duplicates.. I'm just speaking out loud here, but need to see how your data looks like and see how complicate the mapping is going to be.. Collecting is not a problem here. Just give an example of data format you receive in both the files and illustrate what you mean by duplicate records. Then we can give an input based on your exact requirement.

There are other options as well, but requires more development effort (custom)..

Thank you,

Justin.

[Former Member · Accepted solution]

Thank you very much Justin!

The two input files are in different format, such as the first file has 20 data fields, the second one has 400 data fields, the two files should be joined together with the duplicates removed and the output file contains the records which have 400 data fields each.

Do you think PO (PI+BPM) still can achieve this? If so, do you mind give me a rough steps on how to implement it?

Thanks again

Blue

[justin_santhanam · Accepted solution]

Blue,

Collecting the file in BPM is not a problem.. I have did the similar requirement recently , collecting files with two different formats. But my question still revolves around the removal of duplication.. I think it's definitely achievable, but I'm little bit reserve on the removal of duplicate records (without knowing what we are dealing with). Once you start the development I can give more technical info.

(http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/90308598-4b8d-2f10-4a9a-b78973859...)

This is how BPM Pattern looks like to collect the files.

Thank you,

Justin

[Former Member · Accepted solution]

Thank you so much Justin!

I will definitely keep you posted once I start the development work.

Blue

[justin_santhanam · Accepted solution]

You most welcome!!

Cheers,

Justin.

[Former Member · Accepted solution]

Hi Justin,

Just realized we are SAP PI 7.30 on dual stack, can I still using BPM/PO?

Thanks

Blue

[justin_santhanam · Accepted solution]

Hi Blue,

Check with your team to see if they have BPM instance .. If they don't then you can still make use of ccBPM to do the collect pattern. I was thinking if you had BPM instance /license then you can develop in BPM, so that in future if you migrate to single stack you don't have to re-develop this. If not you have to do in ccBPM and later re-develop in BPM (if you migrate to single stack).

Thank you,

Justin.