Remove Duplicate records -File to File

[Former Member]

Hi,

I am working on File to File scenario...I need to remove the duplicate records from sender side and send it across the receiver side...please find the below structure.

Can anybody...give me the suggestion, how can we implement this?!

Source

<Records>

<Row>

<ID>10001</ID>

<Name>Praveen</Name>

</Row>

<Row>

<ID>11901</ID>

<Name>Raj</Name>

</Row>

<Row>

<ID>10001</ID>

<Name>Praveen</Name>

</Row>

</Records>

Target

<Records>

<Row>

<ID>10001</ID>

<Name>Praveen</Name>

</Row>

<Row>

<ID>11901</ID>

<Name>Raj</Name>

</Row>

</Records>

Since ID 10001 Duplicated twice, I need to remove this...

-Esha

[former_member182455 · Accepted solution]

hi esha,

Write a Module for Sender CC where the duplicate records shud not be considred

if u use udf check this udf

vVector vProd = new Vector();

for (int i =0;i.<a.length;i++){

if(!vProd.contains(a)) vProd.add(a);

}

Collections.sort(vProd);

for(i=0;i<vProd.size();i++){

result.addValue((String)vProd.get(i));

}

input is String[] a

Check the following link for reference...

regards

srinivas

[Former Member · Accepted solution]

Hi,

I haven't succeed with this UDF...and I am new to Java....If anybody can give me the adapter module I would appreciate with points.

Or else can anybody suggest me to do with node functions!!!

Can anybody help me on this.

-Esha

[Former Member]

you can follow this approach:-

for target Records node

ID -> removecontext -> sort -> splitbyvalue(value changed)->collapsecontext

for ID field you can also you above mapping.

for name filed you can have same name for differnt ID. for this you can use concat function to concatenate ID and Name and use above mapping ex. and at the end you can right a simple UDF as follows.

you will have two inputs Name + ID using above mapping and Name.

use parseInt with lengthof function to calculate the no characters for Name and Name+ID.

int a = Interger.parseInt(Name.length);

int b = Interger.parseInt(Name+ID.length);

String s = Name+ID.substring(0,a);

return s;

This code is not checked but logic shuold be like that only.

[Former Member]

Hi,

This logic also couldn't help me out....

Can anybody share their experiences to solve this problem.

Esha

[former_member192295]

HI,

Best and simple option is develop UDF in java. If you dont no java language better to choose abap mapping, under execute Adjacent Duplicate statement. Automatically duplicate record will be remove from your list.

[GabrielSagaya]

You can use UDF to remove the duplicate id

id>myudf>id

function myudf(String a[], ResultList result, Container container)

{

Vector vProd = new Vector();

for (int i =0;i.<a.length;i++)

if(!vProd.contains(a))

vProd.add(a);

Collections.sort(vProd);

for(i=0;i<vProd.size();i++)

result.addValue((String)vProd.get(i));

}

Also visit the blog for better idea

/people/jeyakumar.muthu2/blog/2005/12/19/data-mining-using-apriori-algorithm-in-xi-150-part-ii

[Former Member]

Hi,

Can anybody help me on this!!!!!!!

Esha