removing error employee record for huge file

[Former Member]

We have a scenario where we are receiving huge file(140 MB file) with some 25000 employee record.

Due to data issue in one of the  employee record our whole file is getting failed in PI.

Even in mapping log it is saying error on last employee(could not able to generate target element)  rather than actual employee which has error data.

what could be possible solution to remove faulty employee from the file and process rest of it. we thought of below option but not sure if these are good idea.

1. Do data validation at mapping level: this is not feasible as there are lot of business rule and can put all of them in pi to validate xml. also we are already using 2-3 mapping to simplify our business logic implementation.

2. using XML validation: As I know this can only validate xml scheme. it won’t validate the data, such as employee is not an active employee and was terminated in past(employment end date< current date).

what could be other possible solution?

Thanks,

Navneet

[azharshaikh]

Hi Navneet,

Is the message being posted into ECC..can u let us know the message flow and adapters used..

If its Inbound Proxy, you can skip the validation in PI and let Proxy code handle it...if you validate in PI it will fail the entire message incase the validation fails

Regards,

Azhar

[Former Member]

It is  file to  file scenario. PI received xml file from 3rd part and do logic conversion and generate a csv file which is again sent to 3rd party