1 file with 25k records or 25k files?

[Former Member]

Hello,

I have the option of asking my source to send me either 1 file with 25k records or sending 25k records?

this is part of mass update.

my concern is, if the source sends 25k records in 1 file, and even if 1 record fails, will there be an update of other 24999 records? (in mass update what are the chances of an error record)

Also, if source sends me 25k files, i would be calling my proxy so many times :S

thanks,

Vishal

[Former Member · Accepted solution]

Hi,

Suggestion is to use 1 file with 25K record subject to optimum size 1MB (vary depending on your landscape). You can use BPM to avoid failure (Multicast pattern) of all records.

If you are using PI7.1 then suggestion is to use message packaging as well.

Regards,

Gourav

[Former Member · Accepted solution]

Message packaging does not do anything since you're only receiving a single message containing all records: you will only have one persistence step. BPM can be useful, but may make it more complex to handle. You will need a smart BPM pattern to handle the erroneous records, plus the sending application should be capable to resend a single record.

[Former Member · Accepted solution]

Hi,

I suggest message packaging as I was talking about multicast pattern (1 to n) so it will look like:

- Sender to BPM (1 message)

- BPM to Receiver (25K messages) (message packaging recommended here).

Multicast is standard pattern hence can be developed quickly by copying SAP pattern from SAP basis SWC.

Regards,

Gourav

[Former Member · Accepted solution]

no BPM involved...

i just have option of source sending 1 file or 25k files...

[Former Member · Accepted solution]

How big (in bytes) are the 25K record files?

[Former Member · Accepted solution]

Hi,

Send one file (subject to few MBs). even if something goes wrong it can be corrected manually, in case of 25K file it is going to be huge load on system.

Regards,

Gourav

[Former Member · Accepted solution]

say even if the file size is couple of mb's. am sure, performance would still be better than 25k files.

Nextly,

how do i correct this manually?

also, if am not wrong, if there is say a mapping failure, Xi would throw the error as "line itme missing" or somehting like that.

I would not get to know for which record (record 14567 had a mandatory field missing) the error has occureD?

thanks,

vishal

[Former Member · Accepted solution]

Hi,

In trace you will be able to see this (some hint where the error is), it is going to be pain to search and edit 25k records but you proposed very restrictive problems:

Two options I can suggest:

1. Do sanity check (write simple script which perform sanity check of record for you (atleast syntax), if you are using PI7.1 atleast xml validation can be done by communication channel else you need to write simple scripts.

2. If it is possible to club record in batch of 250 files (100 record each) then you can narrow error and rest of records wont be affected, If it is not possible in sender application see if it is possible to write java program to do this for you (clubbing multiple files into one file)

So your scenario will look like:

Sender->25k files->Folder->Java->250 files->XI->Receiver

Hope this helps you,

Regards,

Gourav

[Former Member · Accepted solution]

Configure the alerts for the same to get trigger the failure notificaiton of the same

It may help to certain extent

Rajesh

[Former Member · Accepted solution]

Vishal,

If you want to customize the errors you can see this alex blog and can come up with your own errors:

/people/alessandro.guarneri/blog/2006/01/26/throwing-smart-exceptions-in-xi-graphical-mapping

If something is wrong in one of the record in mapping then if you want to throw error then you have to handle this in mapping.

Else you can write your own module where you can first check the file and if everything is ok then you can pick the file else you can ignore the file. So for validating the file you can see this document:

/people/swaroopa.vishwanath/blog/2005/06/24/generic-approach-for-validating-incoming-flat-file-in-sap-xi--part-1

/people/swaroopa.vishwanath/blog/2005/06/29/generic-approach-for-validating-incoming-flat-file-in-sap-xi--part-ii

/people/ranjit.deshmukh/blog/2008/01/14/validations-using-java-for-those-not-having-pi71

If you want data validations see this thread:

Regards,

---Satish

[Former Member]

In my opinion its better to handle all the records i.e. 25 K in a single file as it will trigger one XI message and which consumes less resources of the same when compared to triggering the 25K files for processing the same

Normally if one record failes to update then mass update should result in error for all 25k records of a file

but I have not come across this

But better to have testcase for the same

Rajesh

[Former Member]

I agree with Rajesh: performance-wise it is better to have a 25K file. Normally spoken (in a well tested environment) the chances of a mapping error are very slim.