File processing thousands of records

[Former Member]

Hello experts,

Scenario - File with thousands of records(Ex 50-100 thousands) and processing thru XI to post IDOC.

Q1. Does it support this much volume ?

Q2. Does it take as one payload ? if yes, if there is mapping error when processing one of the record does it file to process remain ? Does it reject to process entirely or skip that record go thru successive records?

Q3. How to debug this situation as it is one payload? Q4. How to handle errors to address this situations?

[Former Member · Accepted solution]

>>Q1. Does it support this much volume ?

yes

>>Q2. Does it take as one payload ? if yes, if there is >>mapping error when processing one of the record does it >>file to process remain ? Does it reject to process >>entirely or skip that record go thru successive records?

>>Q3. How to debug this situation as it is one payload? >>Q4. How to handle errors to address this situations?

XI will pickup all the records in a single scan. if an error occurs with one of the records, it will drop everything. Out of the box, it cannot skip the errored record and go on. But u can write a module for handling all the errors before in the file adapter.

http://help.sap.com/saphelp_nw04/helpdata/en/cd/5af7c0c994e24fb0d0088443513de2/content.htm

cheers,

Naveen

Q3. How to debug this situation as it is one payload? Q4. How to handle errors to address this situations?

[prabhu_s2]

Hi Kumar

I was searching for some help on the scenario which is much the same as yours. I do have larget volume of data to be retireved. If you have completed this scenario, can you pls let me know the steps involved in processing such larget volumes from a DB.

Thkx

Prabhu

[Former Member]

Hi,

If you are sending the data in a package of 5000 then the data that has been sent will go to SAP system and cannot b e blocked at XI level.

If the BPM goes into error due to huge data nad u are able to restart the process then the data that has been sent will not be sent again.

The only problem here is if u are not able to restart the BPM and u need to process the whole file again then u will be resending the data that has already been sent.

Regards

Vijaya

[moorthy]

Hi Kumar,

<i>***Q1. Does it support this much volume ?</i>Yes, depends on many parameters-

1) Memory - Sizing of hardware

2) Logic used to process the file ( for eg. write a script which will split the file etc)

3) If multiple files at a time having huge volume of data, you can go with EOIO method

If there is a mapping error then you will not get any output. It will not reject only errored records. These you can do making use some logic in the Mapping and BPM etc.

You can trigger Alert from Adapter Engine,Integration Server etc ( from SP14 onwards) Or you can write a logic , in such a way that you can send an email on error. etc.

R/

Moorthy

[former_member91687]

Hi,

Take a look at sravya's blog:

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

Regards,

Chandra

[Former Member]

Thanks for all ur replies. but my question is

if the file is huge, it has to be split using content conversion, (blog says some one did by 5000 records),if this is the case -

i have file with 200,000 records, xi splits for every 5000 (with content conv paramater), and if it failed in the next 5000 records, does it processed first 5000 records into IDOC already? or complete file got rejected?

Since each 5000 records becomes seperate payload? it would cause problem if the file got huge records and few payloads(split of 5000) processed and next few paloads not processed because in between one of the payload has mapping or some problem?

I understand config,heap etc contribute the performance, consider those are take care, i am trying to understand XI capabilities on handling huge file, and how the error handling works if the one of the record got error ?

Thanks,

ravi

Thanks,

[Former Member]

Hi Kumar,

1. The volume supported depends on your XI config.

2. It creates one payload per file when picked up. If you are using IDoc package then it will create one payload while sending the data to SAP. If you are using BPM for processsing one IDoc at a time then it will create seperate payload.

3. If the error is in mapping then it will block the whole file.

4. TO handle errors you can use Alert.

Regards

Vijaya