How to break file into N parts

[Former Member]

Hi All,

For performance tunning purpose, sometimes we need to break big files into small parts before it reaches XI.

Could you tell me how to break a file?

How many parts to break into for best performance

How to handle the broken files at the receiver end?

Regards,

Ashish

[agasthuri_doss · Accepted solution]

Hi,

From sender we can break the files into small size and sent to SAP XI system. for this shell script will help.

Regards

Agasthuri Doss

[Former Member · Accepted solution]

Guys could you please give me the shell script code? and how will these broken files be read then by XI? will I need a single sender adapter pulling files from a directory? and how wouls sender adapter know in what seq it is to puck the files?

Regards,

Ashish

[agasthuri_doss · Accepted solution]

Hi,

This thread will help you

Regards

Agasthuri Doss

[Former Member · Accepted solution]

Ashish,

We dont need to write any script for this. Just check this weblog on how to do this in XI itself:

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

Regards,

---Satish

[justin_santhanam · Accepted solution]

Satish,

I've doubt in that weblog. So u say like if we mention 5000 it will process only 5000 records,even I too agree. But all the file contents first will be loaded into Memory then only the process starts, is it so? Please correct me if i'm wrong. According to my understanding , all the file content will be stored in Memory and only 5000 records will be passed to the Integration Server from AE at a time. Wht do us say?

Best regards,

raj.

[Former Member · Accepted solution]

Raj,

Yes you are correct. If we give this first of all it dont read the entire file. It counts for the no. of records. Till the no. of records equals to 5000 it keeps in memory and then it writes a file. Once again it fetches for the next 5000.

But it is still better than loading the whole file at a time. Sometimes XI breaks down also if it is having high volume of data in the file.

Regards,

---Satish

[Former Member · Accepted solution]

Thanks sathish

[justin_santhanam · Accepted solution]

Satish,

Thanks for the reply. Are u sure that the whole file won't be heaped into memory. I won't think so Satish. If i'm not wrong it will take the whole file into memory then it will send 5000 records at a time.

Ok let we figure it out with an simple example, since I'm at home I can't try it , probably tom I'll try in office. In Content Coversion consider we gave the field names as f1,f2,f3 and if the file has more than 3 fields it will throw an error am I right?

In the No.of records we will give as 10 , and we will give the correct file format for the first 10 records and give some junk data in the 11th record. So the first 10 records will be processed succesfully, the second bunch of records will throw an exception is it?

In the above example if the first 10 records processed successfully then XI is not loading all the file content into memory , its reading only 10 records, if it throws an error initial stage itself its loading all the content into XI?

What do u say Satish?

Best regards,

raj.

[Former Member · Accepted solution]

Raj,

It is not like that. It will first read the 10 records in one shot and the 11 th record in the second. It will create 2 files one with the first 10 records and the other with one record.

Regards,

---Satish

[Former Member · Accepted solution]

Hey satish

so in that case if there is any error in 11th record,then will XI rollback the posting of first 10 records?

Thanx

Ahmad

[satish_chauhan · Accepted solution]

Ahmad,

It will create proper sets and post message to Integration engine and will stop the set/record that has content conversion error.

The further action delete or archive of file will not happen beacuse of this error and you can get alerts if you have configurted.

cheers,

satish

[Former Member]

Hey Ashish,

IF YOU ARE USING LATEST SERVICE PACK OR A PI 7.0 THEN PROBABLY YOU CAN USE THIS

Advanced Mode

To specify additional parameters in the adapter configuration, set the Advanced Mode indicator.

&#9679; <b>Maximum File Size</b>

Specify the maximum file size in bytes.

The adapter only processes files that are smaller than the specified size. An error occurs if files are any larger.

Check out

<u></u><a href="http://help.sap.com/saphelp_nw04/helpdata/en/2c/181077dd7d6b4ea6a8029b20bf7e55/frameset.htm">Sender File Adapter</a>

Hope it was usefull.

Cheers,

<b>RAJ*

*REWARD POINTS IF FOUND USEFULL*</b>

[Former Member]

Hey

you can write a small shell script to break the sender File into multiple chunks,there is no set rule on the size of these small chunks,it depends upon your hardware configuration,and on the receiver side,you can use the append mode to collect all these files again.

If the receiver adapter is such that you can't use append mode,then design a BPM to join all these files together

Thanx

Ahmad