Large file handling issue

[Former Member]

Hi All,

We are working on file a file scenarios and currently testing in the QA environment.

The interface cud handle smaller files (2MB -10MB) successfully.Files are getting picked from the source system and are delivered in the target location.

But When we use larger files (200MB) the interface picks the file from the source directory but is unable to deliver to the target location.<u><b>It throws a system error.</b></u>

Can the group explain how to handle large file.What parameters and settings needs to be changed.

Thanks,

Nandini

[Former Member]

Nandini,

Check out this blog....

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

Nilesh

[Former Member]

Hi Nandini !

We went thru the same. We used a splitting mechanism to convert large files into smaller files and process them as if they where independent.

You can use the "Recordsets Per Message" parameter in the File Adapter sender comm channel by using File Content Conversion, to create a new message for each 1000,10000,etc records in the source file...this could be a way of splitting.

If you scenario works ok for small files, maybe you can develop another scenario that runs previous the current one, that only splits files (e.g. File-XI-File) and then puts the output smaller files in the directory that your current scenario monitors.

Regards,

Matias.

[Former Member]

Wow! 200 mb, that's insane. What is your scenario, what adapters are being used?

[Former Member]

We are movind data from peoplesoft to file based third party system.

We are usinf file adapter on both the sides.

Thanks,

Nandini

[Former Member]

Krishna,

Please check this weblog and try to read the file in chunks of data and see whether it works for you or not:

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

Regards,

---Satish

[Former Member]

Matias,

Thanks for your reply.Your way around looks interesting.

1. We have single source and multiple receivers.When we split the large file into 10 different files will the mapping and transformation work good ?? Please elaborate more on this.

2.But is there any server configuration that cud be changed on either the Java or Abap stacks to handle large files.

Thanks,

Nandini

[Former Member]

Hi Nandini !

1. It all depends on the business logic...if your input file has only one kind of record, there should be no problem..if by the other way, you have header and/or footer..then you will need to use some previous processing that recreates the header and/or footer in each small piece to maintain consistency, or for example if you have any processing that involves data of more than one record to create the output. If you have only one kind of recordset structure, then it probably will not be a problem...the scenario will work as if your input were really 10 smaller files..instead of 1 large 200mb file..you will see 10 different messages in sxmb_moni, each one with its processing status.

If your scenario is file-xi-file, then maybe you should activate the "append" creation mode on the file adapter received comm channel, to "reassemble" the data chunks back to the large file on receiving system..you also will need to activate Exactly Once In Order QOS to ensure that the smaller data chunks are reconstructed in order on the output side (if it matters to the business logic).

2. As far as I know, there is not such a parameter to handle that kind of size.

Regards,

Matias.

[Former Member]

Matis,Satish and Nilesh your replies help us handle our situation.

Big Thanks,

Nandini

[Former Member]

Hi Nandini !!

Great...just one more thing..you will need activate "File Name" in the Adapter Specific Message Attributes of the sender and receiver file adapter channels to ensure that the source file name is used in the target file name reconstruction via append.

Regards,

Matias

[Former Member]

Hi Nandini !!

Check this:

Regards,

Matias.