How to split a big file into several small files using XI

[Former Member]

Hi,

Is there any way to split a huge file into small files using XI.

Thanks

Mukesh

[justin_santhanam]

Mukesh,

What is the source file format, is it csv or xml?

Best regards,

raj.

[Former Member]

It can be a xml file or CSV file.

[Former Member]

Use file adapter ,..Easy option

java can do anything you want;)

regards

krishna

[Former Member]

Thanks.

Can you please explain how to use java in XI to get this done.

[justin_santhanam]

Mukesh,

If its CSV file, using file adapter and split the message before its coming in , but I really don't know its possible for XML using file adapter, I thought the other option would be using BPM to split the message and send it to target, but I'm sure it doesn't solve your issue, before the message is reaching XI u have to split the file am I right?

@Krishna: if it's XML file how you will split the file, could you explain, please. You want me to use any modules?

Best regards,

raj.

[Former Member]

There is a how to guide for using file adapters, an "sdn search" will get you the document.

Based on that , read a file into XI, Use strings to store the file content and do the split

here is some code to get you started

===========

Pseudocode:

===========

// This can be passed in - however many output files to which the source is split

numOutputFiles = 5;

// Create the same number of filehandles as numOutputFiles specifies

open file1, file2, file3, file4, file5;

// Create an Array to hold the references to those filehandles

Array[5] fileHandles = {file1, file2, file3, file5, file5};

// Initialize a loop counter

int loopCounter = 0;

// a temporary holder to "point" the output stream to the correct output file

File currentOutputFile = null;

// loop through your input file

while (sourceFile.nextLine != null)

{

loopCounter++;

if (loopCounter == (numOutputFiles +1) // you've reached 5, loop back

{

loopCounter = 1;

}

currentOutputFile = fileHandles[loopCounter]; // gets the output file at that index of the array

currentOutputFile.write(sourceFile.nextLine);

}

regards

krishna

[Former Member]

currently i am working on 2 scenarios, both CSV and XML.

I was trying to process 80 MB CSV file using file adapter.

I have mentioned 1000 in Recordsets per message just to process 1000 records at a time.

But the server crashed as it is not able to process that big file. I think file adapter loads the whole file into memory not 1000 records from file.

So i really don't think it works. So we have to split the file before file adapter picks it up.

I think it clears my requirement.

Thanks

Mukesh

[Former Member]

>> think file adapter loads the whole file into memory not 1000 records from file.

Exactly. You need to use scripts to split your file. In File adapter specify the name of the script file in the Run OS command before processing.

Regards,

Jai Shankar

[Former Member]

Thanks Jai.

Can you please give me the sample OS code for file splitting.

[prateek]

/people/michal.krawczyk2/blog/2005/08/17/xi-operation-system-command--error-catching

Some approaches

/people/alessandro.guarneri/blog/2006/03/05/managing-bulky-flat-messages-with-sap-xi-tunneling-once-again--updated

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

Regards,

Prateek

[Former Member]

Have alook at this weblog by Michal to understand this.

/people/michal.krawczyk2/blog/2007/02/08/xipi-command-line-sample-functions

BTW, I have no knowlege of writing scripts. But you can get them a lot by a simple googling.

Regards,

Jai Shankar

[Former Member]

Hi Mukesh,

Could you be more specific about the XI version and patch where this issue occured. I am also trying to implement this on our dev server with 200 MB file. I want to see if this issue is related to a particular XI version and has been resolved or fixed in the latest version.

Thanks

Sarvesh