Splitting large message (60MB) based on payload data

[madanmohan_agrawal]

Hi,

I have a file (Flat) to file (xml) scenario. The source flat file is being read by FCC. Since the source flat file is large (upto 60MB) so I have to split it into small files then I have applied "recordset per messages" in FCC level to split large file into small ones. But the clients requirement is to split the large document based on payload data (that is DeliveryDate). That means I cannt split the message based on number of rows of flat file instead I have to split the file on the basis of DeliveryDate so that after splitting into small files, each small file should contain data for exactly one date (say in one file data for 15th NOV and in another file data for 16NOV and so on).

Please suggest some solution to split the large file (60MB) based on payload data(DeliveryDate).

Br,

Madan Agrawal

[madanmohan_agrawal]

?

[rajasekhar_reddy14]

Hi Madan,

in this case split the message in to different messages like 2 mb file,

XI doesn't support 60 mb files, U have to split the flat files based one some condition.

i have same requirement flat file having huge data i split that data using java map.]

then i processed its work fine for me.

I think u can also do it,

But in my case i divided message based on sequence number uniquenumber to diffentiate data.

if there is any sequence number split the message .

Regards,

Raj

[prateek]

For your requirement, you have to split the message in mapping. Perform multi-mapping without using BPM (1:N).

/people/jin.shin/blog/2006/02/07/multi-mapping-without-bpm--yes-it146s-possible

Regards,

Prateek

[madanmohan_agrawal]

Hi Prateek,

The source file is too large (60MB) and if I will not split the message at adapter level then it will throw JCo call error while transfering the large file from AE to IE. Please suggest some other solution.

Br,

Madan Agrawal

[prateek]

You have a requirement that requires splitting at Adapter engine level as well as mapping of entire data at Integration engine level. The only option to satisfy both seems to be BPM. First you split the records using "recordset per messages" = say 50 and then perform the transform step in BPM to merge the similar records in loop. This would be resource consuming but will allow to distribute the load on various engine instead of burdening one engine at a time.

Regards,

Prateek

[madanmohan_agrawal]

Thanks Prateek

but splitting, merging and again splittting in BPM will cosume more memory and time. I will definitely try this one. but s there any solution for the above? any work around?

Br,

Madan Agrawal

[prateek]

Other way would be handling the requirement at the sender side directly writing some OS level script to split the files based on you r requirement of date. You can't think of adapter module as it cannot create multiple messages after split.

Regards,

Prateek

[madanmohan_agrawal]

I think its a good idea!

In our case, OS is WINDOWS so does the script (DOS) will be able to read and divide the file?

or do we have to use ABAP code (batch) for the same?

Br,

Madan Agrawal.