how to handle large files that need payload access & content conversion

[Former Member]

Hi,

We have the following challenge:

We need to process large files with file content conversion in PI 7.3

We cannot just forward the binary flatfiles towards the target in blocks as is possible in PI 7.3 since we need payload access.

The messages are big and cause outofmemory errors and lead to instability of the platform when we try to process them as one message with FCC

We tried to use the FCC option recordset per message but that option still seems to want to convert the entire file to XML before split.

We have a flatfile file structure where we have one header, a number of detail records and a trailer:

Header

  Detail1

  Detail 2

  ...

  Detail n

Trailer

Question 1:

Should the recordsets per message function without reading the entire message first?

Question 2:

Is there another way of splitting the input message before offering it to FCC? Maybe a Java bean that does this?

If we were able to split the message before processing that would really help.\

Thanks!

Tom

[Former Member]

Thanks for your answers so far.

One thing that keeps nagging me is if the Recordsets per message parameter should work in this case without blowing java memory up.

So we only have one header, lots of details and 1 trailer.

The trailer record is not really important to us.

Is it possible with Recordset per message to split such a structure into multiple messages each with their own header? Or is that just not possible with this fuction?

Answers very much appreciated.

Thanks

Tom

[Former Member]

Hi Tom,

If its possible in your case then I would suggest that a simpler approach would be to limit the number of records coming in from the source system. Say, per file 500 records or something.

I am sure you can write a EJB module otherwise to read and split the file.

Best Luck,

V J

[rajasekhar_reddy14]

Definetly record sept per message should work in your case, have you tried using it, give appropriate number and execute end to end.

I dont think there is any predefined bean to use splititng.