Recordset per message configuration in File Sender

[Former Member]

Hello Experts,

I am facing problems in File sender adapter (NFS protocol) while handling huge files in File -> Proxy scenario.  I have configured 10000 lines to be processed per recordset, however i could see only 1 message getting generated even if the file contains around 86,000 rows.

I also tried giving recordset structure like RecordSet,10000 but no luck. Any idea how this configuration works. Is it mandatory to give EOIO as the service although tried with that as well but not working.

Actually we want to process large files with around 200 MB and hence looking for some file splitting options. I am using latest PI 7.5 single stack.

Regards

[anupam_ghosh2]

Hi Ameet,

              Please look in the value of max occurrence of target root node after message mapping. 

If the occurrence has been set to 1 then you will get only one record as output. Secondly check for context mismatch causing one record to be generated in target.

Regards

Anupam

[Former Member]

Thanks Anupam but message mapping is perfectly fine. I am able to post the messages incase of smaller files and infact even larger files of around 14 MB containing around 86K lines. My problem is I cannot limit the size of the files coming in, hence wanted an option to split the files and this option is present since long, hence wondering why it isn't working in this case.

Regards

[anupam_ghosh2]

Then please suggest business to send smaller files which Pi can handle. or use a script to split the file before sender channel starts processing the same. One such script I wrote long back

Optimum File Size for various file scenarios in PI part-2 - Process Integration - SCN Wiki

Also you can set additional parameter in sender adapter

"Specify the maximum file size in bytes."

The adapter only processes files that are smaller than the specified size. An error occurs if files are any larger.

Regards

Anupam

[Former Member]

Hi Anupam,

Actually this is the CSV file so cannot use the above parameter. I guess I should go with some shell script option, however SAP Help mentions that

Recordset per message splits the message into multiple smaller messages and is not doing the same, which is surprising.

Check this link..

Converting Text Format in the Sender File/FTP Adapter to XML - Advanced Adapter Engine - SAP Library

Under Recordsets per Message, specify the number of recordsets to be grouped together in a message.

This entry is optional. The default value is* . In the default setting, all recordsets are included in a message.

If the number of recordsets in a document is greater than the number specified, the adapter creates multiple messages from a document. The last message might then contain fewer recordsets than specified.

Regards

[Former Member]

Hello,

Found that with setting this parameter (Number of recordset), the adapter adds the same number of Recordset tags to the message and actually doesn't splits the message.

I was hoping to have say 10 messages generated for 1000 records file with Records per Message as 100 but unfortunately not happening like that.

Guess would have to write some shell script for splitting the source file into multiple chunks.

Regards

[former_member194786]

Hi Ameet,

Can you please provide screenshots of your channel's content conversion tab and if possible the structure of your file that you are trying to split?

Cheers,

Sanjeev

[markangelo_dihiansan]

Hi Ameet,

I think it will only split the message (Recordset per message) if the csv does not contain a header and a footer.

Regards,

Mark

[anupam_ghosh2]

Hi Mark,

              How will PI distinguish between header and footer.

Can you please elaborate. Are you referring to key field value.

Regards

Anupam

[markangelo_dihiansan]

Hi Anupam,

Yes, key field values are required. The only example I found in sdn that successfully split using Recordset per message only contained rows

Regards,

Mark