Handeling large csv files

[Former Member]

We have a CSV file that is 295MB, when converting this file to XML in the File Adapter the size will only increase (to 4 / 5 times the original size). This means that there is 1GB of network traffic.

Is there a way to split the message in serveral smaller ones (like 10 of 100MB) as to decrease the steep peaks on the network?

Increasing the maximum size of a HTTP request in Java stack and ABAP stack does not suffice for this requirement.

Setting the file Adapter paramet Recordsets per message does not work since the Recordset structure is of format: RECORDSET, *. RECORDSET, 1 is not an option since there are more than 1 recordets in the XML structure.

Please help me to approach this situation. SDN and the SAP Help only have been helpfull for a bit so far.

[former_member187339 · Accepted solution]

Hi

I didn't get why the Recordset per message cannot be used?

Can you paste the input xml stucture which you are expecting from the File Adapter?

it should be of this format, where Recordset is a wrapper for each repeat of records.

<documentName>...

<recordset>

<NameA>

<field-nameA1>field-value</field-nameA1>

<field-nameA2>field-value</field-nameA2>

<field-nameA3>field-value</field-nameA3>

</NameA>

<NameB>

<field-nameB1>column-value</field-nameB1>

<field-nameB2>column-value</field-nameB2>

<field-nameB3>column-value</field-nameB3>

</NameB>

</recordset>

...

<recordset>

...

</recordset>

</documentName>...

If your structure is like above then you can use recordset per message as 1

Regards

Suraj

[Former Member · Accepted solution]

Hi Suraj,

thanks for your remark, i can see it works now, i doubled checked and it the Recordsets per message Is working correct now.

I set the Recordsets per message to 500.

Alas this does not help me as i hoped it would.

I now get this error message in Communication Channel Monitoring for the Sender CC.

Adapter Framework caught exception: error while processing message to remote
system:com.sap.aii.adapter.rfc.core.client.RfcClientException: JCO.Exception while 
calling ZPI_UPLOAD_FCASSORTIMENT inremote system 
(RfcClient[CC_RFC_RCV_PI]):com.sap.mw.jco.JCO$Exception: 
(104) RFC_ERROR_SYSTEM_FAILURE: Overflow when converting from 500

I assume the 500 is the Recordsets per message i defined. Before i defined Recordsets per message this number would be equal to the number of records in the CSV file. I thought the number of records was to big to process at once therefore i wanted to split the sender file into several message.

I cannot believe 500 records is to big to process at once.

Someone please advise me on this error.

[former_member187339 · Accepted solution]

Hello..

indeed 500 can be big number of record. The reason is after adding XML Tags the size increases trmendously.

I will suggest you to further decrease the record set per message value till you get an acceptable value.

if <500 records per message is not acceptable in your scenario then I would suggest you to use java proxy in sender side.

Regards

Suraj

[Former Member · Accepted solution]

You are a hero! Against my thinking it actually was the 500 that was to big, i changed it to 250 and now it works. Well... We now have a problem at the receiver but that is nothing a little implementation cannot solve! Thanks Suraj

[former_member187339 · Accepted solution]

Hello

I am glad I was able to help

Regards

Suraj

[Former Member]

Also check this blog

/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover

[Former Member]

Hi

Handling large files say anything above 100MB is always a problem with File adapter as the data has to be moved from Adapter Engine integration Engine and vice versa.

Third party tools are generally used for that. Conversion Agent by Itemfield is one of the best approaches.

Also, on the Advanced tab of the file sender adapter, select the check box next to Advanced Mode. There you can specify Maximum File Size (Bytes) option.

Edited by: Krish on Sep 1, 2009 11:15 AM