how to handle Big FIles in SAP PI Sender file adapter

[former_member586633]

Hi all ,

I have developed a interface , where it is File to Proxy, it is fine when i do with small and normal files

The structure contain one Header unbounded detail and one Trailer, how to handle when the file size is more than 40 MB

Thanking you

Sridhar

[Former Member]

Hi Sridhar,

Can you please let me know how did you resolve the issue, now I have faced the same isssue for my file to proxy interface.

Thanks,

Venu

[Former Member]

Hi Sridhar and Satheesh,

If you are dealing with large flat files, then the ways to approach this solution are:

1. use 'Recordset per message' option in FCC -- this will reduce stress on the adapter engine

2. set 'tuning' in SXMB_ADM---->EO_MSG_SIZE_LIMIT ------ this will reduce stress of integration engine

you can find many blogs addressing this issue.

https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/2016a0b1-1780-2b10-97bd-be3ac622...

[Former Member]

Hi Sridhar,

You can even pick the whole file in one single go and adjust the server level parameters to accomodate space for it.

But one of the biggest problems that come along is impact on other running interfaces.

My suggestion is to create a dedicated queue for this interface so that impact to other interfaces in minimal.

Regards,

Anurag Mahendru

[Former Member]

Hi,

Refer section 3.2.9.2 of the following document for large file handling.

http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/2016a0b1-1780-2b10-97bd-be3ac62214c7

[rajasekhar_reddy14]

Hi,

is your source data XML or flat file?

if you are doing File content conversion in Sender channel use Record set per message option suggested in previous post.

else if it is XML then you have to write some module.

Regards

Raj

[former_member586633]

Hi ,

Thank you for your reply,,

it is not a XML , it is a file , i have already used record set and set it as 25,000, but the condition is tht it should not be split ted as it is a single set of record to be proceed

[former_member854360]

Hi,

In receiver file adapter set the file creation mode as Append.

so in sender it will splited to multiple file but in receiver it will be appended to the same file.

suppose Your sender file has 50000 records.

then in PI 2 files will be created.

but in receiver adapter specify file construction mode as Append.

So that both the two files will be written to the same target file.

so in receiver system you will have only one file.

Processing Tab Page
Processing Parameters
u25CF      Various options exist for creating the files:

File Construction Mode

[anupam_ghosh2]

Hi Sridhar Gautham,

We can set a limit on the request body message length that can be accepted by the HTTP Provider Service on the Java dispatcher. The system controls this limit by inspecting the Content-Length header of the request or monitoring the chunked request body (in case chunked encoding is applied to the message). If the value of the Content-Length header exceeds the maximum request body length, then the HTTP Provider Service will reject the request with a 413 u201CRequest Entity Too Largeu201D error response. You can limit the length of the request body using the tting MaxRequestContentLength property of the HTTP Provider Service running on the Java dispatcher. By default, the maximum permitted value is 131072 KB (or 128MB).You can configure the MaxRequestContentLength property using the Visual Administrator tool. Proceed as follows:

1. Go to the Properties tab of the HTTP Provider Service running on the dispatcher.

2. Choose MaxRequestContentLength property and enter a value in the Value field. The length is specified in KB.

3. Choose Update to add it to the list of properties.

4. To apply these changes, choose (Save Properties).

The value of the parameter MaxRequestContentLength has to be set to a high value.

The Visual administartor tool may be accessed using this link

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/40a08db9-59b6-2c10-5e8f-a4b2a9aaa...

In short ICM parameters to reset values for this case are

icm/HTTP/max_request_size_KB

icm/server_port_ TIMEOUT

rdisp/max_wprun_time

zttp/max_memreq_MB

Please look into this thread to know more about ICM parameters

http://help.sap.com/saphelp_nw04/helpdata/en/61/f5183a3bef2669e10000000a114084/frameset.htm

Second solution is that you must split the source file, so that each file is less than 5MB in size, then PI would not cause problem for file size between 1MB-5MB. you can insert header and trailer for individual smaller file obtained after split. All this can be done using scripts or conventional programing provided individual records within file are independent of each other. Finally you have to rename each new file created and put them in PI folder in sequential manner. All this can be achieved by simple shell script/batch file, a C code or java code. If you are going for a C or Java code you need a script to call them from PI communication channel parameter "run operating system command before message processing".

regards

Anupam

[naveen_chichili]

HI Gautham,

1.UME Parameters : you need to look into the pool size and poolmax wait parameters - UME recommended parameters (like: poolmaxsize=50, poolmaxwait=60000)

2. Tuning Parameters: you need to look/define the Message Size Limit u201Clike: EO_MSG_SIZE_LIMIT = 0000100u201D under tuning category

3. ICM Parameters: consider ICM parameters (ex: icm/conn_timeout = 900000. icm/HTTP/max_request_size_KB = 2097152)

Regards,

Naveen.

[Former Member]

Hi Raja,

i have requirement as source is XML file (having 10000 service orders) needs to be processed through XI and post idoc to CRM. Can i do it using direct file adapter?

will it cause any performance issue?? could you suggest on this??

Thanks,

Satheesh

[anupam_chakraborty]

Hi,

I would suggest you, that you break the file in the sender adapter channel using the FCC. You can use

Recordset Structure = Record,10000

This would break your message to 10000 record. Use Quality of Service: Exactly Once in Order so that your messages are not jumbled up.

In case you want the entire message to be a part of one proxy, You have to create a File to File Scenario to append the message in receiver and then use another bypass to hit the proxy.