Huge file processing.

[former_member223432]

Hi Experts,

I have file interface where the volume of the data is too huge(in GB)....how do i process this in PI? what are the best approaches in PI 7.1 for handling huge volume of data?

Please advise.

thanks

[Former Member · Accepted solution]

Hi Smith,

  Although its not recommend to process huge data through SAP PI. You may use AAE to process bulky messages and through sizing and performance tuning. Below is the link to sizing performance tuning guide, This may help you.

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/2016a0b1-1780-2b10-97bd-be3ac6221...

-Amol

[former_member223432 · Accepted solution]

thanks for your quick reply...you have always been very helpful

[baskar_gopalakrishnan2]

If your input file is text file, You can use FCC on the sender side & set conversion parameter

Recordsets Per Message:  some number of records

This way you can split the source message in to multiple smaller messages.  Though PI 7.1 says that it does optimum performance up to 5mb, it does not mean that it cannot process high volume messages. You might have to set tuning parameters in SXMB_ADM for the mesg max size limit.

Hope that helps.

[anupam_ghosh2]

Hi Smith,

              Need to split the file with scripts. The scripts may be scheduled to run at definite interval of time or may run in infinite loop in server. Pls take help of BASIS team to put and run the scripts in server. You can look into this article which discuss on splitting of files using shell scripts. In case PI server is in Windows enviornment u need batch files.

Regards

Anupam

[MichalKrawczyk]

Hi,

with 7.1 the only way to prcess files like that would be to create a java proxy or adapter which would split it and send in chunks

as per standard config in 7.3

http://scn.sap.com/people/michal.krawczyk2/blog/2010/10/18/pixi-pi-73-processing-of-large-files--tea...

but with 7.1 you need to do it on your own

Regards,

Michal Krawczyk