Sender File Adapter stop processing all files

[ralf_zimmerningkat3]

Hello all,

the file adapter pick up all files in the directory by default.

if a large number of files are in the directory then this could slow down the pi processing.

is there any way to process only one file per polling??

regards

[Former Member · Accepted solution]

This is the standard behavior of File Adapter, so if you want to change it then may be you have to write your own adapter module.

Other possibilities I can see is to first take all your files into some temp dir and then write a script which will send the files from temp dir to main dir one by one after each polling interval.

[Former Member]

OK.... let me throw some light on few facts..

SAP has designed the File Adapter in such a way that it can process number of files parallely to increase the performance and not to create the performance issue.

Just thing if you have 10 thousand files and File Adapter starts polling one by one then I guess it will take hours or may be days and which I guess is a real performance issue.

So just by having a thought in mind that large number of files will slow down the processing is wrong.

My suggestion is you design your object without any issue, large number of files will not create any performance issue at all.

[Former Member]

Hi Sarvesh Singh / Ralf Zimmerning...

Just to add that maybe there are some benchmarks available from SAP on this .. and you can compare your expected loads to this.

Thanks

Damien

[ralf_zimmerningkat3]

Hey Sarvesh Singh

if you pick up 100 Files each have 5 MB and all are send to IE then in IE by processing you

will run into memory dump in java engine, heap of java engine out of memory, java engine is restarting.

better it will be only to pick up one file per polling time!

[Former Member]

Hi,

I believe that if you use EOIO, though all files are picked by the adapter, still one file at a time will be sent to IE for processing.

By this you can be rest assured that it wont give you a dump.

I would suggest that using EOIO is the best approach.

Regards

Krish

[Former Member]

>

> Hey Sarvesh Singh 
> 
> if you pick up 100 Files each have 5 MB and all are send to IE then in IE by processing you 
> will run into memory dump in java engine, heap of java engine out of memory, java engine is restarting. 
> 
> better it will be only to pick up one file per polling time!

What you are saying is correct, but we have cosider the figures (file size or amount of data) as per the SAP standards.. we cannot just put 100 files each of 5 MB without knowing the fact.

Please go through the link below...

https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/4dc9381e-0a01-0010-c692-f6bf424b...

[former_member184619]

@ Sarvesh - Thanks for this valuable information..

@Ralf - If your source directory is having only these files(all with same name) and you are using NFS (transport protocol) you can choose the

Processing Sequence - By date in sender Comm. Channel

"If you select By Date, files are processed according to their time stamp in the file system, starting with the oldest file."

OR

In Advance mode, you can specify the Maximum file size (in case you are worried about performance)

In this case i think (only one file condition will not me met)

Sachin

[ralf_zimmerningkat3]

the problem is that XI/PI just polls the directory.

Files are send by partner into the directory.

So I have to build a process with third party tools to reduce the files

[Former Member]

Ok.. But now what Sachin has discovered as shown below, it seems to work..

>Processing Sequence - By date in sender Comm. Channel

>"If you select By Date, files are processed according to their time stamp in the file system, starting >with the oldest file."

For more help go through this link..

http://help.sap.com/saphelp_nw70/helpdata/EN/e3/94007075cae04f930cc4c034e411e1/content.htm

Try this and let us know..

[Former Member]

Ralf,

Have you tried the Best Effort option? According to my understand, under Best Effort mode, the adapter picks up a file, sends its to the IE, then waits until it receives the synchronous response before picking up another file, and so on ...

Regards,

Victor

[ralf_zimmerningkat3]

@ Sarvesh Singh

if you select Files by date then the sender adapter will pick up all files ordered by date

but he will pick up also all files in directory at once

[Former Member]

>

> @ Sarvesh Singh 
> 
> if you select Files by date then the sender adapter will pick up all files ordered by date  
> but he will pick up also all files in directory at once

Actually it was my understanding from the very begining that File adapter will always pick all the files at once, but I was not sure about this solution given by Sachin... so I asked you to test it.. hmmmmmm... now what....

See my previous reply about writing an script and moving file one by one from temp dir to main dir.... see if it helps you..

Regards,

Sarvesh

[former_member206760]

Ralf,

just a suggestion..

1.why dont you have a second interface which would be same as the first one ..

only thing is it would be pointing to a second source folder...

now ask you sender application to write some of the files in folder1(say 50) and another 50 in folder 2...

2. you reduce the polling interval to such a level ( say 5 seconds ) that within that time limit there will be no chance that there would be say 100 files in source folder

[Former Member]

My suggestion is to ask your third party (sending system) to restrict the amount of fies if they are huge (follow the quicksizing guide which I send you earlier). By this way you can make sure that there will be no performance issue.

[Former Member]

Hi Sarvesh,

Nice way to expalin As per your solutions my concepts also more clear now..

Regards,

Sameer

[Former Member]

>

> Hello all, 
> the file adapter pick up all files in the directory by default. 
> if  a large number of files are in the directory then this could slow down the pi processing. 
> is there any way to process only one file per polling?? 
> 
> regards

I do not have any proble if you have got the answer. BUT this blog says how to exclude the other files from the same folder.

Your Case: For example, Your Sender CC wants to pick up file ABC.txt from /xyz dir, now suppose there are 10 thousand files of same name in the dir and you want ABC.txt should be picked up one by one, So how this blog is going to help you. Can you Plz explain to me and others too?

@Sachin may be you can throw some light on this... may be I am missing something.

[former_member184619]

Hi Sarvesh,

I agree with you on this, But it might be possibe others ways by combining both options "File Name Mask" and "Exclusion Mask"u2026.

I have just given a suggestion.

[Former Member]

>

> Hi Sarvesh,
> 
> I agree with you on this, But it might be possibe others ways by combining both options "File Name Mask" and "Exclusion Mask"u2026. 
> 
> I have just given a suggestion.

It is not possible even combining both options. You cannot change the standard behavior of File adapter untill to write your won module..

File adapter will simply see if there are 10, 20 or 100 files with same name it will pick them at once.

[ralf_zimmerningkat3]

I try the BLOG with all variants but file adapter picks always all files up

so i have to develop an own file adapter.

Is i t possible to take the sap file adapter sources as example??

[former_member184619]

Hi Ralf,

This might be of some help to you

/people/mickael.huchet/blog/2006/09/18/xipi-how-to-exclude-files-in-a-sender-file-adapter

Sachin

[former_member181985]

You have to use EOIO concept for processing files one by one.

[ralf_zimmerningkat3]

If you configure the EOIO in Sender File Adapter

then the File Adapter will read all files in the directory and sent all files to the IE

In the IE the processing is then EOIO.

I want to pick only one file per polling time