File adapter reading one file and discarding lines yet processed

[Alfredo_Lagunar]

Hi Experts,

I have one external system that works generating one file per day. I need read this file each 10 minutes but I get multiple errors.

Is there any way to only read the pending lines of one file?

Best Regards,

Alfredo.

[Alfredo_Lagunar · Accepted solution]

Hi all,

thanks for your replies, I think the better way to solve the problem is putting a little program that:

     1. opens the file,

     2. checks the lines to take only new lines

     3. copies in a new file in other folder.

This file will be processed with the delete option. My problem is that I can't modify the original file because it is needed to the external system.

Best Regards,

Alfredo.

[iaki_vila]

Hi Alfredo,

There isn't a standard way to read the same file sequentially. The easiest way is to read the file once the day is over because a OS batch that read partially generating a new file to be picked to PI seems to complicated for me.

However, if your scenario is file - RFC (or proxy), the control could  be developed in the endpoint system or during mapping you can  use RFC lookup or jdbc lookup to take the number of lines of the last execution and do another RFC lookup or jdbc lookup to store the new value (total lines).

Regards.

[Former Member]

hi,

stupid question... which option do you have in your Sender CC: Test, Delete or Archive ?

because if you want to really let the source file, and so to try to find which new lines will be "Appended" by your source appli... that means you should have option "Test". Correct ?

BUT... as your source application produce a file per day... that means tomorrow, you will have no more one file, but two files in your soruce folder (!!)... as your option is "Test".... And in one week, you will have 7 files... In another word, this option "Test" cannot be used ! and in fact, your source appli cannot really do an append into a new file per day. They can only append in a "fixed" and unique file !  ... wichic will growth and growth again and again and...

Now, if you, in your Sender CC, you choose option "Delete" or "Archive" (better), what's happen?

you will remove the picked-up file... and so if your source appli want to append new lines, as the file is no more here, this appli will certainly create it (most of application create the file when the file is not yet created and when the option is "Append"... like PI does). With this option, your source appli continue to create the same filename per day, and you, as you have removed the file... in fact that means, every 10 min... you get a new file (!!), with new lines to be processed (!!). Do you really have to try to identify the new lines ?

Other solution (complety different):

At any time get all lines.

The 1st times, you add these lines into a Ztable (see RFCLookup in PI mapping).

for the next process, do the same and compare all lines to thoses already stored in a Ztable, reject the processed lines, process the new, and then stored the processed lines.

But in my mind, it should be better to discuss with your source appli in order to know exactly why they absolutely need to create only one file per day and to append the whole day into this file.

regards.

Mickael

[former_member184720]

Hi Alfredo - May be you can provide some more details

If i understand your scenario - you have a system which generates one file per day and it keeps on appending it through out the day. You need to read it for every 10min and send only the new records if any.

AFAIK, There is no directly way for reading delta from a file.

for some reason if they cannot generate a new file evrytime, ask them to add the timestamp for each line.Atleast we need some identifier to find the new records right

Ofcourse you read the full file everytime but in the mapping while sending it to target, apply some logic like

Current datetime - timestamp from the file <11min then create the record