File Processing Suggestion

[Former Member]

Hi,

My source system will create a file ULYY.dat and start appending the same file within seconds.

XI needs to poll the files every minute .

The design strategy is that the source system will append the records if it finds the file else it will create a new file with the same filename ULYY.

The current design strategy that we have decided is that XI needs to first pick up the filename and rename it before processing.

Later it will archive it. By this, XI can guarantee that it does not process partial data.

Is this good design strategy? or is there any other way out.

I was thinking that what would happen if the source system tries to append records when xi is renaming the file? Since everything is happening within seconds, i am confused as of what will be the good design approach

[Former Member · Accepted solution]

Hi Krish,

Suppose the file that PI picks currently is having extension as .dat. Now whenever PI sees a new file with extension .dat in the directory it will pick it up (no matter whether it is a completed file or a file which is just getting updated with the source process). So if the source creates a file with extension .tmp (and then keep updating it) and later when the file content is completely transferred (ie the complete file is created) then rename it from .tmp to .dat.

Like this PI will not be able to pick in the incomplete file. Another approach will be to keep creating the file in some other directory and when it is completely created then move it to the polling directory of PI. (for doing this a schedule batch program can be used)

Thanks,

[Former Member · Accepted solution]

Hello Hareen,

The problem is that the source system will keep on appending records.

So, there is never a question of complete file.

But when XI picks up it should process only those records which are already there.

[Former Member]

Hi,

to guarantee that XI does not process partial data, you can configure some time in Msecs to Wait Before Modification Check.

His is a parameter of your sender communication channel --> advanced mode --> additional parameters.

Regards,

Carme