Error attempting to process incomplete inbound FTP file

[mark_meilleur]

We are currently receiving some large files into PI via FTP. Here is the communication channel definition:

Adapter Type: File

Transport Protocol: File System (NFS)

Message Protocol: File

Adapter Engine: Integration Server

Poll Interval (secs): 600

Msecs to Wait Before Modification Check: 10000

Our issue is that at times PI attempts to process the file before it has completed uploading, and throws this error: "Attempt to send file failed because file was modified during processing. 2428928 bytes expected, 2675033 bytes found. Trying to resend file". The next time the file system is polled for new files, PI selects and processes the file successfully.

Two questions:

1) Is it possible for PI to wait until the upload has fully completed before it attempts to process the file? For example, can I run an operating system command before message processing which attempts to put a lock on the file and waits until it is successful before the file is processed?

2) We have implemented alert monitoring from Solution Manager for errors in PI. These cases are throwing an alert upon the initial error, but these turn out to be false since the message is eventually processed successfully. Can I configure Solution Manager to bypass this type of error?

Thank you very much for your help with this.

[Former Member · Accepted solution]

Hi Mark,

For your first question. PI is waiting until the file is written out completely. It's doing that with the modification check setiing. It's not really an error that is getting thrown in Solman, it's more like an information alert as a result of the modification check.

Preeti's idea is a good one to minimise impact on third party systems. Leave the current setup as is so that the third parties carry on with business as usual. Create a new folder. Point PI file adapter to the new folder. Write a script to move the file to this newly created folder once it's written out completely. The only issue you might have is making sure that the file is complete before you move it. Maybe if there is something like an end-of-file marker at the end of the file or a record count you could use that to ensure completeness when scripting. That would be more bulletproof than a straight modification check in case there might have been a network failure while the file was being written out.

Regards, Trevor

[mark_meilleur · Accepted solution]

Thank you Trevor (and Preeti). This does seem to be the best solution (outside of disabling the alerts). Guess now I have to figure out how to trigger this script to run when a file arrives. I've done something similar on an IBM I-Series but programming on Intel is a relatively new experience for me.

[Former Member]

Mark,

Solution to your issue has been given in SAP Note 821267 question #3.

Please try to tweak your Msecs to wait parameter.

Thanks

Aamir

[mark_meilleur]

Hi Aamir,

Thanks for the reply. We have played around with this setting, but this is what's leading to the alert being thrown. PI polls the directory, finds a file and records the number of records, waits the specified msecs, then finds that the number of records in the file has changed and throws the alert.

Mark

[mark_meilleur]

Thank you for the suggestion Michal. As these files are coming from third parties, I'd prefer to minimize the impact on them, and it seems that your suggestion would require them to handle the renaming of the file, no?

[MichalKrawczyk]

Hi Mark,

>>>1) Is it possible for PI to wait until the upload has fully completed before it attempts to process the file? For example, can I run an operating system command before message processing which attempts to put a lock on the file and waits until it is successful before the file is processed?

there is an easy way to achive it

put the file with a different name (let's say .lfg extension for flag)

after the application finishes uploading make sure it will also change the name to .xml etc.

inside PI file adapter specify to upload only .xml extensions - this way it will only take the whole file

>>>>2

as for solman question if you do like I suggest you will not have the errors in solman

Regards,

Michal Krawczyk

[Former Member]

Another alternative is to place the file in a temp folder that PI does not poll. Once the file is ready to be picked up, transfer it to the polling directory using a script on the host system. This saves changes at the end system. This is based on the assumption that transfer within local folders are quicker than the over the network transfers.