File Adapter to picking multiple files in sequence based on condition

[Former Member]

Hi PI Experts,

In my scenario, SAP BW would be writing 3 files in one of NFS location (not in PI File system).

.dat file ( Which contains the records or data )

.man file ( Which contains total number of records in .dat file, run data and time of .dat file etc )

.done file ( This is the empty file, which means other two files .dat and .man are completed and its ready to pickup )

SAP PI need to pick the above files only if the .done file is completely written to the NFS location.

Currently we are using the Integrated Configurations, and the SAP PI picks the .dat file before .done file is written in NFS.

Some .dat files sizes are more then 3GB and it takes several hours to write the files.

Requirement is in our case is that

PI should pick .dat file from the folder only if .done file is available.

and it should pick in the below in sequence

.dat

.man

.done

I have posted this query earlier in SDN and experts suggested to go for custom module in file adapter.

Can any one provide me the logic / code to do the above task.

Thanks,

Kiran

[Shabarish_Nair · Accepted solution]

in your case the best way would be to use a script that will move the required file from original folder to say folder <PI> once the done file is created.

Thus PI can pick the required file from the new folder.

I dont think a module will also be of any help to you in this case

Another option would be to go for a java proxy which i think will only be complex for you to create.

ref:

[Former Member · Accepted solution]

Hi Vijay,

Thanks for your suggestion.

But we have decided not to go with the script, we have some restrictions so we left the option of script.

hoping the java module could help and earlier some of our friends suggested to go with the custom java module.

Thanks,

Kiran.

[Shabarish_Nair · Accepted solution]

>

> hoping the java module could help and earlier some of our friends suggested to go with the custom java module.
> 
> Thanks,
> Kiran.

i wonder how this can be achieved using a module. will wait to see other responses.

[former_member200962 · Accepted solution]

the JAVA module you are planning to use will be used in your channel and will be triggered when the channel picks up the file....but then you have to decide first which file has to be picked.....so the module may not be useful here....even i feel script is useful here

Regards,

Abhishek.

[Former Member · Accepted solution]

Hi Abhishek,

Thank you.

yes, so when the channel picks up the file the module will trigger and should be able to check if there is a done file in the folder.

thats what i was expecting, i know script should help but in our case we are not allowed to use the script because of our internal reasons so i was trying to know if custom module can help.

if not then we need to look for some thing else.

Thanks,

Kiran.

[stefan_grube · Accepted solution]

You could use "additional files" in channel. Then the file will not be read until all required files are available.

But in this scenarios all files are processed within one message, which might be not what you want.

[Former Member · Accepted solution]

Since BW is writing the files, initially write the files as temp.dat, temp.man, temp.done.

When writing the files is completed, from the program trigger a job in BW which run temporarily to rename the files.

In the job initially check if temp.done file is available? Then rename temp.dat to pickup.dat, check the file count in the job, rename temp.man and so on it will control the sequence.

Now incase of error on adapter, file should go to error folder so we are sure job will not run for ever or for too many loops than expected abend the job for manual intervention and rerun.

or

After the files are written start the File Channel externally from BW so that it polls for the files (3 channels) and then stop by scheduling a job later in the day with appropriate checks.

Regards,

[Former Member · Accepted solution]

Hi Srihari,

Thanks for your suggestions but we don't have any control from BW end, they just write files to their file server location.

we have to make sure all the files are completely written and then pick the files.

This is how we make sure that the huge file (*.dat) writing s completed is when we see .done file in the file location

as the .done file is created on to folder only when the remaining files are completed.

hence we just need to make sure that we start picking the files from file location only when we see .done file and then can pick all the files at a time.

as Stefan Gurbe mentioned that additional files option could be useful but we don't have message in this scenario, this is just integrated scenario in PI7.1 it justs picks file from one location and places in an other location.. there is no XML conversion or any thing.

so still looking for a solution other than script ..........

Thanks,

Kiran.

[Former Member · Accepted solution]

any help on this ???

[Former Member · Accepted solution]

Hi Kiran,

We had developed one JAVA module which checks if file is completly written or not. It is used in Sender communiation channel. Logic for module is that it tries to rename the file, if file is written completely, renaming takes place else it gives error. I our case when renaming is successful, we revert the renaming (to original Source File Name) & exit the module so normal processing takes place however if renaming is not allowed, it indicates that files is still being writtne, in this case, we terminate the process & file is not picked up in PI.

Another logic here could be check the file size at first time & after some time again check the same. If there is some mismatch, it indicates that file is being written however we observed that our source application was taking lot of time to write the file & chekcing the size of file in few seconds gap was not giving desired result hence we adopted above approach.

This is working fine & we are using this from more tan 50 interfaces.

Your requirement is not exactly same however u can also build similar logic in which if .done doesnt exist roll back whole proess. If it exist, just contiune with normal processing.

Hope it helps.

Regards

Sushil

[Former Member · Accepted solution]

Hi Sushil,

Your solution should be able to resolve my problem to maximum extent, if not we can tweek a bit and try.

but for that can you please provide me the JAVA code in both the cases you have mentioned.

that would be a great help for me.

Thanks,

Kiran.

[Former Member · Accepted solution]

Hi,

Why dont use the Advanced Mode option "Msecs to Wait Before Modification Check"???

[Former Member]

Hi Kiran,

Creata a sender Comm Channel and Mention the filename as <filename>.dat and XX<filename>.man

In the Before OS command check for .done file, one line script command, no need to call a seperate script in OS level

If .done file present it will process the .dat file

In After OS command, write the command to rename the <filename>.man file to XX<filename>.man

Hence in the second poll .man extension file will get pick

Best Regards,

Chandra Shekhar.

[RaghuVamseedhar]

Hi Kiran,

I would divide this requirement into 4 scenarios. I will use concept of controlling channels externally [Blog Link|http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6456] [original link is broken] [original link is broken] [original link is broken];.

1. Scenario1 (File to File for .done): - Every 1 minute poll for .done file in test mode ( not in delete mode ). If file is present take it using SenderChannel1 and place in some target folder using ReceiverChannel1. ReceiverChannel1 should have an Adapter module with code to start SenderChannel2 of Scenario2 and stop SenderChannel1.

2. Scenario2 (File to File for .dat): - SenderChannel2 for is already started by ReceiverChannel1 adapter. Pick the file (delete mode) using SenderChannel2, place in target folder using ReceiverChannel2. ReceiverChannel2 should have an Adapter module with code to start SenderChannel3 of Scenario3 and stop SenderChannel2.

3. Scenario3 (File to File for .man): - SenderChannel3 for is already started by ReceiverChannel2 adapter. Pick the file (delete mode) using SenderChannel3, place in target folder using ReceiverChannel3. ReceiverChannel3 should have an Adapter module with code to start SenderChannel4 of Scenario4 and stop SenderChannel3.

4. Scenario4 (File to File for .done): - SenderChannel3 for is already started by ReceiverChannel3 adapter. Pick the file (delete mode) using SenderChannel4, place in target folder using ReceiverChannel4. ReceiverChannel4 should have an Adapter module with code to start SenderChannel1 of Scenario4 and stop SenderChannel4.

It is a loop: - 1,2,3,4,1,2,3,4,1. First time 2,3,4 channels should be stopped.

You can try to write generic adapter module, read sap help link in blog, which can start and stop a channel by taking all parameters to form HTTP request.

Regards,

Raghu_Vamsee

[Former Member]

Hi Kiran,

Develop a custom module, let any of the file get pick by the Adapter

1. check the picked file name extension i.e. is .done or not

2. if .done then exit from the module

3. else open a connection to access FTP server, once connection is established move to the respective directory where files are present

4. check for the .done file, if present close the connection and exit from the module else raise the exception

Best Regards,

Chandra Shekhar.