Create one file when the Source directory is empty

[Former Member]

Hi Experts,

I am creating one scenario MDM--> PI --> Teradata. MDM scheduler is scheduled once in a day and MDM will send maximum 600 data at a time. PI will then pick the data from MDM using MDM adapter and post the data to TERADATA using JDBC adapter.

Now, the requirement is: Once PI completes the data processing (uploading all the data successfully in TERADATA system) for the day, it will simply create one file or some notification to start the Teradata Jobs which is scheduled in Tivoli system.

Is it possible?

Thanks & Regards,

Moumita

[yeeloon-khoo]

Hi Moumita,

I don't know about MDM adapter, but based on this statement:"MDM scheduler is scheduled to create the files of MDM changes for last one day and this scheduler will run once in a day", By the time scheduler start run, the number of file should be known because it is past 1 day data and won't be change anymore, although it is not fixed.

If the files is one-by-one, and still want to ensure all data origin from MDM is 100% matched with data sent to TERADATA, I see must do a compare between MDM and TERADATA.

The idea:
1) Find a way to get number of file from MDM. Probably a script at file adapter to count number of files after all file generated. Or ask MDM guy any other way.

2) Find a way to get number of record saved in TERADATA. Probably JDBC query table with last_change_date="today" and success="true".

3) Periodically run (2) and check if count of (1) and (2) matched or equal 600, then start the Teradata Jobs.

Hope i won't make thing complicate.

Regards,

Yee Loon

[Muniyappan]

Hi,

How about this? this is kind of suggestion. you can fine tune it if it suits.

interface 1 : MDM-->PI-->TerraData

interface 2 : to start the terra data job( you can have something like File-->PI-->Terradata)

now schedule the interface 2 in such way, it will run after 30 mins with respect to interface 1.

so interface 1 will run first and after 30 mins interface 2 will run to kick start the job.

keep file processing mode as test for interface 2.

Regards,

Muni

[Former Member]

Hi Muni,

Same here. Some data inconsistency can happen.

Please advise.

Thanks,

Moumita

[Former Member]

Hi Satish/Raghu,

Thanks for your help.

I have one doubt here. MDM is sending one record in a file and at a time, MDM can send maximum 600 files. As per my understanding, PI will pick the files one by one and process it. In that case, after processing each and every file, PI will call the stored procedure (as suggected by Statish) or create the file (as suggested by Raghu). But, my requirement is, after uploading the chunk of files which has come from MDM at a time, PI will send one notification or something like that PI has successfully uploaded the file. Because, MDM scheduler is scheduled to create the files of MDM changes for last one day and this scheduler will run once in a day. So, PI will get all the files once in a day but the file number is not fixed.

Please advise and also please let me know if my understanding is not correct.

Thanks & Regards,

Moumita

[Former Member]

Hi Moumita,

I didn't work on MDM adapter, but may be there should be option on MDM adapter to retrieve all data sets at one time during particular time frame. Please check that too.

My above suggestion is recommending for one time call.

Thanks,
Sateesh

[Former Member]

Hello Momita ,

If you know daily when the MDM main interface is going to process the files i.e. if you know the end time daily. Then you can create another simple file-file interface and then use Adapter availability planning on that interface to start after the scheduled end time.

Even if you do not know the end time for MDM interface then you can discuss with the business and use ATP on this interface too and stop the interface. Any pending data will be processed in the next run.

You can have a number of ATP schedules based upon your requirement. May be 4-5 in a day.

Thanks.

[Former Member]

Hi Sateesh,

Yes, your suggestion will definitely help for one time call. But this is not the case here .

Here, MDM is also creating one xml file. I have asked MDM guy to combine all the records in a file. But, no luck there. Technically, they can do it but it may cause some performance issue from their side. So, they are not ready to combine all the records in a file.

Can we do something with the polling properties of the sender communication channel? Like, can we check the source folder maybe upto 5-6 times and if it is empty then we can send some notification to the target side. Is it possible?

Thanks,

Moumita

[Former Member]

Hi Gaurav,

If I schedule any channel in a particular time, it may happen that any failure reported in PI at the time of uploading then also PI is sending the green signal to start the Job to do further processing. In that case, some data inconsistency will happen.

Thanks & Regards,

Moumita

[Former Member]

Hi Moumita,

Yes possible.

My understanding as you said you are using JDBC adapter to Insert/post the data to Tera Data.

There are two options,

1st option:

What Tera data jobs do? please confirm with your database Guy, If Tera data jobs execute the stored procedure, then PI can execute same stored procedure immediately once Insert done into Table on same database call.

Your Message Type should be like this.

Statement1

--dbTableName

---action

---table

---access

----field1

----field2

Statement2

--storedProcedureName

---action

---table

---field1

---field2

First Statement for Insert into table and second statement for call the Stored procedure.

Import Note:  This option is only for if Tera database schedule jobs need to execute any kind of stored procedure after Insert from PI, Instead of Tera Database schedules just do from PI.

I am n't sure about requirement on Database end, but mostly this is case. so you can just check with your database guys about purpose  of schedule jobs.

This way you ignore 2nd option.

2nd option

After succesfully Inserted into Terabase table from PI, then take response(response from database for ex: Insert_count) back from the table to PI, if response come back to PI that means Insert successful, take the reference of this Database Response and create the file (or) Mail notification.

ccBPM is always best for these steps.

Hope you understand, Put me your doubts. Thanks.

Best Regards,
Sateesh

[RaghuVamseedhar]

Moumita,

Create two target Service Interfaces (same Message Type). For first Service Interface select JDBC receiver and for second Service Interface select File receiver. Maintain Order in Interface determination. File will be placed after JDBC channel runs (success/failure).