Problem in Sending Bulk Records to Database.

[former_member644654]

Hi All,

I'm doing an interface which is using RFC as a sender and JDBC as a receiver.

The requirement is:

1) No. of records changed per day should be sent to the database as new records for updating the table.

2) Total No. of records is more than 45000 per day.

3) There shouldn't be any records which are already present in the database ( i.e. duplicate entries should be avoided)

Kindly suggest me if I can do any changes or modifications in XI in order to avaoid such duplicate entries and also handle the bulk data volume on a daily basis.

Correct me if I'm wrong but can I use BPM for this purpose? Is there a provision in BPM to stop sending duplicate entries also alongwith sending the bulk data?

If I can use BPM, please suggest me how to implement it for this requirement of mine.

Thanks in Advance.

Sreedhar.

[former_member187339 · Accepted solution]

Hi Sreedhar,

If you wan to update few records, make sure you run RFC at some particular time of the day.. Like this PI will only receive file only at a specific time and it can insert it in DB..

Also to avoid multiple insertion of duplicate data you can try to use the update_insert option of jdbc adapter,. like this if records are already present it will be updated otherwise insertion will happen.

Also the frequency of running this interface will be a crutial one, because both RFC and JDBC adapter are not good for handling huge data size.. so try to run this interface as frequently as you want (so that data size will be less)

I am not sure if the availability planning will be effective with RFC sender adapter (i have never tried it), because if you are making this adapter available say after an interval of 6 hrs.. then till that time there will be a communication failure between SAPECC and PI..

So my suggestion is :

1. Schedule the rfc to run every half hour..

2. use update insert option in receiver jdbc adapter

Regards

Suraj

[former_member644654 · Accepted solution]

Hi Suraj,

Thank you for the methods you suggested.

I'll schedule the Adapter for every half an hour but in JDBC adapter where should I mention it as update_insert?

Thanks in Advance.

Sreedhar.

[Former Member · Accepted solution]

Hi Sreedhar,

As Suraj mentioned 'Availability Time planning' might not be a feasible option here (sorry for my earlier response) as the RFC does a 'PUSH' of data from R/3 to PI whereas schedulling channel is more effective for scenarios where PI 'PULL's data from source system (ex:file,jdbc etc). Unavailabilty of RFC sender channel during RFC call to PI can prone to errorrs.

So its better to schedulle RFC at R/3 end.

The update_insert should be given for 'action' attribute in the mapping for the receiver JDBC structure.

Edited by: nagarjuna _s on Dec 8, 2009 7:59 AM

[former_member187339 · Accepted solution]

Hi Sreedhar,

It is not here in jdbc communication channel, you need to specify update_insert as the action

Check this link

http://help.sap.com/saphelp_nw04/helpdata/en/2e/96fd3f2d14e869e10000000a155106/content.htm

Updated

Don't use Availability planning for RFC Adapter.. For testing purpose you can but not in production

Regards

Suraj

[former_member644654 · Accepted solution]

Hi Suraj,

Thanks for the suggestions. I've handled the duplicate entries in the ABAP side i.e. in RFC and scheduled the JDBC adapter for every 1 hour.

Sreedhar.

[former_member200962 · Accepted solution]

scheduled the JDBC adapter for every 1 hour. 

Scheduling is done for Sender channel and not for receiver! You may end up with error in SXMB_MONI saying channel stopped by administrative task.

Regards,

Abhishek.

[former_member644654 · Accepted solution]

Thank you Abhishek for the update. I was about to implement that in receiver. Thank you once again.

I'll change it to the Sender channel or schedule a background job for every 1 hour.

Sreedhar

[former_member200962]

Is there a provision in BPM to stop sending duplicate entries also alongwith sending the bulk data?

BPM does not have any such functionality....nor any standard method in XI/ PI can do this.

This needs to be implemented at the RFC end itself that no duplicate record should be send.

To check if the record is already present in DB (using XI) is not a good option as XI has to do a lookup (in DB) for each record send from RFC and then map the new entries.

So best option will be to program the RFC to only send the new additions and not all the available data.

Regards,

Abhishek.

[former_member644654]

Hi Abhishek,

Thank you for the suggestion provided.

If I can check for the duplicate entries in the RFC, is it possible in XI using BPM to send the data in and every hour or every 6 hours to send the data instead of sending all the data at one go?

If yes, please suggest me how to give a break of hours.

Thanks in Advance.

Sreedhar.

[rajasekhar_reddy14]

yes you can do it,schedule an adapter for every 6 hours..so that iw will execute accordingly,

select communication channel monitoring in component monitoring,select availability planing.

Regards,

Raj

[former_member200962]

Again BPM is not needed

As suggested by Raj you have to implement a concept called Availability Time Planning (available from PI7.0 SP19....if not wrong in mentioning the SP)....there is also a blog on the same written by Shabz....even help has documented this.

Regards,

Abhishek.

[Former Member]

You can invoke the RFC with a periodically schedulled background job on SAP R/3 system or you can use availability time planning option in PI cc monitoring to activate the channel at particular times.

And regarding the handling of throughput on JDBC adapter, set maximum concurrency and poolwaiting time parameters in receiver JDBC channel to handle concurrent calls to dB.Increase thread count for JDBC adapter if neccessary(this has to be done thr visaul admin).

To prevent duplicates set a unique key field for table and use update_insert in SQL query. But its good to handle duplicates in ABAP program itself to prevent unneccesary load on PI server.