cancel
Showing results for 
Search instead for 
Did you mean: 

Specifying the "batch size" in JDBC pull

Former Member
0 Kudos

Hi all,

I am working on an interface in PI 7.0 in which a JDBC sender channel pulls data from JDE database table once in a day. Recently, the JDBC channel gave error while fetching 26000 records (which is a lot more than typical no. of records fetched normally).

Can someone pls advise if there is any easy way by which we can specify the no. of records to be pulled in each read operation ? This will take care of such occasional spikes.

Regards,

-Shankar

Accepted Solutions (0)

Answers (1)

Answers (1)

MichalKrawczyk
Active Contributor
0 Kudos

Hi,

depending on the DB you can specify TOP (MSSQL) UP TO and things like that

it's not jdbc adapter generic

Regards,

Michal Krawczyk

Former Member
0 Kudos

Thanks for replying.. The database is MS SQL. Looks like TOP command retrieves only specified no. of records, but not all records, correct ?

Just to clarify again, I want to specify a parameter, say 1000, so that if there are 26000 records in all, there will be 26 read operations, each fetching 1000 records.

former_member206760
Active Contributor
0 Kudos

Hi Shankar,

The answer given by michel is correct.

suppose you have specified TOP 1000. where status eq NEW...then 1st time it will fetch top 1000 records with status new ...next time when it will poll it will fetch another 1000 records with status NEW

so u fetch top 1000 records everytime with status NEW and may be in the update satatement update those records with status say OLD...so that u do not fetch the same records again

Former Member
0 Kudos

Thanks Tarang for confirming. Yes, there is a flag field which is blank before reading and the update statement makes it "Y" after that record has been read.

One query. Will the subsequent reads happen immediately one after the other, OR they will happen only after the polling interval of channel ?

I will anyways try out this option and revert if any issues.

Regards,

-Shankar

former_member206760
Active Contributor
0 Kudos

it will happen only after the polling interval specified in the sender channle