DI MySQL Job stops early

[Former Member]

I'm trying to load data into a reporting table, but can only get the DI job to load the first 10,000 rows or so before the state of the job goes to "stop". I am joining two mysql tables together that should give a result set of 9.6 million records or so.

This should be the resulting sql of the table joins, and when run in a mysql query browser, the result set is 9589630 records.

SELECT

Table1.ID,

Table2.ID

FROM

Table1,

Table2

WHERE

Table1.col1 = Table2.col1 and

Table1.col2 = Table2.col2 and

Table1.ins_date <= $ins_date

but when the job runs, it will cache Table2, and only bring in the first 10,000 rows of Table1 before it just stops. I can't find anything on the mysql server that would prevent this from running, and no errors are returned from DI.

Has anyone seen this problem? Anyone have any advice?

DI Version: 11.7.3

MySQL version: 5.0.445

[scott_broadway]

Hi Brett,

Two questions:

- When you say that the dataflow stops after 10,000 rows from Table1, what do you mean? Does the job crash? What is shown in the Trace, Monitor, and Error log files after it "stops"?

- What SQL is being pushed down to the database? For instance, with the dataflow open on the canvas, go to "Validate" - "Display Optimized SQL". This will show you the sql clause(s) pushed down to the datastore connection(s).

Thanks,

~Scott