Usage many transforms in the same data flow
I have a case, where I need to remove duplicate record, and retain most recent date record. So, I used 3 Query transform,
2) Generate ROW-NUMBER for each group and
3) Filter record based on row number
Apart from that, I used 3 more query transforms for some other functionality. Finally I end up having 6 query transforms in the same data flow.
Is there is any restrictions in using more transforms in the same data flow? If yes, will it affect the performance?
Kindly show me some light on it.
Dirk Venken replied
There's no theoretical restriction on the complexity of a dataflow. As a rule of thumb, for practcial reasons, I try to make my flows fit the size of a my screen.
Performance is affected by many parameters. Check Efficient extraction of most recent data from a history table for a technique to improve the performance when removing duplicate records.