Slow performance issue

[Former Member]

Here I go again. I am just getting used to the product, so any help would be appreciated. The hardware is a physical box with 2.4g CPU and 4gb ram running windows server 2003. While the job is running, cpu is approx 40% concistently with physical memory: Total 3931532, Available, 1881300, System Cache 1001500.

From the above, I would say I am not constrained by hardware limitations. My dataflow is a flat file with 3831930 rows. I am selecting one policyid off the file (49 rows) which is being written to another flat file. The job is taking 7 minutes to complete!!!

I ran a comparison to the ETL tool we currently use and it only takes 28 seconds to run.

Once again, any tune up tips you may provide would be greatly appreciated!!

Thanks,

Dan

[Former Member]

I believe it was due to my bad building skills. I had a Validation transform selecting off the rows I needed. I should have used the Query transform to do this. Build time is now down to 11 seconds.

Like I said, I am just getting to know the product. Look for more questions from me in the future!!! Hopefully, I will be able to answer some of your questions in the near future.

[werner_daehn]

Also, I am not particularily proud of our file format performance. We could do a better job here in partitioning the file. All we do parallelize yet is the file parsing when setting the "number of parallel threads" in the file format reader object. On databases we do a much better job....