BusinessObjects Data Quality 11.7 (DQXI) performance problem

[Former Member]

Hi,

I am responsable for the 1st implementation of DQXI on Brazil and I am having some performance problems. I cannot use DI because we are using a HPUX server and also because it is not localized for Brazillian Address and Names yet.

I got a HPUX 11.52 server with 16x1GHz processors and 32 GB RAM and when I run some projects on it, only 1 CPU is being really used and all the others are idle. Do I need any special setup on the bootstrap file so that I can use more than 1 cpu?

Thks for the help,

Reinaldo

[Former Member]

Anagha,

Thks for the answer. Right now the Thread parameter at the boostrap is set to 12 and the cache is 512.

I have created a very simple job using just a DataCleanse object and set the Thread to 12 just to check if could get the server to use more then 1 cpu but I am getting no improvment.

Any other clue? Right now I am running this on a HP-UX server.

Thks a lot,

Reinaldo

[Former Member]

The NUM_OF_THREAD setting in bootstrap comes in picture when you are using DQXI in real time mode. I don't think that is the case here.

Did you try increasing the NUM_OF_THREADS on Data cleanse?

[Former Member]

I'm trying to understand performance with DQXI 11.7 on Windows. I have a four way box, but can't seem to get DQXI to use more than 10% of CPU. Will the CACHE_SIZE, MAX_COMMAND_SIZE or NUM_OF_THREADS settings in the bootstrap file have any effect?

[Former Member]

I have solved my problem: We used to have too many tranforms on a project, and I could not alocate enought threads. Now I am trying to use as few transformations as possible for each project.

Right now, I am using the following:

Reader(1) --> Formater_Pre(2), Search_Replace_Pre(2) (for words), Global Address(4), and Formatter_Pos(2), Writer(1).

The numbers inside the () are the total treads I have set for each transformation

On the Formatter Pre I am doing all tranformations I need like concatenating fields, treating some address issues (in brazil the address goes like Street Atlanta). As lots of people writes the worng type of address, DQXI was doing lots of bad address corrections.

The Replace_Pre is used to do search replace of Words. Can I do this type of action inside a Formatter? What function should I use?

The Global Address is a standard one

The Formatter Pos concatenate|parse some fields so that they are written on the propoer column as well as format the Address that Global Adress could not treat.

The performance is getting better and I hope to get a 600 lines/second thoughtput.

Remembet to delete every variable or object inside a Formatter that you are not using anymore.

Other important thing is to delete all the fileds you are not using (the global address has lots of output filds that I am not using so I have deleted then and saved some resources.)

Thks for all the help,

Reinaldo

[NielsWeigel]

Hi Reinaldo,

I wanted to reply with the remark that with Data Services XI 3.1 and the All Platform release teh Unix platfroms are supported. This shoul give you a way to upgrade from the DQ XI R2 (11.7) to our new solution.

Regarding the localization for Brazil on Names and Addresses and your comments on Global Address cleansing capabilities I would like to get more information from you on that.

Níels Weigel

[Former Member]

Reinaldo,

You should need any tweaking in the bootstrap file. But you can bump up the NUM_OF_THREADS setting on transforms in your project(dataflow) to more than 1. By default these are set to 1. See if that helps with CPU utilization.

CPU utilization also depends on how the project is built. For instance, if the project has a sort transform doing a full sort, during sorting process you may not be able to make use of multiple CPUs as DQXI full sort works on single thread.

Anagha