Import Manager Usage : Approaches for developing Import file structure and text validations

[Former Member]

Hi Experts,

We are having 50+ import maps. We have provided option to users to drop the files for data import. Currently Import Manager(7.1 Sp08) does not have capability of pre-import validation of file data such as

a. file structure - number of columns specific to import map

b. file text validations - special characters, empty lines, empty cells

c. Uniqueness of the records in the file

For this, we are planning to build temporary folder(port specific) in which user drops in the file. We use the custom development to do above mentioned validations and then move the files to actual import ports.

Please let us know if you have worked on similar requirements and how you have fulfilled such requirements.

Regards,

Ganga

[Former Member · Accepted solution]

Hi Ganga,

If you use a well defined xml as source format and have xsd with validations you can take care of most of above concerns.If you still want to make checks you can trigger a WF for records on import and subject them to approval and review.Alternatively,you can import the data into a staging table and run validations and assignments on that data.Post this data manipulation you can syndicate this data and re-import it into the initial destination table.

Also,PI , DS or other tools can also be used to transform data into desired format.

Regarding point C - If source has same record repeated 5 times and MDM imports it then only one record will be imported based on Parametric import feature of MDM. If you wish to enforce uniqueness of one or few fields then you can set that in the target table in MDM. The records not obeying uniqueness criteria will be rejected.

So there are a lot of ways you can achieve this,choice can be made based on availability and scalability of solution.

Thanks,

Ravi

[Former Member · Accepted solution]

Hi Ravi,

Please share some more information on "data into a staging table and run validations and assignments on that data" i.e. scenarios you used this approach.

If I introduce staging table; can I hide the table from data manager users?

Regards,

Ganga

[Former Member · Accepted solution]

Hi Ganga,

Assuming you have a well defined xsd and are getting valid xmls from source in the Inbound Port of MDM. Also,you have a Primary key in form of External ID (say).

So just by making and defining a XSD you get most of what you want in your questions a and b.

Now if you wish to use PI to drop files in the inbound port then you can build all the validations in PI itself and you would not need Staging table.

Otherwise,you can have another table (preferably Main table) in the same repository or other dummy repository where records are created on import based on External ID.

Here you can launch an MDM workflow on import of these records and run assignments to replace unwanted characters and Validations to give error for rejecting some records based on the data quality level desired.Once unwanted characters are removed and data is validated it can be syndicated using a syndication step in the Workflow.So records which fail are not sent and which pass are sent to a outbound port.

From the outbound port PI or some other job can pick the file from this outbound folder and drop to Inbound folder of the same repository which imports to the required Primary Main table.Here again you have the option to leverage validations in PI and further check if data is fine.

Once this activity is done you can delete the records from the staging table.

Thanks,

Ravi

[Former Member · Accepted solution]

Thanks for your post as I had a similar issue.

Regards,

Salaheddine Mawloud