Cannot Profile File Data

[Former Member]

I'm running into an issue where I am not able to profile file data in Information Steward 4.0. I believe this is related to the fact that Data Insight is not returning any rows when I click view data for the specific file that I have loaded. However, I do not receive any error messages as to why the view data is not returning any records.

Here is what I'm doing:

1. I've created a file format using the actual file I want to load to propose the schema
2. I've adjusted some of the columns data structures in that proposed schema from int to varchar (I've also left these alone and my results are the same)
3. I save schema
4. Add file to the project using the schema
5. Once file is loaded into the project,
   1. I select the file and view data -> Zero records returned
   2. Try to profile the data -> Failed due to the following message from the log:

Subtask [run id=1] Table WorkFiles on EDM03.DS-US-NewAdds.txt: Dataservices execution failed for (14.0) 01-18-13 13:10:06 (E) (5664:1364) FIL-080124: |Data flow DF_SPF_STATS_427_1|Transform PROFILE_SOURCE1__AL_ReadFileMT_Process                                                     The bad row limit for file <//server/WorkFiles/DS-US-NewAdds.txt> is met. The limit specified is <1>.. Error (14.0) 01-18-13 13:10:06 (E) (5664:1364) FIL-080124: |Data flow DF_SPF_STATS_427_1|Transform PROFILE_SOURCE1__AL_ReadFileMT_Process                                                     The bad row limit for file <//server/WorkFiles/DS-US-NewAdds.txt> is met. The limit specified is <1>. (COR-10690) (TSK-10008)

I suspect something is wrong with the schema, but I cannot pin point it. Any help would be appreciated.

Tom

[paula_augedahl · Accepted solution]

Tom,

Since the default setting for Max errors to stop processing is set to 1, one row not meeting the defined layout will cause the file to error.  When you define the flat file format, you can indicate to log errors to a file. This is found under More Options on the File Format defintion.  Try setting Write errors to error file to yes and indicate a path/filename for Error file and review what is written there to try to understand the problem in the format.

Hope this is helpful.

Thanks,
Paula

[former_member221211]

Hi Thomas,

If your results are still failing you can also try to set "Max errors" to no limit. That would allow you to profile the data. I have attached screeen shots of the steps but they are:

1) Go to file format and click on more options

2) In the pop up windows go all the way day and select No limit instead of one in the "Max error" option.

If you set it that way the profile will be able to complete.

Hope this helps.