Is there a record limitation with flat file with a csv extension?

[former_member697230]

I have migrated a flat file load in BI 7.0 and now it seems there is a record limitation of 65k rows in the file. I am getting the following error mesage in the monitor: option =2. CSV split errorr: 2 =other problem with the escape character.

If I use a file with less than 65K rows ( and I mean 1 less than ) I get no error and if I use a file with more than 65K ( and I mean 1 more ) I get the error. Is there a setting somewhere that is setting the load limit to 65K or is this a real limitation?

[Former Member · Accepted solution]

Hi

There is no record limitation in BW side. The limitations are ruled by the limitations of the flat file... In Old excel, only 65k rows allowed and in MS-2007 upto 1 million rows possible and you can extract them

Regards

N Ganesh

[former_member697230 · Accepted solution]

Hello N Ganesh,

I don't mean to doubt you but can you explain this scenario? In our production system using a 3.x datasource I am able to load a csv file with 85K rows. In our dev system using the same csv file I am not able to load using the migrated BI 7.0 datasource unless I reduce the number of rows in the file to 65K. The interesting thing is that the error described earlier is showing up in the first 3 data packages ( of 20K recs each) and the 4th and 5th data packages are green. I would think it would be the opposite. Anyway I am playing around with the cache per Gopi's suggestion, so far with no luck.

Thanks for your input.

Peter

[Former Member · Accepted solution]

Some more thing you can try.

Try to reduce the Inpackage Packet Size.

If all these things are not working then we have to split the file and load it. Espescially with BI 7.0 I have also faced this problem and finally I split the file and loaded the data.

Regs

Gopi

[former_member697230]

Yes I realize that 65K is a limitation in excel, however we have no problem loading csv files in excess of 65K records using a 3.x datasource. This appears to be a new limitation with BI 7.0 datasource.

[Former Member]

What I observe from your post is that the CSV file is having some problem after 65k records.

Or it may be a problem for Caching the input file. You can set Cache settings using "RSCUSTV...." transactions.

BASIS guys will be able to help you if the problem is with Caching.

Regs

Gopi

Assign points if it is helpful

[former_member697230]

Hello Gopi,

Thanks for taking time with my issue and I will certainly assign points. I have discussed your recomendations with out basis people and they are unfamiliar with the caching transactions. Could you please list the specific transactions for us.

Thanks,

Peter

[former_member697230]

Hi Gopi,

Nevermind I have found the transaction. I will let you know how things work out.

Thanks,

Peter

[Former Member]

This is the limitation of MS Excel. I don't think we can increase it.

Regards

Pankaj

[Former Member]

If you open the CSV in excel then you have the limitaition.

Try to open the CSV in Notepad.

Regs

Gopi