Character encoding failing with large files

[former_member187447]

Hi Folks,

I have a some characters coming in the flat file such as Ë, é  etc. The problem is a bit strange. For large files which are of the size in MB. the character encoding ISO 8859-1 when used in sender channel is working fine to retain all the special characters, but it fails for the smaller files of the size in MB. The XML after FCCfrom the sender channel is changing for example for Ë to Ã

When I use UTF-8 encoding in sender channel, it works for small files of the size KB, but for larger files the special characters are getting changed to some kind of square box. 

I am trying to understand what kind of encoding need to be applied in the sender channel to retain the characters irrespective of the file size. I tried to implement the modules specified in the following link but its the same issue. I can implement Java mapping but I wonder if it makes any difference as the behavior is dependent on the file size. Please suggest.

[stefan_grube · Accepted solution]

For me it sound as your large files are encoded in ISO 8859-1 and your small files are encoded in UTF-8. Use a hex editor and check the the hex values for É and é and compare to a code page chart. If the encoding of the files is different, you cannot process those files with the same sender file adapter. Make sure with your project team, that you expect all files in same encoding.