Encoding issue - having UTF-16LE BOM "FF FE"

[Former Member]

Hello Experts,

The scenario is as follows:

SAP sends IDOCS -> SAP-PI (collects IDOCS & creates IDOC xml) -> Content conversion done at Receiver CC -> Text File having pipe delimited is placed in an FTP location.

Requirement :

Currently SAP R/3 is sending Balkan and Cyrilllic chracters to PI. Both SAP-R/3 and PI are Unicode compliant. SAP-PI version being used is SAP-PI 7.1. BPM is used to collect the idocs based on time.

The SAP-PI while converting the IDOC to idoc XML, it has header as "encoding=UTF-8".

The text file that is getting created at the FTP location is an ANSI file. (If you open the text file with EDITPLUS(ver3) tool, you can check the file type as ANSI. )We need to change this as UTF-16LE.

In the receiver CC, in the first Target tab, we have maintained Transfer Mode as "Binary" in FTP connection parameters.

In the Processing tab, we have maintained the File type as Text and Encoding as "UTF-16LE".

We also switched from Binary-Binary, Binary-Text and Text-Text in both the tabs, but the file that is getting put, is still an ANSI file in the FTP location.

In PI all the characters are coming correctly. But the time of creating the file, the file is getting created as ANSI.

We need to have the file type as UTF-16 having BOM(Byte Order Mark) as "FF FE". If you open in the EDITPLUS text editor, it should show as UTF-16.

Please if any of you experts have come across any solution for this issue, please let me know the steps. It an issue in production and need your help asap.

Points to be awarded to the best answer and an answer that helps us solve the problem.

Thanks.

Deb.

[stefan_grube]

from another discussion in SDN forum, I have learned that PI does not add a BOM.

UTF-16LE and UTF-16BE do not have a BOM, as the byte order is clear from declaration.

So you have to add the BOM with an OS script.

When you put UTF-16LE in receiver channel, the target file should be in UTF-16LE. if this does not work, check if UTF-16LE is installed in server, where PI is running. But if it is missing, an error message would happen in channel monitor.

You have to check the encoding of the file with a hex editor. You cannot verify this with with Notepad or any other text editor.

[Former Member]

Yes we are checking the encoding with the HEX editor also but the BOM is different than the expected FFFE.

Do you know the OS script to add the BOM. If yes, pls let me know.

No error is seen in the channel monitor when we add the UTF-16 in the receiver channel.

[stefan_grube]

> Yes we are checking the encoding with the HEX editor also but the BOM is different than the expected FFFE.

Besides the BOM, is the encoding correct?

[Former Member]

Hello Stefan,

No the encoding is not correct. It is getting encoded as ANSI.

Is there a possibility that is the FTP server is not a Unicode, then files will get placed as ANSI. Or this(Unicode or non-unicode compliance) does not matter for FTP servers.

[stefan_grube]

> No the encoding is not correct. It is getting encoded as ANSI.

As ANSi is not an encoding, I would like to know, which encoding the files really have.

In principle the files could be UTF-8 or UTF-16. You can distinguish this easily with a hex editor, as in UTF-16 almost every second byte is zero.

FTP servers do not change encoding of a file.

[Former Member]

In that case, we are getting the UTF-16 as we can see that every second byte is zero. But the BOM "FFFE" is not present.

One more thing.

When we place the file in the AL11 directory of PI, the file that we see using the editplus tool shows as UTF-8+.

[stefan_grube]

But the BOM "FFFE" is not present.

i am thinking about adding the BOM with FCC parameters.

You could try:

NameA.beginSeparator = BOM (if you have a distinct header line)

NameA.addHeaderLine = 3

NameA.headerLine = BOM

BOM = ´0xFF´´0xFE´

[stefan_grube]

by the way: report this issue to SAP. UTF-16 is a standard which should be supported. maybe there is already a solution available.

[Former Member]

We do not have any header line in our data record.

We hav reported to SAP on this, but the BOM issue is still pending.