cancel
Showing results for 
Search instead for 
Did you mean: 

Issue with the sender file interfaces when using UTF-8 files

former_member186078
Active Participant
0 Kudos

Hello experts,

We are having number of File to SAP interface scenarios in our business process. In the past, we are having the files coming to us in ASCII text format. Recently, because of the business process change(to handle multiple languages like Chinese), we are getting now the same files in UTF-8 format. So, we have changed the attribute File Type in the sender communication channel from Binary to Text and used encoding format as UTF-8.

Now, when we see the processed files in PI, we are missing the Header record in the message. For example, we have the PO interface in the following format:

Identifier Information

H PO header information

I PO Item information

I PO Item information

Once the file adapter picks this format kind of file and when we see the XML message in PI, it is just having only two Items information. The header part is not coming to PI at all and the message is trying to get processed inside SAP and therefore interface is failing.

Additional observations made:

u2022 We have checked the Document Offset field value too; it is initial for the interface.

u2022 We added an empty line in the UTF-8 text file, it worked fine. But, this is not an ideal solution for us, because the system which generates these files, canu2019t handle it.

Does any one has observed this kind of problem before? If so, can you please help me ...

Thanks,

Adithya K

Accepted Solutions (0)

Answers (5)

Answers (5)

Former Member
0 Kudos

HI Adithya ,

can you once test with using encoding UTF 16 it will resolved the BOM related issue . If the source team is sending txt file and you are converting it to XML using content conversion then use file type as text and encoding as UTF 16 . However if you are getting the XML file from source then use XML ananomyzer bean as suggested by Stefan and just at place of encoding use UTF 16 .

Secondly UTF 16 is assupe set of UTF 8 so you wwill not face any issue and hope your BOM problem will get resolved

Regards,

Saurabh

Former Member
0 Kudos

Thanks for all the information. Currently i've tried to use both modules (TextCodepageConversionBean and XMLAnonymizerBean) in my Sender File Adapter, but for now without any result...

To describe the situation: i am using flatfiles which i am fetched from the file system (NFS) by the Sender File Adapter. To translate the characters to the XML strucutre, the File Content Conversion protocol is been used.

The content conversion defines the structure base on the first two characters on each row in the flat file. This worked fine till one of our suppliers is delivering BOM characters like "EF BB BF" in the beginning of the file.The content conversion is not able to recognize my header characters in this situation, which will normally start with "01".

Any suggestions?

baskar_gopalakrishnan2
Active Contributor
0 Kudos

Modules like TextCodePageConversionBean and XMLAnonymizer will not solve your issue. As Stefan suggested talk to sender team and ask them to send without BOM characters. Otherwise before you pass the file to PI, You might have to write some java coding to remove those characters first.

stefan_grube
Active Contributor
0 Kudos

Let me repeat that:

SAP PI does not support "EF BB BF", as this is Microsoft specific and no standard.

You cannot do anything inside PI with standard modules.

Fix the issue on sender side. Maybe you can write an OS script to remove the BOM.

By the way: This issue has no practical impact as this behaviour shows up only when you build test files with Microsoft Notepad. When the files are generated by an application program automatically, then it never has a BOM.

So do not waste your time to try to fix it.

Former Member
0 Kudos

Hi All,

Most of the time the Sender system donot understand what is BOM and trust me it's difficult to convince them to remove the BOM. I resoved the issue using simple substring function.

The BOM char ( ) when UTF-8 always appear in the beging of the file.So used substring to remove the first char.

u can try that

Former Member
0 Kudos

Hi Adithya,

please try the XMLAnonymizerBean , see link attached.

this should help.

[XMLAnonymizerBean|http://help.sap.com/saphelp_nw04/helpdata/en/45/d169186a29570ae10000000a114a6b/content.htm]

Former Member
0 Kudos

Dear Adithya,

Did you solve this situation? I am facing the same issue by the Sender (File) Communication Channel. Hereby my header information is not fetched from the file because of the Byte Order Mark (BOM) characters in beginning of the file.

I allready tried text/binary filetype and encoding UTF-8, but without result.

Best regards,

Joost Weghorst

Former Member
0 Kudos
stefan_grube
Active Contributor
0 Kudos

> Did you solve this situation? I am facing the same issue by the Sender (File) Communication Channel. Hereby my header information is not fetched from the file because of the Byte Order Mark (BOM) characters in beginning of the file.

A BOM is no standard, it is Microsoft special.

Ask the sender not to use a BOM.

stefan_grube
Active Contributor
0 Kudos

In principle you need not apply file encoding UTF-8, as this is default value.

The reason for the missing header line is a so-called BOM which is added to the file.

The BOM is a Microsoft specific feature and regularily causes errors in file adapter.

Ask the sender of the file not to add a BOM.

http://en.wikipedia.org/wiki/Byte_order_mark

stefan_grube
Active Contributor
0 Kudos

Could you check, if it works when you apply file type = binary?