Encoding issue in SOAP Receiver_UTF-8/ISO-8859-1

[Former Member]

Hi Experts,

I have been facing encoding issue while using SOAP Adapter. The scenario is a synchronous one from Proxy <--> SAP PI <--> SOAP.

1. I have not used any Java mapping and it is normal one to one mapping, so the encoding should be UTF-8

2. I have used ISO-8859-1 encoding in the adapter engine which has worked, but still on the target webserver, the data seems to be corrupted and could see '?' symbols.

3. I have gone through various SAP Forums links, but couldnt find suitable exact answer to drill down the issue.

Does it has something to do with the encoding setting in SAP ECC System?

Looking forward to hear from you.

Regards,

N. Jayanth Kumar.

[baskar_gopalakrishnan2 · Accepted solution]

Please check with data fields any weird characters coming from the sender system. UTF 8 is right encoding type. You normally dont need to change. Are you getting any data like foreign characters or image or so? If so, then correct it in the sender system or need the appropriate encoding type to send. Send the same data via soapUI and see how it works

[Former Member · Accepted solution]

Hi Bhaskar,

When tested from SOAP UI Client, the target webserver when viewed, it has the characters rightly appearing. Below are characters we have been using.

Radox „2In1” Dušas Želeja Un Šampūns Vīriešiem Εε ΖζΜμΠπΣσΤτЛлščфûâΞξ,ΧχΣσΘθגÔô • Ʒʒ • ǮǯŊŋ• ÑñDždžǼǽ • ǢǣĤĥ Ľľ • Ȑȑ •й йд дÖö - ק20130811
ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóŠ æä،о®SHY§¥‰ ÇƉ¶€0712A - 20130811

Even when we pushed from PI RWB manually, we are getting response that the file has been accpeted, but still there are '?' mark symbols.

Is it something where the encoding has to be set in ECC System as well? Request you to please help.

[ambrish_mishra · Accepted solution]

Hi Jayanth,

Did you check the ABAP connection type 3 in ECC. You need to check unicode on the 4th tab. In PI, the encoding should not change since it is UTF-8 by default. Don't think a Java mapping is required.

Hope it helps!

Ambrish

[ambrish_mishra · Accepted solution]

Don't test with RWB since data with encoding is assured only from test through ECC.

[stefan_grube · Accepted solution]

Radox „2In1” Dušas Želeja Un Šampūns Vīriešiem Εε ΖζΜμΠπΣσΤτЛлščфûâΞξ,ΧχΣσΘθגÔô • Ʒʒ • ǮǯŊŋ• ÑñDždžǼǽ • ǢǣĤĥ Ľľ • Ȑȑ •й йд дÖö - ק20130811
 ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóŠ æä،о®SHY§¥‰ ÇƉ¶€0712A - 20130811

What do you expect, when you use characters, which are not part of ISO-8859-1?

Please check wikipedia first!

[Former Member · Accepted solution]

Hi Stefan,

Yes I have used the UTF-8 initially and I didnt use any other character set coding. When I pushed the message from SOAP UI Client, the characters appear correctly on target system with successful response coming back.

When we pushed the same from ECC, we are getting successful response back, but characters on target system Portal doesnt seem to appear correctly. There are '?' symbols.

We are connecting to target system via SOAP Receiver which has a proxy configured with a specific host and port. Does it have any difference?

More over, the characters from the Integration engine also seem to appear correct when extracted. Not sure where the exact issue lies in. Can you please help us here Stefan.

Regards,

Jayanth Kumar

[Former Member · Accepted solution]

Hi Ambrish,

I checked the Type 3 connection in ECC system and it is as below. Do we have to change anything here?

[ambrish_mishra · Accepted solution]

That's fine Jayanth

[Former Member · Accepted solution]

Hi Ambrish,

We are connecting to target system via SOAP Receiver which has a proxy configured with a specific host and port. Does it have any difference?

Or do we need anything else to do in SOAP Receiver channel to set the encoding?

[ambrish_mishra · Accepted solution]

Hi Jayanth,

If you want to change the encoding from UTF-8 to ISO-8859-1, here is the excerpt from

http://www.stechno.net/sap-notes.html?view=sapnote&id=856597

• Q: What character encoding is supported by the SOAP receiver adapter?

           A: The SOAP receiver adapter can use any character encoding supported by the local JDK. The request message from the SOAP receiver is normally encoded in UTF-8. If you want to change this encoding, for instance to iso-8859-1, you can set parameter XMBWS.XMLEncoding to iso-8859-1 in the module configuration for the SOAP adapter module. This setting is for the outgoing SOAP message and has no effect on the incoming SOAP message. For the incoming SOAP message, any code page supported by the local JDK is accepted.

Worth a try....

Hope it helps!

Ambrish

[stefan_grube · Accepted solution]

In the beginning of the thread you said:

" I have used ISO-8859-1 encoding in the adapter engine which has worked, but still on the target webserver, the data seems to be corrupted and could see '?' symbols."

Now you say:

"Yes I have used the UTF-8 initially and I didnt use any other character set coding."

Maybe you can describe the whole scenario more precisely.

Can you figure out which characters are represented as "?" in the target? The greek, cyrillic or any others?

[Former Member · Accepted solution]

Hi Stefan,

Sure, I will explain the scenario more precisely. The flow is a synchronous flow as below.

SAP ECC (Proxy) <--> PI <--> Target system (SOAP).

We are connecting to target system via SOAP Receiver which has a proxy configured with a specific host and port. Does it really have any difference in handling special characters?

Should the WSDL File also have the "UTF-8" encoding when we upload into PI system?

This is a simple one to one mapping 'without' any specific encodings that have been set. So usually it should be UTF-8

When the special characters are being triggered from ECC which goes via PI, the target server gives us error response because of the special characters in the file which it is unable to recognize.

I have put all remaining characters aside and tested with below special characters that were triggered to see if it works: I think there are cyrillic characters as well in the input file.

ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóŠ

When the response comes back to ECC from target system, it has below errors that the request has been rejected due to the below.

ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóÅÂ

When I open the error XML in taken from PI the above characters appear.

I have added then ISO-8859-1 seeburger Bic module in SOAP Receiver channel and it didnt throw error response, but the characters on the target portal were appearing as '?' symbols.

When we push the same characters from SOAP UI Client using UTF-8, it works correctly and the characters appear rightly on portal with successful response back.

I am not sure where is the exact issue. I have also tried to do below things.

1. Used Java mapping to return the data in UTF-8 encoding as below but not success.

                                        private final String utf ="UTF-8";

                                        byte[] buffer  = new byte[in.available()];

                                        int bblen = in.read(buffer);

                                        String str = new String(buffer, utf

                                        System.out.println(str);

                                        out.write(str.getBytes(utf));

                                        out.close();

2. I have also used various modules anonymizer bean, XMBWS.XMLEncoding to UTF-8, checked the check box to keep encoded headers, but still no response.

3. The ABAP Connection has the unicode settings already.

4. There is another inbound interface from same webserver SOAP --> PI --> Proxy which successfully handles special characters in PI and they appear correctly in ECC.

Is it something needs to be done in ECC while sending the data to PI, must be encoded in UTF-8 in ABAP Code? When it works for inbound without anything, not sure why is it not working for outbound. Request you to please help me here.

Regards,

N. Jayanth Kumar.

[Former Member · Accepted solution]

Hi Bhaskar,

When pushed from SOAP UI Client the characters appear correctly on target system portal. When pushed from ECC, they dont appear correctly. The source is an ECC system via proxy connectivity. Does the ABAP code needs to be amended to send everything to PI in UTF-8 format.

We are connecting to target system via SOAP Receiver which has a proxy configured with a specific host and port. Does it really have any difference in handling special characters?

Regards,

N. Jayanth Kumar.

[Former Member · Accepted solution]

Hi Ambrish,

I have tried to use XMBWS.XMLEncoding to UTF-8 and ISO-8859-1 to handle special characters, but it didnt work out as well

Regards,

N. Jayanth Kumar

[stefan_grube · Accepted solution]

Remove all modules and java mappings and check this:

- Do you see the request message correctly in SXMB_MONI? Are characters correct?

- Do you see response message correctly?

Is ECC a unicode system?

[Former Member · Accepted solution]

Hi Stefan,

I could see the characters correctly appearing in sxmb_moni of PI system. The response message simply says if it is accepted or rejected. We are directing the message to Target system using a SOAP URL as well as by configuring a proxy additionally. Will that cause a difference Stefan?

when I used ISO-8859-1, it worked but '?' symbols appear on CPNP Portal.

when I dont use ISO-8859-1 the messages completely gets rejected

The ECC system is a unicode system as per below screenshot.

[stefan_grube · Accepted solution]

Of course greek characters are represented as "?" when you use ISO-8859-1.

What does the specification say? Does the webservice request the data in UTF-8 or ISO-8859-1?

Does your scenario require texts in Greek or Russian?

[Former Member · Accepted solution]

Hi Stefan,

The scope which we are implementing is for entire Europe which should cover for all special characters is what had been informed to us. Target webservice provider has informed that they would need  UTF-8. The strange thing here is, when we push a message from SOAP UI Client with all the characters it works well. But when pushed from PI, it throws error on target system

Is it something that needs to be done in ECC side Stefan.

I will find out what are the exact characters that are required here.

Regards,

N. Jayanth Kumar.

[stefan_grube · Accepted solution]

 ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóÅÂ
When I open the error XML in taken from PI the above characters appear.

How do you open this? In fact that are UTF-8 characters displayed as ISO-8859-1.

What just came into my mind: How do you generate test data in ECC? Do you type the data yourself or do they come from the database?

[Former Member · Accepted solution]

Hi Stefan,

I have got the response form target application and when I saved it as XML file and open, it looks as below.

but when opened in ECC system , they appear as below.

Currently from ECC team, I have an updated that the data is being copied and then being triggered.

[stefan_grube · Accepted solution]

You open the XML with what? With Internet Explorer or Notepad?

How is the XML encoding header?

[Former Member · Accepted solution]

Hi Stefan,

I open the XML file in internet explorer and it has the header UTF-8.

[stefan_grube · Accepted solution]

It seems there was an ISO -> UTF-8 transformation on a file which was already UTF-8 and so it was corrupted by this transformation.

Did you remove the Java mapping from the scenario?

[Former Member · Accepted solution]

Hi Stefan,

I have included the mapping and included the below module in SOAP Receiver channel.

localejbs/AF_Modules/MessageTransformBean     Local Enterprise Bean     transform

  transform     Transform.ContentType     text/xml;charset=utf-8

It has worked for below characters

Varušas Želeja Un Šampuns
ViΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłos

but fails with these characters
ΖζΜμΠπΣσΤ τЛлščфûâ Ξξ,ΧχΣσΘθגÔô

Regards,

Jayanth Kumar.

[stefan_grube · Accepted solution]

Could you remove the mapping as well as the adapter module?

I do not see any differenc between your first and your second string. Both are a mixture of latin and greek characters.

[Former Member · Accepted solution]

Hi Stefan,

I had removed the java mapping and module and tested which has thrown errors already.

                                        private final String utf ="UTF-8";

                                        byte[] buffer  = new byte[in.available()];

                                        int bblen = in.read(buffer);

                                        String str = new String(buffer, utf

                                        System.out.println(str);

                                        out.write(str.getBytes(utf));

                                        out.close();

The Java mapping with latest Messagetransform Bean module is giving success for the special characters. I will make tests below and will let you know.

1. Remove Java mapping and modules and test again

2. Remove Java mapping and include MessageTransformBean and test.

On the other side, I had made some tests from SOAP UI Client with UTF-8 encoding set, where I had pushed the characters where no PI or ECC would be involved.

ΖζΜμΠπΣσΤ τЛлščфûâ Ξξ,ΧχΣσΘθגÔô

The target system has thrown errors from SOAP UI client as well, which says there is an issue on Target application?

Regards,

Jayanth Kumar.

[stefan_grube · Accepted solution]

Check with a hex editor. There might be invisible ascii characters inside the string.

[Former Member · Accepted solution]

Hi Stefan,

There are unwanted characters that are not in scope like vietnamese also there were really some spaces that were added in ECC and when they removed the spaces and sent the messages it worked. I have removed the Java mapping and included the Message transform bean which worked. I will keep you updated if we still face any errors. Thanks a lot for your support Stefan. Your inputs have been really very useful.

Regards,

Jayanth Kumar.

[Former Member · Accepted solution]

Hi Jayanth - please let me know your solution.

I'm facing the same problem.

Thanks.

[Former Member]

Hi ,

Can anyone please let me know that how to check that ABAP proxy setup is UTF-8 encoding enabled or not . Also how to where I need to check the Unicode settings for communucation channle as I can see in SM59 , unicode there are only two radio buttons as shown here in screen shot

Thanks ,

Ashish

[Former Member]

Hi Jayanth,

As i understand, the words coming as part of your payload are latvian and these characters are not supported by ISO-8859-1. You might need to refer to specific code page for baltic language (Cp1257).

Please try using Cp1257 encoding in your SOAP sender channel and lets see if it works.

Regards,

Anurag

[Former Member]

This message was moderated.

[Former Member]

This message was moderated.

[Former Member]

Hi Jayanth - Not sure you got solution to issue.

Below Module configuration In the SOAP receiver adapter worked for me. I was in the same situation.

1

sap.com/com.sap.aii.af.soapadapter/XISOAPAdapterBean

Local Enterprise Bean

soap

2

localejbs/AF_Modules/MessageTransformBean

Local Enterprise Bean

transform

3

AF_Modules/TextCodepageConversionBean

Local Enterprise Bean

convert

convert

Conversion.charset

UTF-8

transform

Transform.ContentType

text/xml

Thanks.

[Former Member]

Hi All,

I made changes in the Java code as below

private final String utf ="UTF-8";

byte[] buffer  = new byte[in.available()];

                                        int bblen = in.read(buffer);

                                        String str = new String(buffer, utf);

                                        //str = str.replaceAll(utf, latin);

                                        System.out.println(str);

                                        out.write(str.getBytes(utf));

                                        out.close();

and pushed the data to target system without any encoding in Adapter level in any of module configuration, but I still get to see errors in response that the target system is unable to recognize the special characters. Can you please help

[iaki_vila]

Hi Jayanth,

Have you checked that the data is coming right to PI and is outgoing right in the ECC?. if the sender adapter, or the abap proxy in your case, is changing the codification, the java mapping will be useless.

Regards,

[Former Member]

Hi Inaki,

For confirmation purposes, I had pushed the data from PI RWB directly and still the response threw errors. In parallel can you pelase help m ein how do we see the encoding that has been set in ECC whilst the data is being triggered.

Regards,

Jayanth

Message was edited by: Jayanth Kumar

[iaki_vila]

Hi,

If you are using an ABAP proxy to send the message, go to sxmb_moni in the ECC system. With abap proxy the connection is direct between integration engines (dual PI system), with RWB also uses http protocol, im not sure, but the test is a bit different for the real one. I mean, even if you get a right answer with a java mapping or xsl mapping, you should check if the data is coming rigtht to PI from ECC.

Regards,

[Former Member]

Hi Inaki,

When I saw the special characters sxmb_moni from inbound message in PI, the characters look o, strangely they are throwing errors yet

The special characters appear as below in PI.

ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóŠ

and when it seem to reach the target system via SOAP Receiver channel (I havent not used any module as it is UTF-8, I have used only to convert the content in java mapping in UTF-8)

they appear to be as below when we got the response back with errors as below.

ΣτοματικοΔSæ åΠερПастаъбиý žÉψç㚎eŻőá šľΘρεψηςłosóÅÂÂ

[iaki_vila]

Hi,

If i understand you right when the message come to PI looks right. Have you tested to use trace.addInfo instruction before and and after of the treatment in the java mapping?, it's to check the rightness in that step. May be the problem is in the SOAP adapter, try to put Dont use SOAP Envelope and you make with an XSL or the same java mapping the SOAP envelope.

Regards,

[Former Member]

Hi Inaki,

I have already unchecked the parameter Do Not use the SOAP Envelope. I will check on the trace for java mapping. We are actually using a proxy Host and Port to hit the target systems with in SOAP Receiver Adapter. Does it have something to do?

Regards,

N. Jayanth Kumar

[anupam_ghosh2]

Hi Jayanth,

                  The input to this java mapping

                                        private final String utf ="UTF-8";
                                        byte[] buffer  = new byte[in.available()];
                                        int bblen = in.read(buffer);
                                        String str = new String(buffer, utf);
                                        //str = str.replaceAll(utf, latin);
                                        System.out.println(str);
                                        out.write(str.getBytes(utf));
                                        out.close();

might not be in correct encoding. You have mentioned encoding scheme as "UTF-8" (line 01) and reading the input in the same encoding (line 04).

I would suggest to change the value of the variable "utf" to some other encoding(if this does not work use other encoding too) as shown below

                                        private final String utf ="ISO-8859-1";
                                        byte[] buffer  = new byte[in.available()];
                                        int bblen = in.read(buffer);
                                        String str = new String(buffer, utf);
                                        str = str.replaceAll(utf, "UTF-8");
                                        System.out.println(str);
                                        out.write(str.getBytes("UTF-8"));
                                        out.close();

Regards

Anupam

[iaki_vila]

Hi Jayanth,

If you have read this document http://scn.sap.com/docs/DOC-16138 and you still get the problem you can try with this note Note 945570 - PI Sheet Input characters in Unicode system changed to #

Regards.

[Former Member]

Hi Inaki,

I have gone through the Note 94550. It has a comment as below.

The Internet Explorer encodes POST data in accordance with HTML page

encoding. The following article from Microsoft explains this:

http://support.microsoft.com/default.aspx?scid=kb;en-us;303612

In the current case the PI Sheet encoding is UTF-16. But as described in

the article the POST data is sent in UTF-8.

Is it something which we can force to use "UTF-8" by using a java mapping while returning the output file as below?

finalOutput.getBytes("UTF-8")

where the finalOutput is a string which has the data.

[iaki_vila]

Hi Jayanth,

I think a java mapping wouldn't resolve your problem because the data could be reaching to this point corrupted. Have you checked in the ECC that the data is outgoing correctly?, check the sxmb_monitor in the ECC. Check how the XML is coming to PI in the first pipeline step, if the data is corrupted, you can't resolve anything with a java mapping, if the data is correct, you can try with a java mapping  or xsl transformation.

<xsl:output method="xml" encoding="UTF-8"/>

Regard.

[anupam_ghosh2]

Hi Jayanth,

                  You can convert entire payload to UTF-8 using java mapping. Please check this wonderful "how to" guide  http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/502991a2-45d9-2910-d99f-8aba5d79f... by Stefan Grube .   Additionally you can check this thread https://scn.sap.com/thread/2068827

Regards

Anupam

[Former Member]

Hi Anupam,

I am using the below code to return the output in UTF-8

public void execute(InputStream inputStream, OutputStream outputStream)

                    throws StreamTransformationException {

                    // TODO Auto-generated method stub

                    try {

                              // Initializing Trace

                              docInput =           new BufferedReader(new InputStreamReader(inputStream));

                    } catch (Exception e) {

                              throw new StreamTransformationException("Unable to parse input document - ".concat(e.getMessage()));

                    }

                    try {

                              while ((inputLine = docInput.readLine()) != null) {

                                        inputData.append(inputLine);

                              }

                              finalOutput = inputData.toString();

                              outputStream.write(finalOutput.getBytes("UTF-8"));

[anupam_ghosh2]

Hi Jayanth,

                 I beleive this is not the complete code. Could you please explain the presence of variable "inputData".

Regards

Anupam

[Former Member]

Hi Anupam,

I have used the below code in simple.

private final String utf ="UTF-8";

byte[] buffer  = new byte[in.available()];

                                        int bblen = in.read(buffer);

                                        String str = new String(buffer, utf);

                                        //str = str.replaceAll(utf, latin);

                                        System.out.println(str);

                                        out.write(str.getBytes(utf));

                                        out.close();