Trying to send unicode characters with PI 7.0/SP 15 ( or 25 )

[bruce_hartley]

We are using XI 3.0/PI 7.0 here, I think it's either SP 15 or 25, SAP_BASIS has SP 25, PI_BASIS has SP 15.

We have the need to send uncode characters inside IDOC's from our SAP sytem to another non-SAP system.  From SAP to XI is IDOC's - from XI to the remote system is SOAP.

The SAP to XI part appears to be fine.  The data shows up and we can see the data properly in both SXMB_MONI and SXMB_IFR's Runtime Workbench ( RWB ).

The part where we are having an issue is the XI to remote system end.  It's defined as a SOAP adapter, transport protocol is HTTP, message protocol is SOAP 1.1, adapter engine is Integration Server".

We didn't understand at first what was going wrong ( and it's been going wrong for a long time, I recently was put in charge of supporting this ), so I enabled error logging.  Whenever an invalid character got sent, here's what we get as an error as follows ( some parts changed for security reasons ) in blue

Alert ID: ##46411##

Error:  : The sender of the concerned message is <SENDER>.

The erronous message belongs to <interface name> interface to interface.

The receiver service of this message is .

The Adapter type used for this message is SOAP and the error reported by the adapter is SOAP: call failed: java.io.IOException: invalid content type for SOAP: TEXT/HTML .

Use transaction SXMB_MONI in the XI System to see the erronous transaction.

Possible Subsequent Activities

Message Monitor <some long IP address for RWB>

Message Monitor (Secure Connection) <some long secure IP address for RWB>

Communication Channel Monitoring on <development system ID>

What we tried to do for this was to be very sneaky and add CDATA to the fields.  That got rid of the above, but then the adapter engine decided to take revenge on us and convert our leading and trailing < and > to &lt; and &gt; thus ruining the idea

I have been told to update to PI 7.1, but currently that is not possible.

Anyone have any idea how to do this?  I'm open to writing some code to fix this and the CDATA trick would have worked

[bruce_hartley · Accepted solution]

The almost final answer

As I stated - we logged a case with SAP on this - and for our version of XI ( buyer beware - other versions of XI may NOT  have this issue ), the use of various characters in the XML are NOT supported if it is in the material number field.  I am trying to ascertain with them if this is a bug that was fixed in a future version or if it is something with the adapter engine.  They even stated that slashes are ok if you use ALE to transfer materials from one system to another - they claim that in some "dictionary" that it's not valid.  If anyone has a clue as to what dictionary that is - I'm all ears.

I didn't get a complete list from them, but  in a past issue I found that apersands ( & ) cause issues as well.  What has me totally ticked off about this is that if I put the / in the description it works just fine.

Don't even get me started it took a month for them to tell us this plus system access - there is no way any person would need access to our system to give us that answer.

I am calling them out on this to get a better answer plus explicit documentation on what is allowed and what isn't - to me this is a bug that they fixed that they don't want to port backwards.  I do realize we have an old version of XI - but this should work no matter what the version - / is NOT one of the 5 special XML characters so unlike ampersands that isn't an issue either.

[stefan_grube · Accepted solution]

This is no PI problem.

The receiver does not accept a / in the material number field.

The message fails on receiver side, not inside the PI.

Restrictions on Field level should be part of the interface specification, if there is a WSDL for the interface, it should be described inside this. Maybe you can ask for it.

[Former Member · Accepted solution]

This message was moderated.

[bruce_hartley · Accepted solution]

Hey Stefan;

SAP finally got back to us after 2 months of traces and other stuff and told us the same thing.  I am now going to work with the vendor to get them to fix this.

Thank you for all of your efforts and replies and helping us with it.  I should have just gone back to the vendor with your answer but now that I have SAP's official answer as well I have a lot of ammunition to get the vendor to fix it.

[bruce_hartley · Accepted solution]

Hey everyone;

We finally got to the very bottom of this - and I wanted to tell everyone what we found because sometimes the issue ISN'T where you think it is and you really need to look through the ENTIRE communication chain when confronted with something like this.

As Stefan pointed out - it was in the receiving application - and of all places it could be it was in the area where they log the data coming over from our side - it turns out they had an audit buried in it that was checking to see if the material number was alphanumeric.  That was throwing an error that wasn't being logged ( the irony of that point is what made this issue drive us crazy ) and thus made the error all but invisible to everything but the XI soap channel which detected the error and then returned the HTTP 500 Internal Server Error the receiving application was throwing in their soap receiver code.

Thank you all who contributed and thank you Stefan for coming up with the right answer two months before SAP support did.  Where SAP support helped in additio into Stefan was to

For those of you who want to know how we got to see that it was the receiving side, it was SAP support who had us install the latest version of XPI and go through example 50.  That OSS note is 1514898 and it always has the latest on how to install and use XPI.  We also used the TCP Gateway trace tool to trap the traffic and report to us what was going on at each point.

I hope this information helps others who are stuck in a similar situation.  All I can say is "trust the trace".

[former_member190681 · Accepted solution]

thank you. helpful information.

[bruce_hartley]

Stefan;

I did what you suggested - we took a material that bombed, copied it and removed the slash, and it went to the remote application just fine.  We then copied the working one and added a slash, and it failed.  In both cases it was a brand new material just to make sure we weren't introducing another isuse.

I then had the person who wrote the remote system to look in their log for any attempt to send both, they found the first one but no evidence of the second one was found.  I have worked with their company for a number of years and if there was an issue they would readily admit it and take steps to fix it.  That's not to say it isn't their issue, it's just none of us could find any evidence of it ever being sent thus our take that it is failing in XI somewhere.

As a summary to help those following this now and in the future, here's what we have found so far:

1 - The connection between SAP and XI was not set to Unicode.  What was strange was that it was wrong on our test system but correct on our production system - this fixed our unicode issue on our test system which was the subject of why this got posted originally.  As pointed out by others, also make sure that your receiving system can handle UTF-8 ( ours can as we found out ).

2 - We have opened up an OSS note with SAP on the issue where a slash in the material number causes the interface to fail.  I will keep this space posted on the progress, we just sent a trace of the channel to them yesterday.

Thank you again to everyone who has posted on this and helped, I now have a much better understand of how SAP handles unicode characters and now we understand what our issue is.

[stefan_grube]

Maybe you can trace the http stream, then you know if the message reached the service.

Check this blog:

http://scn.sap.com/people/stefan.grube/blog/2007/03/29/troubleshooting-soap-http-and-mail-adapter-sc...

[bruce_hartley]

Stefan - I read the link - thank you for posting that - what we used for a trace was a tool that SAP provides to do that, our consulting partner ran it and sent the trace they asked for.

If they need more, I would be more than willing to try it.

[bruce_hartley]

To everyone;

I may have posted the wrong information on this, after testing with two different materials we found out that the chinese characters are NOT the issue but what is the issue is a slash.

This may be an already reported bug by someone besides me, but if there is a slash in the material number the SOAP adapter fails.  All that we are doing in the mapping is a straight one-to-one copy of every field, so that shouldn't be an issue.  I reported this to SAP as issue number 432212/2013 and also I remembered an old ticket where there were ampersands in the data that caused an issue too, and what we were told by SAP at the time was to change them all to &apos; ( thank you SO much SAP for not fixing the issue in our version ).

I have not seen the ampersand issue in some time and we gutted and rebuilt our XI systems a year ago, so maybe that issue is gone.  Also, we are planning an update from SAP-BASIS from SP 25 to SP 27 and PI BASIS from 15 to 17, so maybe that will solve it too.

I am hoping to get to the bottom of both issues within the next few days.

[stefan_grube]

Let me tell you that you are talking about two different issues. First of all it is not the SOAP adapter who complains about the slash, it is your receiving service. It expects plain numbers instead of a slash. The description of your error message clearly states that it comes from the server, not from the SOAP adapter itself.

The other thing is the ampersand issue. There are two characters which are not allowed within an XML element: ampersand & and less <

A & must be replaced with &amp; not &apos; There is nothing to fix from SAP side, it is XML standard.

[bruce_hartley]

Hey Stefan;

I was told by SAP when I put in an OSS note for the ampersand issue ( and you are correct - it's &amp;, not &apos - totally my bad in that I wasn't paying close attention ) that it is a "feature" of our version of XI and that if we wanted it fixed to go to version 7.1 - whch got us very upset.

Thank you for the info this though - I knew about the ampersand - I didn't know about the < being affected as well - although we don't generally use that for anything material related - the ampersand issue though did impact us and like you said - we had to work around it.

[bruce_hartley]

Jens and Stefan - I'm now working on this issue again, now that I can get the full text I will be resending the failures and see what it says for them.

[Former Member]

Hi Bruce, I'm coming in late to this thread but I'm curious about your payload before you altered it with CDATA tags in the mapping.

I'm sorry if you already understand this point, but in RWB one can see the payload at two different steps, Integration Engine (which will match the payload you see in SXMB_MONI) or Adapter Engine (what the SOAP adapter is seeing).

Does the payload of the SOAP Adapter in RWB (Adapter Engine) look like valid XML? What is the encoding there?

SOAP Adapter defaults to UTF-8 so it is an interesting problem. Stephan is of course right you should know what the receiving system expects. But maybe the error is occurring during the SOAP parsing before delivery. Looking at the payload from the adapter rather than the Integration Engine may help you isolate.

[bruce_hartley]

Aaron;

I agree with you about examining the output - we need to see what the payload is at the various steps - what I found out was that IE was "interpreting" the output so what I thought was invalid XML may actually have been valid XML and that what we did with CDATA made it worse.

For example, we thought that our version of XI wasn't outputting <, >, ', ", and & correctly because we would see it in the output as a single ', but after examining the output from what we did with CDATA, I found that it turned the leading < into &lt; and the trailing > into &gt; which would be correct behaviour.

I am having my co-partner in this undo the CDATA change and then we are going to start creating test cases and one of the things will be to "intercept" the output and try sending the receiver new output with a different codepage to see if it will handle Chinese and other unicode characters.

Sorry this has taken so long to investigate, we had an XI issue that we had to resolve and now this week we can get back into the investigation into this and I do promise to let everyone know what we find as this topic seems to come up a lot but I haven't seen any solid solutions.

[Former Member]

Yes, if you can isolate the problem inside the SOAP adapter step you might be able to use the charset parameter or a module bean to change encoding whatever the case may be. Good luck!

[JaySchwendemann]

The blue stuff you mentioned as "error logging" is actually called alerting (alert framework). Maybe this will help when talking about things in PI environment. There are possibilities to enhance the mostly boilerplate texts with your own texts and variables.

However, to really dig into this problem I would suggest either:

1. Have a look at communication channel monitoring in RWB (located in first tab) and locate the SOAP communication channel. It should show you some (oftern rough) error information on what was going wrong

2. Go to RWB and locate an erroneous message and open the audit log (clicking on detail first). There you may see some additional error information, but when it comes to SOAP error information might be limited

3. Last you should really check transaction SXMB_MONI within the ABAP stack (SAP GUI). Locate a erroneous message and search for Error section under "SOAP-HEADER". If this doesn't put you in the right place, have also a look at "Trace" section which might contain a stack trace with additional information.

HTH

Cheers

Jens

[bruce_hartley]

Jens - thank you for correcting my naming of the feature, it is the Alert framework that I was referring to.  I also found out through another recent post that OSS note 1294312 covers how to get the complete text of an error which really will help - we were getting that cutoff issue too.

[bruce_hartley]

Stefan;

I had never thought to think that the other system might be causing the failure nstead of XI - thank you!  I will check into that, it may take a bit ( it may be as much as 3 weeks ) but I think you are well on track to having the right answer because it makes total sense.

I will ask the person who "owns" the software on that system what code page(s) we can use and if that solves it I will definitely give you the credit and will add a further explanation.

Thank you again for the idea and I hope it fixes this.

[stefan_grube]

Before you try to find a work around, you should undesrtand, why the message fails.

As the SOAP adapter reports an HTML response instead of in XML response, the remote system behaves odd. So check the error log on the remote system, if there is any issue with the message.

You are talking about unicode data, but did you check, which code page is expected by the remote system? Maybe it does not accept UTF-8 at all. Figure out which code page is accepted and then change it.