cancel
Showing results for 
Search instead for 
Did you mean: 

"Invalid UTF-8 start byte 0xbf" when changing universes

Joe_Peters
Active Contributor
0 Kudos

I'm attempting to change a report to another universe using REST, and I am getting the following error after submitting the mapping:

<error_code>RWS 00070</error_code>
<message>Internal server error. (RWS 00070)</message>
<stack_trace>javax.ws.rs.WebApplicationException: java.lang.RuntimeException: [was class java.io.CharConversionException] Invalid UTF-8 start byte 0xbf (at char #54921, byte #51299)&#13;

    at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:165)&#13;

    at com.sap.webi.raylight.ExtendedJAXBElementProvider.readFrom(ExtendedJAXBElementProvider.java:63)&#13;

    at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:1030)&#13;

    at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:610)&#13;

    at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:575)&#13;

    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:239)&#13;

    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:242)&#13;

    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:242)&#13;

    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:242)&#13;

    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:91)&#13;

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)&#13;

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)&#13;

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)&#13;

    at java.util.concurrent.FutureTask.run(FutureTask.java:138)&#13;

    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)&#13;

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)&#13;

    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)&#13;

    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:113)&#13;

    at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:102)&#13;

    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:464)&#13;

    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:149)&#13;

    at com.sap.bip.rs.server.servlet.BIPServletController.invoke(BIPServletController.java:93)&#13;

    at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)&#13;

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)&#13;

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)&#13;

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)&#13;

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)&#13;

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)&#13;

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)&#13;

    at com.businessobjects.sdk.actionfilter.WorkflowFilter.doFilter(WorkflowFilter.java:45)&#13;

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)&#13;

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)&#13;

    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)&#13;

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)&#13;

    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)&#13;

    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)&#13;

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)&#13;

    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)&#13;

    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)&#13;

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)&#13;

    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)&#13;

    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)&#13;

    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)&#13;

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)&#13;

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)&#13;

    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)&#13;

    at java.lang.Thread.run(Thread.java:763)&#13;

Caused by: java.lang.RuntimeException: [was class java.io.CharConversionException] Invalid UTF-8 start byte 0xbf (at char #54921, byte #51299)&#13;

    at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)&#13;

    at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)&#13;

    at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)&#13;

    at com.ctc.wstx.sr.BasicStreamReader.getTextCharacters(BasicStreamReader.java:830)&#13;

    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleCharacters(StAXStreamConnector.java:312)&#13;

    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:176)&#13;

    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:351)&#13;

    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:323)&#13;

    at org.apache.cxf.jaxrs.provider.JAXBElementProvider.unmarshalFromInputStream(JAXBElementProvider.java:211)&#13;

    at org.apache.cxf.jaxrs.provider.JAXBElementProvider.doUnmarshal(JAXBElementProvider.java:177)&#13;

    at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:146)&#13;

    ... 46 more&#13;

Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0xbf (at char #54921, byte #51299)&#13;

    at com.ctc.wstx.io.UTF8Reader.reportInvalidInitial(UTF8Reader.java:302)&#13;

    at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:188)&#13;

    at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)&#13;

    at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)&#13;

    at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)&#13;

    at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)&#13;

    at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)&#13;

    at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)&#13;

    at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)&#13;

    ... 54 more&#13;

Any idea what might be causing this?

Changing the universe manually (via WebI) works fine.

Accepted Solutions (1)

Accepted Solutions (1)

former_member197386
Active Contributor
0 Kudos

Dear Joe,

It seems the message body you sent with your POST request didn't have the correct encoding. Please, could you verify you are using UTF8 charset to encode your request?

Best regards,

Anthony

Joe_Peters
Active Contributor
0 Kudos

Thanks for your reply, Anthony.

Yes, I'm setting the XML transformer to UTF-8 with:


transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");

And setting the content-type as:


application/xml; charset=utf-8

I narrowed down the cause of he problem as a non-printable character (0xBF) in the prompt LOV.  It does not appear to be getting escaped.

former_member197386
Active Contributor
0 Kudos

Hello Joe,

To sum up, you have an issue when you send a POST request to perform a change source with parameters, right?

Do you know if this issue can be easily reproduced?

Regards,

Anthony

Joe_Peters
Active Contributor
0 Kudos

I did some more research into this issue.

It also occurs when I attempt to refresh the report normally (i.e., not when changing universes), if the prompt value includes non-printable characters.  This happens even though the value was received via an earlier call to the server.

I make a call to /biprws/raylight/v1/documents/5974/parameters.  This is a portion of the response:


<parameters>

    <parameter dpId="DP0" type="prompt" optional="false">

        <id>0</id>

        <technicalName>Test</technicalName>

        <name>Test</name>

        <answer type="Text" constrained="false">

            <info cardinality="Multiple">

                <lov searchable="true" refreshable="true" partial="true" hierarchical="true">

                    <id>UNIVERSELOV_DS0.DO3e7</id>

                    <updated>2014-12-29T08:34:31.000-05:00</updated>

                    <values>

                        <value>WORKERS¿ COMP</value>

                    </values>

                </lov>

The "¿" character is actually $C2BF.  When I return this value to the server via another parameters call, I get the same error as before.

former_member197386
Active Contributor
0 Kudos

Hmmm ... Is this character (inverted question mark) voluntary inserted?

You confirm it is this one which causes trouble:

http://www.fileformat.info/info/unicode/char/00bf/index.htm

Regards,

Anthony

Joe_Peters
Active Contributor
0 Kudos

Well...  whether the person who entered the data into the front-end system intended to use this character -- probably not.  But it is in the database, and it works correctly in the WebI prompt panel.

former_member197386
Active Contributor
0 Kudos

Thanks Joe, that allows to have a better understanding of the usecase.

Regards,

Anthony

former_member197386
Active Contributor
0 Kudos

Hey Joe,

I tried to reproduce this issue using Azot with the attached script (I just add a prompt on Year to Formatting Sample). I send the '¿' character (I enter ALT + 168) as answer value.

However, I didn't reproduce the issue for now.

Anthony

eric_festinger
Contributor
0 Kudos

hi Joe,

Like Anthony, I would say that it is a character encoding issue on your side. It seems that the result may also depend of the stream or writer you are using... (see for example Producing valid XML with Java and UTF-8 encoding - Stack Overflow)

Can you show us more than just the transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8") part?

Regards,

eric festinger

Joe_Peters
Active Contributor
0 Kudos

Hi, Anthony.

I took your xml and just tweaked it to point to my test report.  Here's the error that I get:


java -jar azot.jar baddate.xml

Exception in thread "main" com.sap.azot.AzotException: javax.xml.bind.UnmarshalException

- with linked exception:

[com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.]

        at com.sap.azot.Launcher.launch(Launcher.java:228)

        at com.sap.azot.Launcher.main(Launcher.java:87)

        at com.sap.azot.Launcher.main(Launcher.java:61)

Caused by: javax.xml.bind.UnmarshalException

- with linked exception:

[com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.]

        at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:206)

        at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:175)

        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)

        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:162)

        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:171)

        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:189)

        at com.sap.azot.Launcher.launch(Launcher.java:226)

        ... 2 more

Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 seq

uence.

        at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)

        at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)

        at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)

        at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(XMLEntityScanner.java:1252)

        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(XMLDocumentFragmentSc

annerImpl.java:1654)

        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocument

FragmentScannerImpl.java:3020)

        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)

        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)

        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScanne

rImpl.java:510)

        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)

        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)

        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)

        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)

        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:648)

        at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:203)

        ... 8 more

Different message, but seems related.

former_member197386
Active Contributor
0 Kudos

Please, could you attach the "baddate.xml" to your last post?

Anthony

Joe_Peters
Active Contributor
0 Kudos

Here it is.  I also ran it with a legitimate value just to make sure the error wasn't from something else.

Joe_Peters
Active Contributor
0 Kudos

Sure, here it is:


transformer = XMLUtilities.transformerFactory.newTransformer();

transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");

transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");

StringWriter writer = new StringWriter();

transformer.transform(new DOMSource(node), new StreamResult(writer));

return writer.getBuffer().toString().replaceAll("\n|\r", "");

I saw that same post on SO.  I don't think that's applicable since I'm writing to a String, not a file.  But if you can suggest anything, I'll try it.

eric_festinger
Contributor
0 Kudos

I didn't try (not even to compile), but maybe something like that would work:

...

ByteArrayOutputStream baos = new ByteArrayOutputStream();

transformer.transform(new DOMSource(node), new StreamResult(baos));

return baos.toString("UTF-8").replaceAll("\n|\r", "");

eric

Joe_Peters
Active Contributor
0 Kudos

It compiled, but I got the same error.


transformer = XMLUtilities.transformerFactory.newTransformer();

transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");

transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");

ByteArrayOutputStream baos = new ByteArrayOutputStream();

transformer.transform(new DOMSource(node), new StreamResult(baos));

return baos.toString("UTF-8").replaceAll("\n|\r", "");

ted_ueda
Active Contributor
0 Kudos

I just downloaded your baddate.xml and looked at it with a hex editor, and the upside-down question mark is encoded 0xbf and not 0xc2 0xbf.

The xml may have been modified using an ANSI and not UTF-8 text editor.

Sincerely,

Ted Ueda

Joe_Peters
Active Contributor
0 Kudos

That actually did catch my eye.

When I change it to 0xC2BF, it does work.  However, I wasn't able to compile the source to azot, so I wasn't able to watch the traffic in Fiddler.

ted_ueda
Active Contributor
0 Kudos

If I were to guess, the actual character is supposed to be a fancy right-close-single-quote copied-and-pasted from Microsoft Word. 

But that couldn't get translated to the encoding used by the database or the database driver, so it got represented by an upside-down-question-mark (characters not representable by a encoding scheme usually gets represented by a question mark).  And likely the driver or some intermediate isn't correctly encoding that upside-down-question-mark to UTF-8. 

Sincerely,

Ted Ueda

Joe_Peters
Active Contributor
0 Kudos

I believe that's probably correct.  But I'm simply sending back the text that I received from a /parameters call.  If I need to translate it to something, I will, but I don't know how to return it in a subsequent call.

It works in Azot, but I can't see enough of the code to know what it's doing differently.

ted_ueda
Active Contributor
0 Kudos

That's the thing - azot shouldn't translate UTF-8 to ANSI, so I'm assuming there's something in your workflow that causes that translation from the time the /parameters is received to when it's sent back.

If you consume /parameters using some other tool, do you see 0xbf or 0xc2 bf?

If you have a SAP Support Contract, I can certainly help you directly.

Sincerely,

Ted Ueda

Joe_Peters
Active Contributor
0 Kudos

I think I fixed it.  This seems to do it:


return new String(writer.getBuffer().toString().replaceAll("\n|\r", "").getBytes("UTF-8"));

<value>A WORKERS?? 5

former_member197386
Active Contributor
0 Kudos

Hi Jo,

I looked at the Azot source code to remember and here is how the content is sent to the server:


final String contentLenght = String.valueOf(singleContent.getBytes().length);

connection.setRequestProperty("Content-Length", contentLenght);

final DataOutputStream out = new DataOutputStream(connection.getOutputStream());

out.write(singleContent.getBytes(Charset.forName("UTF8")));

out.flush();

out.close();

Regards,

Anthony

Joe_Peters
Active Contributor
0 Kudos

Thanks, Anthony.

I did resolve it, with a similar solution.

I found that although I was transforming the XML to a String with the proper UTF-8 encoding, my String was later being decoded when it was being written to the output stream of the URLConnection. So now I just transform the encoded node directly to the OutputStream:


requestXML.toOutputStream(connection.getOutputStream());


public void toOutputStream(OutputStream os)

throws XMLUtilityException

{

    Transformer transformer;

    try

    {

        transformer = XMLUtilities.transformerFactory.newTransformer();

        transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");

        transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");

        transformer.transform(new DOMSource(node), new StreamResult(os));

    }

    catch (TransformerException e)

    {

        throw new XMLUtilityException("Unable to render XML.",e);

    }

}

seems to be working.

Never had to worry about this stuff with REBean!

Answers (1)

Answers (1)

former_member197386
Active Contributor
0 Kudos

Hi Joe,

Is this question answered now or do you need additional help?

Best regards,

Anthony

Joe_Peters
Active Contributor
0 Kudos

I'm all set now, Anthony, thanks.