Encoding Issue : JMS and Mapping : utf-8 iso8859-1

[former_member192079]

Hi All,

I am facing some problem with encoding issue.

Scenario : JMS --> SAP PI --> JMS

Requirment : Input plain text file contain some special characters,"©®" . Based on this condition,In Java Mapping

we check the Payload and changed the 'encoding' tag to UTF-8 or ISO8859-1.

: <?xml version="1.0" encoding="UTF-8"?> in the target XML output.

While testing in Operation mapping our Java mapping works fine. as the encodeing tag changes from

UTF-8 to ISO8859-1 if the special character exists.But if I test the same in Integration Directory(Test Configuration)

or did a end to end testing. The encoding tag did'nt changes.

For testing we had to a set of Plain Text files with UTF-8 and ISO8859-1 .

I tried the options of using beans in Adapter modules in Sender JMS channel.

MessageTransformBean, TextCodepageConversionBean, XmlAnonymizerBean

These doc & threads ,was also referred[How to Handle Encoding in PI|http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/502991a2-45d9-2910-d99f-8aba5d79fb42]

Regards,

Ashutosh R

[GabrielSagaya · Accepted solution]

Did you try with XSLT mapping ?

<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" encoding="ISO-8859-1"/>
<xsl:template match="/">
<xsl:copy-of select="*" />
</xsl:template>
</xsl:stylesheet>

[stefan_grube]

Why you change the encoding flage based on a value?

I assume that the encoding is always iso8859-1.

The standard beans do not work as they assume the encoding in the content-type of the message (something like "text/plain;charset=iso8859-1")

But as this is an HTTP header parameter, it does not exist in JMS sender adapter.

As workaround you can use the MTB twice.

For the first you set Transfer.ContentType to text/plain;charset=iso8859-1

For the second (where you use also FCC) you set Transfer.ContentType to text/plain;charset=UTF-8

[former_member192079]

Hi Stefan.

Sorry for the late reply....

Why you change the encoding flage based on a value?

we need to send target XML file with appropriate tag value based upon

special character " © ®..

if © ®. is present in data, encoding should be iso-8859-1 else UTF-8.

The issue i am facing is : The required functionality work by Java mapping and tested out of PI box

Tested same result in Operation mapping Test ..

But while testing end to end or Integration directory ( Test configuration)

the result varies... i mean for data not having " © ®." it create the encoding tag value as

Iso-8859-1. which should be utf-8...

Is in one of pipeline steps after operation mapping , the Source file is Flat text file, with no encoding details, Only PI " © ®" if else option will decide the target xml file encoding...apart from other mapping transformations.

Regards,

Ashutosh

[former_member192079]

and what add to my confusion is the SXMB_MONI trace steps snapshots

<Trace level="2" type="T"> Call method execute of the application Java mapping ESADeploy.ModifyOrg123456 </Trace>

<Trace level="1" type="T">*** START APPLICATION TRACE ***</Trace>

<Trace level="2" type="T">Process Started</Trace>

<Trace level="1" type="T">*** END APPLICATION TRACE ***</Trace>

<Trace level="3" type="T">Search esawarehouse/input_propertyfile.xml (http://xx.com/Logistics/ESAWarehouse, -1) in swcv 1c74ad40-cf77-11de-9c50-d0f1346358c2.</Trace>

<Trace level="3" type="T">Loaded resource esawarehouse/input_propertyfile.xml</Trace>

<Trace level="2" type="T"> Java mapping ESADeploy/ModifyOrg123456 completed. (executeStep() of ESADeploy.ModifyOrg123456). </Trace>

<Trace level="3" type="T">Dynamic Configuration ( http://sap.com/xi/XI/Message/30/general senderAgreementGUID b2465081fad9321----- )</Trace>

<Trace level="1" type="T"> ContentType text/plain;charset=UTF-8 </Trace>

</Trace>

</Trace>

The last step is no where Sync with the result of the Java mapping output of <encoding> tag value...

no idea why this step executes...

Regards,

Ashu

[stefan_grube]

> Is in one of pipeline steps after operation mapping , the <encoding> tag value is overwritten...

Operation mapping by graphical mapping tool sets encoding alwasy to UTF-8. You cannot do anything against this behaviour.

[former_member192079]

Yes, Stefan u are correct.

But I am using Java Mapping, and testing it in Operation mapping level.

Regards,

Ashutosh

[stefan_grube]

> But I am using Java Mapping, and testing it in Operation mapping level.

Maybe I did not understand the issue correctly.

After your Java mapping, the encoding tag of the XML header should be the same as you have coded.

Is this not the case? Besides mapping the payload is not changed inside XI runtime.

[former_member192079]

Hi stefan,

Yes, the Java Mapping changes the <encoding> tag value as coded.only in , while testing in operation mapping tab.

BUT when i execute the scenario end to end with same set of test data, the <encoding> tag value is not , what it is expected.

even i tried to test the interface in integration directory ( Test configuration).. the <encoding> tag value is unexpected....

Regards,

Ashu

[former_member472138]

Hello Ashu,

You need to get into Visual administrator and change the default parameter for encoding as you want. This is the way you can make your output to be expected. So, catch up the basis guys to perform this.

Regards

Pothana

[stefan_grube]

I think in test tab you use a different codepage for the characters compared to the original files.

You have to consider in your Java program that the "©®" characters come in ISO codepage, so you cannot simple do a string compare.

You have to copy the inputstream to a byte array and check for byte values of the iso codepage.

Maybe you post your java program, so we can discuss it.

[former_member192079]

Your concern is viable but One thing to recall.

Result is ok in Operation mapping test tab

Result unexpected :

1. Integration directory ( Test configuration ) : Test data is same as for Operation map testing...

2. End to end testing.

i.e why, my concern was with any pipeline steps since Integration directory ( Test configuration ) test the whole interface..

will paste the code... and flow details...

regards,

Ashu

[former_member192079]

Hi,

The code excceds the max limit of 150000chars. i can mail it to u..

still the fraction snapshot is :

//******************Start of Code Parse the input XML******************//


        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
		Document documentIn = null;
		DocumentBuilder builder = null;
		try {
				builder = factory.newDocumentBuilder();
			} catch (ParserConfigurationException e1) {
				e1.printStackTrace();
			}
        try {

            documentIn = builder.parse(in);

        } catch (SAXException ex) {
            Logger.getLogger(ModifyOrg1234567.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(ModifyOrg1234567.class.getName()).log(Level.SEVERE, null, ex);
        }
                        NodeList list_f01=documentIn.getElementsByTagName("body");
                        String str_f="";
				for(int k=0; k<list_f01.getLength();k++){
                                  	Node f=list_f01.item(k);
					f=f.getFirstChild();
					str_f= str_f+f.getNodeValue();

				}
      //******************End of Code Parse the input XML******************//

*************Start propery file lookup***********


FileInputStream inPropFile1 = new FileInputStream("C:/Documents and Settings/rawata1/Desktop/ESA WareHouse/input_propertyfile.xml");//
//InputStream inPropFile1 =  this.getClass().getClassLoader().getResourceAsStream("esawarehouse/input_propertyfile.xml");

                Properties properties = new Properties();
                try {

                    properties.loadFromXML(inPropFile1);

                    keytagvalueTokenOne = properties.getProperty(this.tokenOne);
                    keytagvalueTokenTwo = properties.getProperty(this.tokenTwo);
                    keytagvalueTokenThree = properties.getProperty(this.tokenThree);
********************end look up**************************

******Call search method******** 

encodingType =   fixSpecialCharforWeb(fileContent);

**********Setting the encoding tag value :**************

TransformerFactory tf = TransformerFactory.newInstance();
			Transformer transform = tf.newTransformer();

                        if (encodingType == true) {
                           
                             transform.setOutputProperty(OutputKeys.ENCODING, "iso-8859-1");
                        } else if (encodingType == false)
                        {
                             transform.setOutputProperty(OutputKeys.ENCODING, "utf-8");
                        }

                        System.out.println("ashutosh transformer executed");
			transform.transform(new DOMSource(documentOut),new StreamResult(out));
                      //out.write(inptxml.getBytes());

Read the input xml file, the whole data is stored in a String "fileContent".

The "fileContent" data is also passed to method for Special Character Searching

fixSpecialCharforWeb();which decides the encoding tag value.

Their is a property File lookup is done to get some token values

then an whole series of String concatenation creates a target xml.

The

transform.setOutputPropert

y is used, based upon the Search method output.

I even use even o

ut.write(inptxml.getBytes(utf-8 or iso8859)

inplace of

setOutputProperty.program works but again end to end testing and ID(test config ) give unexpected

results.

So i am back to zero...

Regards,

Ashu

[stefan_grube]

> fixSpecialCharforWeb();which decides the encoding tag value.

This method would be more interesting.

[former_member192079]

Hi

public static boolean fixSpecialCharforWeb(String text) {

int i = 0;

Character c = null;

char[] ctext = null;

StringBuffer newText = new StringBuffer("");

//boolean encodingType = false;

if ((text == null) || (text.trim().length() == 0)) {

return encodingType;

} else {

try {

for (i = 0; i < text.trim().length(); i++) {

ctext = text.trim().substring(i, i + 1).toCharArray();

c = new Character(ctext[0]);

//Single quote

if ((text.trim().substring(i, i + 1).equals("'")) || (c.hashCode() == 8217) || (text.trim().substring(i, i + 1).equals("?")) || (c.hashCode() == 146) || (c.hashCode() == 145)) {

//newText.append("'");

encodingType = true;

return encodingType;

}

//Double quotes

if ((c.hashCode() == 8220) || (c.hashCode() == 8221) || (c.hashCode() == 147) || (c.hashCode() == 148)) {

//newText.append(""");

encodingType = true;

return encodingType;

}

// bullet point

if ((c.hashCode() == 8226) || (c.hashCode() == 149)){

encodingType = true;

return encodingType;

}

// tilde

if ((c.hashCode() == 732) || (c.hashCode() == 152)){

encodingType = true;

return encodingType;

}

// Soft Hypen

if (c.hashCode() == 173){

encodingType = true;

return encodingType;

}

// En-Dash

if ((c.hashCode() == 8211) || (c.hashCode() == 150)) {

encodingType = true;

return encodingType;

}

// Em-Dash

if ((c.hashCode() == 8212) || (c.hashCode() == 151)) {

encodingType = true;

return encodingType;

}

// Euro Sign

if ((c.hashCode() == 8364) || (c.hashCode() == 128)) {

encodingType = true;

return encodingType;

}

// Yen Sign

if (c.hashCode() == 165) {

encodingType = true;

return encodingType;

}

// Pound Sign

if (c.hashCode() == 163) {

encodingType = true;

return encodingType;

}

// 1/2 sign

if (c.hashCode() == 189) {

encodingType = true;

return encodingType;

}

// 1/4 sign

if (c.hashCode() == 188) {

encodingType = true;

return encodingType;

}

// 3/4 sign

if (c.hashCode() == 190) {

encodingType = true;

return encodingType;

}

// Sword/dagger

if ((c.hashCode() == 8224) || (c.hashCode() == 134)) {

encodingType = true;

return encodingType;

}

// Trademark

if ((c.hashCode() == 8482) || (c.hashCode() == 153)) {

encodingType = true;

return encodingType;

}

// Ampersand &

if ((text.trim().substring(i, i+1).equals("&")) || (c.hashCode() == 38)) {

encodingType = true;

return encodingType;

}

//Registered mark

if ((text.trim().substring(i, i + 1).equals("?")) || (c.hashCode() == 174)) {

//newText.append("®");

encodingType = true;

return encodingType;

}

//Copyright mark

if ((text.trim().substring(i, i + 1).equals("?")) || (c.hashCode() == 169)) {

encodingType = true;

return encodingType;

}

// Question.

if (c.hashCode() == 63 && c.toString().equals("?")){

//newText.append("?");

encodingType = true;

return encodingType;

}

//handling symbol ?

if ((text.trim().substring(i, i+1).equals("?")) || (c.hashCode() == 233)) {

encodingType = true;

return encodingType;

}

//?

if ((text.trim().substring(i, i+1).equals("?")) || (c.hashCode() == 232)) {

encodingType = true;

return encodingType;

}

if (c.hashCode() == 144) {

encodingType = true;

return encodingType;

}

}

} catch (Exception e) {

e.printStackTrace();

}

}

return encodingType;

}

[stefan_grube]

When you do something like this:

> ctext = text.trim().substring(i, i + 1).toCharArray();

> c = new Character(ctext[0]);

you have to keep in mind that the substring does not work for byte but for characters.

That could produce errors when the original text is ISO codepage, but Java count this as UTF-8. All non-ascii iso characters might be grouped with other bytes (up to 4 bytes) and so your code might not work.

When your original payload is iso codepage, you should first du a conversion to UTF-8

That could be done in just one line of code.

[former_member192079]

the UTF-8 conversion is happening at Sender JMS channel (Message transformation Bean).

so Java mapping receives the UTF-8 format only....

As per ur input, in ur first reply to this thread...i used it...

-> Is it possible to get the source code of Message transformation Bean to check how it tranform the message from iso to utf8.

because i guess, i m only changing the encoding tag value -- not exactly transforming the payload, as to

relevant <encoding> tag value...

regards,

ashu

Edited by: ashutosh rawat on Nov 22, 2010 1:55 PM

[stefan_grube]

When your XML is in UTF-8, then you cannot compare the characters with their ISO values.

But you cn compare it with the corresponding literal.

[former_member192079]

Hi Stefan,

When your XML is in UTF-8, then you cannot compare the characters with their ISO values.

Search is handled with String and Array of characters. in the method fixSpecialCharforWeb

if ((text.trim().substring(i, i + 1).equals("'")) || (c.hashCode() == 8217).

The problem identified : we used a Static field in JavaMapping.

Thanks for your inputs.....

Ashu