encoding issue

[Former Member]

Hi Experts,

I need some insights for the character encoding.

The legacy system is sending nordic characters ( öä) in a fixed length flat to PI. We are using JMS adapter to receive the data.

The character set of Legacy system is

WE8MSWIN1252.

we have some outbound interfaces also with the same legacy system.

In receiver JMS communication channel I have used 

CCSID = 1252 

under module ( AF_Modules/MessageTransformationBean) 

Transform.ContentType  = text/xml;charset = UTF-8

With this setting Legacy system is able to view the nordic characters .

But for the inbound interfaces it is not working .

I have used following CCSID's

00819

1208

1252

used TextCodepageConversionBean but unsuccesfull.

We are getting junk characters in place of nordic when we see in comm channel monitoring, MONI or in IDoc.

Can anyone tell why its not working,

Regards

Biplab Das

Edited by: biplab das on Jul 28, 2011 2:57 PM

[stefan_grube · Accepted solution]

Don't use TextCodepageConversionBean , that won't help.

Are you sure, that the sender provides correct data?

Dump the incoming message into a file and check with a hex editor.

The CCSID should be 1252

[Former Member · Accepted solution]

Hi Stefan,

Thanks.

I have checked the file content , it is looking fine.it is having nordic characters

I have used CCSID 1252 and UTF-8 in the module but.. it is not working.

when i chenged the encoding of the dumped file to UTF-8 , and used a file adapter it worked fine.

any other workaround?

[stefan_grube · Accepted solution]

Now I see the problem. You have an XML file.

In this case you should not change the encoding, as the XML encoding declaration does not change to the encoding of the file.

Try:

- no CCSID

- CCSID 1208 or 1209 (UTF-8)

[Former Member · Accepted solution]

HI Stefan,

I am sorry but the input is not XML.

thats a flat file and PI is reading it from MQ. ( MQ is not doing any transformation).

[stefan_grube · Accepted solution]

What do mean with: "when i changed the encoding of the dumped file to UTF-8 , and used a file adapter it worked fine."?

How do you change the encoding of a flat fle?

[Former Member · Accepted solution]

Ok let me explain.

the end to end flow is

Legacy system (oracle DB) -> MQ->PI->SAP

i have asked the MQ guys to hold the message in MQ and they did it and saved the text file as default encoding type (ANSI).

then i used a file adapter to pick the file from local server. it was not working with all combination i used.

Then i opend the file and changed the encoding to UTF-8 while saving. Then it worked.

But i can understand this is not a proper testing as i will get the data from Legacy system with char set WE8MSWIN1252.

What i am not getting here is outbound one worked... why the inbound not working.

Stefan - what u suggest?

Regards

Biplab

[stefan_grube · Accepted solution]

Are you doing a conversion to XML with MessageTransformBean?

Which encoding do you use with this bean?

[stefan_grube · Accepted solution]

Could you upload the file to a share so I can have a look?

[Former Member · Accepted solution]

Hi Stefan...

Legacy system is pulling the data from database, see below the record.

000000010103463348010220050511äåö 101

[Former Member · Accepted solution]

Yes .. i am using Message Transformation bean.

i have tried using

Transform.ContentType  = text/xml;charset=UTF-8
Transform.ContentType  = text/xml;charset=ISO-8859-1

i have also tried without using the parameter Transform.ContentType .

Also tried with CCSID 1208 and 1209.

Not working

[stefan_grube · Accepted solution]

I wonder if the file in MQ has the correct encoding.

It is just a test file, so it might have wrong characters.

When the file has encoding 1252, you can use encoding cp1252 or iso-8859-1 in file adapter.

But you have to verify with a hex editor. Just to "see" the characters corectly does not mean that they are correct.

[Former Member · Accepted solution]

I agree with Stefan. The MQ might not be using the correct encoding on their queue.

Please ask your MQ team to check the 'MQMD.Format' for the queue from where PI is picking the message.

If possible, stop the sender channel in PI and ask your MQ team to retrieve the message from queue and verify the data.

http://publib.boulder.ibm.com/infocenter/wasinfo/beta/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2F...

Edited by: nagarjuna _s on Aug 12, 2011 2:23 PM

[Former Member · Accepted solution]

Hi Stefan,

I have just noticed, only English and German language are installed into it ( t code SMLT).

could this be the reason, ?

I have checked in ECC all the language are installed.

but in IDoc also those are showing junk ###

[stefan_grube · Accepted solution]

> I have just noticed, only English and German language are installed into it ( t code SMLT).

>

> could this be the reason, ?

No, definitely not.

But I really have no idea why this does not work for you.

The cp1252 code page is the most common codepage in the world.

It is Windows latin code page.

[Former Member · Accepted solution]

i have searched almost all related thread in sdn for this issue, seems like nobody has a straight forward solution.

Some has used custom module and some has used java mapping to achieve this.

But i have more than 10 interfaces and donot have timeline to implement any of the above.

Edited by: biplab das on Sep 4, 2011 1:10 PM

[stefan_grube · Accepted solution]

Please validate following:

- the characters in database (are they real cp1252)

- the characters in MQ: use a tool to download data from queue

- the characters in PI without any conversion

validate the date with a hex editor see if the hex values of the characters are correct according to code page

It is useless to use Notepad, Word or other plain text editors to validate the characters.

You find free hex editors and code page charts in internet.

[Former Member · Accepted solution]

Hi Stefan and all,

We have not found any concrete solution for this in PI.

We have used workarounds

MQ guys change their configuration and reading the message as Jms/text instead Jms/byte and they set the encoding as 1252.So now Pi can easily read the messages with european characters or any special characters in it.

Thanks all for your time.

Regards

Biplab Das

[stefan_grube · Accepted solution]

> MQ guys change their configuration and reading the message as Jms/text instead Jms/byte and they set the encoding as 1252.So now Pi can easily read the messages with european characters or any special characters in it.

This not a work around, this is the correct setting.

[former_member472138]

Give a try with unix script for converting.

Regards

Pothana

[former_member854360]

Please see this thread

It might be helpfull to you.

http://en.wikipedia.org/wiki/ISO/IEC_8859-10