File with ASCII Control Characters causes runtime exception in mapping

[Former Member]

Hi,

We are using XI/PI 3.0 SP12 and have flat file to ABAP proxy scenario. We are using file content conversion to convert the file data to XML and then a basic map to ABAP proxy. The file is being generated from a legacy system and we have observed that sometimes it contains ASCII control characters (characters less ASCII space i.e. < hex #0x32).

The content conversion works fine and converts the file data to XML but we get error during mapping.

e.g. if a Hex 03 ( End of Text / Ctrl - C ) is received in the file it causes following error.

-

---

Runtime exception occurred during execution of application mapping program com/sap/xi/tf/_MM_EPlusRaw_To_SAPRaw_: com.sap.aii.utilxi.misc.api.BaseRuntimeException; Fatal Error: com.sap.engine.lib.xml.parser.ParserException: Invalid char #0x3(:main:, row:7, col:28)

-

---

This is understandable as XML specifications do not allow for such characters to be present in the tags as values. The only UTF control characters that the W3 XML specification [http://w3.org/TR/2004/REC-xml-20040204/] allows are horizontal tab (#0x09), carriage return (#0x0D) and line feed (#0x0A). The File Adapter Content Conversion parser however, does not appear to conform to the specs. It should be throwing an exception, or remove the offending characters. The XML parser however is trapping the error in the mapping step.

We have asked the source system to fix up this issue but ideally we would also like to know if this can also be fixed at XI end and how difficult will it be. We are reading a big file and are getting the file adapter to split it in chunks for us. If we get the runtime exception midway through the process then this holds up out qRFC queue and means a lot of clean up has to be done.

The options that I think we could have are :

1) Standard parameter in the Sender File Adapter that takes care of replacing the control characters ?

2) Adapter Module on the File adapter to trap these characters. Of course we do not want to filter the "nl" / "0x0D" CR or "0x0A" LF characters indicating the record breaks.

3) By running a unix script. If so then I would like an example script that is efficient.

4) Any way that the data from the file adapter is passed into CDATA[[ tag.

5) Some way to send the file data to ABAP proxy without converting to XML

Any suggestions on how to approach this are welcome.

Thanks in advance.

Charu

Edited by: Charu Kulkarni on Feb 20, 2008 9:09 PM

[Former Member · Accepted solution]

Charu:

I think the reason why file adapter did not throw any exception while reading the file may be due to "File Encoding" format you have selected (US-ASCII, ISO-8859-1, UTF-8 etc..) which ever better suits your requirement. I have seen some threads similar to  your requirement and learned that CDATA might help you to resolve. Following is one thread which is similar to your req:

[Former Member · Accepted solution]

Hi Guru,

Thanks for your suggestion and the link. I have tried various combinations of the file encoding parameter in the File Adapter. I even tried to change the file type parameter to binary.

XML special characters like &amp; etc are getting correctly encoded by the Flat File Adapter. The issue is that if ASCII characters less than hex 20 are existing in the file then these do not get replaced with some dummy characters and the mapping fails at runtime.

I guess anyone can recreate the problem by introducing these characters in the file ( e.g. by using Text pad and choosing some control characters ).

We could look at a quick way to convert the output of the file adapter content conversion into CDATA[[ tags. Or do I have to apply a XSLT map.

 <xml version="1.0" encoding="utf-8" ?> 
 <ns:MT_RawData xmlns:ns="http://abc.co.nz/SalesBilling">
 <record>
  <row>TD09000000000089880020070929DUMMY</row> 
  </record>
 <record>
  <row>TDOR07AI00000100001020070424 LGSIW0161035276E Newton &amp; Einstein CO LTD INTERNET-100006053 NYN</row> 
  </record>
</ns:MT_RawData>

Thanks

[Former Member · Accepted solution]

Charu:

Can you look at the following links. second is little elobrated but I guess you fill something related to you

http://help.sap.com/saphelp_nw04/helpdata/en/78/759f3cad1e3251e10000000a114084/content.htm

http://help.sap.com/saphelp_nw04/helpdata/en/0d/00453c91f37151e10000000a11402f/frameset.htm

I will try to find other options in the mean time

[Former Member · Accepted solution]

We have settled on a workaround so that the source system will be cleaned up not to send the control characters in the file.

[bhavesh_kantilal · Accepted solution]

In my case we wrote a Java Mapping that replaces these control characters as my Source was in no position to clean up thee characters. The Java Mapping all it does was convert input stream to a string and do a search for control charctrers and replace the same.

We made this a generic class that can be reused in all SWCV's and hence find it a useful class to have.

Regards

Bhavesh

[Former Member · Accepted solution]

Thanks Bhavesh. This does sound a Elegant Solution !

Bhavesh can you please post the Java Class and steps to create the mapping and make it Generic for use in other SWCVs?

Looks like we can use this Java Mapping as the first map in the interface mappings where ever there is a chance to have these control characters in XML.

Actually this should be a standard feature of the file Adapter as I said before.

Thanks Again,

Charu

[ravi_raman2 · Accepted solution]

Charu,

Put this in a mapping...

/***************************

import java.util.regex.*;

import java.io.*;

public class Control {

public static void main(String[] args)

throws Exception {

//Create a file object with the file name

//in the argument:

File fin = new File("fileName1");

File fout = new File("fileName2");

//Open and input and output stream

FileInputStream fis =

new FileInputStream(fin);

FileOutputStream fos =

new FileOutputStream(fout);

BufferedReader in = new BufferedReader(

new InputStreamReader(fis));

BufferedWriter out = new BufferedWriter(

new OutputStreamWriter(fos));

// The pattern matches control characters

Pattern p = Pattern.compile("{cntrl}");

Matcher m = p.matcher("");

String aLine = null;

while((aLine = in.readLine()) != null) {

m.reset(aLine);

//Replaces control characters with an empty

//string.

String result = m.replaceAll("");

out.write(result);

out.newLine();

}

in.close();

out.close();

}

}

*********************************/

Hope that helps

Regards

Ravi Raman

[bhavesh_kantilal · Accepted solution]

Code is posted by Ravi. A google search will provide multiple such codes.

Meanwhile, in terms of the generic, Create a SWCV, import it in the Imported Archive and then created a SWCV Dependency in the SLD. A SDN search on creating Software Component Dependencies should definitely help . You will find a few of my old answers describing this in detail as well.

As for a standard solution from SAP. I wish so as well. Maybe a module, maybe a genric java class anything should help.

Regards

Bhavesh

[ravi_raman2 · Accepted solution]

Hi,

Is your issue fixed...if it is then can you award points...that would help to motivate us to reply the next time, you post a thread.

Points & Interesting issues motivate answers..!!!!!!!!

Thanks and Regards

Ravi Raman

[Former Member · Accepted solution]

Bhavesh / Ravi. I have awarded points. Thanks !

We were not able to use regular expression for pattern matching for some reason. We even tried to escape the curly brackets but the replace was not working.

However our Java expert researched and found that another regular expression to find the characters that are not in range (space to tilde) "[^ -~]" and this did the trick.

This is the code snippet with the runtime Java Mapping class that our Java expert came up with. Your comments on it would be appreciated.

 
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationException;

public class ReplaceCtrlChars implements StreamTransformation {

	private Map param = null;

	public void setParameter (Map param) {
		this.param = param;
		if (param == null) {
			this.param = new HashMap();
		}
	}
	
	public void execute(InputStream inStream, OutputStream outStream) throws StreamTransformationException{
		try{
			BufferedReader in = new BufferedReader(new InputStreamReader(inStream));
			BufferedWriter out = new BufferedWriter(new OutputStreamWriter(outStream));
	
			// The pattern matches control characters
			Pattern p = Pattern.compile("[^ -~]");
	
			Matcher m = p.matcher("");
			String aLine = null;
			while((aLine = in.readLine()) != null) {
				m.reset(aLine);
				//Replaces control characters with an empty string.
				String result = m.replaceAll("?");		
				out.write(result);
				out.newLine();
			}
			in.close();
			out.close();					
		}catch(Exception e){
			e.printStackTrace();
		}
	}
}

Best Regards and see you on the forum.

Charu

[ravi_raman2 · Accepted solution]

Charu,

Did you test it..with valid and invalid values..

Regards

Ravi Raman

[Former Member · Accepted solution]

We have tested our java mapping and it does replace the control characters and the subsequent message mapping also works. A 100 MB file is filtered in about 5 seconds. It would probably need a QA on the exception handling.