cancel
Showing results for 
Search instead for 
Did you mean: 

XML data compression using gzipoutputstream

Former Member
0 Kudos

Hi,

I'm using copy xml feature (return as xml) in message mapping to get the xml content into a string. and passing the string through a JAVA UDF to compress it using gzipoutputstream class. This interface is processing around 500 messages per day, and one message size is 7MB. Message processing and data compression works correctly without any issues. But we have noticed PI database is growing very fast after this solution is implemented. only difference in message mapping is, use of a Java UDF to do the compression. all other field mapping including "return as xml" feature is same as before. Message archiving and deletion is activated, background jobs are working without any errors as usual. After this solution is implemented , database is running out space after three days of processing. We're on PI7.3 AEX (Java Only version). Retention period for processed messages is 1 day. Database should be able to hold message load from one day as it worked before. Following is the JAVA UDF, I'm using. Could you please let me know, if this can cause any issues in PI database. Any sugessions or ideas, greatly appreciated.


public String msgcompress(String str, Container container) throws StreamTransformationException{

ByteArrayOutputStream out = new ByteArrayOutputStream();

try
{

//Compress Message
        GZIPOutputStream gzip = new GZIPOutputStream(out);
        gzip.write(str.getBytes("UTF-8"));
        gzip.close();
        byte[] theZByteArray = out.toByteArray();

//Convert Compressed binary into BASE64 string
        str = DatatypeConverter.printBase64Binary(theZByteArray);


}

catch (IOException uee) {
uee.printStackTrace();
}

return str;
}

Accepted Solutions (1)

Accepted Solutions (1)

former_member181985
Active Contributor
0 Kudos

Hi Sanka,

Can you provide what is the message size after message transformation step with the help of message SOAP runtime header and as well the actual message size from target system or by downloading payload from java monitoring.

Regards,

Praveen Gujjeti

Former Member
0 Kudos

Hi Praveen,

We're using PI7.3 AEX (Java only) version. I can't see SOAP runtime information in SOAP header. I'll check if there's a configuration parameter to enable runtime information. I have downloaded payload before and after mapping step.

Payload size from Source Message           = 6795 KB

Payload size after Transformation step      = 97 KB

Thanks & Regards,

Sanka.

Former Member
0 Kudos

Hi Praveen,

Under the message details tab, I can see content-length. It's close to the payload size.

Message Headers : content-type=multipart/related; boundary=SAP_bec3ad31-e855-11e2-bd79-00002d30b15b_END; type="text/xml"; start="<soap-51DAC66EB0321550E10080000A003220@sap.com>"

http=POST

content-length=103044

Thanks & Regards,

Sanka.

former_member181985
Active Contributor
0 Kudos

Hi Sanka,

Really strange issue. Seems a bug in new PI version. What the time taken for message execution in PI (specially time taken for mapping transformation)

Regards,

Praveen Gujjeti

Former Member
0 Kudos

Hi Praveen,


Transformation step takes around 4 seconds. Basis have added more space into the database and they'll monitor the database growth for next few days. I'll update you after tormmorow's archiving job.


9/07/2013 3:09:04.918 PM Information Message status set to DLNG

9/07/2013 3:09:04.919 PM Information Executing Request Mapping

9/07/2013 3:09:07.129 PM Information Trying to put the message into the send queue

9/07/2013 3:09:07.374 PM Information Message successfully put into the queue

Thanks & Regards,

Sanka.

Former Member
0 Kudos

Hi Praveen,

I got the access to the PI Database and I have been monitoring the PI database size over the last week. Had no issues with DB Size. So it's confirmed data compression is not involved in sudden database increaze. Also analyzed the DB size variation over the last year. There were sudden increase in DB (around 5 to 7 GB) on some months, that's even before we start data compression. Thanks a lot for your information and experience, that helped me to be more confident that compression shouldn't cause any issues.

Thanks & Regards,

Sanka.

Answers (0)