Handling a large message(>100MB) in a Proxy to File scenario in PI 7.31
We have a proxy to SFTP interface where PI is receiving an XML proxy message with message size between 100 MB - 150 MB.
The message has reached the maximum size limit and is unable to get out of the ECC system; it is throwing the below error:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
To add a little more context to the issue, we are on PI 7.31 (single stack). We do not want to dump the PI system with such huge size of data, rather thinking to split it into smaller chunks in ECC and stitching them back in PI into one single file. This is because, the target legacy system can not process multiple files; it can process one full load file per day.
I have come across the below blog where splitting and stitching methods are explained:
But, the blog involves OS level commands and writing the file to a temporary folder and I assume this blog was written keeping PI 7.1 in mind.
Is there any other optimum solution using standard methods to handle the large messages in PI 7.31, bearing the below points in mind:
1. We do not want to change the system level parameters(this might impact the overall performance of the system)
2. BPM is not a preference(can go for it if there is no other option). This interface runs just once in a day, hence just thinking if developing a BPM interface is really worth it.
Awaiting for a solution to this problem.