Performance Issue - Idoc_Sender messages in scheduled status

[Former Member]

Hello,

We are facing a performance issue in our production environment SAP PI 7.31 single stack ICO(SAP Idoc - Third party). We have 6 nodes for the production server and each node is having default 5 worker threads for processing.

Problem Statement :

1. All the interfaces(around 10) having source as Idoc are going to scheduled status. The messages are processing very slowly and that too for only 2-3 interfaces and rest all are in idle status.This issue started occurring as some interfaces are sending heavy master data load due to year end(Around 70-80K messages).

2. Please note that all these interfaces are using same IDOC_SENDER communication channel.

3. We have also checked the dispatcher queue from adapter engine status and it is having a large backlog of queued messages.Further all the worker nodes are full for the Idoc_send i.e 5 worker threads.

What we have done/analysed so far :

1. The system CPU and memory parameters are fine and that is not a problem.

2. Gone through the links :

Possible solutions in mind :

1. Create different Idoc_sender channels each for one interface so that one channel is not overloaded with processing for multiple interfaces.

2. Set "queueParallelism.maxReceivers" property of messaging system to 2-3 which is currently set as default "0" as per sap note 1493502 to assign only a particular number of worker threads.

Queries :

1. Will the solution 1 mentioned above will be helpful in coping this situation? Does havign only a signle Idoc_sender communication channel slows down the message processing?

2. As per SAP standard guide the parallelism for Idoc_sender depends upon configuration. So whether the parameter "queueParallelism.maxReceivers" will be applicable to Idoc_sender channels or not? Or it is only applicable for other channels like RFC?

3. "queueParallelism.maxReceivers" property of messaging system is applicable per server node or the entire system level?For e.g. if we have a total of 30 worker threads(5 per node * 6 nodes) then if we set the parameter value to for e.g. 5 , then at run time 5 threads will be occupied per interface and 25 will be free for others or (5-5) zero threads will be free for others?

Any other pointers on this situation will be helpful.

Thanks.

[engswee · Accepted solution]

Hi Gaurav

I've mentioned in my reply on my blog to let us know what is the receiver adapter type for these scenarios. Is it some file based adapter, i.e. File, FTP, SFTP?

From the description of your issue, I don't think the problem is with the IDoc sender side. ICO processing are completely done on the sender adapter's worker thread side so it might have given a false impression that the issue is at the sender.

If the bottleneck is at the IDoc sender, the messages would not even be in your PI system yet, there would instead be a backlog on the source ABAP system (backlog can be seen in t-code SM58).

Since the messages are already in PI's messaging system, the backlog in scheduled status indicates that it is due to bottleneck on the receiver side. It is a case of sender system sending volume of messages that is more than the receiver system is capable of handling. Read up the whole of section 6.2 of the performance check document.

In general I think there is not much you can do since this is a one-off activity. Unless there is a way for the receiving system to increase the speed at which it processes the messages PI sends to it.

Regarding the maxReceiver settings, note that this will not help the throughput of these 10 IDoc interfaces, i.e. changing it will not speed up processing of these interfaces since the limitation is due to the receiver system. What maxReceiver does it to prevent slow processing of any one interface from blocking all other interfaces that are running in PI that uses IDoc sender adapter.

Rgds

Eng Swee

PS: Hope this helps. I'll be away so won't be able to answer any further queries on this.

[Former Member]

Hi Gaurav,

As you mentioned both blocks found in dispatcher queue and all work threads, you may consider tuning approach based on below points:

1. Blocks in dispatcher queue means all threads for the specific MS(message system) queue of receiver adapter are NOT available. For this issue, increasing threads or nodes is the straightforward solution.

2. However blocks also found in all threads, which means the throughput the backend receiver system has to be improved, and in this case, solution 1 about increasing threads or nodes will Not help.

Regarding your queries:

1. different sender idoc channel will not have performance improvement, which has been explained by Eng Swee.

2. "queueParallelism.maxReceivers" is not applicable for ICOs, since MS receiver queue is skipped. instead "messaging.system.queueParallelism.queueTypes" can be configured to "IcoAsync" as SAP note 1493502 for the same function.

3. based on your example, in extreme situation, 0 thread will be available for other interface if all 5 threads are occupied by specific interface due to slow processing at receiver side.

So in general you may try to improve the idoc receiver thruput rather than tunings in PI for your issue, and to avoid this interface blocking all other interfaces, "messaging.system.queueParallelism.queueTypes" should be configured together with "queueParallelism.maxReceivers" set to value less than 5.

Regards,

Hailong

[former_member186851]

Increasing the processing threads and server nodes can help you more.