SAP PO | EOiO problem

[Dimitri]

Hi All,

We are using a SAP PO 7.4 system and one of our processes is receving an IDOC from the SAP ERP system.

This IDOC is mapped to the target structure and delivered to 25 different receiving applications.

In the sender IDOC adapter, EOiO is enabled. In the ICO object, receiver interfaces tab, the option Maintain Order at Runtime is disabled.

What we see in the monitoring is messages in status holding. And this without and error holding up messages.

I opened an OSS incident and SAP replied with the messages that we need to speed up message delivery on the receiver side.

This is not that easy and I'm really looking for a solution on the SAP PO system. Perhaps extended EOiO error handling?

Other suggestions are very welcome.

Thanks a lot.

Dimitri

[bhavesh_kantilal]

Hello Dimitri,

If I understand this right, messages are going into Holding state inspite of no errorrs? Well if thats the case then that definitely is not right..

What I can think of is as a workaround would be - to check out of thew 25 which truly need a E2E EOIO and only for those have a common receiver with "Maintain Interface Order at Runtime". For others I would just let it pass through with another receiver without Maintain Interface Order at Runtime.

Ofcourse, ideally if there is no issue, I would expected PO to provide this EOIO queuing automatically without the statement that you have just described from SAP. Might be worth escalating as well ( I assume this would have already been done on your end )..

Regards

Bhavesh

[Dimitri]

Hi Bhavesh,

Your understanding is correct. And indeed, that is not OK.

Some of the receivers indeed do not need messages being received in the correct order. The majority of course definitely needs it. For the moment being, I will leave them all with the Maintain Order at Runtime disabled. It does not make any difference.

I already asked SAP for possible solutions on SAP PO itself.

And in the meantime, I'm trying to figure out if extended EOiO error handling could be an option for us.

Kind regards,

Dimitri

[Ryan-Crosby]

Hi Dimitri,

I would agree with the response from SAP regarding the delivery on the receiving end since EOIO is guaranteed to be delivered in a particular order.  Here is the help information regarding the two modes for extended EOIO - I suppose use of it depends on what your exact requirement is in regards to message delivery for the 25 applications.

Regards,

Ryan Crosby

[bhavesh_kantilal]

Hello Ryan,

This is interesting. Can you share the source of this info so we can read the entire context to this?

Regards,

Bhavesh

[Ryan-Crosby]

Hi Bhavesh,

Here is a link to the help for 7.4 - my screenshot was for 7.31 because it was the first search result but they look to be nearly identical in content.

https://help.sap.com/saphelp_nw74/helpdata/en/f3/9d36d0144145b2953d223fb689c9a4/content.htm

Regards,

Ryan Crosby

[Dimitri]

Hi Ryan,

For some receivers, the order is not important. As long as they get all the messages delivered.

For other receivers, the order is a must. Therefore I need to eliminate EOiO for some receivers (the slow ones) to avoid the dependency between parent message (the idoc) and the child messages to be delivered. I already asked SAP about the extended EOiO error handling, but no answer yet.

I need to do this directly on production because I cannot simulate this on acceptation.

Therefore I want to be (almost) sure it will work.

Kind regards,

Dimitri

[vadimklimov]

Hi,

Extended EOIO error handling functionality makes sense if you would like to introduce specific handling of failed messages for EOIO scenarios (either to move them to a specific queue, or to remove them completely - in both options, allowing processing of successor messages assigned same standard serialization context). My understanding from Dimitri's initial post was, that there are no failures / error messages that would block the queue in PO. If so, extended EOIO is not suitable here.

Dimitri, when you notice pending holding messages, can you figure out what is status of a most predecessor message in corresponding serialization context? Is there a message that is being delivered, or all predecessor messages are delivered successfully and others are pending in holding status? If the predecessor message in a serialization context is being delivered (so it is long-running step of message processing), then I would tend to agree with recommendation from SAP and proceed investigation of possibilities to optimize message processing (depending on where delay happens - is it some step within PO pipeline processing that takes much time? or transmission of a message to a receiver? or processing on a receiver side?). On the other hand, if predecessor messages were processed successfully and successors are holding (meaning, no predecessor in delivering or error status), then I would rather consider this as a PO specific issue, considering its failure to acknowledge completion of predecessor message processing and triggering successor message processing by EOIO sequencer.

Regards,

Vadim

[Ryan-Crosby]

Hi Dimitri,

In that case the option of "Remove from Queue" may work for you because it adjusts the QoS to "Exactly Once" and it appears that in the configuration you can set this per receiver.  Based on what I can see for how it works then this should give you the stuff that you need.

Regards,

Ryan Crosby

[Dimitri]

Hi Vadim,

All messages are in holding status. That's the tricky part.

No other status in that particular queue.

Kind regards,

Dimitri

[Dimitri]

Hi Ryan,

What about, in the ICO object, Maintain Order at Runtime option in the receiver interfaces tab?

Disabled/enable for all receivers or align with the Extended EOiO error handling?

Once that is clear I will give it a try directly on production.

Kind regards,

Dimitri

[Ryan-Crosby]

HI Dimitri,

The help indicates that Extended EOIO Handling cannot be used in conjunction with Maintain Order at Runtime.  However, after re-reading the mention of no errors I don't think this is an option for you as I believe it is only triggered in the case of failure.  I think you would have to look at one sender for EO vs. a separate one for EOIO.

Regards,

Ryan Crosby

[vadimklimov]

Dimitri,

Can you find some relevant evidences in developer traces of the PO system? Any error that would relate to EOIO message handling? There were similar issues some time ago (for example, see SAP Note 2177278), which were fixed by code corrections - and causes of those issues were commonly reflected in developer traces.

And if there is no predecessor message in delivering status, I would not correlate it to slow receiver system, since absence of messages being delivered means to me, there is no invocation of receiver system (being it slow or fast) at this time and dependency on its processing capabilities.

Regards,

Vadim

[Dimitri]

Hi Ryan,

The sender, in this case the SAP ERP system, will send IDOCs in EOiO. This is enabled in the sender idoc adapter. That is a requirement because for the majority of receivers, the EOiO is very important.

I will review all comments again and try something out. The current situation must be solved asap.

[UPDATE 11.05.2016]

• Extended EOiO error handling will not solve the problem because it acts on messages in error state. That is not the case.
• Maintain Order at Runtime option is enable again. I checked the SAP help again and I saw the recommendation.

The problem still persists, but only 1 or 2 receiving applications have messages in HOLDING status.

I'm currently looking at parameter sets and values based on 2 OSS notes: 2177278 and 1678427.

Kind regards,

Dimitri

[Dimitri]

Hi Vadim,

Interesting note you mention. I check the developer traces, but I could not find the exact error mentioned for that location. But I found something for this location with eoio in the name:

com.sap.engine.messaging.impl.core.queue.consumer.AsyncConsumer.processEOIOLoop(QueueMessage, MessageController)

This is the error:

Catching com.sap.engine.messaging.runtime.ClusterException: Could not trigger cluster event TRIGGER_SEQUENCERS for all nodes. Reason: com.sap.engine.frame.cluster.message.ResponseTimedOutException: Participant [252093351] did not return a response in the specified timeout of [120000] ms

at com.sap.engine.messaging.runtime.j2ee.sapengine.SAPJ2EEClusterController.triggerAllNodes(SAPJ2EEClusterController.java:411)

at com.sap.engine.messaging.runtime.j2ee.sapengine.SAPJ2EEClusterController.triggerSequencers(SAPJ2EEClusterController.java:555)

at com.sap.engine.messaging.impl.core.queue.consumer.AsyncConsumer.processEOIOLoop(AsyncConsumer.java:509)

at com.sap.engine.messaging.impl.core.queue.consumer.SendConsumer.onMessage(SendConsumer.java:145)

at com.sap.engine.messaging.impl.core.queue.Queue.run(Queue.java:1109)

at com.sap.engine.messaging.runtime.MSWorkWrapper.run(MSWorkWrapper.java:58)

at com.sap.engine.core.thread.impl3.ActionObject.run(ActionObject.java:37)

at java.security.AccessController.doPrivileged(Native Method)

at com.sap.engine.core.thread.impl3.SingleThread.execute(SingleThread.java:185)

at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:302)

Caused by: com.sap.engine.frame.cluster.message.ResponseTimedOutException: Participant [252093351] did not return a response in the specified timeout of [120000] ms

at com.sap.engine.core.cluster.impl6.ms.MSMultipleAnswerImpl.getAnswer(MSMultipleAnswerImpl.java:143)

at com.sap.engine.messaging.runtime.j2ee.sapengine.SAPJ2EEClusterController.triggerAllNodes(SAPJ2EEClusterController.java:387) 

Now, I could imagine SAP claims to speed up message processing at the receiver side. See bold part.

If I'm not mistaken, the basis team check 2 parameters specified by SAP and they both have the correct value.

Kind regards,

Dimitri

[vadimklimov]

Dimitri,

Was one of parameters that was suggested to be checked, a property messaging.cluster.timeout of a Java service XPI Service: Messaging System?

Can you please check if there is large backlog of holding messages on the specified cluster node (252093351)?

Can you try resending first holding messages manually and checking if they get processed?

My idea is, if there is huge backlog of EOIO messages on a specific cluster node, to reduce it by manually resending affected messages, and then checking if sequencer resumes to normal processing of EOIO messages after backlog is reduced / has gone.

Regards,

Vadim

[Dimitri]

Hi Vadim,

If I'm not mistaken, the basis team modified parameter messaging.cluster.timeout. For that parameter, I see a default calculated value of 60000 and a custom calculated value of 120000.

I'm not sure about the backlog being on 1 cluster node. Initially, this was the case, but I'm not sure if this was also the case in the past days.

We can easily process queues being in holding status. That is not the problem.

But we need to do it manually. That is not feasible 24/7.

Currently, I see 2 options:

• wait for feedback from SAP. I asked them about 2 oss notes
• have a look at the extended EOiO error handling

Kind regards,

Dimitri