message stuck in Integration Server

[xiaofang_liu]

Hi export

Today we met a urgent problem.

The message was stuck for 2 hours on the early morning.

Although, finally,PI resent the message automatically 2 hours  later.

But we have no idea why the queue execute the IDOC so slow and lead to the delay of

all IDOC. Around 4:30 AM, system back to normal , all delayed IDOC had

been sent normally.

During this period,beside IDOC ,some synchronous interface got error

too. reason is "ICM_HTTP_TIMEOUT"

We don't now how should we begin to investigate this problem.

Can anyone give advice?

[xiaofang_liu]

Some update of the issue.

Actually, we don’t know whether the system was  stuck or not stuck( only with performance worsen).

The exact issue can be described as  ‘one day morning(2:10-4:00am), all the message in PI system was not be processed efficiently, the max process time reach to 2 hours( we can see begin time and end time in SXI_MONITOR), whereas basis believed  the network and system workload was OK”.

Let’s say “ system  can receive IDOC normally from 2:10am to 4am, and we believe the message stuck in ABAP side from 2 hours, so from 2:10am to 4 am, system didn't distribute any message”.

After 4am, everything comeback, system resent  the failed messaged.

ICM_HTTP_TIMEOUT  402  and JCO_COMMUNICATION_FAILURE Error in some message

No data was sent to Adapter Engine between 2:12-3:35

Biggest pipeline is mapping request.

We have asked some exports, they suggest we enlarge parameter ‘rdisp/max_wprun_time’  from 600 to 1200 or bigger, but we don’t think it can help us, we don’t think this can be the cause for speed worsen and sudden comeback. We have checked the data volume at morning, it was not big.

Also, someone suggest we check MaxThreadCount,. It is also fine, the value is 300.

We believe the delay happened at ABAP side, the data was stuck from ABAP to JAVA side.

We don’t have Wily Introscope.

Anyone can give some advice on how to investigate the problem?

[Former Member]

before sometime I got the similar problem.

It looks like the messages are too big.

You can try to rarse the time out and message volumne in ICM parameter.

[ambrish_mishra]

Hi,

please check the system for the volume of messages flowing through the system at the time this IDoc got stuck. Was it a peak time where the number of messages in PI was very high.

please ask the basis team to check the memory consumption at the specified time.

Assuming that the mentioned IDoc went in error, check from the logs for the reason of failure in the first execution.

You can also check SM58, SMQ1, SMQ2 to see if everything is clear (already suggested).

What we are possibly looking at is some performance tuning required in your system:

http://scn.sap.com/community/pi-and-soa-middleware/blog/2011/01/26/tuning-the-pi-messaging-system-qu...

You can many links on SCN for overall system fine-tuning, housekeeping jobs which keep the PI system in a healthy state.

Hope it helps!

Ambrish

[Former Member]

Hi,

You would need to first find out where was the delay caused. Was it in Integration engine on ABAP side or Adapter Engine on Java side?

You can check the hop timing in SXMB_MONI for a message.

On the Java end, there are specific threads allocated to each adapter. By default it is 5/server node.

In general backlogs on queues can be recognized using the Engine Status (RWB -> Component Monitoring -> Adapter Engine -> Engine Status)

You can see here the number of messages in the queue and also the threads available and currently in use. This page only represents a snapshot for one server node. It is not possible to look at historical data or verify the thread usage on several server nodes simultaneously.

Therefore Wily Introscope is highly recommended. In the PI Triage dashboard you can see all the information in one screen and also analysis of historical data (per default 30 days) is possible.

So if you have Wily, use that to do your analysis.

If not go through the hop timing in sxmb_moni and then also compare when was that message processed in Adapter engine.

Thanks,

Yomesh

[nabendu_sen]

Hi Liu,

Did you check SM58 for any failure? What was the status of the queues in SMQ2?

For Timeout you can refer to the below document:

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/c059d583-a551-2c10-e095-eb5d95e03...