All messages in PI stuck in "To be Delivered" state

[Former Member]

Hi,

We are facing an issue with message flow in PI 7.31.

The messages once received by PI are scheduled for delivery, but after an interval of time either go in "error" or remain "to be delivered / waiting".

There are multiple problems identified for same :

1. No logs getting created for messages showing in "Delivered" state. Also, unable to see the target payload for successful messages. This is happening for all flow pattern (file/idoc/http)

2. The messages for Receiver SOAP are in error state show error as "HTTP 0 null. connection reset. below snap will show the same.

3. Some messages are in "To be delivered" and no message log shown as above. on attempt to reprocess, log shows administrator task initiated, but status remains as is.

4. Messages in "SOAP adapter :general error" show following log.

5. messages in "Messaging general error: show following log.

Mapping <OM> failed to execute: MappingException: Creating MappingInterfaceLocal instance failed while executing mapping, IllegalStateException: service [com.sap.aii.adapter.xi.svc], CSN component [BC-XI-CON-AFW] should not use an unregistered loader for operation [loadClass of java.lang.reflect.InvocationTargetException].
[This loader has been unregistered as part of the following operation:
    |com.sap.engine.core.classload.impl0.LoadContextImpl.unregister(LoadContextImpl.java:912)
    |com.sap.engine.core.service630.container.LoadContextWrapper.unregister(LoadContextWrapper.java:131)
    |com.sap.engine.services.deploy.server.utils.LoadContextUtils.ensureLoaderIsUnregistered(LoadContextUtils.java:95)
    |com.sap.engine.services.deploy.server.utils.LoadContextUtils.releaseSharedApplicationLoader(LoadContextUtils.java:413)
    |com.sap.engine.services.deploy.server.DeployServiceImpl.removeApplicationLoader(DeployServiceImpl.java:1493)
    |com.sap.engine.services.deploy.server.application.StopTransaction.commonCommitFinished(StopTransaction.java:250)
    |com.sap.engine.services.deploy.server.application.StopTransaction.commitCommon(StopTransaction.java:285)
    |com.sap.engine.services.deploy.server.application.StopTransaction.commitLocal(StopTransaction.java:277)
    |com.sap.engine.services.deploy.server.application.ApplicationTransaction.makeAllPhasesLocal(ApplicationTransaction.java:574)
    |com.sap.engine.services.deploy.server.application.ParallelAdapter.makeAllPhasesLocal(ParallelAdapter.java:335)
    |com.sap.engine.services.deploy.server.DeployServiceImpl.stopApplicationLocalAndWait(DeployServiceImpl.java:2409)
    |com.sap.engine.services.deploy.server.LifecycleController.stopConsumer(LifecycleController.java:800)
    |com.sap.engine.services.deploy.server.LifecycleController.stopAllConsumersOf(LifecycleController.java:747)
    |com.sap.engine.services.deploy.server.LifecycleController.componentIsGettingUnavailable(LifecycleController.java:716)
    |com.sap.engine.services.deploy.server.application.StopTransaction.beginCommon(StopTransaction.java:177)
    |com.sap.engine.services.deploy.server.application.StopTransaction.beginLocal(StopTransaction.java:155)
    |com.sap.engine.services.deploy.server.application.ApplicationTransaction.makeAllPhasesLocal(ApplicationTransaction.java:554)
    |com.sap.engine.services.deploy.server.application.ParallelAdapter.makeAllPhasesLocal(ParallelAdapter.java:335)
    |com.sap.engine.services.deploy.server.DeployServiceImpl.execute(DeployServiceImpl.java:1710)
    |com.sap.engine.services.deploy.server.remote.RemoteCommandFactory$StopRemoteCommand.execute(RemoteCommandFactory.java:762)
    |com.sap.engine.services.deploy.server.remote.RemoteCommandFactory$RemoteCommand$1.run(RemoteCommandFactory.java:139)
    |com.sap.engine.services.deploy.server.remote.RemoteCommandFactory$RemoteCommand$1.run(RemoteCommandFactory.java:134)
    |java.security.AccessController.doPrivileged(Native Method)
    |javax.security.auth.Subject.doAs(Subject.java:396)
    |com.sap.engine.services.deploy.server.remote.RemoteCommandFactory$RemoteCommand.executeInSubjectDoAs(RemoteCommandFactory.java:134)
    |com.sap.engine.services.deploy.server.remote.RemoteCaller$1.run(RemoteCaller.java:589)
    |com.sap.engine.core.thread.impl3.ActionObject.run(ActionObject.java:37)
    |java.security.AccessController.doPrivileged(Native Method)
    |com.sap.engine.core.thread.impl3.SingleThread.execute(SingleThread.java:185)
    |com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:302)
    --------------------------------------------------------------------------]
ClassLoader name: [sap.com/com.sap.xi.services]
Loader hash code: 306c875e
Living status: unregistered
Direct parent loaders:
   [system:Frame]
   [interface:webservices]
   [interface:cross]
   [interface:security]
   [interface:transactionext]
   [library:webservices_lib]
   [library:opensql]
   [library:jms]
   [library:ejb20]
   [service:p4]
   [service:ejb]
   [service:servlet_jsp]
   [library:engine.security.core.lib.facade]
   [library:com.sap.xi.util.dbprof]
   [sap.com/com.sap.aii.sec.prx]
   [sap.com/tc~esi~uddi~sr~proxy~ear]
   [library:sharecheck.lib.facade]
   [library:com.sap.xi.mapping.tool.lib]
   [library:tc~di~dtrclient~protocol~plb]
   [service:engine.security.facade]
   [service:com.sap.aii.af.cpa.svc]
   [library:tc~di~scm~transport_api_impl]
   [service:com.sap.aii.af.svc]
   [service:com.sap.xi.mapping.svc]
   [service:com.sap.aii.proxy.svc]
   [library:com.sap.xi.adminweb.sbeans]
   [service:tc~je~webcontainer~public~api]
   [sap.com/com.sap.xi.ib.resources]
This loader has been unregistered and should not be used anymore. Remove all the references to classes loaded by this loader

Response is appreciated.

[former_member190681]

Dear Harshith,

1) if those messages are in tobedelivered means there are so many reasons.

a) in the log check what are steps in audit log.

b) if the receiver is also not able to receive the data.
c) if it is going to error check the error log like connection timed out/ system error or some other.

if it is connection timed out/connection reset then the problem is at receiver end.

2) i have observed the error log "Http 0 null" in communication channel audit log as per your screen shots that means receiver application server is stopped or not available.

Please ask to chekc at receiver application.

Regards,

Kiran Polani

[anupam_ghosh2]

Hi Harshit,

                 Please discuss this issue with BASIS team. It seems server is running out of memory.

They might need to extend the memory limit of the PI server. Please also check the cache status.

Regards

Anupam

[Former Member]

Hi Anupam,

I checked with BASIS team, the server has good amount of available memory. Also, we had undergone a PI system restart and full cache refreshes.

Regards

Harshit

[azharshaikh]

Hi Harshit,

Please check SAP Note # 1986690 if it helps.

https://websmp230.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/sno/ui_entry/entry.htm?param=69765F6D6F6465...

Regards,

Azhar

[Former Member]

Hi Azhar,

The value set in system is MS = 3.

As per note, it should be set as 1 or 3.

[engswee]

Hi Harshit

Have you gone through the blog below to help identify what is causing the issue in your system.

TBDL messages are caused by hanging threads, which normally are due to connectivity issues.

Rgds

Eng Swee

[Former Member]

Hi Eng Swee Yeoh,

I had referred your blog and identified the messages are stuck in Dispatcher. I tried with stopping the receiver channels, verified the connectivity but still the issue is there.

Also, we have undergone a system restart for PI as well as target legacy system.

Regards,

Harshit

[engswee]

Hi Harshit

Did you manage to find any messages in Delivering status? These will be the messages that are causing the rest to be held up in To Be Delivered status. They will also correspond to the blocked threads as shown in the Queue Information of the Adapter Engine.

Once you find these, check the log for the messages to see if there are anything that can indicate any possible issues.

Also, is your 7.31 system on single stack, or are you using ICO scenario if it is on dual stack? For ICO-based scenario, the threads used are on the sender side, so you will need to stop the corresponding sender channel to release the thread.

Do you still have this issue after a PI restart?

Rgds

Eng Swee