Constant CPU consumption by HTTP Worker

[Former Member]

Hello,

We faced the following issue with SAP Sourcing 9 SP11 on Windows 2008 and Oracle 11.2.0.2.

There are 5 servers in the landscape: central instance+database and 4 additional application servers.

On one application server jstart process begins to consume 13% of CPU constantly. Even when there are no any users or active processes in the system.

Analyzing of thread dump showed that the one of HTTP Worker works abnormally.

Appreciate any help.

Problem HTTP Worker from thread dump:

"HTTP Worker [@345538115]" cpu=236824906.25 [reset 236603281.25] ms allocated=43782970136 B (40.78 GB) [reset 2499752 B (2.38 MB)] defined_classes=623

io= file i/o: 12377188/17529687 B, net i/o: 104051913/47287720 B, files opened:10514, socks opened:308  [reset file i/o: 1354/751 B, net i/o: 685/869 B, files opened:20, socks opened:0 ]

user="" session="-642691623" request="97964" application="sap.com/E-Sourcing-Server" isapplicationthread="true" prio=6 tid=0x000000003c48f000 nid=0x1c8c / 7308 runnable  [_thread_blocked (_call_back), stack(0x0000000055870000,0x0000000055a70000)] [0x0000000055a6c000]

   java.lang.Thread.State: RUNNABLE

                at com.sap.eso.doccommon.ont.allocation.ScenarioLineItemConfigBo.getScenario()Lcom/sap/eso/doccommon/ont/allocation/ScenarioBo;(ScenarioLineItemConfigBo.java:889)

                at com.sap.eso.doccommon.ont.allocation.ScenarioLineItemConfigBo.markDemandModified()V(ScenarioLineItemConfigBo.java:1281)

                at com.sap.eso.doccommon.ont.allocation.ScenarioLineItemConfigBo.setDemand(Lcom/sap/odp/common/types/Amount;)V(ScenarioLineItemConfigBo.java:935)

                at com.sap.eso.doccommon.ont.allocation.ScenarioLineItemConfigBo.markDemandModified()V(ScenarioLineItemConfigBo.java:1287)

                at com.sap.eso.doccommon.ont.allocation.ScenarioLineItemConfigBo.setDemand(Lcom/sap/odp/common/types/Amount;)V(ScenarioLineItemConfigBo.java:935)

                at sun.reflect.GeneratedMethodAccessor8364.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Method.java:597)

                at com.sap.odp.comp.busobjedit.util.ReflectionHelper.invokeMethodForKeyPath(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Class;[Ljava/lang/Object;)Ljava/lang/Object;(ReflectionHelper.java:116)

                at com.sap.odp.comp.collectionlist.CollectionListFilter.createFromFormData()Ljava/lang/Object;(CollectionListFilter.java:720)

                at com.sap.odp.comp.busobjedit.util.FieldUtils.executeSetterFilter(Lcom/sap/odp/common/platform/SessionContextIfc;Lcom/sap/odp/common/db/PersistentBo;Lcom/sap/odp/doc/display/util/DisplayStateManager;Lcom/sap/odp/doc/display/util/DisplayState;Lcom/sap/odp/api/display/IMBOParameterIfc;Ljava/lang/reflect/Method;Ljava/lang/Class;Ljava/lang/Class;Ljava/lang/String;Lcom/sap/odp/common/db/metadata/MemberMdIfc;Lcom/sap/odp/comp/busobjedit/UIField;Ljava/lang/String;Lcom/sap/odp/comp/busobjedit/FilterProcessor;)I(FieldUtils.java:853)

                at com.sap.odp.comp.busobjedit.util.FieldUtils.processUIField(Lcom/sap/odp/common/platform/SessionContextIfc;Lcom/sap/odp/doc/display/util/DisplayStateManager;Lcom/sap/odp/doc/display/util/DisplayState;Lcom/sap/odp/common/db/PersistentBo;Lcom/sap/odp/comp/busobjedit/UIField;Lcom/sap/odp/api/display/IMBOParameterIfc;Ljava/lang/String;Lcom/sap/odp/comp/busobjedit/FilterProcessor;)I(FieldUtils.java:802)

                at com.sap.odp.comp.busobjedit.imbo.BusObjEditRQ.processList(Ljava/util/List;ZLcom/sap/odp/api/display/IMBOParameterIfc;)Z(BusObjEditRQ.java:577)

                at com.sap.odp.comp.busobjedit.imbo.BusObjEditRQ.updateBizDoc(Lcom/sap/odp/api/display/IMBOParameterIfc;)Z(BusObjEditRQ.java:545)

                at com.sap.odp.comp.busobjedit.imbo.BusObjEditRQ.initialize(Lcom/sap/odp/api/display/IMBOParameterIfc;)I(BusObjEditRQ.java:258)

                at com.sap.odp.display.util.IMBOTransformer.executeIMBO(Lorg/w3c/dom/Document;Lorg/w3c/dom/Node;Lcom/sap/odp/display/util/IMBOParameter;)Lorg/w3c/dom/Node;(IMBOTransformer.java:436)

                at com.sap.odp.display.util.IMBOTransformer.processNode(Lorg/w3c/dom/Document;Lorg/w3c/dom/Node;Lcom/sap/odp/display/util/IMBOParameter;)Lorg/w3c/dom/Node;(IMBOTransformer.java:223)

                at com.sap.odp.display.util.IMBOTransformer.processNode(Lorg/w3c/dom/Document;Lorg/w3c/dom/Node;Lcom/sap/odp/display/util/IMBOParameter;)Lorg/w3c/dom/Node;(IMBOTransformer.java:275)

                at com.sap.odp.display.util.IMBOTransformer.getTransformedXML(Lcom/sap/odp/display/util/DisplayXMLProcessor;Lcom/sap/odp/display/util/IMBOParameter;)Lorg/w3c/dom/Document;(IMBOTransformer.java:161)

                at com.sap.odp.display.util.IMBOTransactionThread.processRequest()V(IMBOTransactionThread.java:373)

                at com.sap.odp.display.util.IMBOFrameworkProcessor.processRequest(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(IMBOFrameworkProcessor.java:352)

                at com.sap.odp.display.servlet.IMBOFrameworkServlet.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(IMBOFrameworkServlet.java:79)

                at javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(HttpServlet.java:754)

                at javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(HttpServlet.java:847)

                at com.sap.engine.services.servlets_jsp.server.Invokable.invoke(Ljavax/servlet/Servlet;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;ZZLcom/sap/engine/interfaces/security/SecuritySession;)V(Invokable.java:152)

                at com.sap.engine.services.servlets_jsp.server.Invokable.invoke(Ljavax/servlet/Servlet;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/security/auth/Subject;Z)V(Invokable.java:38)

                at com.sap.engine.services.servlets_jsp.server.HttpHandlerImpl.runServlet(Ljava/lang/String;Lcom/sap/engine/services/httpserver/interfaces/HttpParameters;Lcom/sap/engine/services/servlets_jsp/server/application/ApplicationContext;)V(HttpHandlerImpl.java:457)

                at com.sap.engine.services.servlets_jsp.server.HttpHandlerImpl.handleRequest(Ljava/lang/String;Lcom/sap/engine/services/httpserver/interfaces/HttpParameters;)V(HttpHandlerImpl.java:210)

                at com.sap.engine.services.httpserver.server.RequestAnalizer.startServlet(Ljava/lang/String;)V(RequestAnalizer.java:441)

                at com.sap.engine.services.httpserver.server.RequestAnalizer.startServlet(Lcom/sap/engine/services/httpserver/lib/util/MessageBytes;)V(RequestAnalizer.java:430)

                at com.sap.engine.services.servlets_jsp.filters.DSRWebContainerFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(DSRWebContainerFilter.java:38)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.servlets_jsp.filters.ServletSelector.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ServletSelector.java:81)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.servlets_jsp.filters.ApplicationSelector.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ApplicationSelector.java:276)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.WebContainerInvoker.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/HostChain;)V(WebContainerInvoker.java:81)

                at com.sap.engine.services.httpserver.chain.HostFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(HostFilter.java:9)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.ResponseLogWriter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/HostChain;)V(ResponseLogWriter.java:60)

                at com.sap.engine.services.httpserver.chain.HostFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(HostFilter.java:9)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.DefineHostFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/ServerChain;)V(DefineHostFilter.java:27)

                at com.sap.engine.services.httpserver.chain.ServerFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ServerFilter.java:12)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.MonitoringFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/ServerChain;)V(MonitoringFilter.java:29)

                at com.sap.engine.services.httpserver.chain.ServerFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ServerFilter.java:12)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.SessionSizeFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/ServerChain;)V(SessionSizeFilter.java:26)

                at com.sap.engine.services.httpserver.chain.ServerFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ServerFilter.java:12)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.MemoryStatisticFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/ServerChain;)V(MemoryStatisticFilter.java:57)

                at com.sap.engine.services.httpserver.chain.ServerFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ServerFilter.java:12)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.filters.DSRHttpFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/ServerChain;)V(DSRHttpFilter.java:43)

                at com.sap.engine.services.httpserver.chain.ServerFilter.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;Lcom/sap/engine/services/httpserver/chain/Chain;)V(ServerFilter.java:12)

                at com.sap.engine.services.httpserver.chain.AbstractChain.process(Lcom/sap/engine/services/httpserver/chain/HTTPRequest;Lcom/sap/engine/services/httpserver/chain/HTTPResponse;)V(AbstractChain.java:78)

                at com.sap.engine.services.httpserver.server.Processor.chainedRequest(Lcom/sap/engine/services/httpserver/server/Processor$FCAProcessorThread;Lcom/sap/bc/proj/jstartup/fca/FCAConnection;II)V(Processor.java:475)

                at com.sap.engine.services.httpserver.server.Processor$FCAProcessorThread.process()V(Processor.java:269)

                at com.sap.engine.services.httpserver.server.rcm.RequestProcessorThread.run()V(RequestProcessorThread.java:56)

                at com.sap.engine.core.thread.execution.Executable.run(Z)V(Executable.java:122)

                at com.sap.engine.core.thread.execution.Executable.run()V(Executable.java:101)

                at com.sap.engine.core.thread.execution.CentralExecutor$SingleThread.run()V(CentralExecutor.java:328)

   Locked ownable synchronizers:

                  - None

Thank you!

[Former Member · Accepted solution]

Problem has been solved with SAP Sourcing 9 SP16.

We checked it on our DEV system.

[petr_solberg · Accepted solution]

Hi Alexey,

thanks for taking the time to give this update and closure to this issue.

I will look into it.

Nice weekend and best regards,

Andy.

[petr_solberg · Accepted solution]

Hi Alexey,

there is an OSS Note for SP16 -

     Support Package 16 of E-Sourcing Server 9.0, note 1956257

but it's not released yet.

Do you know if there is an OSS Note specific to this problem.

Thanks and regards,

Andy.

[Former Member · Accepted solution]

Hi Andy,

I haven't seen any notes specific to this problem. I think the problem will be mentioned in note 1956257.

Regards,

Alexey

[Former Member]

The problem was reported to SAP. SAP is working on fix.

[petr_solberg]

Hi Alexey,

is there any news on this issue ?

Kind regards,

Andy.

[Former Member]

Hi Andy,

I've got reply from SAP that fix will be provided in future patches.

Have you faced the same problem?

[petr_solberg]

Hi Alexey,

we might have something like this.

And what's the solution in the short term, while you wait for the patch, did SAP point you to any useful OSS Notes ?

Thanks and regards,

Andy.

[Former Member]

Hi Andy,

I'm waiting the answer about workaround from SAP. I'll let you know here when I get it.

[petr_solberg]

thanks Alexey,

Andy.

[Former Member]

Hi Andy,

unfortunately there is no workaround as a code fix is required.

We simply restart the node in this situation during the weekend when workload on the system is minimal. Of course it will work for you if there more than one node in your system.

Alexey.

[petr_solberg]

Hi Alexey,

thank you very much for the feedback.

I will look into this.

Kind regards and happy new year,

Andy.