ERP - Dialog Response shoots very high at night time

[Former Member]

Dear All,

We are currently facing a very weird and non-understandable performance problem.

We have Cluster CI + DB ( Linux , Oracle 11.2.0.2) + 10 Application instances. Our Target SLA for dialog response time is around 700ms, earlier before 2 weeks response time was normal and suddenly response time at 21:00 is shooting very high, nearly 3 times of that between day time( morning 8am-9pm).  Its actually not understandable what is happening in system at 21:00 , we noticed/observed below till now -

1) backup run at 21:00 , we postpone backup to 23:00 ( still response was very high between 21:00 - 23:00 )

2) Redolog backup happens at 22:00 - postpone the same to 24:00 ( still response was very high between 21:00 - 23:00 )

3) We have not noticed huge number of IDOCS passed during respective time ( 4000-5000)

4) We collected trace for multiple transactions but not much help. Is there any other way to check load by idocs?

We observed that Database response time increases and causing dialog response time increased too.

As we observed we noticed Task type "BUFFER SYNC" is also increased during respective period.

if DB response time considered for Dialog steps , we observed huge difference but we didnt understand why suddenly after 21:00 to 04:00 its shooting so high. We will appreciate if any ways to check and reduce this dialog response/db response time.

We will appreciate if any further ways to investigate on this.

We have also raised this to SAP but not yet any good suggestion.

To give a whole day scenario about how response time flows - can claerly see between 21:00 - 03:00 response time is getting very high.

--

Regards,

[evgeny_turovsky]

Hello!

Check T-code STAD for your period of time from 21:00 to 23:00 and you can find all big response times for each transaction step.

Then check DBACOCKPIT transaction for db locks and performance problem in you DB.

As I see there is a problem in DB instance, not in the Application server and you should check DB performance problem first of all now.

[Former Member]

Dear ,

we didnt observed any lock entries in DB instance. We have asked DBA team to check AWR report and provide top used statements and analysis for same.

Our Emphasis is on DB performance only, we suspect DB performance is getting degraded during repsective affected time period , although not sure why yet...

Regards

[manumohandas82]

Hi

For 1 ) You can optimise the outbound flow using  transaction SMQS . Each entry in SM58 will take one Dialog workprocess , You can restrict the no of Dialog work process to be used for the Idocs flowing to HR using SMQS also RZ12

2 )is there any backup scheduled on the system with the issue at 9:00 PM ?

Thanks ,

Manu

[magexposito]

Hello James,

Do you know if there is any process being executed on a different SAP or non-SAP system that is connected with your system? About checking the load from the Idoc, are they inbound or outbound Idoc?

What you can do is :

- Do a system trace using /SDF/MON transaction in order to check free Work Processes, CPU use, Memory use, SQL statements, etc. during that period of time.

- Do a performance trace using ST12 transaction. I think you can get the response time per user for a period of time, the idea is to do a performance trace for that specific user so you can get all the RFC calls, SQL statements, etc.

Regards.

[Former Member]

Dear Miguel,

Thanks for hint on /SDF/MON system trace transaction. I will perform the same at night time during 20:50:00 and will monitor the same.

As well as for ST12 I am concerned is that, we notice multiple users with multiple transactions, hence its not sure how to proceed ahead with it. Still I will check the same during respective affected duration.

Regarding - Connected systems , there are multiple systems connected to this ERP PRD instance. Hence I am not sure about what exactly is causing respective dialog and DB response time shift.

IDOCs are not much only 4000-5000 , mostly outgoing IDOCs in system at respectie duration.

Can background jobs cause DB load time to increase, which is affected dialog response time.

Regards,

[Former Member]

Dear Migel,

We ran system trace using /SDF/MON transaction, as per logs , all application servers have many free resources and no memory bottle necks. Resources are ample.

Observations -

1) We have observed around 4000 IDOCs ( outbound IDOCs to HR prd system) during 21:00 - 24:00

     How can we check IDOC flow happening for this? is there any transaction/job log where we can check/post pone it?

2) another observations about jobs starting at exactly at 21:00. We noticed that DB% time increased to around 89% at 21:00. also we have noticed Average time for logical DB changes (ms) have increased quite alot comparing to other hours through out the day.

Average time for logical DB changes (ms) -

any idea about why respective is increased so much and how can we reduce the same?

Regards,

Bond

[magexposito]

Hello Bond,

About the information you added on your last replies, it is quite interesting that the DB% increased around 21:00. Did you checked on transaction SM37 the jobs scheduled from 21:00 to 22:00 to see if any of them has a high execution time?

On transaction ST03N it is possible to check which processes and users expend more time on CPU, Database, etc. so what you can do is to check which one has the highest Average Response Time on Database and compare it within a few days. Also you can get a ranking list with the top response time per processes. The idea is to get which user and program is raising the Response Time and try to find a solution for it.

Regards.

[Former Member]

Hello Bond

I would like also to point out that sometimes this behavior could be a result of a totally different system (not your SAP) that hogs all the network or storage resources.

In some cases the storage machine that holds the DB disks of your SAP system also hosts other DBs of other systems.

If a different system is getting backed up at that time - it could cause heavy i/o on the disks and even cause network bottlenecks.

Could be worth checking...

[Former Member]

Dear , thanks for suggestion.

We have tried to move database backup from 21:00 to 23:00 , still we have not seen any update on respective performance concern. We have around 6-7 backups running for production systems at night time, I will check with my DB team for backup timings for other systems.

Although we observed there is no I/O concerns on DB node.

[Former Member]

Try to generate AWR report for the same period of time to analyze overall state of DB instance. If possible upload it here.

[Former Member]

AWR report is too big to upload here.. not sure how to upload a file here..

is there any specific part of that report you like to copy-paste?