high CPU response times, altough low CPU utilization

[Former Member]

Hi Friends,

We have a performance problem after we migrate our basis system,

The current system is :

Database server : Sun SPARC Enterprise T5240 Server - 2 CPU's with 6 core, 8 thread, 1.2 Ghz

32 GB RAM

and we use another identical configured server as an application server.

Database is Oracle 10.2.0 and operating system is Solaris 10.

The problem is average CPU response time is 450 ms, and max. CPU load is %5 percent.

Pre-migration configuration with old servers, we had CPU response : 150 ms and max CPU load: 50%.

Has anyone of you, have experinced a similar problem with this new CPU's?

Thanks in advance

[Former Member · Accepted solution]

Hi. We have almost a similiar environment as yours. We also recently faced this issue. Check out at OS level the prstat -t -c 1 10 command and see if the response is coming too slow or late. Also check the system parameters (/etc/system) file. I am enclosing the values which have been set at our end

set rlim_fd_cur=8192

set semsys:seminfo_semmni=100

set semsys:seminfo_semmsl=256

set shmsys:shminfo_shmmax=4294967295

set shmsys:shminfo_shmmni=100

set noexec_user_stack=1

Also try to restart the servers. Restart means complete reboot not just SAP. Give init 6

The restart had our issue resolved but as of yet have not been able to justify the cause. Most probs it is an OS problem.

Maybe this works

Regards

Lokesh

[markus_doehr2 · Accepted solution]

> set rlim_fd_cur=8192

> set semsys:seminfo_semmni=100

> set semsys:seminfo_semmsl=256

> set shmsys:shminfo_shmmax=4294967295

> set shmsys:shminfo_shmmni=100

> set noexec_user_stack=1

Despite the first (rlim_fd_cur) all those parameters are no more valid for Solaris 10.

Read Note 724713 - parameter settings for Solaris 10

On Solaris 10 the OS limitations are set by resource controls (/etc/project, /etc/user_attr).

> Most probs it is an OS problem.

This is no an OS problem but expected behavior. Those servers are "Cool Threads" servers. Since ABAP is a single threaded application the computation isn't done in parallel and hence you don't get the expected performance.

Markus

[Former Member · Accepted solution]

Thank you Lokesh and Markus, both of you seem to have huge experience.

Lokesh, the server restarted many times in the last month, and there is no development. Thank you for your system parameters, but it seems to be different in Solaris 10. Have you got any data on CPU response times for benchmark?

Markus, I appreciate your words on CPU and ABAP processing logic, we should not have much expectations, you are right. However what we expect is at least same response times with old servers. there is no change in SAP utilization side, same modules, same # of users.

Old server was: HP rp3440, 2 X PA 8800 CPU (2 core, 1.0 GHz )

New server is: Sun SPARC T5240, 2 X UltraSPARC T2 Plus CPU (6 core, 1.2 GHz and 8 thread)

so, do you really think that it is normal for CPU response time to rise there times from 150 ms to 450 ms. I think something should be done at least to have the same response times.

Thank you for your contributions.

Uzan

[markus_doehr2 · Accepted solution]

> Old server was: HP rp3440, 2 X PA 8800 CPU (2 core, 1.0 GHz )

> New server is: Sun SPARC T5240, 2 X UltraSPARC T2 Plus CPU (6 core, 1.2 GHz and 8 thread)

Oh - you even switched the platform...

> so, do you really think that it is normal for CPU response time to rise there times from 150 ms to 450 ms. I think something should be done at least to have the same response times.

The interesting question is: Where is the time getting lost? Is it really a CPU bound issue or are there other factors influencing?

What do you see if you execute

vmstat 1

and

mpstat 1

for some time?

If you post the output here, put code tags around them (mark the posted text and press on the << >> sign next to the underline buttom), this makes the output readable here.

Markus

[Former Member · Accepted solution]

Thank you very much Lokesh and Markus,

I have found an Oracle metaink document. Exactly tells the case. This is a common CPU performance problem with SPARC T CPU's.

Doc 781763.1

Lokesh, what is your system's average CPU response time, so that I can benchmark?

and Markus do you know any company migrated to this "cool threaded CPU's" and not happy with the performance and decided to change the servers again. I need real world cases to convince management for replacing the servers.

Thank you again

Uzan

[markus_doehr2 · Accepted solution]

> and Markus do you know any company migrated to this "cool threaded CPU's" and not happy with the performance and decided to change the servers again. I need real world cases to convince management for replacing the servers.

I don't think the Oracle is basically your issue but the SAP (ABAP) application. If you use those machines for Java-AS'es you will see a significant difference (much better performance).

I don't know any customer who did that, I can just point to the official benchmarks...

I'd talk to your SUN sales rep, he may trade those machines in for AMD-Opteron or Intel boxes instead, those machines don't show that behavior (and are much cheaper).

Markus

[Former Member · Accepted solution]

Thats the best move.

Go to Intel or AMD x86_64 hardware.

We moved lots of HP UX machine's (even itaniums) and SUN E450 systems to x86_64 hardware running linux, Oracle and SAP.

Don't go to Intel Itanium ! Its not value for money ! At least this is my opinion.

Great performance compared with the old HPUX / SUN platforms. For less $ and cheaper in support to.

We use the small HP BL460C (2 dual or 2 quad core cpu's and 16 or 32 GB RAM) a lot with EVA6000 storage. And get great performance out of it.

[Former Member]

Can you check st06 transaction and find out what is taking maximum cpu?

[markus_doehr2]

> The problem is average CPU response time is 450 ms, and max. CPU load is %5 percent.

> Pre-migration configuration with old servers, we had CPU response : 150 ms and max CPU load: 50%.

Is the CPU response time you see in ST03N? How long is the system up and running on the new server?

Markus

[Former Member]

Hello Markus,

New system has been running for a month. The figures given are from EarlyWatch Alert Report.

450 ms, is weekly avarage of last week, and 150 ms was weekly average of the last week with old servers.

Thank you

[markus_doehr2]

> New system has been running for a month. The figures given are from EarlyWatch Alert Report.

> 450 ms, is weekly avarage of last week, and 150 ms was weekly average of the last week with old servers.

Is the database time included? Did you adapt the instance parameters for the new system? I can imagine of many reasons why this is happening...

Markus

[Former Member]

Thank you for your interest Markus,

The database times are not included, only CPU times. There is an improvement in database response times, as expected.

But CPU times went worse altough it was expected to improve.

I am aware that there might be lots of reasons, but what I guess is, there should be a parameter wrong with the CPU settings. This SPARC server CPU is a new tech. 6 core 8 thread one, which is expected to work much faster than the old simple dual core CPU.

However its response time is very high, although it is never utilized more than %5 percent.

The problem might be at database settings(Oracle 10.2.0) or at Solaris ? ? ?

Have you got any experience with this new CPU's? there is no parameter at RZ10 for the use of 6 core CPUs.

Thank you

Uzan

[markus_doehr2]

> I am aware that there might be lots of reasons, but what I guess is, there should be a parameter wrong with the CPU settings. This SPARC server CPU is a new tech. 6 core 8 thread one, which is expected to work much faster than the old simple dual core CPU.

Is it?

ABAP is basically a virtual machine and what is most important: ABAP runs single threaded. This means in consequence that the processing time of a program relies on the single processing power of one core. More cores means more parallelism but the speed of the single statement always relies on the power of one core and how fast it can get the data over the bus and back. So the significant number for speed is basically the Mhz (for ABAP, Java is very different).

Your old machine may have had a DualCore SPARC 1,2 Ghz and the new machine has one 1,4 Ghz 6core CPU (assuming so).

> However its response time is very high, although it is never utilized more than %5 percent.

see above.

> The problem might be at database settings(Oracle 10.2.0) or at Solaris ? ? ?

Well - no - I don't think so

Since ABAP is a single threaded application it can't leverage the CPU power due to the fact, that a parallelism in the program itself does not take place and hence the machine scales not linear with the number of cores. So you may have factually a machine with about 1 dot something CPUs and not 6 as you would expect.

This is not specific to SPARC CPU design but for all multi-core systems. A single threaded application is only as fast as the CPU speed. ABAP programs tend to be huge so you will also see effects of cache displacement and bus congestions. Less cores and more physical CPUs perform much better than any multicore CPU.

For Java the world is very different because Java works as one process (in the SAP case jlaunch) which has many many threads that can be executed in parallel on different pipelines on the CPU.

Unfortunately this is a design problem and there's not much you can do about it.

Markus

[Former Member]

Hi

Is your installation ABAP+ JAVA ?

Regards

Anish

[Former Member]

Only ABAP