Slow System Performance on SunOS 5.10

[Former Member]

Dear all,

Our system experienced very slow response time a few days ago. After checking the wp trace file, we found the oracle is out of memory and there were some malloc failed(not enough space). Our system is ECC6 SR3 SP 16 and Oracle 10g dbase with RAM 4GB and swap 16 GB on container of SunOS 5.10.

My assumption when the process running out of memory, it should retrieve from virtual memory. At command vmstat -s the output of swap in/out and pages swapped in/out is zero. Guess problem with vm of container. Anyone can help? Thanks.

[Former Member]

Currently my oracle is still using ISM (pg_contig has been disabled). Still don't know the impact between ISM vs DISM usage.

We have increased the DIA work process to answer slow system response due to wait time process is very long. By this time we prefer to go on this way first...

[markus_doehr2]

Yes - ISM is still used. However, the system time should decrease dramatically if pg_contig is diabled (and the server was rebooted).

Markus

[former_member227283]

Hi,

Pls paste your output of following,

projects -l $(projects -d <userid>)

Thanks

Anil

[markus_doehr2]

> My assumption when the process running out of memory, it should retrieve from virtual memory. At command vmstat -s the output of swap in/out and pages swapped in/out is zero. Guess problem with vm of container. Anyone can help? Thanks.

What hardware are you using? Is this a T-Server from Sun?

Markus

[Former Member]

To Markus,

No, our machine is M4000. Thanks

[markus_doehr2]

> No, our machine is M4000.

Ok.

Did you create an /etc/project for the application in the zone and did you add the users to /etc/user_attr?

Markus

[Former Member]

Yes Markus,

We did exactly as per described in SAP Note 724713 - parameter setting for Solaris. FYI, the system was fine before. This happens a few days ago (2 times on different days).

At that time, all logon user faced the very slow response, close to hanged, and when I logged to OS level, each time I type a command such as ls, df -k, etc, the OS feedback is slow as well.

I checked the trace file of some work process and see out of memory for Oracle (ORA-04030) and malloc failed as well. I though it would retrieve the resource from swap area(disk). Then I checked vmstat 5, and the value on po column is 0.

Please advice. Thanks.

[markus_doehr2]

> We did exactly as per described in SAP Note 724713 - parameter setting for Solaris. FYI, the system was fine before. This happens a few days ago (2 times on different days).

> At that time, all logon user faced the very slow response, close to hanged, and when I logged to OS level, each time I type a command such as ls, df -k, etc, the OS feedback is slow as well.

You have

set pg_contig_disable=1

set in /etc/system?

> I checked the trace file of some work process and see out of memory for Oracle (ORA-04030) and malloc failed as well. I though it would retrieve the resource from swap area(disk). Then I checked vmstat 5, and the value on po column is 0.

ORA-04030 does not necessarily mean that the database couldn't allocate memory from the OS. It can also happen if

- your shell limits (for the user running the Oracle database) are reached

- value pga_aggregate_target is reached

Markus

[Former Member]

I haven't set pg_contig_disable=, what is and why I need to set parameter that param? Thanks.

[markus_doehr2]

> I haven't set pg_contig_disable=, what is and why I need to set parameter that param?

pg_contig_disable means, that the system will, at memory allocation time, "rearrange" the memory internally to find a big contiguous amount of memory (so copying memory). Depending on the amount of memory you have in your machine and the amount of fragmentation due to many processes this may take some time and almost bring the system to halt.

If that parameter is set, the system won't try that and allocates the memory non-contiguously.

You either need to set that parameter in /etc/system and reboot or do

echo "pg_contig_disable/W 1" | mdb -kw

Markus

[Former Member]

Thanks Markus,

I didnt notice that this parameter mentioned in the note, I only notice the limit parameter.

Considering the role of this parameter, I'm a bit worry actually, if the contiguous feature being disabled, I guess the memory swap in/out to database will happen more often, that will affect the performance as well right? But I think I will try to set this parameter, hopping the simultaneous processes will not hang my system next time.

Unfortunately, my main issue about why the container cannot used swap area when the memory is running out still a question, so I still open to suggestion on this swap issue. Anyone? Thanks a lot.

Edited by: sprout009 on Oct 16, 2009 7:03 PM

[markus_doehr2]

> Considering the role of this parameter, I'm a bit worry actually, if the contiguous feature being disabled, I guess the memory swap in/out to database will happen more often, that will affect the performance as well right?

The only difference will be that the system will allocate the memory as it is and not trying to move memory around to get contiguously memory.

> But I think I will try to set this parameter, hopping the simultaneous processes will not hang my system next time.

Yes, I'd do that.

> Unfortunately, my main issue about why the container cannot used swap area when the memory is running out still a question, so I still open to suggestion on this swap issue.

On Solaris Oracle uses (D)ISM (Dynamic Intimate shared memory) which means, that this kind of memory class is not swapped in case of memory shortness.

Again I'm not convinced that the ORA error is caused by not having enough memory available on OS but because an internal Oracle memory structure was full. If you limit the memory for pga_target to 2 GB there can be 100 GB of OS memory left, Oracle won't use it. So I'd first check those parameters.

Markus

[Former Member]

Thanks Anil,

I believe my project setting is correct, I've tested and compare the output to be same as per mentioned in above SAP Note

To Markus,

About your definition that Oracle on Solaris uses DISM therefore it doesn't swapping to disk, I found a statement on the following link:

[http://download.oracle.com/docs/cd/B19306_01/server.102/b15658/appe_sol.htm|http://download.oracle.com/docs/cd/B19306_01/server.102/b15658/appe_sol.htm] saying this:

On Solaris 8 and Solaris 9 systems, dynamic/pageable ISM (DISM) is available. This enables Oracle Database to share virtual memory resources between processes sharing the segment, and at the same time, enables memory paging.

I tot paging word on the statement refer to page in/out to swap area, kindly correct me if I am wrong, and it will be helpful if you can provide me SAP Note on statement that Oracle system with DISM/ISM doesnt do swap to VM. Thanks in advance.

Edited by: sprout009 on Oct 20, 2009 8:52 AM

[markus_doehr2]

> On Solaris 8 and Solaris 9 systems, dynamic/pageable ISM (DISM) is available. This enables Oracle Database to share virtual memory resources between processes sharing the segment, and at the same time, enables memory paging.

>

> I tot paging word on the statement refer to page in/out to swap area, kindly correct me if I am wrong, and it will be helpful if you can provide me SAP Note on statement that Oracle system with DISM/ISM doesnt do swap to VM. Thanks in advance.

You misunderstood that.

The concept of ISM is to not swap the allocated memory. In the early versions the memory could only be allocated on application start (Oracle startup) and could not be adapted. With DISM you can configure the memory usage during runtime of the database and so change the size of the shared memory segment. If you then decrease the size of that segment, other applications can use that memory and hence it can be swapped/paged out.

Without seeing what's happening on your machine if you think it's "slow" it's just wild guesses. I suggest you open a call at BC-OP-SUN and let the support give you a dtrace script to trace down the root cause of the error.

I still think that your Oracle parameters are too small and hence the memory is not used.

Markus

[MPGraziano]

Hey Markus...(or anyone!)

On WIndows OS, if we were to increase Memory from ..say 32G to 64G how would we config in Profile to make the application make the most effeicient usage of that additional memory?

Also if you have notes, papers would really appreciate it...

maria

[markus_doehr2]

> On WIndows OS, if we were to increase Memory from ..say 32G to 64G how would we config in Profile to make the application make the most effeicient usage of that additional memory?

I suggest opening a new thread for a new question

Neverless, you configure the parameter PHYS_MEMSIZE to tell the system how much memory should be used for the SAP application. If a database is running on that machine also I'd reconfigure the memory there too.

Markus

[MPGraziano]

Thanks Markus...

I've posted another thread under sam forum , only for SQL on Windows....

btw the database is on same server as appl

[Former Member]

Hi Markus,

Thanks for your explanation so far.

I agree that I have to setting up the oracle parameter regarding the ORA04030-out of memory error since oracle won't use virtual memory in any reason but only the allocated memory for Oracle.

But what about the error for malloc failed in work process trace file? The instance memory should use virtual memory (swap area on disk) in this case. From your explanation about DISM / ISM, I conclude in this case, no matter how much swap I set up in my zone, it useless coz the swap to disk won't work out (Please correct me if I'm wrong about this).

FYI, this behaviour doesn't occur on my global zone. I use vmstat -s in both global and container zone, the pages swap in/out in global is not 0 but in container, where the malloc failed occur, all swap values are 0

[markus_doehr2]

> But what about the error for malloc failed in work process trace file? The instance memory should use virtual memory (swap area on disk) in this case.

If other limits are hindering - no. These can be the project settings, the shell (limit/ulimit -a) or defined processor sets/pools in the zone.

> From your explanation about DISM / ISM, I conclude in this case, no matter how much swap I set up in my zone, it useless coz the swap to disk won't work out (Please correct me if I'm wrong about this).

What's the output of

swap -l

Markus

[Former Member]

Markus,

We've set the limit to unlimited for both SID and ORA admin user as below.

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
nofiles(descriptors) 4096
vmemory(kbytes)      unlimited

Hereby the output of swap -l as of now

swapfile             dev  swaplo blocks   free
/dev/swap            0,0      16 52447088 52447088

[markus_doehr2]

> We've set the limit to unlimited for both SID and ORA admin user as below.

Ok.

If you still think that this is an OS problem I'd contact Sun. I don't know enough of your machine and your configuration to judge anything.

Markus

[Former Member]

OK, Thanks for your help so far.

[former_member524429]

Hi,

Please refer the following SAP Notes:

[ Note 570375 - SAP on Sun Solaris: Swap Space and Paging|https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=570375]

[Note 39412 - How many work processes to configure|https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=39412]

[Note 425207 - SAP memory management, current parameter ranges|https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=425207]

[Note 830576 - Parameter recommendations for Oracle 10g|https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=830576]

[ Note 838240 - Oracle Database 10g: Operating Systems Requirements Solaris|https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=838240]

[ Note 618868 - FAQ: Oracle performance|https://websmp230.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=618868]

Regards,

Bhavik G. shroff

[Former Member]

Thanks for your reply.

I've read relevant note you've given, 570375 about swap and paging.

The thing is, vmstat 5 command on our container during slow response time(when memory is running out) show no po value, the container cannot paged out the processes. What I know is the swap area are shared between global zone and container zone.

I need the answer if anyone ever experienced the same thing on container of SunOS 5.10 and what are the solution for this? Thanks.