Performance on Solaris 10 - Operating system paging

[Former Member]

Has anyone experience performance issues after an upgrade from Solaris 9 to Solaris 10 on SAP systems with limited memory?

We have many systems that are on servers with 4 Gig of memory and ran well on Solaris 9. After an upgrade to Solaris 10 we are experiencing very high OS system paging rates. The response times of the SAP systems are very poor when this occurs. It seems to take very little load to cause this.

I realize more memory or decreases in Oracle or SAP memory parms will solve this but am wondering if there is anything on the Solaris OS that could resolve this?

Thanks,

Dan

[markus_doehr2]

Without knowing comparable numbers, it´s difficult to give an advise.

Did you create project files (in /etc/project and /etc/user_attr) for each of those systems on Solaris 10? Most parameters in /etc/system became obsolete with Solaris 10.

If you did not, then the system will use the default values which may or may not be right for your environment.

--

Markus

[Former Member]

We did set up projects. The paging numbers were near 0 before the upgrade and vary widely with load under Solaris 10. For example when a heavy I/O job is started paging rates sometimes exceed 100,000 pgscan/sec with over 10,000 being typical.

[markus_doehr2]

Do you use VxFS or do you use UFS for the data storage?

--

Markus

[Former Member]

We use VxFS.

[markus_doehr2]

What comes into my mind is

Note 892631 - Solaris: mount options for best performance

just a quick guess, if you don´t already know that note and mount options.

Also check that Oracle has "filesystemio_options=setall" set.

Also check, if you use (D)ISM for Oracle, which will "lock" pages in memory exclusively so the application can´t make use of that:

http://sunsolve.sun.com/search/document.do?assetkey=1-9-72952-1&searchclause=dism%2420large%2420page

In any case, I´d suggest opening a call under BC-OP-SUN and let the Sun people look at the system.

--

Markus

[Former Member]

We are well aware of Solaris mount options as this caused us some issues many years ago. Good thought though. We have "filesystemio_options = setall" set.

I am not familiar with (D)ISM for Oracle. We have a pretty typical Oracle set up for SAP so I doubt that is set. How can you tell?

Thanks,

Dan

p.s. We have an open case with Sun.

[markus_doehr2]

I was thinking about Sunsolve article:

http://sunsolve.sun.com/search/document.do?assetkey=1-9-72952-1

Check that article, it gives a pretty good information what it is, how to detect, if it´s used and how to enable large pages, which will increase performance significantly.

You can also have a look at

http://sunsolve.sun.com/search/document.do?assetkey=1-9-79713-1

where you can read how to find out, if ISM or DISM is used.

What you can also check is the amount of memory really used in the system (using transaction OS04 and ST04 and SM04) and if that only occurs if someone is using a very big amount of memory or if the database is causing trouble.

--

Markus

[Former Member]

I will check those out. Thanks Markus.

[Former Member]

DISM can be used but in global zone only (according to Sun document "Best Practive for Running Database in Solaris Containers" , the proc_lock_memory privilege which is required to run the ora_dism_ process is not available in non globale zone)

The doc i got is from 2005, so don't know if the Sun recommendations has been updated since then.

In order to activate DISM (if you are in a global zone), sga_max_size should be set up larger than the sum of sga components: db_cache_size, shared_pool_size ...)

Also look for the Sun Blueprint "Dynamic Reconfiguration and Oracle 9i Dynamic Resizable SGA" on http://www.sun.com/blueprints

If you use ISM because in a non-global zone, you can use oracle parameter lock_sga to ensure the SGA is loaded into the RAM and useism_for_pga = true to ensure PGA is loaded into the RAM.

Make sure you have enough RAM to hold filesystem cache (OS memory) , oracle memory, and applications memory

Make sur your PGA and SGA are correctly sized size, since you won't be able to dynamically change the ISM allocation. (see v$shared_pool_advice, v$db_cache_advice, v$pga_target_advice ...)

Take the usual precautions:

- have a successfull backup first

- do the change on a test machine

- and/or ask your vendor before proceeding

Other Doc to read ...

Note 697483 - "Oracle Dynamic SGA on Solaris" (recommends to read Sun doc n°230653)

Note 724713 - parameter settings for Solaris 10, here is an extract :

Only one parameter from SAP note 395438 should remain in file

etc/system

set rlim_fd_cur=8192

As described in SunSolve document 215536, the "Large Page Out Of the Box" (LPOOB) feature of the Solaris 10 memory management, first implemented in Solaris 10 1/06 (Update 1), can lead to performance problems when the system runs out of large chunks of free contiguous memory. This is because the Solaris 10 kernel by default will attempt to relocate memory pages to free up space for creating larger blocks of contiguous memory. Known symptoms are high %system CPU time in vmstat, high number of cross calls in mpstat, and Oracle calling mmap(2) to /dev/zero for getting more memory.

Memory page relocation for satisfying large page allocation requests can be disabled by setting the following Solaris kernel parameter in /etc/system

set pg_contig_disable=1

This will not switch off the LPOOB feature. Large memory pages will still be used when enough free space of contiguous memory is available, so the benefits of this feature will remain

Note 870652 - Installation of SAP in a Solaris Zone

Note 1246022 - Support for SAP applications in Solaris Zones

Edited by: Emmanuel TCHENG on Oct 13, 2009 12:02 PM

[Former Member]

Hi

This is an update to my previous post since i have checked the last blueprint from sun : the proc_lock_memory privilege is now available also in non-globale zone (if your solaris 10 patches are up to date) so you can now take advantage of the DISM for the dynamic SGA management as of orace 10 in non globale zone as well as in globale zone.