Memory leak in Solaris 10?

[former_member84399]

Dear all,

I have installed and configured a Solution Manager 4.0 system SPS 15 on a Solaris 10 T2000 SPARC hardware with 8 core CPUs and 32 GB of RAM. The database is Oracle 10.2.0.2. My Java virtual machine version is 1.4.2_17. When I start the system and go to st06, I see that I have 26 GB of RAM free

A few hours (sometimes even one hour) later and without any activity on the system whatsoever (it is brand new so nobody but myself is using it) I execute st06 and it looks like I only have 1 GB of memory left.

My saposcol is consistent with my kernel level (patch 133, which is the patch for the Support Stack 15). I had the same problem with support stack 13 as well ( so upgrading kernel didn’t make any difference) and Java virtual machine 1.4.2_13

I have the same problem even when I start the system without starting the Java Stack (set parameter rdisp/j2ee_start to 0), so I do not believe that it has to do with the Java virtual machine or the Java Stack in general

Any ideas ?

Many thanks

Andreas

[markus_doehr2 · Accepted solution]

The memory is not "leaking".

Solaris uses the available (= free) memory for filesystem cache. If the memory is needed, the system will "free" the memory out of the filesystem buffer and will pass it to the application. This makes sure the system is using the memory as needed and at the same time keeping filesystem accesses to a minimum.

See http://www.solarisinternals.com/si/reading/sunworldonline/swol-07-1999/swol-07-filesystem3.html

Markus

[former_member84399 · Accepted solution]

Markus, thanks, very interesting data

However, I have other installations ( ERP 2004 with Oracle 10.2.0.2 running on Solaris 10) in same as before SUN T2000 hardware with 32 GB RAM and 8 core CPUs where I have consistently 19 GB free memory free. The only difference between this and my Solution Manager system is that the Soluion Manager system is in a Solaris zone

Many thanks

Andreas

[markus_doehr2 · Accepted solution]

Are any other instances running on the machine (more zones or an instance in the global zone)?

Are you using the same release of Solaris 10? (cat /etc/release on both machines)

Saposcol is not "zone-aware" - it will see the full machine always.

Markus

[former_member84399 · Accepted solution]

Markus,

PRD app server, where I see 19 GB free memory consistently

root@prdap001 # cat /etc/release

Solaris 10 6/06 s10s_u2wos_09a SPARC

Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.

Use is subject to license terms.

Assembled 09 June 2006

SOL system where I see 1 GB free memory

root@solman # cat /etc/release

Solaris 10 8/07 s10s_u4wos_12b SPARC

Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.

Use is subject to license terms.

Assembled 16 August 2007

I do not have any other zones (at the moment) nor any other instances running in that hardware, but it was purchased to host solution manager and some XI systems (not installed yet). If Solution Manager appears to consume all memory, I am not sure if any more systems will cope in there .

Many thanks

Andreas

[markus_doehr2 · Accepted solution]

Don´t worry - SolMan is not using all the memory.

What do you see when you execute

prstatt -Z

in the global zone on the box with the Solman? How much memory is used there for the zone?

Markus

[former_member84399 · Accepted solution]

Hello,

Looks like Solution Manager is only using 4.4 GB from what I see in prstat. Shouldn't I worry at all then ?

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

8879 root 3664K 3656K cpu0 59 0 0:00:00 0.1% prstat/1

8859 root 9040K 7000K sleep 59 0 0:00:00 0.0% sshd/1

4585 100 8032K 3584K sleep 59 0 0:09:10 0.0% saposcol/1

8858 root 5672K 4888K sleep 59 0 0:00:00 0.0% ssh/1

6873 100 1397M 1276M sleep 59 0 0:02:45 0.0% oracle/1

7162 100 2609M 698M sleep 59 0 0:12:57 0.0% jlaunch/200

7161 100 431M 266M sleep 59 0 0:03:52 0.0% jlaunch/116

1833 noaccess 294M 86M sleep 59 0 0:02:51 0.0% java/55

6869 100 1398M 1277M sleep 59 0 0:01:48 0.0% oracle/1

1874 noaccess 211M 89M sleep 59 0 0:02:50 0.0% java/55

945 daemon 5104K 3848K sleep 59 0 0:00:01 0.0% kcfd/5

882 noaccess 283M 87M sleep 59 0 0:02:49 0.0% java/55

135 daemon 4632K 2728K sleep 59 0 0:00:00 0.0% kcfd/5

290 root 2080K 128K sleep 59 0 0:00:00 0.0% smcboot/1

251 root 9512K 4072K sleep 59 0 0:00:03 0.0% inetd/4

264 root 2504K 848K sleep 59 0 0:00:00 0.0% ttymon/1

250 root 2168K 680K sleep 59 0 0:00:00 0.0% sac/1

289 root 2080K 120K sleep 59 0 0:00:00 0.0% smcboot/1

210 root 2536K 920K sleep 100 - 0:00:09 0.0% xntpd/1

238 daemon 2408K 456K sleep 60 -20 0:00:00 0.0% nfs4cbd/2

201 root 2920K 1528K sleep 59 0 0:00:00 0.0% cron/1

233 daemon 2800K 560K sleep 59 0 0:00:00 0.0% rpcbind/1

132 root 6896K 1752K sleep 59 0 0:00:00 0.0% devfsadm/8

123 root 2184K 104K sleep 59 0 0:00:00 0.0% drd/2

ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE

0 50 287M 260M 0.8% 0:04:46 0.1% global

2 133 13G 4491M 14% 1:18:58 0.1% solman

1 35 273M 186M 0.6% 0:04:24 0.0% content

Another prstat shows the dis+work process, not sure why it was not shown in the previous prstat

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

7022 100 5293M 591M sleep 59 0 0:24:00 0.1% disp+work/1

4585 100 8032K 3584K sleep 59 0 0:09:15 0.0% saposcol/1

8978 root 3664K 3656K cpu24 59 0 0:00:00 0.0% prstat/1

7162 100 2609M 698M sleep 59 0 0:12:59 0.0% jlaunch/199

7161 100 431M 266M sleep 59 0 0:03:55 0.0% jlaunch/116

6873 100 1397M 1276M sleep 59 0 0:02:47 0.0% oracle/1

1833 noaccess 294M 86M sleep 59 0 0:02:52 0.0% java/55

6869 100 1398M 1277M sleep 59 0 0:01:49 0.0% oracle/1

1874 noaccess 211M 89M sleep 59 0 0:02:51 0.0% java/55

882 noaccess 283M 87M sleep 59 0 0:02:50 0.0% java/55

457 root 8136K 1832K sleep 59 0 0:00:03 0.0% sendmail/1

363 root 5080K 272K sleep 59 0 0:00:00 0.0% automountd/2

751 root 16M 5568K sleep 59 0 0:00:07 0.0% svc.startd/12

364 root 5512K 3392K sleep 59 0 0:00:00 0.0% automountd/2

397 root 3592K 560K sleep 59 0 0:00:00 0.0% rpc.metad/1

268 root 2560K 464K sleep 59 0 0:00:00 0.0% ttymon/1

407 root 15M 6648K sleep 59 0 0:00:04 0.0% fmd/18

287 root 2088K 320K sleep 59 0 0:00:00 0.0% smcboot/1

290 root 2080K 192K sleep 59 0 0:00:00 0.0% smcboot/1

251 root 9512K 4072K sleep 59 0 0:00:03 0.0% inetd/4

264 root 2504K 848K sleep 59 0 0:00:00 0.0% ttymon/1

250 root 2168K 680K sleep 59 0 0:00:00 0.0% sac/1

289 root 2080K 184K sleep 59 0 0:00:00 0.0% smcboot/1

210 root 2536K 920K sleep 100 - 0:00:09 0.0% xntpd/1

238 daemon 2488K 2048K sleep 60 -20 0:00:00 0.0% nfs4cbd/2

201 root 2920K 1528K sleep 59 0 0:00:00 0.0% cron/1

233 daemon 2800K 560K sleep 59 0 0:00:00 0.0% rpcbind/1

132 root 6896K 1816K sleep 59 0 0:00:00 0.0% devfsadm/8

123 root 2184K 104K sleep 59 0 0:00:00 0.0% drd/2

113 root 2640K 160K sleep 59 0 0:00:00 0.0% syseventd/14

244 daemon 2392K 520K sleep 60 -20 0:00:00 0.0% lockd/2

135 daemon 4632K 2728K sleep 59 0 0:00:00 0.0% kcfd/5

Many thanks

Andreas

Edited by: support team on Mar 7, 2008 1:15 PM

[markus_doehr2 · Accepted solution]

No - not at all.

As said - the system uses the available memory as filesystem cache; this is done by the VM layer.

From the book "Solaris Internals" (btw: SUPERIOR book!)

<quote>

One of the major advantages of using the VM to manage file system caching is that all free memory in the system is available as a cache, providing significant performance improvements for applications that use the file system and removing the need for manual tuning of the size of the cache. The VM system can allocate all free memory for file system cache, meaning that on a typical system with file system I/O, almost all of the physical memory will be advantageously used.

<endquote>

The difference to the other system you stated can be a result of using e. g. VxFS as filesystem or running Oracle using DISM.

Your system is behaving totally normal - as expected.

Markus

[former_member84399 · Accepted solution]

Ok, cool!

I suppose st06 is not of much use any more with this setup to decide how well the Unix box is doing with memory as far as SAP is concerned. Any suggestions? top and prstat I suppose?

Many thanks

Andreas

[markus_doehr2 · Accepted solution]

This is not a new "problem", almost all operating system work like that (also HP-UX and Linux, can't state for AIX).

If you want a "hard" memory border, you should consider using LDoms instead of zones (http://www.sun.com/blueprints/0207/820-0832.html).

To see the actual memory of an instance you can use ST02.

And yes - prstat (-Z) and "top" are the tools of choice - I use them all the time.

Markus