usr/sap/SID is 100% full but actual usage is only around 500MB

[Former Member]

Hi experts,

We received a ticket regarding usr/sap/SID 100% full

On checking allocation used indeed was 100%

nlxdsm29:deqadm 24> df -k .

/usr/sap/DEQ (/dev/vgdeq_1/lv_sap ) : 1560576 total allocated Kb

0 free allocated Kb

1560576 used allocated Kb

100 % allocation used

Probing further i found that actual usage is just around 500MB by data directory

nlxdsm29:deqadm 31> pwd

/usr/sap/DEQ/DVEBMGS01/data

nlxdsm29:deqadm 30> du -sk *|sort -nr|head -10

457608 PAGFIL01

1 rslgssta

1 rslgspid

1 rslgcpid

1 ENQSTAT

0 astat

0 ROLLFL01

Am stuck after this!!

kindly suggest

[Former Member · Accepted solution]

Hello Ameya,

If you do not mind can you please tell us how you resolved your problem..I am also getting the same error ..

Thanks in advance..

[Former Member · Accepted solution]

Hi Ashley,

I will tell u exactly what happened

Our AIX team found that some processes were using up some space on this file system. Dont know the command .

/usr/sap/DEQ: 12349co(deqadm) 12374co(deqadm) 12358co(deqadm) 12348co(deqadm) 26411c(root) 12353co(deqadm) 8212co(deqadm) 12319co(deqadm) 12361co(deqadm) 12362co(deqadm) 19175co(deqadm) 12351co(deqadm) 12360co(deqadm) 12371co(deqadm) 17619c(deqadm) 12363co(deqadm) 12347co(deqadm) 12354co(deqadm) 12373co(deqadm) 12355co(deqadm) 12352co(deqadm) 26279co(deqadm) 29673c(root) 12366co(deqadm) 12350co(deqadm) 12372co(deqadm) 12359co(deqadm)

A mail was sent to the senior member regarding the situation.

After which he told to increase the file system by 3GB.

i really dint get the reason to increase the file system but thats how it was resolved.

it is a logical volume. i dont have any idea about logical volumes.

/usr/sap/DEQ (/dev/vgdeq_1/lv_sap ) : 3046856 total allocated Kb

1484334 free allocated Kb

1562522 used allocated Kb

51 % allocation used

[Former Member · Accepted solution]

Hi

I cannot completely solve your problem from my place, but i have some input. The command that your AIX admins were using is fuser

root# fuser -uc /usr/sap/SID
/usr/sap/SID:    23668co(sidadm)   20775co(sidadm)   20743co(sidadm)   18241co(sidadm)  ... 

This are of course the SAP workprocesses. On some unixes you have to be root to execute fuser. There are various reasons for a du and a df to differ (filesystem, allocation block sizes, sparse files...). In your case i suspect a big file got deleted, but an existing process was still having a file handle open to it. That way the space on the filesystem couldn't be freed up. So if you stopped your SAP system (or only that process) and restarted it, then the problem should be gone.

Best regards, Michael

[Former Member · Accepted solution]

Thanks Michael,

Woah its quite difficult to understand

So process rslgcoll(co) had a file handle open to file in usr/sap/SID directory.

During this time the file was removed from the directory.

But as the process was writing to that file in the directory the space though the file was deleted couldn't be freed up , reflected in df command but not in du command.

Correct me if am wrong

Its really difficult for a novice lyk me to understand this concept

I would really appreciate if u could provide me some links to understand this better

Or if you can explain it in more simplified manner if u dont mind obviously!!

[Former Member · Accepted solution]

I just browsed the AIX forums, but did not find a good thread there, so here is some stuff from the HP-UX forums:

[Files deleted but the filesystem is still full|http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1129437]

[Filesystem full issue|http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=183186]

[Disk full but no files..|http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=994355]

Regards, Michael

[Former Member · Accepted solution]

Hi Michael, yes on some of our HP-UX, we will see same situation, usually after reboot all are OK.

Regards.

[Former Member · Accepted solution]

Thanks Michael

Now its quite clear

[Former Member]

Hi,

You can reduce the size of the trace file to make the space free. To reduce the trace files, go to SM50--> process (on the top of menu..on the most corner)--> Trace--

---

>Reset -

---

>All Files

This will give you immediate relief and make your system running. Now more over this, you can delete all *.old and core files safely from the path /usr/sap/DEQ/DVEBM/work directory.

With Regards,

Saurabh

[Former Member]

Hi Saurabh,

i know the the default procedures to free up some space on this directory,i also know u can remove .stat and .astat from data directory

I think u havent got the question Sir

My question is

Why are the 2 commands showing different output

This command shows usage is 1.5 GB

nlxdsm29:deqadm 24> df -k .

/usr/sap/DEQ (/dev/vgdeq_1/lv_sap ) : 1560576 total allocated Kb

0 free allocated Kb

1560576 used allocated Kb

100 % allocation used

while this command shows usage is just 500MB

nlxdsm29:deqadm 28> *du -sk |sort -nr|head -10

457613 data

24285 log

689 work

51 sec

nlxdsm29:deqadm 29> cd data

nlxdsm29:deqadm 30> du -sk *|sort -nr|head -10

457608 PAGFIL01

1 rslgssta

1 rslgspid

1 rslgcpid

1 ENQSTAT

0 astat

0 ROLLFL01

I hope you got the question now!!!

please try to understand the question and then answer

regards

Edited by: Ameya Joshi on Jun 6, 2009 8:09 PM

[Former Member]

Hi Ameya,

There is definately something wrong with command du -sk. You can discuss this with your OS Admin.

Because work directory can not be only of size 689 KB !

Can you please provide the output of work directory files in KB?...use command ls -altr in work directory and provide the output ?

With Regards,

Saurabh

Edited by: Saurabh.Arora on Jun 6, 2009 2:24 PM

[Former Member]

there are 192 files in the work

nlxdsm29:deqadm 39> ls -altr|wc -l

192

and i dont think there is any problem with the command du

and the files are in the size of kb trust me on this

nlxdsm29:deqadm 45> pwd

/usr/sap/DEQ/DVEBMGS01/work

nlxdsm29:deqadm 33> du -sk *|sort -nr|head -10

93 dev_ftp

42 pxastat

30 \DEQ2200000165606

18 CPICTRC6738

18 CPICTRC6073

18 CPICTRC3685

18 CPICTRC3101

18 CPICTRC27653

18 CPICTRC26456

18 CPICTRC1816

[Former Member]

Hi Ameya,

As per this output, it means that trace files of WP are less than 18 KB....Very Hard to beleive !

Well, you can take a restart of your SAP server because any way your /usr/sap/<SID> is 100% full and your system is not in usable state.

After making a restart, all the WP will be restarted and you may see some difference....

With Regards,

Saurabh

[JPReyes]

I seen this behaviour before i sorted by unmounting and remounting the file system.

[Former Member]

Well our AIX team dint do that

they extended the file system by 3GB

But i still dint get the inconsistency!!

[Former Member]

Hi Ameya,

Here are my 2 cents:

Maybe you have 2 different partitions:

u201C/usr/sap/SID/u201D (1.5 GB) and u201C/SID/DVEBMGS00/datau201D (with a different size)

And the first partition is the mount point of the second.

Please, ask to your UNIX SysAdmins to type the following command:

df -k (without the final u201C.u201D) and post the result.

Regards,

Federico Biavati

[Former Member]

hi,

once check the virtuval memory in that drive.

[Former Member]

Please check the logs in the work directory and the trace generated there.you can also check that if some coredump is generated in the work Directory.

/usr/sap/DEQ/DV*/work

Thanks Rishi Abrol

[Former Member]

Hi Rishi,

du -sk * |sort -nr |head -10 command sorts directory according to their usage, with high usage directory on top

so if core dump had been generated then work directory would have been 1st in the list

Being a novice i havent seen such thing

Is there any chance that any process is using some space from this directory???

Edited by: Ameya Joshi on Jun 6, 2009 7:55 PM