Wednesday, May 8, 2013

Newbie help: Why is my filesystem full?

If a filesystem fills up and you have no idea where to start, try the disk usage command. This will at least help you find breathing room, and it may help you find the overgrown or unexpected file that caused the problem.

If you haven't already, use df to make sure you know which filesystem is full.  Many systems only have one filesystem, but it's more common to have a few.  For the purpose of this discussion, you're only concerned about files below the mount point of one overutilized filesystem.


Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vgroot     30G  3.6G   25G  13% /
tmpfs                 1.9G  4.0K  1.9G   1% /dev/shm

/dev/mapper/vgapp      20G  6.7G   13G  96% /app

nfssrv:/vol/devshare   10G  192K   10G   1% /app/dshare
nfssrv:/export/home/fred
                      102G   90G   12G  89% /home/fred


cd to the mount point or a point in the directory tree where you suspect things might have gone awry, then do this:

du -sk ./* | sort -n 

In English, that's "disk usage -specific_file -kilobytes, then sort -numeric." These flags will be the same on any *nix.

Non-root users will receive a lot of "Permission denied" errors when running this command in system or shared directories. This can be a good thing; if you're interested in finding files created by a certain user, the errors show you where not to look. On the other hand, if you're really in the dark and/or troubleshooting a multiuser system, you may need to become root.

You can use du to identify the largest directories, which may help you find your problems. 

An example:

$cd /app
$ls -atlr
total 8167528
dr-xr-xr-x. 30 root   root      4096 Mar 20 03:51 ..
drwxr-xr-x.  7 lruser lrgrp     4096 Mar 26 20:04 common
drwxrwxr-x.  9 lruser lrgrp  1490944 Apr 23 16:24 .
drwxr-xr-x. 16 lruser lrgrp     4096 May  1 16:52 calendar
drwxr-xr-x. 14 lruser lrgrp     4096 May  7 18:16 liferay

Not much help, right?  But du makes it easy to decide where to look first.
$du -sk ./* | sort -n
644116 ./common
1563248 ./calendar
8613876 ./liferay

So the Liferay directory's pretty chubby.  Why is that?
$cd liferay
$du -sk ./* | sort -n
0 ./jdk-liferay
4 ./deploy
520884 ./data
1092408 ./logs
1232588 ./backup
5492700 ./appserver

Unsurprisingly, the JVM accounts for most of Liferay's 8gb.  I should also look for things to delete from the backup directory, but let's stick with the big fish for now.
$cd appserver
$du -sk ./* | sort -n
...
149408 ./temp
419056 ./web
4759364 ./logs

And again unsurprisingly, the logs directory accounts for most of the JVM's utilization.  Du works on files as well, so you could use it to quickly find the largest file in a directory.
$cd logs
$du -sk ./* | sort -n
54340 ./applog.20121001.gz
71796 ./applog.20120418.gz
349252 ./applog
422952 ./applog.20120613
2816980 ./applog.20130204

Wait, I'm lazy.  How big was that last file?
$ls -lh applog.20130204
-rw-r--r--. 1 lruser lrgrp 2.7G Feb  4 16:55 applog.20130204

Many commands that show file sizes, such as df,  du, and ls, have a -h (-human_readable) flag.  We use du -k because the -h output is not easily sortable.

If you haven't freed enough space or found your culprit, pick another place in the tree and du down from there.



No comments: