29-10-2017 | Remy van Elst
In my box of sysadmin tools there are multiple gems I use for troubleshooting servers. Since I work at a cloud provider sometimes I have to fix servers that are not mine. One of those tools is
ncdu. It's a very usefull tool when a server has a full disk, both full of used space or full of used inodes. This article covers ncdu and shows the process of finding the culprit when you're out of disk space or inodes.
On the homepage ncdu describes itself as:
Ncdu is a disk usage analyzer with an ncurses interface. It is designed to find space hogs on a remote server where you don't have an entire graphical setup available, but it is a useful tool even on regular desktop systems. Ncdu aims to be fast, simple and easy to use, and should be able to run in any minimal POSIX-like environment with ncurses installed.
When the root disk is full, a server will behave strangely. Some daemons still work, others will work for a bit and some refuse to start at all. If you use
ext4 the server will boot up most of the time since there is a bit of reserved space in the filesystem.
A disk can be full because of too much space being used. You can check that with the
df -h command:
root@s1:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/vda1 20G 19G 0 100% /
A disk can also be full because there are to many files and folders. You've ran out of inodes.
df -i shows you that:
root@s1:~# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda1 1305600 1305600 0 100% /
In this case one of my own servers has ran out of inodes. I know where they are, one of the users has no quota and has synced an entire 10 years of Maildir, which resulted in too many small files. If I wouldn't have known where the storage was being used, ncdu would be my go to tool for finding out.
Yes you can use
du takes a long time and it doesn't tell you where the inodes are used. A shell loop with find for finding inodes has the same problem. You also can't easily navigate the results.
ncdu gives a graphical overview of the files and folders, sorted by most space used. In the lower right corner there is a counter for the number of files in the current folder. Using that counter, you can navigate around the filesystem and see where the millions of inodes are used. Sorting by inode usage can also be done with the
C key. No other tool I know of has that option.
ncdu also allows exporting the results. You can then see them on another machine. Usefull if you have a weird ARM nas with terminal display issues. ncdu runs remotely without the TUI,
scp over the results file and use the local terminal to find out what's wrong.
ncdu is available for almost every operating system imaginable and probably in your package manager:
apt-get install ncdu yum install ncdu
If you have trouble with a full disk or your package manager doesn't work, the developer even has static binaries on the page:
Sometimes you can remove a few files to make space for this download, where the package manager doesn't work.
Start up ncdu. If you don't give any parameters, it will scan the current folder. If you give a filesystem path, it will scan that:
Will start a scan of the entire system. When the scan is finished, use the arrow keys and ENTER to navigate around. Files and folders can be deleted right away using
d. You can change the display from the bar to percentage or both using the
g key. Sorting based on filesize with
s, by name with
n and by inodes with
C. That option is very usefull when your disk has run out of inodes.
ncdu is not real time. If you remove files outside of ncdu you need to run a rescan.