Running out of space is the most common reason a server stops working with no ill intent involved. The site returns 500, the database will not write, mail will not go out — and df -h meanwhile reports several free gigabytes. Let us go through all three cases where that happens, and through what is worth knowing about SMART.

Case one: out of inodes

A filesystem holds two limited resources: room for content and records about files. The second runs out independently of the first:

df -h
df -i

If IUse% in the second command is 100, inodes are the problem. There is space, but not a single file can be created, not even an empty one.

The culprits are always the same: millions of tiny files. PHP sessions in /var/lib/php/sessions whose garbage collection broke. An application cache that is never cleared. A stuck mail queue. A directory of thumbnails. You can find them like this:

sudo find / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20

The command lists the directories with the most files. On a large system it runs for minutes — that is normal.

Case two: the file is deleted and the space did not come back

An even more treacherous situation. Somebody removed an overgrown log with rm, but the process writing to it is still holding it open. The file is no longer in the directory, the space is still taken, and it will stay that way until the process is restarted.

sudo lsof +L1

The command shows files with zero links in the filesystem but a process still holding them. The cure is restarting or gently reloading that process, not a hunt for "where did the gigabytes go".

Hence the rule: an overgrown log is not deleted but truncated, which keeps the open descriptor usable:

sudo truncate -s 0 /var/log/huge.log

And immediately afterwards, set up rotation — otherwise the whole thing repeats within a week.

Case three: the logs ate it

Where exactly the space went is shown by walking down level by level:

sudo du -x -h --max-depth=1 / | sort -h
sudo du -x -h --max-depth=1 /var | sort -h

The -x flag keeps it from wandering onto other filesystems; without it the count crawls into /proc and network mounts. All of this is more comfortable in ncdu, if it is available.

The usual devourers:

  • journald. One command settles it: journalctl --disk-usage. It is capped by the line SystemMaxUse=500M in /etc/systemd/journald.conf and cleaned once with journalctl --vacuum-size=200M;
  • security tool logs. Suricata with its stock configuration writes three dozen event types and sets up no rotation for itself — on a real server that produced 15 GB in two days. The same happens with audit.log and with debug logs in nginx;
  • backups written to the same disk and never removed;
  • the apt cache — apt clean occasionally returns a couple of gigabytes.

A word about /boot

A small partition that fills up with old kernels. It does not bring the site down by itself, but it stops updates dead: the new kernel will not install, and the whole package queue stalls behind it. Cured with apt autoremove --purge, and prevented by enabling automatic removal of unused kernels in the automatic-updates settings.

SMART: which attributes matter

A caveat first: on a virtual server SMART is usually unavailable — the disk is virtual and you do not see the physical one underneath. That is not a fault, there is simply no data. Everything that follows concerns dedicated servers and your own hardware.

sudo smartctl -a /dev/sda

The line SMART overall-health self-assessment test result: PASSED is no reason to relax: it stays green practically until the drive dies. What to look at are the specific counters:

  • 5, Reallocated_Sector_Ct — reallocated sectors. Not zero means the drive is already degrading; growing over time means replace it without delay;
  • 197, Current_Pending_Sector — sectors that cannot be read and await a decision. The most alarming of the lot: it usually means part of the data is already unrecoverable;
  • 198, Offline_Uncorrectable — the same, confirmed by a check;
  • SSD: Percentage Used / Media_Wearout_Indicator — the write endurance consumed. A predictable figure, and the one replacement is planned against.

Temperature and power-on hours, on the other hand, say nothing on their own: a drive with five years of runtime and zero error counters is more dependable than a new one with a 1 in attribute 197.

The point of watching

All of the above is a reaction to something that has already happened. Yet both a filling disk and SSD wear are slow, entirely predictable processes: a two-week occupancy graph shows the date the space will run out long before it does. The difference between "the site has been down since three in the morning" and "the logs need clearing on Thursday" is nothing more than having the figure in front of you. What that looks like assembled is on the demo page below.