The question is almost always the same one: last night the site took twenty seconds to answer, and by morning it had cleared up on its own. There is nothing left to look at — top answers for the current second, and what you are being asked about is three in the morning. Below is what the load figures actually mean, which of them have to be read in pairs, and why a single reading misleads more often than it helps.

Load average: three numbers and one common misreading

uptime
cat /proc/loadavg
nproc

The three numbers are averages over one, five and fifteen minutes. They are not to be compared against zero but against the core count that nproc reports. A value of 8 on an eight-core machine is full but healthy load: there is exactly as much work as the server can carry out. A value of 2 on a single-core VPS is a queue twice as long as the machine can serve, and every request waits its turn.

How the three relate to each other gives you the direction. The one-minute figure well above the fifteen-minute one means load is climbing right now. The other way round means the peak has passed and you are looking at its tail.

Now the misreading that spoils these numbers more than anything else. Load average on Linux is not "CPU utilisation". Unlike other Unix systems, Linux counts into it not only the processes running or ready to run, but also those in state D — uninterruptible sleep. That means waiting on a disk or on a network filesystem. Hence the server sitting at a load of 12 with a nearly idle CPU: no work is getting done, everyone is queuing.

CPU or disk

Telling those two apart is the first thing worth doing:

vmstat 1 5
iostat -x 1 3

In the vmstat output, three columns matter. r is how many processes are queued for the CPU, b how many are blocked waiting on I/O, and wa the share of time the CPU spent idle waiting for the disk. A steady wa above 10–15 % with modest us and sy means the server is disk-bound: adding cores is pointless, the ones you have are already free.

Which processes are waiting can be listed by name:

ps -eo state,pid,comm,wchan:30 | awk '$1 ~ /^D/'

An example from our own practice. Suricata with the stock configuration writes some thirty event types and sets up no rotation for them — on a working server that produced 15 GB of logs in two days. The load it caused showed up not as a busy CPU but precisely as I/O wait: the system was writing continuously and everything else queued behind it. The fix is not a bigger plan but a shorter list of event types and a rotation policy.

The opposite case, purely CPU-bound and far less obvious. A web page was calling apt as the user PHP-FPM runs under. Root keeps a binary cache at /var/cache/apt/pkgcache.bin — 70 MB on a host with a dozen repositories — and maps it into memory for free. An ordinary user cannot write to that directory and rebuilds the cache from scratch on every call. Measured on one and the same machine: 0.01 seconds of CPU time as root against 4.2 seconds as an unprivileged user. The same command, a difference of four hundred times, multiplied by every page view.

The conclusion from both cases is the same: load has to be measured, not guessed. Timing each suspect command separately takes half an hour and usually points somewhere other than where the suspicion started.

Memory: free does not show what it appears to

free -h

The used column on its own says very little, and the free column is actively misleading: Linux hands unused memory to the page cache and gives it back to applications on demand. The column to read is available — how much can be claimed without going into swap. A large buff/cache is not a problem but a sign of a system working as intended.

With swap, what matters is the shape rather than the current figure. Up and back to zero means there was a brief peak. Up once and staying there means the peak has already happened, pages were evicted and nothing brings them back: the server looks calm although at some point it ran out of memory. Whether swapping is happening right now shows in the si and so columns of vmstat; non-zero values there are the form of slowdown users notice most.

If a process simply disappeared during the night, the explanation is usually here:

journalctl -k --since yesterday | grep -i "out of memory"
dmesg -T | grep -i "killed process"

A line reading Out of memory: Killed process 1234 (mysqld) settles the question better than any graph: the database did not "fall over", the kernel stopped it because memory ran out.

Who is doing it

ps aux --sort=-%cpu | head -10
ps aux --sort=-%mem | head -10

One caveat: %CPU in ps is an average over the whole life of the process, so a short burst is lost in that list. For a burst you need top or pidstat 1 5, which measure over an interval.

When load becomes a security question

A flat 100 % of CPU at night, with a process carrying a meaningless name and a working directory in /tmp or /dev/shm, is the classic picture of a miner rather than of a site that has grown. A spike in inbound traffic together with a rise in log writes is a password attack in progress. A sudden jump in log volume is usually a storm of false positives from a filter.

Security tooling itself is a category of its own. On one of our servers the load sat at around three with no visitors at all, and the top of ps by accumulated CPU time held not the site and not the database but CrowdSec, fail2ban, Falco and Suricata. That is neither a failure nor a reason to turn them off, but the cost of protection is worth knowing as a figure: on a small VPS it is noticeable.

The point of watching

Everything above answers the question "what is happening now". The morning question — what happened at three in the morning — is not covered by these commands: there is no data for the night just gone unless something recorded it. And standing up Prometheus and Grafana for a single VPS is questionable, since the observing stack ends up heavier than the server being observed.

A row in a database every five minutes and one page that draws the last twenty-four hours from it are enough: load average, CPU utilisation and I/O wait separately, memory and swap, disk reads and writes, partition usage, inodes, file descriptors, connections. On one time axis the pairs can be read by eye: high wa with a calm CPU, swap that never returned to zero, descriptors approaching the limit — the coming too many open files error becomes visible hours before it happens. What that looks like assembled is on the demo page below.