Integrity monitoring answers a question neither an antivirus nor a firewall answers: what changed on this server since last week. A signature scanner looks for known-bad things; AIDE knows nothing about badness, it knows that this file was different yesterday. For finding a backdoor appended to an existing file, that is the only approach that works.

AIDE installs in two commands. Configuring it so that its reports get read is the harder part — and that is usually what kills it: the first report arrives with ten thousand lines, nobody opens the second, and a month later the job is deleted.

Installation and the first database

sudo apt install aide aide-common
sudo aideinit

Initialisation takes from a few minutes to half an hour: hashes are computed for every file. The result is placed next to the working database with a .new suffix, and has to be put into service:

sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

From then on a check is run with:

sudo aide --check

What to exclude to keep the report readable

The settings live in /etc/aide/aide.conf and in the directory /etc/aide/aide.conf.d/. The stock Debian configuration checks too much, and the first thing to do is remove what changes by itself all the time:

  • /var/log — changes every second;
  • all of /var/lib — databases, package state, service state;
  • /var/cache, /tmp, /proc, /sys, /run;
  • the site's upload and cache directories — visitors change their contents.

What is worth checking with the strictest settings is a narrow list:

  • /bin, /sbin, /usr/bin, /usr/sbin — the system's executables;
  • /lib, /usr/lib — libraries;
  • /etc — configuration;
  • /root/.ssh and the .ssh directories of your users;
  • the site's code, but without the upload and cache directories.

The benchmark: an ordinary daily report should fit on a screen. If it is longer, the exclusions are insufficient and it will stop being read.

Where to keep the database

An important point that often gets missed. If somebody else has root, replacing a file and immediately updating the AIDE database costs them nothing — and the check afterwards will report that all is well. A database sitting on the same machine, writable, protects only against accidents.

A sensible minimum for a single server:

  • copy the database to another machine after every update, and run the check with that copy put in place;
  • or at least keep the database's checksum separately and verify it before checking:
sha256sum /var/lib/aide/aide.db

Even a measure that simple turns a silent substitution into a noticeable event.

Updating the database is a deliberate act

After legitimate changes — a system update, a new version of the site deployed — the database is rebuilt:

sudo aide --update
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

When that is done matters. Right: read the report, satisfy yourself that every change is explainable, and only then update. Wrong, and very common: run aide --update on a schedule so the reports come out clean. In the second case the system runs, the reports arrive, and changes are recorded as normal at exactly the moment they occur — which means the point has been lost entirely.

Schedule and load

The aide-common package installs a daily job by itself. The check loads the disk and processor for several minutes, so it is worth running at a quiet hour and with lowered priority:

0 4 * * * ionice -c3 nice -n19 /usr/bin/aide --check

How to read the report

Three sections: files added, removed and changed. For each change it shows what exactly differs: contents, permissions, owner, time.

What to react to first:

  • a change to any file in /bin, /sbin, /usr/bin outside an update window;
  • new files in system directories;
  • changes in authorized_keys, /etc/passwd, /etc/sudoers, /etc/crontab and /etc/cron.d;
  • a file /etc/ld.so.preload that has appeared — it never appears on its own.

A hundred changed files right after apt upgrade is normal, and the report's timestamp confirms it. Three changed files in /usr/bin on a Wednesday with no updates is a reason to stop and look into it.

AIDE and debsums

The tools solve neighbouring problems, but their reference differs. AIDE compares against a snapshot of your own — so it sees changes in any files, the site's code included. debsums compares against the checksums in distribution packages — so it sees system files replaced with no prior setup at all, but knows nothing about what you installed outside packages. It is reasonable to keep both; in practice debsums usually goes in first, because it needs neither configuration nor a database.

They share one weakness as well: the report arrives by mail, and mail gets lost. So what matters is not that the check ran but that there is a place where its latest result is visible. What that looks like is on the demo page below.