Falco is known as a tool for Kubernetes, and nearly every guide to it is written for clusters. Yet it is not tied to clusters: it watches system calls, and on an ordinary server with a website that works in exactly the same way. It is simply that hardly anybody writes about that case.

It is useful where the other tools stay silent. A web shell on a site violates no permissions, produces no failed logins and matches no signature if it was written by hand. But it has behaviour that is abnormal for a web server: the PHP-FPM process starts a shell. That is what Falco sees.

What it notices

Typical events on an ordinary server:

  • a shell spawned by the web server or PHP process — practically an unambiguous sign of a web shell;
  • a program started from /tmp, /dev/shm or /var/tmp;
  • sensitive files (/etc/shadow, private keys) read by a process that has no business doing so;
  • modification of system binaries;
  • an outbound connection from a process that should not use the network.

Installation

The key choice is made at installation: how Falco obtains system calls. The modern option is based on eBPF and requires neither building a kernel module nor kernel headers:

sudo falcoctl driver config --type modern_ebpf
sudo systemctl restart falco

The classic kernel module needs headers and is rebuilt after every kernel update — on a server where updates install automatically, that is a recurring source of a dead service. If the kernel is recent enough (5.8 and above), choose eBPF and forget the problem.

To check that events really are arriving:

sudo systemctl status falco
sudo journalctl -u falco -n 50

Noise, and removing it

This is the main work. The stock rule set is aimed at container environments, and on an ordinary server a significant part of it is either inapplicable or fires constantly.

The shipped rules (/etc/falco/falco_rules.yaml) are not edited — the file is replaced on update. Your own changes go into /etc/falco/falco_rules.local.yaml, and unwanted rules are switched off there as well:

- rule: Terminal shell in container
  enabled: false

What usually needs adjusting on a server without containers:

  • every container rule — with no containers they only take up room in the report;
  • "Write below etc" — fires at every package installation and at any edit of yours to a configuration file. It needs an exception for apt, dpkg and unattended-upgrades, otherwise the events arrive in a stream;
  • "Read sensitive file untrusted" — fires on monitoring agents, backup tools and audit tools such as Lynis;
  • launches from temporary directories — legitimate exceptions exist: building an application, or an automated browser unpacking a driver into a temporary directory. Such events look alarming but are explainable, and it is worth adding an exception for the specific path straight away rather than working it out afresh every time.

The sensible order is the same as with any other detection tool: for the first week only watch and add exceptions, and only then treat an event that appears as a signal. The rule is simple — if the report regularly contains events you do not read, it is doing nothing for you.

Where to send the events

The output is configured in /etc/falco/falco.yaml: a file, the system journal, or piping to an external program. For a single server a file with rotation afterwards is enough — do not forget the rotation, the event file grows like any other log and by default nobody watches it.

Priorities are worth using to separate things out: critical events go where you will see them immediately, the rest into the general journal for later review.

Falco and auditd are not the same thing

Both watch system calls, but with different aims. auditd records what happens so that the picture can be reconstructed later: it evaluates nothing and reports nothing, it keeps a journal. Falco applies rules at the moment of the event and says "this looks suspicious" — that is, it gives a signal rather than a record.

Keeping both is reasonable: the journal for reconstruction and the signals for reaction. If you have to choose one, auditd is more useful on a server where reconstructing the sequence of events after an incident matters most; Falco where you want an early signal about a running miner or a web shell.

Is it worth the room on a small server

The honest answer: not always. Falco processes system calls and is noticeable on the processor on a busy machine. If the server hosts one site and you still have neither integrity monitoring nor decent automatic updates, it is not the place to start.

Its moment comes later — once the basics are done and the remaining question is what happens on the server that the logs do not show. One class of event it covers better than all the others put together: a shell started by the web server process. What the events look like split by priority and rule is on the demo page below.