Monit watches that services are running and brings them back when they fall over. The first fair question: why, when systemd can do the same with a single Restart=always line?

The answer defines the whole setup. systemd sees that a process has exited and starts it again. It does not know whether the service answers requests. PHP-FPM that has hit its process limit is alive and well as far as systemd is concerned, while the site returns 502. MySQL that has run out of disk keeps running as a process and executes not a single query. Monit checks not the fact that a process exists but its behaviour: whether the port answers, what an HTTP request returns, how much memory is in use.

Hence the rule: leave systemd as it is and add Monit for the functional checks. There is no point duplicating the restart of a crashed process.

Configuration

sudo apt install monit

The main file is /etc/monit/monitrc; your own checks go as separate files into /etc/monit/conf.d/. The usual opening:

set daemon 60
set logfile /var/log/monit.log
set mailserver localhost
set alert admin@example.com

Polling once a minute is a sensible balance. More often adds load and raises the risk of a false restart over a one-second delay.

The web interface: local only

Monit has a built-in web page, and it usually gets enabled roughly the way the first guide you find shows it — on all addresses. The result is a control panel for the server's services, reachable from the internet. The correct variant:

set httpd port 2812 and
    use address localhost
    allow localhost
    allow admin:'a-long-password'

Access from outside goes through an SSH tunnel:

ssh -L 2812:localhost:2812 user@203.0.113.25

After that the page opens on your own computer at localhost:2812, and nothing faces outwards.

Checks that make sense

The web server — not by the fact of a process but by its answer:

check process nginx with pidfile /run/nginx.pid
    start program = "/bin/systemctl start nginx"
    stop program  = "/bin/systemctl stop nginx"
    if failed host 127.0.0.1 port 80 protocol http
        request "/" status = 200
        for 3 cycles then restart
    if 3 restarts within 10 cycles then unmonitor

Three lines here matter more than the rest.

protocol http ... status = 200 is the check the whole thing was set up for: the server has to return a page, not merely hold a port open.

for 3 cycles means do not react to a single failure. Without it Monit will restart the service over one slow response under load, which only makes matters worse.

if 3 restarts within 10 cycles then unmonitor is a mandatory line. If a service will not come up because of a configuration error, Monit will restart it forever, adding load and filling the logs. This line means: three failed attempts, stop and leave a message. What is needed next is a person; automation is past helping.

Disk space is checked with no automation at all, simply as an alert:

check filesystem rootfs with path /
    if space usage > 85% then alert
    if inode usage > 85% then alert

The inode line is needed separately: they run out independently of space, and without it that situation goes unnoticed.

Checking the configuration

sudo monit -t
sudo systemctl reload monit
sudo monit summary
sudo monit status

monit -t checks the syntax before it is applied. monit summary gives a short table of states — the place to start when looking into any problem.

And check that mail is being sent. Monit reports events by email, and if delivery is not configured there will be no notification: the service will restart and you will not know about it — while repeated restarts are precisely the main signal. They can also be seen in the log: /var/log/monit.log.

What Monit does not do

It does not remove the cause. A restart is a postponement, and a service being restarted constantly means memory is short somewhere, or space is running out, or the application leaks. The value of the tool is not in the restart but in the counter: three nginx restarts in a day is a diagnosis that would otherwise have gone unnoticed, because the site was working the whole time.

So what to look at is not the current state — it is nearly always green — but the history: how many times over the past week something came back up by itself. What that looks like on one page is on the demo below.