The question rarely comes out of nowhere. The server got slower, the hosting provider sent a note about outbound traffic, or there is a login notification in your mail for a login you never made. What follows is the unpleasant part: it is not obvious what to look at or in what order, and the first instinct — wipe it and start again — is almost always premature.

Below is a checking order that takes about twenty minutes and, in most cases, gives a clear answer. It runs from the cheapest thing to the most expensive: first what is visible straight away, then what has to be compared against a reference.

Step one: who logged in

Start with logins. If someone else got in, they almost certainly came through SSH, and there is a trace.

last -20
lastb | head -20
who

last shows the most recent successful logins, lastb the failed ones, who whoever is connected right now. What matters is not the count but the shape. Thousands of failed attempts from addresses that never come back are ordinary background brute force; it runs against every public address without pause and means nothing.

What should worry you is different:

  • a successful login from an address where none of your people are;
  • a login under a username you never created;
  • failed attempts against a username that actually exists on this machine — that means somebody learned who your users are rather than working through a dictionary;
  • a session open right now that you did not open.

Check separately whether any unfamiliar keys have appeared. The file ~/.ssh/authorized_keys is the most common way to stay in: the password can be changed as often as you like, the key remains.

cat ~/.ssh/authorized_keys
sudo cat /root/.ssh/authorized_keys

Every line there is somebody's access. If you cannot say whose, treat it as somebody else's.

Step two: what changed in the system

An intrusion nearly always leaves traces on disk: a replaced binary, an extra line in a config, a new file in the web server's directory. Checking that by eye is hopeless — you need a reference.

On Debian and Ubuntu the reference is already there: every package knows the checksums of its own files.

sudo apt install debsums
sudo debsums -c

The command lists files that differ from what the distribution installed. Some of the findings will be legitimate — configuration files under /etc exist to be edited. A modified executable in /usr/bin, /usr/sbin or /bin on a server you never touched by hand is another matter entirely.

The second source is recent files where none should be. A web shell usually sits in an uploads directory and looks like an inoffensive .php:

find /var/www -type f -name '*.php' -mtime -14 -ls

Fourteen days is only a starting point; use the period over which you know you published nothing.

Step three: what is going out

A compromised server is rarely broken into for its own sake. It gets used: to send spam, to mine, to reach into other networks, to host somebody else's files. All of that creates outbound connections that were not there before.

ss -tulpn
ss -tp state established

The first command shows what is listening, the second what is connected right now. Read the process column. What raises questions: an unfamiliar process listening on 0.0.0.0; outbound connections to high ports at addresses your application has no business with; and above all a process started from /tmp or /dev/shm — nothing legitimate runs out of those directories.

Look at the load while you are there. A miner gives itself away by keeping the processor busy on a site that enjoys no popularity whatsoever.

If the signs are there

The first impulse is to clean up quickly: delete the foreign key, kill the process, remove the file. Do not — you would be destroying the very thing that could later explain how they got in. And if that stays unexplained, they will be back, possibly tomorrow.

An order that preserves both the data and the picture:

  1. Take a disk snapshot at your provider if that is available. It is the one step that cannot be repeated later.
  2. Cut the machine off the network, or close everything except your own IP — but do not power it off. Shutting down loses the process list and the open connections, and that is half the evidence.
  3. Copy the logs off the machine: /var/log/auth.log, the web server logs, the output of the three commands above.
  4. Only now work out how they got in.

A clean reinstall is the right ending if access was obtained with root privileges. No amount of cleaning guarantees that nothing was left behind. But reinstalling without understanding the cause is pointless: you will put the same hole back on a fresh system.

How to keep the question from arriving out of the blue

Everything above is a one-off manual check, and it answers the question "what is happening right now". The trouble is that the question tends to be asked late: once the provider has written or the site has gone down.

Each of these checks exists as a separate tool that can watch continuously: failed logins — fail2ban, file changes — AIDE, package integrity — debsums, open ports — a regular snapshot of ss. Installing them one by one is not hard; what is hard is getting into the habit of logging in daily to read six different outputs, which is why in practice nobody reads them.

That is exactly the point of a panel that collects them: the same data, but on one page and with history, so that "this is not how it was yesterday" is visible without a special trip to find out. Below are the demo pages showing what that looks like assembled.