Getting access is half the job; the other half is not losing it. That is why nearly all automated malicious software first of all arranges to be started again: after a reboot, after the file is deleted, after the password is changed. Hence the familiar story — "we cleaned it up and two days later it came back".

There are not that many places where this is arranged, and all of them can be checked in a few minutes. Below is the full round in order.

1. SSH keys

The simplest way back: a line in authorized_keys survives a password change, a system update and a reboot.

sudo find /home /root -name authorized_keys -exec ls -l {} \; -exec cat {} \;

Every line is somebody's permanent access. If you cannot say whose, treat it as somebody else's. Note the comment at the end of the line as well: it is arbitrary text, and matching your own name proves nothing.

2. Cron jobs of every user

It is not only your own crontab that needs checking:

for u in $(cut -f1 -d: /etc/passwd); do echo "== $u"; sudo crontab -u "$u" -l 2>/dev/null; done
sudo ls -la /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/
sudo cat /etc/crontab

Signs worth looking for: @reboot lines, long encoded commands, calls to curl or wget piped into a shell, anything started from /tmp, /dev/shm or /var/tmp. Nothing standard runs out of those directories.

3. systemd timers and services

The modern equivalent of cron, and checked noticeably less often:

systemctl list-timers --all
systemctl list-units --type=service --state=running
sudo ls -la /etc/systemd/system/ /run/systemd/system/

Separately, user services, which run without root privileges and do not appear in the general list:

systemctl --user list-units --type=service
sudo ls -la /home/*/.config/systemd/user/

One more detail: lingering enabled for a user (loginctl enable-linger) lets their services run with no active session. Check it with loginctl list-users.

4. Shell start-up files

Code appended to the end of a shell start-up file runs at every login:

sudo tail -5 /root/.bashrc /root/.profile /home/*/.bashrc /home/*/.profile
sudo ls -la /etc/profile.d/

Look at the end of the file specifically — that is where things are appended so as not to stand out during a quick glance.

5. ld.so.preload

A file that makes the system load a given library into every process started. It is never created under normal circumstances:

ls -l /etc/ld.so.preload

Its presence is practically an unambiguous sign of serious compromise, and such a library usually hides files, processes and network connections alike. Nothing you see on that machine afterwards deserves trust.

6. Package manager hooks

A method rarely remembered: apt can run commands before and after every package operation.

sudo ls -la /etc/apt/apt.conf.d/
sudo grep -r 'DPkg::Pre-Invoke\|DPkg::Post-Invoke\|APT::Update' /etc/apt/apt.conf.d/

Such a hook fires at every update installation — that is, regularly and as root.

7. The message of the day on Ubuntu

The directory /etc/update-motd.d/ holds executable scripts that run at every SSH login and compose the greeting text. The place is convenient precisely because it looks like part of the system:

sudo ls -la /etc/update-motd.d/

8. Deferred at jobs

sudo atq
sudo ls -la /var/spool/cron/atjobs/ 2>/dev/null

An old and rarely used mechanism, and therefore the one checked least of all.

What to do if you find something

The first impulse is to delete the find immediately. That is a mistake: the information about how it got there disappears with it, and without an answer to that question the whole thing will happen again.

  1. Keep a copy of the file or job and its modification time.
  2. Using that time, look into the web server logs and auth.log for what was happening in the same minute. That is usually where the entry point is.
  3. Check all eight places from the list, not only the one where something turned up. A backdoor is almost never left in a single copy.
  4. Only then clean up and close the vulnerability itself.

Knowing what normal looks like

The main difficulty in this check is not the commands but the fact that an unfamiliar line in a job list does not look suspicious if you do not remember what the list used to be. On a server configured by somebody else, or configured a year ago, telling yours from theirs is nearly impossible.

Hence the practical conclusion: taking an "as it is" snapshot makes sense today, while the server is in order. The list of cron jobs, timers, keys and services on a healthy machine is the reference everything is later compared against. Collected and kept with history, it turns the hunt for a backdoor from a multi-hour exercise into a comparison of two lists. What that looks like is on the demo pages below.