Linux — Frequently Asked Questions
Practical answers for running Linux servers — which distribution, SSH hardening, automatic patching, diagnosing a full disk or high load, systemd services, swap, and what to check first when a server misbehaves.
Choosing and building#
Which distribution should we use?#
The one your team already knows, provided it has a long support window and a security update stream you can rely on. The differences between mainstream server distributions matter far less than familiarity and the support end date.
Check that date before building anything. Migrating a host off an end-of-life distribution is always more work than choosing a longer-supported one at the start, and it always becomes urgent at an inconvenient moment.
Should servers be built by hand?#
No, or at least not twice. Hand-built hosts cannot be reproduced, and the knowledge of how they were configured leaves with the person who built them.
Use an image, a configuration tool or a script — even a modest shell script kept in version control is a large improvement over memory. The test is simple: if this host were lost, could you rebuild it without it?
How much should we harden?#
The high-value items are short: disable SSH password authentication, disable direct root login, firewall everything not deliberately exposed, apply security updates automatically, ship logs off-host, and remove services nobody uses.
That list removes most realistic risk. Extensive benchmark hardening beyond it has diminishing returns for most organisations and a real cost in things that mysteriously stop working.
Access#
What is the right SSH configuration?#
Keys only, no password authentication, no direct root login, and access restricted by source where possible — a bastion, a VPN, or an allowlist. Exposing SSH to the whole internet is survivable with keys, but it fills your logs with attempts and removes a layer for free.
Record who holds keys, as named individuals. "The team has access" is not an access record, and it makes removing access on departure unreliable.
What happens if we lose SSH access?#
This is worth answering before it happens. A second key held by a different person on a different machine costs nothing today and is impossible to arrange on the day you need it. If the host is a cloud instance, know where the serial or console access is and confirm you can actually use it.
The failure to avoid: one key, one laptop, one person.
Do we need fail2ban or similar?#
It reduces log noise and blocks the least sophisticated attempts. With password authentication already disabled it is a convenience rather than a control. Useful, not foundational — and it introduces its own way to lock yourself out, so allowlist your own addresses.
Patching and updates#
Should security updates be automatic?#
For most servers, yes, with a defined reboot window for kernel updates. The objection is that an update might break something; the alternative is running known-vulnerable, internet-facing packages for months because patching is somebody's spare-time task.
If a host is too sensitive for automatic updates, that is a legitimate position — but then patching needs a named owner and a schedule, not an intention.
Do we have to reboot?#
For kernel and core library updates, yes, unless you are using live patching. A host that has not rebooted in a year has both unapplied fixes and an untested boot path.
Rebooting deliberately, on a schedule, is how you find out that a service was started by hand and never enabled at boot — a discovery best made at 10:00 on a Tuesday rather than during a power event.
Diagnosing problems#
The server is slow. What do I check first?#
Establish which resource is exhausted before changing anything. Load average and CPU, memory and swap activity, disk I/O wait, and disk space — in that order, and quickly.
High load with low CPU usually means processes are blocked on I/O rather than computing. That distinction sends you to entirely different causes, and it is the first fork in almost every performance investigation.
The disk is full but I cannot find the files. Why?#
Usually one of three things. A deleted file still held open by a running process still occupies space until that process is restarted — that is the classic case, often a log file rotated incorrectly. Or the space is under a mount point, hidden by something mounted over it. Or you have run out of inodes rather than bytes, typically from millions of small files.
Check for deleted-but-open files, check inode usage, and check what is mounted where.
A process was killed and there is nothing in the application log. What happened?#
Very likely the kernel's out-of-memory killer. It terminates a process abruptly when the system runs out of memory, so the application never gets a chance to log anything — it simply ceases.
The kernel log records it. The process killed is often not the one that caused the problem, which is why the incident can look unrelated to the recent change.
Should we have swap?#
A modest amount is usually helpful: it gives the kernel somewhere to put pages that are genuinely idle, which is not the same as running your workload from disk. Zero swap makes memory pressure fail more abruptly.
What you should not do is rely on swap to run a workload that does not fit in memory. The symptom is a machine that responds to nothing while technically remaining up, which is harder to recover from than a clean failure.
Services and operations#
How should a service be run?#
Under the init system, as a unit with a dedicated non-root user, restart policy, resource limits, and logging to the journal or stdout. Not from a shell script started in a terminal, and not from a session that dies when someone disconnects.
The test: reboot the host and confirm everything comes back without human intervention. If it does not, that is a defect, not a quirk.
Where should logs go?#
Off the host, centrally, with local rotation configured so a runaway log cannot fill the disk. Logs that exist only locally are lost in exactly the incidents where you most need them — compromise, disk failure, or an instance that was replaced.
What should be monitored on a Linux host?#
Disk space (the most common preventable outage), memory pressure, load, and whether the host is reachable at all. Then the specific services it runs, checked by doing something real rather than by confirming the process exists.
A process can be running and completely stuck. "The service is up" and "the service is working" are different questions, and only the second one matters to a user.