Lesson 18 of 19 · Level 3 — Advanced: kernel & security
Host hardening
Reduce a server's attack surface step by step: fewer services and packages, hardened SSH, default-deny firewall, automatic security updates, audit logging, kernel settings, and CIS benchmarks to measure the result.
Hardening is subtraction
A default installation is built to work for everyone. A hardened server is built to do one job and nothing else. Most of hardening is removing, disabling and restricting.
Protecting a house isn't one big lock. You close windows you never open (unused services), put a strong lock on the front door (SSH), build a fence with one gate (firewall), fix broken locks quickly (updates), fit a camera on the safe (audit logs), and ask an inspector to check it all against a list (CIS benchmark).
1. Shrink the attack surface
$ sudo ss -ltnup # what is listening, and on which address?
$ systemctl list-unit-files --state=enabled --type=service
$ sudo apt purge -y telnetd rsh-server 2>/dev/null # remove what you don't need
$ sudo systemctl disable --now cups avahi-daemon # desktop services on a server? disable
Everything listening on 0.0.0.0 must have a reason. Databases and admin UIs should listen on 127.0.0.1 or a private interface.
2. SSH
In /etc/ssh/sshd_config (or a drop-in in /etc/ssh/sshd_config.d/):
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowGroups ssh-users
MaxAuthTries 3
X11Forwarding no
$ sudo sshd -t && sudo systemctl reload ssh # the unit is 'sshd' on the RHEL family
Keep your current session open and test a new login before logging out.
3. Firewall: default deny
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow OpenSSH
$ sudo ufw allow 443/tcp
$ sudo ufw enable
$ sudo ufw status verbose
On the RHEL family, firewalld uses zones: sudo firewall-cmd --permanent --add-service=https && sudo firewall-cmd --reload. In clouds, security groups are a second, independent layer. Use both.
4. Patch automatically, reboot deliberately
- Ubuntu/Debian:
unattended-upgradesinstalls security updates daily (dpkg-reconfigure -plow unattended-upgrades). - RHEL family:
dnf-automaticwithapply_updates = yes(andupgrade_type = securityif you only want security updates) in/etc/dnf/automatic.conf, then enablednf-automatic.timer. - Kernel updates need a reboot to take effect: check
/var/run/reboot-required(Ubuntu) orneeds-restarting -r(RHEL family), and schedule rolling reboots.
5. Audit: who changed what?
auditd records security-relevant events. Rules in /etc/audit/rules.d/hardening.rules:
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd
$ sudo augenrules --load
$ sudo ausearch -k identity -ts today
Ship audit and auth logs to a central system: logs that stay only on a compromised host can be erased.
6. Kernel settings
# /etc/sysctl.d/60-hardening.conf
kernel.kptr_restrict = 2 # hide kernel addresses
kernel.dmesg_restrict = 1 # only root reads the kernel log
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.log_martians = 1
Check against the host's role
Some hardening settings conflict with routing roles. Kubernetes nodes need net.ipv4.ip_forward = 1, and some CNIs need reverse-path filtering in loose mode (rp_filter = 2) or disabled on their interfaces. Apply hardening profiles per role, and test.
7. Measure it: CIS benchmarks
The CIS Benchmarks are detailed, community-agreed hardening checklists per distribution (Level 1 = broadly safe, Level 2 = stricter). Scan instead of guessing:
- OpenSCAP (
oscap) with the SCAP Security Guide content, available on most distributions; it produces an HTML report of passed and failed rules. - Ubuntu Security Guide (
usg) for Ubuntu Pro systems. - For Kubernetes nodes, kube-bench checks the node and control-plane settings (see Kubernetes Security & Hardening).
Treat the report as a to-do list: fix, document exceptions with a reason, re-scan.
Try it: harden a fresh VM, then prove it
- Record the "before":
ss -ltnup, enabled services, setuid files, and (if available) an OpenSCAP scan score. - Apply the SSH settings (keeping a session open), default-deny firewall, automatic updates, audit rules and sysctls above.
- Try an SSH password login from another machine (should fail); edit
/etc/sudoers.d/testand find the event withausearch -k sudoers. - Re-scan and compare. Write down each remaining failure as "fix" or "accepted, because…".
Going deeper: hardening at fleet scale
- Bake hardening into golden images (Packer or image pipelines) and enforce it continuously with configuration management. Hand-hardened servers drift.
- Immutable OS distributions (read-only root, transactional updates, e.g. SUSE SLE Micro, Fedora CoreOS, Bottlerocket) remove whole classes of tampering and drift, which is why they're popular for Kubernetes and edge nodes.
- Add file integrity monitoring (AIDE) and time sync (chrony). Accurate timestamps matter for audit trails and TLS.
- Hardening supports security; it doesn't replace least privilege, network segmentation, secrets management and monitoring.
Recap
- Hardening = subtraction: fewer packages, services and open ports.
- SSH keys only, no root login, restricted groups; default-deny firewall; automatic security updates and planned reboots.
- auditd watches for sensitive files; central logging; sysctl hardening checked against the host's role.
- Measure with CIS benchmarks (OpenSCAP, usg, kube-bench) and bake it into images.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.