Prasanta Kumar Behera
Linux — Level by Level›19 · Capstone: harden & tune a host
Learning Hub / Linux & Scripting / Linux — Level by Level

Lesson 19 of 19 · Level 3 — Advanced: kernel & security

Capstone: harden & tune a host

Combine all three Linux levels: take a fresh server, baseline it, harden it, tune it for a web workload, prove every change with before/after evidence, and hand it over with a runbook.

Advanced
Key wordscapstonebaselinehardeningtuningevidencerunbook

The brief

You've been given a fresh Ubuntu (or Rocky/Alma) VM that will serve a public website with nginx. Your job:

  1. Baseline it: record its state and performance.
  2. Harden it (lesson 18) without breaking the website.
  3. Tune it for serving many concurrent HTTP requests.
  4. Prove every change with before/after evidence.
  5. Hand over a one-page runbook.

It's like getting a new bicycle ready for a race. First you write down how fast it goes today. Then you fit the lock and lights (security), pump the tyres and oil the chain (tuning), and time it again. The notes you hand to the next rider (runbook) say what you changed, why, and what to check if something feels wrong.

Part 1: baseline

$ mkdir -p ~/capstone/before && cd ~/capstone/before
$ sudo ss -ltnup > listen.txt
$ systemctl list-unit-files --state=enabled > enabled.txt
$ sudo find / -xdev -perm -4000 -type f 2>/dev/null > setuid.txt
$ sudo sysctl -a 2>/dev/null > sysctl.txt

Install nginx, then load-test it from another machine (so the load generator doesn't compete for the server's CPU):

$ wrk -t4 -c200 -d60s --latency http://<server>/
Running 1m test @ http://<server>/
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max
    Latency    …
  Latency Distribution
     50%   …
     99%   …
Requests/sec: …

While it runs, record vmstat 1, mpstat -P ALL 1 and iostat -xz 1 on the server. Save everything.

Part 2: harden (Level 3, lesson 18)

Apply, one at a time, testing the website after each:

  • Remove or disable unneeded packages and services.
  • SSH: keys only, no root, AllowGroups.
  • Default-deny firewall allowing only SSH and HTTP/HTTPS.
  • Automatic security updates.
  • auditd rules for identity, sudoers and sshd config.
  • Hardening sysctls appropriate for a web server.
  • SELinux Enforcing (RHEL family) or AppArmor profiles active (Ubuntu), with no denials for nginx.

Part 3: tune for many connections

Candidate changes. Make one at a time, re-run the same load test, and keep only what helps:

Area Change Why it might help
nginx worker_processes auto; and a higher worker_connections Use all CPUs; more concurrent connections
nginx keepalive_timeout, sendfile on;, gzip for text Fewer new connections; less work per request
File descriptors LimitNOFILE=65536 via systemctl edit nginx Each connection uses a file descriptor
Kernel net.core.somaxconn, net.ipv4.tcp_max_syn_backlog Deeper accept queues under bursts
Kernel net.ipv4.ip_local_port_range (on proxies making many outbound connections) More ephemeral ports

Tuning isn't a list of magic numbers

Many "Linux tuning guides" are outdated or aimed at other workloads. If a change doesn't move your measured numbers, revert it. Fewer settings are easier to operate.

Part 4: prove it

For every change, one line in a table:

# Change Before (req/s, p99) After (req/s, p99) Keep?
1 worker_processes auto … … ✅ / ❌

Also diff the security evidence:

$ diff ~/capstone/before/listen.txt ~/capstone/after/listen.txt
$ diff ~/capstone/before/enabled.txt ~/capstone/after/enabled.txt

Part 5: the runbook

One page, for the next person on call:

  • What the server does and who owns it.
  • What we changed, and why (link the evidence table).
  • How to check it's healthy: the three commands you'd run first.
  • Known exceptions: hardening rules deliberately not applied, and why.
  • How to roll back each change.
Reviewer's checklist
  • Nothing unexpected listens on 0.0.0.0.
  • SSH password login is refused; root login is refused.
  • Firewall default is deny, with only the needed ports open.
  • Updates apply automatically; there's a reboot plan.
  • Audit events appear for sudoers and identity changes.
  • MAC is enforcing, with no denials for nginx.
  • Every tuning change has before/after numbers, and unhelpful changes were reverted.
  • The runbook lets someone new check health and roll back.

You've finished Linux: Level by Level

From your first whoami to kernel parameters, cgroups, eBPF, SELinux and a hardened, measured server. This is the foundation under every other course here. Next: Bash Scripting and Python for Infrastructure to automate it, then Kubernetes Administration.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.