Checkpoint: five broken servers · Prasanta Kumar Behera

Five realistic broken servers, each presented as symptoms and evidence. Diagnose each one from what you see, as you would on call, then compare with the worked solution.

How to use this lesson

Each case gives you the symptom a user reports and the evidence you'd collect. Before opening the solution, write down:

Which layer is failing (from what you've learned in Levels 1–2).
The one command that proves it.
The fix, and how to prevent it next time.

This is like a detective game. Each case has clues. You don't get points for guessing. You get points for pointing at the clue that proves your answer.

Case 1: "The API returns 502 after the deploy"

Evidence:

$ systemctl status orders-api
× orders-api.service - Orders API
     Active: failed (Result: exit-code) since Sat 2026-09-27 11:02:44 UTC
$ journalctl -u orders-api -n 5 --no-pager
… orders-api[4410]: Error: listen EADDRINUSE: address already in use 0.0.0.0:8080
… systemd[1]: orders-api.service: Main process exited, code=exited, status=1/FAILURE

Solution

Layer: port. Something already holds port 8080.

$ sudo ss -ltnp | grep ':8080 '
LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=3120,fd=21))
$ ps -o pid,lstart,cmd -p 3120
  PID                  STARTED CMD
 3120 Fri Sep 26 09:14:10 2026 node /srv/orders-api/old/server.js

An old copy, started by hand yesterday outside systemd, is still running. Stop it (sudo kill 3120), then sudo systemctl start orders-api.

Prevention: run services only through systemd, and make the deploy script check the port before starting.

Case 2: "Disk full, but nothing big on it"

Evidence:

$ df -h /
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vg0-root     40G   40G     0 100% /
$ sudo du -xsh /
14G /

Solution

Layer: filesystem. df ≫ du means deleted but still open files.

$ sudo lsof +L1
COMMAND  PID   USER FD TYPE DEVICE    SIZE/OFF NLINK NODE NAME
java    2211 appsvc 5w  REG  253,0 27917287424     0 1311 /var/log/app/app.log (deleted)

Someone ran rm on a 26 GB live log. Restart the service (or truncate via /proc/2211/fd/5 as an emergency measure) to release the space.

Prevention: logrotate with copytruncate (or a reopen signal) for this app, and alerts at 80% disk.

Case 3: "Everything is slow since noon"

Evidence:

$ uptime
 12:18:40 up 40 days,  load average: 31.40, 28.02, 12.77
$ nproc
8
$ vmstat 1 3
 r  b   swpd   free   buff  cache   si   so    bi     bo   in   cs us sy id wa st
 2 29      0 412300  81220 9120440    0    0  1200 188000 2100 3300  4  3 12 81  0

Solution

Layer: disk. Look at b = 29 (processes blocked on I/O) and wa = 81%: the CPUs are mostly waiting. The high load comes from I/O wait, not computation.

$ iostat -xz 1 2 | grep -E 'Device|sd|nvme'
Device  r/s    w/s   wkB/s  w_await  aqu-sz  %util
sda     8.0  1450.0 188000.0  210.30   305.10 100.00
$ sudo pidstat -d 1 1
UID  PID   kB_rd/s   kB_wr/s  Command
0   9981      0.00 185000.00  tar

A backup (tar) moved from 02:00 to 12:00 is saturating the disk. Stop or ionice -c3 -p 9981 it now; move the schedule back.

Prevention: run backups off-peak with ionice/nice, ideally from a snapshot, and alert on disk await.

Case 4: "The deploy user can't SSH in any more"

Evidence (from the client):

$ ssh deploy@web-02
deploy@web-02: Permission denied (publickey).

From the server's log:

$ sudo journalctl -u ssh -n 3 --no-pager
… sshd[7718]: Authentication refused: bad ownership or modes for directory /home/deploy

Solution

Layer: permissions. Someone ran chmod -R 777 /home/deploy "to fix a permission problem". sshd's StrictModes now ignores the keys because anyone could have edited them.

$ sudo chmod 755 /home/deploy
$ sudo chmod 700 /home/deploy/.ssh
$ sudo chmod 600 /home/deploy/.ssh/authorized_keys
$ sudo chown -R deploy:deploy /home/deploy/.ssh

Also review what else the recursive 777 exposed, and whether authorized_keys was modified while it was world-writable.

Prevention: never use 777 as a fix; manage user homes and keys with configuration management.

Case 5: "Nothing can resolve any hostname"

Evidence:

$ curl https://api.github.com
curl: (6) Could not resolve host: api.github.com
$ cat /etc/resolv.conf | grep nameserver
nameserver 127.0.0.53
$ ping -c1 1.1.1.1
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=4.1 ms

Solution

Layer: DNS. The network works (ping by IP succeeds). 127.0.0.53 is the systemd-resolved local stub.

$ systemctl is-active systemd-resolved
inactive
$ sudo systemctl enable --now systemd-resolved
$ resolvectl query api.github.com
api.github.com: 140.82.121.6

Someone disabled systemd-resolved while "cleaning up services".

Prevention: keep a list of essential services in configuration management, and a synthetic DNS check in monitoring.

Going deeper: turn cases into habits

Each case maps to one habit: 1 check ports and orphans, 2 df vs du, 3 load with idle CPU = I/O, 4 never 777, 5 test by IP before blaming DNS. Keep a personal runbook of cases like these. It's the most valuable document you'll own on call.

You've finished Linux Level 2

You can now troubleshoot services, disks, networks, performance and SSH from evidence. Level 3 goes under the hood: the kernel, cgroups and namespaces, NUMA and CPU pinning, eBPF tracing, SELinux and host hardening.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.