Level 2 — Intermediate: troubleshooting · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Units
systemctl cat nginx | Show the unit file(s) actually in use |
sudo systemctl edit nginx | Create a drop-in override (survives package upgrades) |
sudo systemctl daemon-reload | Reload unit files after changing them |
systemctl list-dependencies multi-user.target | What starts at boot |
systemctl --failed | Everything that failed |
Journal
journalctl -u nginx -f | Follow one service's logs |
journalctl -u nginx --since '1 hour ago' -p warning | Warnings and worse, last hour |
journalctl -b -1 -p err | Errors from the previous boot |
journalctl -k | Kernel messages (like dmesg) |
journalctl --disk-usage / sudo journalctl --vacuum-size=500M | Journal size / shrink it |
Boot
systemd-analyze blame | head | Slowest units at boot |
systemd-analyze critical-chain | The chain that delayed boot |
Where did the space go?
df -h | Free space per mounted filesystem |
df -i | Free inodes (file slots) per filesystem |
sudo du -xh --max-depth=1 / | sort -rh | head | Biggest top-level directories on the root filesystem |
sudo lsof +L1 | Deleted files still held open (space not yet freed) |
sudo find / -xdev -size +1G -type f 2>/dev/null | Files over 1 GB on this filesystem |
Disks & mounts
lsblk -f | Disks, partitions, filesystems, mount points |
findmnt /var/lib | Which filesystem a path lives on |
sudo mount -a | Mount everything in /etc/fstab (test after editing it!) |
LVM grow
sudo pvs; sudo vgs; sudo lvs | Physical volumes, volume groups, logical volumes |
sudo lvextend -r -L +10G /dev/vg0/data | Grow a logical volume AND its filesystem |
Interfaces & routes
ip -br addr | Interfaces and IPs, one line each |
ip route | Routing table (look for 'default via') |
ip route get 8.8.8.8 | Which interface and gateway a destination would use |
ip -s link show eth0 | Packet and error counters |
Sockets & connectivity
sudo ss -ltnp | Listening TCP ports and their processes |
ss -tn state established | Current TCP connections |
nc -vz db.internal 5432 | Can I open a TCP connection to this port? |
curl -v https://api.example.com/health | Full HTTP/TLS exchange |
DNS & packets
dig +short api.example.com | Resolve a name |
resolvectl status | Which DNS servers this host uses (systemd-resolved) |
sudo tcpdump -ni any port 5432 | Watch packets to/from port 5432 |
sudo nft list ruleset / sudo iptables -S | Firewall rules |
The 60-second checklist
uptime | Load averages: 1, 5, 15 minutes |
sudo dmesg -T | tail | Kernel errors: OOM kills, disk or network errors |
vmstat 1 5 | Run queue, memory, swap, I/O wait, context switches |
mpstat -P ALL 1 3 | Per-CPU usage: one hot CPU? |
pidstat 1 3 | Which processes use CPU |
iostat -xz 1 3 | Per-disk utilisation and latency (await) |
free -h | Memory: look at 'available' |
sar -n DEV 1 3 | Network throughput per interface |
top (or htop) | Everything live |
Install the tools
sudo apt install -y sysstat / sudo dnf install -y sysstat | Provides iostat, mpstat, pidstat, sar |
Keys
ssh-keygen -t ed25519 -C 'asha@laptop' | Create a key pair (protect it with a passphrase) |
ssh-copy-id asha@web-01 | Install your public key on a server |
ssh-add ~/.ssh/id_ed25519 | Load the key into the agent (type the passphrase once) |
ssh -v asha@web-01 | Verbose: see why a login fails |
Getting around
ssh -J bastion.example.com asha@10.0.5.20 | Jump through a bastion host |
ssh -L 8080:localhost:80 asha@web-01 | Local port forward: laptop:8080 → web-01:80 |
rsync -avz --progress ./site/ asha@web-01:/srv/site/ | Sync a directory (only changed files) |
scp backup.tar.gz asha@web-01:/tmp/ | Copy one file |
tmux
tmux new -s work | Start a named session |
Ctrl+b d | Detach (the session keeps running) |
tmux attach -t work | Re-attach later |
Ctrl+b % / Ctrl+b " | Split pane vertically / horizontally |
Your first five commands on any sick server
uptime; free -h; df -h; df -i | Load, memory, space and inodes at a glance |
systemctl --failed | Failed services |
journalctl -p err -b --no-pager | tail -30 | Recent errors this boot |
sudo ss -ltnp | What's listening |
sudo dmesg -T | tail -20 | Kernel complaints (OOM, disk, network) |
