Real-world incident scenarios · wrap-up
Cheat sheet & self-check
39 questions across 13 lessons. Each answer links back to the lesson it came from.
Pick an answer to see if you got it, and why.
Q1. What does the etcd NOSPACE alarm mean?
Show answer
B. It's a safety limit inside etcd. A full filesystem is a different problem with a different fix, although they can happen together.
From lesson 07 · etcd out of space at 2 a.m.Q2. Why defragment members one at a time?
Show answer
B. Start with followers, finish with the leader, and check health between members.
From lesson 07 · etcd out of space at 2 a.m.Q3. After recovery, what's the most important follow-up?
Show answer
B. Compaction fixes tonight; finding the source and alerting on size prevents the next night.
From lesson 07 · etcd out of space at 2 a.m.Q4. A batch job without limits runs on the same node as the API. API latency spikes and some pods are evicted. Why?
Show answer
B. Without requests the scheduler doesn't reserve room; without limits the job can take everything.
From lesson 08 · The noisy neighbourQ5. Which pods are evicted first under node memory pressure?
Show answer
B. QoS classes and usage relative to requests decide eviction order.
From lesson 08 · The noisy neighbourQ6. A service shows high latency but low CPU usage. What might a CPU limit be doing?
Show answer
B. Check `container_cpu_cfs_throttled_periods_total`; many teams avoid tight CPU limits on latency-sensitive services.
From lesson 08 · The noisy neighbourQ7. After `kubeadm certs renew all`, kubectl still shows the expired certificate. Why?
Show answer
B. kubeadm's docs note the control-plane static pods need restarting after renewal. Also update any copied kubeconfigs.
From lesson 09 · Certificates expired after a failed rotationQ8. How long do kubeadm leaf certificates last by default, and what renews them routinely?
Show answer
B. Clusters left un-upgraded for a year are the classic expiry victims. CAs last 10 years by default.
From lesson 09 · Certificates expired after a failed rotationQ9. One worker is NotReady with x509 errors in the kubelet log, others are fine. Likely cause?
Show answer
B. Kubelet client certs rotate automatically when working; a node offline for long or with rotation disabled can miss it.
From lesson 09 · Certificates expired after a failed rotationQ10. Why can an edge server reach 172.17.20.1 (its gateway on eth0) but not 172.17.5.40 at the customer site?
Show answer
B. The /24 on eth0 covers only 172.17.20.x; the rest of 172.17.0.0/16 now points at docker0.
From lesson 10 · Docker network collides with the site networkQ11. How do you move Docker's default bridge network?
Show answer
B. Configure it permanently; deleting the interface by hand doesn't survive restarts.
From lesson 10 · Docker network collides with the site networkQ12. What prevents this class of problem across hundreds of sites?
Show answer
B. Collect site ranges up front and validate the plan in CI (see Cluster Design, lesson 06).
From lesson 10 · Docker network collides with the site networkQ13. Why might a node lose its static routes after a network restart?
Show answer
B. Anything not in netplan/NetworkManager/systemd-networkd config disappears on restart.
From lesson 11 · Node unreachable after a network restartQ14. After `ifdown -a` on an older-style config, etcd and the API server fail on the node even though eth0 is back up. What's a likely cause?
Show answer
B. If lo is defined in the interfaces file, 'all' includes it. Check `ip -br link` for lo.
From lesson 11 · Node unreachable after a network restartQ15. What's the safest way to change networking on a remote node?
Show answer
B. Plan for losing the connection you're using to make the change.
From lesson 11 · Node unreachable after a network restartQ16. Pods on a rebooted node fail with 'MountVolume.MountDevice failed … already mounted or mount point busy' for Longhorn volumes. A known cause?
Show answer
B. multipath is enabled by default on some distributions. Check `multipath -ll` and Longhorn's knowledge base.
From lesson 12 · Longhorn volumes won't attach after a rebootQ17. Why does Longhorn need `iscsid` on every node?
Show answer
B. If iscsid isn't enabled at boot, volumes can't attach after a reboot.
From lesson 12 · Longhorn volumes won't attach after a rebootQ18. A pod can't start because its RWO volume is 'already attached' to a node that is powered off. What's the safe approach?
Show answer
B. Two writers on one block device corrupts data. Verify before forcing.
From lesson 12 · Longhorn volumes won't attach after a rebootQ19. A freshly issued certificate is rejected with 'x509: certificate has expired or is not yet valid'. The CA and cert look fine. What should you check?
Show answer
B. Time drift masquerades as certificate problems.
From lesson 13 · Clock drift: NTP sync failedQ20. Why is a big, sudden clock step risky on a running cluster?
Show answer
B. chrony slews small offsets gradually; large steps deserve a controlled approach.
From lesson 13 · Clock drift: NTP sync failedQ21. What makes time sync robust at edge sites?
Show answer
B. Edge sites often lose internet; time must keep working locally.
From lesson 13 · Clock drift: NTP sync failedQ22. Why can one lookup of 'api.example.com' from a pod generate several DNS queries?
Show answer
B. Search-path expansion can turn one lookup into many. Lower ndots for such workloads, or use FQDNs with a trailing dot.
From lesson 14 · Cluster-wide DNS timeoutsQ23. Intermittent 5-second DNS delays that disappear with NodeLocal DNSCache often point to what?
Show answer
B. The 5-second delay is the resolver's default timeout before retrying.
From lesson 14 · Cluster-wide DNS timeoutsQ24. CoreDNS logs show timeouts to the upstream resolvers. What's the likely scope?
Show answer
B. Split the problem: kubernetes.default works? Then look upstream.
From lesson 14 · Cluster-wide DNS timeoutsQ25. A webhook with failurePolicy: Fail is unreachable. What happens to matching API requests?
Show answer
B. Fail-closed is safer for security policies but can block the cluster if the webhook isn't highly available.
From lesson 15 · A broken admission webhook blocks every deployQ26. The webhook's own pods can't start because pod creation is blocked by that same webhook. How do you avoid this deadlock by design?
Show answer
B. Webhooks must never gate the resources needed to recover themselves.
From lesson 15 · A broken admission webhook blocks every deployQ27. Why record every manual change to a webhook configuration during an incident?
Show answer
B. Emergency changes are debts; write them down and pay them back.
From lesson 15 · A broken admission webhook blocks every deployQ28. What does the DiskPressure node condition cause?
Show answer
B. Eviction thresholds (e.g. nodefs.available) trigger reclaim; know your thresholds.
From lesson 16 · Node disk full: images and logsQ29. Which is a safe first cleanup step on a full node?
Show answer
B. Never delete runtime directories by hand; use the runtime's tools.
From lesson 16 · Node disk full: images and logsQ30. One pod writes huge temporary files to emptyDir and fills the node. What prevents a repeat?
Show answer
B. Local disk is a resource like CPU and memory; give it requests and limits.
From lesson 16 · Node disk full: images and logsQ31. Why can EKS run out of pod IPs when nodes have plenty of CPU and memory?
Show answer
B. Pod density on EKS is bounded by IP address capacity, not just compute.
From lesson 17 · EKS: pods can't get IP addressesQ32. What does prefix delegation change?
Show answer
B. More pods per node, but plan subnet space: prefixes need free /28 blocks.
From lesson 17 · EKS: pods can't get IP addressesQ33. Warm-pool settings make every node pre-allocate many IPs. What's the effect in a small subnet?
Show answer
B. Defaults favour fast pod start; small subnets need tighter warm pools.
From lesson 17 · EKS: pods can't get IP addressesQ34. Why can a broken aws-auth ConfigMap make nodes NotReady as well as locking out humans?
Show answer
B. aws-auth (legacy) carries both human and node identity mappings; one YAML error affects both.
From lesson 18 · EKS: locked out after editing aws-authQ35. What do EKS access entries improve?
Show answer
B. Access entries plus access policies (or RBAC groups) are the recommended way forward.
From lesson 18 · EKS: locked out after editing aws-authQ36. During the lockout, what's the first thing to find?
Show answer
B. Recovery depends on a working identity. Know it before you need it (break-glass).
From lesson 18 · EKS: locked out after editing aws-authQ37. Why do 502s appear when old pods terminate during a rollout?
Show answer
B. Termination and deregistration happen in parallel; a preStop delay gives the ALB time to stop routing.
From lesson 19 · AWS: ALB 502s during every deploymentQ38. What do pod readiness gates from the AWS Load Balancer Controller add?
Show answer
B. Without them, the rollout can outrun the load balancer's health checks.
From lesson 19 · AWS: ALB 502s during every deploymentQ39. How do you tell whether a 502 came from the ALB or from your application?
Show answer
B. Separating ALB-generated errors from app errors avoids chasing the wrong layer.
From lesson 19 · AWS: ALB 502s during every deployment