Lesson 12 of 18 · Automate & Recover
Capstone: own, harden & operate
Put the whole course together: build a production-ready EKS platform from Terraform, harden it, prove it with checks and drills, and hand it over with runbooks, SLOs and a cost report.
The brief
Deliver an EKS platform for a team called Shop that another team could take over tomorrow. Everything is code, everything is proven, and everything is documented.
You've built a whole theme park, one ride at a time. Now you hand the keys to new managers. They get the map, the safety manual, the fire-drill plan, the opening checklist, the budget, and proof that every ride passed inspection. If they can run it without phoning you, you've done it right.
Part 1: build (from this course)
| Area | Deliverable | Lesson |
|---|---|---|
| Structure | Layers network/, cluster/, platform/ with remote state and locking |
02, 03 |
| Networking | VPC across 3 AZs, endpoints, prefix delegation and/or 100.64 pod CIDR | 04 |
| Cluster | Private endpoint, control-plane logs, pinned add-ons | 01, 09 |
| Identity | SSO roles via access entries; Pod Identity per app; IMDSv2 hop limit 1 | 05 |
| Capacity | System node group + Karpenter NodePools (Spot + on-demand), limits | 06 |
| Storage | gp3 default, gp3-fast, EFS for shared files, snapshot class | 07 |
| Edge | LBC with ALB IngressGroup + ACM + WAF, or NLB + gateway | 08 |
| Delivery | PR plans, approved applies via OIDC, drift job | 10 |
| Security | Controls from Kubernetes Security (PSA, policies, secrets, audit) | — |
| Recovery | Velero to cross-region S3, secrets replication, rebuild runbook | 11 |
Part 2: prove it
Collect evidence for each claim:
- Rebuild: destroy and recreate the platform layer from code; time it.
- Upgrade: upgrade the cluster by one minor version with a synthetic check running; record errors (target: none).
- Scale: scale a test workload ×10; Karpenter provisions; consolidation removes nodes afterwards.
- Spot interruption: simulate one (e.g. with AWS Fault Injection Service) and confirm pods move gracefully.
- Identity: a developer role sees only its namespace; an app pod reads only its own S3 prefix.
- Restore: Velero restores the Shop namespace with data into a new namespace.
- Drift: a manual console change is detected by the nightly job.
Part 3: hand over
A short operations handbook:
- Architecture diagram and the ADRs behind major choices (Karpenter vs node groups, CNI mode, edge pattern).
- Access model: who has what, how to request access, break-glass.
- Runbooks: node NotReady, pods Pending (IP exhaustion, capacity), ALB 5xx, failed upgrade step, state lock stuck, restore from backup, region failover.
- SLOs: API availability, deploy success (synthetic probe), ingress availability; the dashboards and alerts that measure them.
- Upgrade calendar: next versions, owners, and the procedure (lesson 09).
- Cost report: monthly cost by component (control plane, nodes by capacity type, NAT and data transfer, load balancers, storage), and the top three savings opportunities.
Reviewer's checklist
- A fresh engineer can create a dev copy of the platform from the README alone.
- No long-lived AWS keys anywhere (CI via OIDC, people via SSO, pods via Pod Identity).
- Control plane endpoint is private or tightly restricted; IMDSv2 enforced on nodes.
- Every NodePool has limits; Spot is used only where interruption is acceptable.
- Upgrade and restore have both been done at least once, with timings recorded.
- Alerts reach an on-call rota, and each alert links to a runbook.
- The cost report names owners for each major cost line.
You've finished Amazon EKS in Production
You can design, build, secure, scale, upgrade, recover and hand over an EKS platform, all as code. Next: GitOps with Argo CD to manage everything inside the cluster declaratively, or Observability with OpenTelemetry to see it all running.
Keep going with the Playbooks, challenges & practice section: run-in-order playbooks for creating and operating this stack, the laptop-to-app flow, real-world challenges, recovery runbooks, and a $0 simulator.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.