Capstone: hardened cluster from scratch

The brief

Build a small production-like cluster (kubeadm on 3 VMs, or kind with Calico for a lighter version) that hosts two tenants, shop and payments, and harden it using every lesson in this course. Then attack it and prove what each control stops.

Part 1: build (checklist)

Area	Control	Lesson
PKI	Certificates checked; expiry monitored; break-glass cert stored separately	01, 05
Certificates	cert-manager with a private CA (and ACME if public)	02
Identity	OIDC login for humans; no shared admin kubeconfig	03, 04
RBAC	Personas → IdP groups; bindings in Git; default ServiceAccounts powerless	06
Edge	Gateway API (or Ingress) with TLS from cert-manager	07, 08
Network	Default-deny + DNS + explicit flows in both tenant namespaces	09
Workloads	PSA restricted enforced; image-tag and required-label policies	10
Secrets	Encryption at rest; secrets via External Secrets or Sealed Secrets	11
Supply chain	Only signed images from your registry admitted	12
Detection	Audit policy on; Falco installed; alerts routed	13
Baseline	kube-bench run; FAILs fixed or documented	14

Part 2: attack (and record)

For each attempt, record: what you tried → expected result → actual result → which control acted → evidence (log line, event, error).

1. Steal the default token: from a shop pod, use its ServiceAccount token to list Secrets. Expected: 403 (RBAC).
2. Run a privileged pod: kubectl run pwn --image=busybox:1.36 --privileged -n shop -- sleep 1d. Expected: rejected (PSA).
3. Mount the host: a pod with a hostPath of /. Expected: rejected (PSA baseline/restricted).
4. Lateral movement: from shop/web, call payments/api. Expected: timeout (NetworkPolicy).
5. Exfiltrate: from shop/web, wget an external site. Expected: blocked by egress policy (if designed so).
6. Unsigned image: deploy an image you pushed without signing. Expected: rejected (verifyImages).
7. Shell in a container: kubectl exec into a pod and read /etc/shadow. Expected: allowed by RBAC for on-call, but detected (Falco alert + audit log entry).
8. Read secrets from etcd: on the control plane, read a Secret directly from etcd. Expected: ciphertext (encryption at rest).
9. Old credentials: try a revoked OIDC user after disabling them in the IdP. Expected: denied once the token expires.
10. Privilege escalation via RBAC: as a developer, try to create a RoleBinding to cluster-admin. Expected: forbidden (no bind/escalate).

Part 3: the security runbook

Write 2–4 pages for the team that will operate the cluster:

• Architecture: layers and where each control lives (a diagram helps).
• Access: how people log in; how access is granted and removed; break-glass procedure.
• Policies: which are enforced, how to request an exception, how to test a manifest before deploying.
• Detection & response: which alerts exist, who receives them, first steps for each (e.g. "Falco: shell in production container").
• Maintenance: certificate expiry, key rotation (encryption at rest, cosign), benchmark schedule.
• Evidence table from Part 2.

Reviewer's checklist

• All ten attack attempts have evidence, and any that succeeded unexpectedly have a fix or an accepted-risk entry.
• No human uses a shared, long-lived admin credential.
• Default ServiceAccounts in tenant namespaces can do nothing useful.
• Both tenants are isolated from each other at the network and RBAC level.
• Secrets are encrypted at rest, and none are in Git in plain form.
• Only signed images from the internal registry can run.
• Alerts reach a human, and the runbook tells them what to do.

You've finished Kubernetes Security & Hardening

From the cluster's certificates to OIDC logins, policy, supply chain and runtime detection, you can build a cluster that's secure and prove it. Next: Networking Deep Dive for the layers underneath, or Amazon EKS in Production to apply these controls in the cloud.