Prasanta Kumar Behera
Amazon EKS in Production with Terraform›11 · Disaster recovery
Learning Hub / Cloud — OpenStack, AWS & EKS / Amazon EKS in Production with Terraform

Lesson 11 of 18 · Automate & Recover

Disaster recovery

Recover from the two EKS disasters that matter: broken or lost Terraform state, and a lost cluster or region. Prevention guard-rails, state restore from S3 versions, import, and a tested rebuild path from Git and backups.

Advanced
Key wordsTerraform state recoveryS3 versioningimport blocksprevent_destroycluster rebuildGitOpsVeleromulti-region

Two kinds of disaster

Disaster Examples Recovery ingredients
State disasters Wrong-workspace destroy, corrupted state, someone deleted resources in the console, a botched refactor S3 versions, import, careful state surgery, prevention guard-rails
Cluster/region disasters Cluster deleted, add-on upgrade broke everything, region outage Terraform (cluster), GitOps (apps), backups (data), secrets manager, a second region

Two ways to lose your homework. Either you lose the list of what you've done (the state), but the work itself is still in your bag, so you find it and write the list again. Or you lose the whole bag (the cluster), and you need the recipe (code) to redo the work and a photocopy of your notes (backups) to fill in the answers.

Prevent first

  • prevent_destroy on the cluster, VPC, state bucket and KMS keys; deletion protection where resources support it.
  • Separate states per environment and layer (lesson 02), so one mistake has a small blast radius.
  • CI-only applies; no human applies to production from laptops.
  • IAM denies for destructive actions (e.g. eks:DeleteCluster) on CI roles, except through a break-glass path.
  • Plan review with policy checks on deletions (lesson 10).

Recovering Terraform state

  1. Freeze: stop pipelines and announce it; take the lock if needed.
  2. Preserve: terraform state pull > before-repair.tfstate.
  3. Compare: what does state say vs what exists in AWS (console, CLI, terraform plan output)?
  4. Recover, by case:
Case Recovery
State was overwritten or corrupted Restore an earlier S3 object version of the state file (then plan to confirm no surprises)
Resources exist, but state forgot them import blocks in code (reviewed), then plan/apply
Resources were deleted, state still lists them plan shows them as to-create; apply recreates them (data inside is a separate problem)
Refactor moved addresses moved blocks rather than destroy/create 
import {
  to = module.eks.aws_eks_cluster.this[0]
  id = "prod"
}
  1. Verify: a clean plan (no unexpected changes) before re-enabling pipelines. Write the incident up.

Recovering a lost cluster or region

The EKS control plane is managed, and there's no etcd snapshot for you to restore. Your cluster must be rebuildable from code:

  1. Terraform: apply network → cluster → platform layers (in another region if the region is down; keep region-parameterised code and pre-provisioned network where RTO demands it).
  2. GitOps: bootstrap Argo CD; it syncs platform add-ons and applications from Git.
  3. Secrets: External Secrets pulls from Secrets Manager (enable cross-region replication for secrets you need in DR).
  4. Images: ECR replication to the DR region, or images from a registry available there.
  5. Data: Velero restores namespaces and volumes (with snapshot data moved to S3 and replicated cross-region); databases restore from their own backups or failover replicas.
  6. Traffic: Route 53 failover or weighted records switch users to the new cluster.

Time each step in a drill. The sum is your real RTO.

Try it: two drills (sandbox account)

State drill

  1. Create a small stack (S3 bucket + IAM role) with remote state and versioning.
  2. Simulate corruption: terraform state rm the bucket. Run plan: it wants to create a bucket that already exists.
  3. Recover with an import block, and alternatively by restoring the previous S3 state version. Confirm a clean plan.

Cluster drill

  1. With your EKS stack in code and a GitOps-managed sample app (plus a Velero backup of its namespace), destroy the platform and cluster layers.
  2. Rebuild: apply the layers, bootstrap GitOps, restore data with Velero. Time every step.

Going deeper: resilience strategy

  • Decide per application whether cross-region DR is needed. Many internal tools can accept "rebuild within a day"; customer-facing checkout may need active-active.
  • Keep the state bucket and backups in another account (and region), with Object Lock where appropriate.
  • Rehearse DR quarterly and after big platform changes, and rotate who leads the drill.
  • Watch for hidden single points: an IdP, DNS provider, container registry or CI system that lives only in the failed region.

Recap

  • Prevent: prevent_destroy, small states, CI-only applies, IAM denies, reviewed plans.
  • State recovery: freeze → preserve → compare → restore S3 version / import / moved → clean plan.
  • Cluster recovery: Terraform + GitOps + secrets manager + image replication + Velero/database backups + DNS failover.
  • Drill it and time it. That's your RTO.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.