Recovery playbook · Prasanta Kumar Behera

Step-by-step runbooks for the bad days on this stack: a stuck state lock, corrupted or overwritten state, resources deleted outside Terraform, resources that exist but aren't in state, a destroy that hangs on network interfaces, lost cluster access, and rebuilding a lost cluster from code.

Before any runbook

Freeze: pause pipelines for the affected environment; announce in the incident channel.
Preserve: terraform state pull > before-repair-$(date +%F-%H%M).tfstate.
Compare: what state says vs what exists in AWS.
After the fix: a clean plan before unfreezing; write it up (see Incident Handling, lesson 05).

Concepts behind these runbooks are in lesson 11.

A fire drill card on the wall: one card per kind of emergency, short, numbered steps, written when everyone was calm. When the alarm rings, you follow the card instead of inventing a plan.

RB-1: stale state lock

$ terraform plan -var-file=prod.tfvars
Error: Error acquiring the state lock … Lock Info: ID: 5c1d…  Who: runner@ci-7  Operation: OperationTypeApply  Created: …

Check CI: is that pipeline run still going? If yes, wait.
If the run crashed or was cancelled: terraform force-unlock 5c1d….
Run plan: if it shows partial changes, the crashed apply left work half done; apply to converge (after review).

RB-2: state overwritten or corrupted

$ aws s3api list-object-versions --bucket acme-tfstate-prod --prefix eks/prod/cluster/terraform.tfstate \
    --query 'Versions[].[VersionId,LastModified,Size]' --output table
$ aws s3api get-object --bucket acme-tfstate-prod --key eks/prod/cluster/terraform.tfstate \
    --version-id <good-version-id> /tmp/good.tfstate
$ aws s3api copy-object --bucket acme-tfstate-prod --key eks/prod/cluster/terraform.tfstate \
    --copy-source "acme-tfstate-prod/eks/prod/cluster/terraform.tfstate?versionId=<good-version-id>"
$ terraform plan -var-file=prod.tfvars          # expect no (or only understood) changes

With DynamoDB-based locking, Terraform also stores a checksum of the state; after restoring a version by hand, Terraform may report that the state doesn't match the expected digest. Follow the error's instructions (update or remove the digest item for that state in the lock table). With S3-native locking (use_lockfile), there's no DynamoDB digest to fix.

RB-3: resources deleted outside Terraform

Someone deleted a node group or security group in the console.

terraform plan shows them as to create.
Review, then apply to recreate. Data inside (volumes) needs its own restore.
Find out how it happened (CloudTrail) and remove that access path.

RB-4: resources exist but aren't in state

A lost state file, or something created by hand that should be managed:

import {
  to = module.eks.aws_eks_cluster.this[0]
  id = "prod"
}

terraform plan shows the import; review carefully that the configuration matches reality (to avoid replacement), then apply. Repeat for each missing resource.

RB-5: destroy hangs on the network

$ aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc> \
    --query 'NetworkInterfaces[].[NetworkInterfaceId,Description,Status,InterfaceType]' --output table
$ aws elbv2 describe-load-balancers --query 'LoadBalancers[?VpcId==`<vpc>`].[LoadBalancerName,LoadBalancerArn]'

ENIs described as ELB → delete the load balancer (ideally via the Kubernetes object while the cluster exists).
Instances from Karpenter → terminate them (or delete NodePools before destroying the cluster).
aws-K8S-… ENIs from the VPC CNI → they go away with their instances; detached leftovers can be deleted.
Then re-run the destroy.

RB-6: nobody can access the cluster

Find an identity outside the broken path: break-glass access entry, or the cluster creator (in legacy ConfigMap mode).
Restore access config from Git (access entries via Terraform, or aws-auth from the last good version).
Details: Incident Handling, lesson 18.

RB-7: rebuild a lost cluster from code

Confirm the scope (cluster deleted, region impaired).
Apply network → cluster → platform from Git (lesson 13's playbook) in the same or another region.
Let Argo CD restore applications from the GitOps repo.
Restore data: EBS snapshots, Velero backups (velero restore create --from-backup <name>), databases from their own backups.
Switch traffic (DNS/Route 53), verify SLOs, then write up measured RTO/RPO.

Try it: practise the runbooks (sandbox)

Start an apply, kill it mid-way, and practise RB-1 (inspect lock info, force-unlock, re-plan).
Deliberately overwrite state with a bad apply in a toy stack and restore a previous S3 version (RB-2).
Delete a security group in the console and recover with plan/apply (RB-3).
Create an ALB via an Ingress, then destroy the network layer without deleting it first; clean up with RB-5.
Time a full rebuild of a small environment (RB-7).

Command summary

terraform state pull > before-repair.tfstate
terraform force-unlock <LOCK_ID>
aws s3api list-object-versions --bucket <b> --prefix <key>
aws s3api copy-object --bucket <b> --key <key> --copy-source "<b>/<key>?versionId=<id>"
terraform plan -var-file=prod.tfvars
# import { to = … id = … } → plan → apply
aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc>
aws elbv2 describe-load-balancers
aws eks list-access-entries --cluster-name prod
velero restore create --from-backup <name>

Recap

Always freeze, preserve (state pull), compare, fix, verify with a clean plan.
Locks: force-unlock only when nothing runs. Corruption: restore an S3 version (mind the DynamoDB digest).
Missing resources: plan/apply; unmanaged ones: import blocks.
Stuck destroys: leftover ENIs/LBs/instances. Access: break-glass. Lost cluster: rebuild from code + GitOps + backups.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.