Remote state: S3, locking, env-per-tfvars

State is the source of truth, so protect it

Terraform's state maps your code to real resources. Lose it, corrupt it, or let two people apply at once, and Terraform can't safely manage the cluster any more. State also contains sensitive values, so treat it like a secret.

The S3 backend

terraform {
  required_version = ">= 1.10"
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "eks/prod/cluster/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true      # S3-native locking (Terraform 1.10+); older setups: dynamodb_table = "tf-locks"
  }
}

The state bucket itself (created once, often by a small bootstrap stack):

• Versioning on: every state change is recoverable (lesson 11).
• Encryption (SSE-KMS), Block Public Access, and a bucket policy limiting access to CI roles and a few admins.
• Ideally in a separate account from the clusters it manages.

One state per environment and layer

Don't put everything in one giant state. Split by environment and by layer:

live/
├── prod/
│   ├── network/      (VPC, subnets, endpoints)      → key: eks/prod/network
│   ├── cluster/      (EKS, node groups, IAM, add-ons) → key: eks/prod/cluster
│   └── platform/     (Helm: LB controller, Karpenter…) → key: eks/prod/platform
└── dev/
    └── …same layers, own keys (and ideally own account)
modules/
└── eks-cluster/  vpc/  karpenter/ …

Smaller states mean faster plans, smaller blast radius, and clear ownership. Layers read each other's outputs through terraform_remote_state data sources (or SSM parameters).

Per-environment tfvars

# prod.tfvars
cluster_name       = "prod"
kubernetes_version = "1.32"
system_node_count  = 3
endpoint_public    = false

$ terraform init -backend-config=backend-prod.hcl
$ terraform plan -var-file=prod.tfvars -out=prod.plan
$ terraform apply prod.plan

Workspaces vs directories: Terraform workspaces share one configuration with separate states. They're simple, but it's easy to apply to the wrong one. Separate directories (or tools like Terragrunt) make the environment explicit in the path, and let environments differ when needed.

Drift

Someone changes a security group in the console, and now reality differs from code. Detect it on a schedule:

$ terraform plan -var-file=prod.tfvars -detailed-exitcode
$ echo $?
2

A nightly CI job that alerts on exit code 2 catches drift early. Then either codify the change or revert it.

Recap

• Remote state in S3: encrypted, versioned, access-restricted, locked (use_lockfile, or DynamoDB on older setups).
• One state per environment and layer; share outputs deliberately.
• tfvars per environment and explicit init per backend; be careful with workspaces.
• Detect drift nightly with plan -detailed-exitcode.