Prasanta Kumar Behera
Amazon EKS in Production with Terraform›16 · Challenges on this stack
Learning Hub / Cloud — OpenStack, AWS & EKS / Amazon EKS in Production with Terraform

Lesson 16 of 18 · Playbooks, challenges & practice

Challenges on this stack

Real-world problems you'll meet running this EKS stack, each with the symptom, how to confirm it and how to fix it for good: a plan that wants to replace the cluster, console drift, Karpenter not launching nodes, pods stuck on EBS volumes in another AZ, Ingresses without an address, surprise NAT costs and more.

Advanced
Key wordsforces replacementprevent_destroydriftKarpenter not launchingNodeClaimsEBS AZ affinityWaitForFirstConsumerIngress without addresssubnet tagsNAT costVPC endpoints

How to use this lesson

Each challenge: symptom → confirm → fix → prevent. Many link to deeper lessons. (Generic incident method: Incident Handling, lessons 01–02.)

A field guide for a hiking trail: "if you see this sign, here's what it means and which path to take". You won't meet every problem on every hike, but you'll recognise them when you do.

1. The plan wants to replace the cluster

  • Symptom: module.eks.aws_eks_cluster.this[0] must be replaced.
  • Confirm: find the line marked # forces replacement in the plan (name, some network settings, encryption config changes).
  • Fix: revert the change, or plan a real migration (new cluster, move workloads).
  • Prevent: lifecycle { prevent_destroy = true } on the cluster; CI fails plans containing replacements of critical resources.

2. Console drift

  • Symptom: a plan you didn't expect shows changes (a security group rule, node group size).
  • Confirm: terraform plan diff; CloudTrail shows who changed it.
  • Fix: decide: revert (apply) or codify (update Terraform to match).
  • Prevent: scheduled plan -detailed-exitcode drift checks (lesson 02); read-only console for humans in prod.

3. The chicken-and-egg bites on apply or destroy

  • Symptom: kubernetes/helm provider errors connecting to a cluster that doesn't exist yet or is being destroyed.
  • Fix: separate cluster and platform layers; destroy platform first (lesson 03).

4. Karpenter doesn't launch nodes

  • Symptom: pods Pending, no new nodes.
  • Confirm: kubectl get nodeclaims, kubectl describe nodeclaim <n>, Karpenter logs (no compatible instance types, subnet/SG not found, insufficient capacity, IAM errors).
  • Fix: widen NodePool requirements (families, sizes, zones, capacity types); fix EC2NodeClass subnet/security-group selector tags; check the node IAM role/instance profile; check NodePool limits.
  • Prevent: alert on pods pending > N minutes; test NodePools in staging (lesson 06).

5. Pods stuck on EBS volumes in another AZ

  • Symptom: volume node affinity conflict, pod Pending after a node replacement.
  • Confirm: kubectl get pv -o wide shows the volume's zone; no schedulable node in that zone.
  • Fix: make capacity available in that AZ (NodePool zones), or restore to a new volume in another AZ from a snapshot.
  • Prevent: StorageClasses with WaitForFirstConsumer; capacity in every AZ; EFS for data that must move between AZs (lesson 07).

6. Ingress has no ADDRESS

  • Symptom: kubectl get ingress shows no address; no ALB appears.
  • Confirm: LB controller logs: missing subnet tags (kubernetes.io/role/elb), IAM permission errors, wrong ingressClassName, invalid annotations.
  • Fix: tag subnets in the network layer; fix the controller's IAM (Pod Identity/IRSA); correct the Ingress.
  • Prevent: smoke test in the creation playbook (lesson 13).

7. Pods can't get IP addresses

Subnet or ENI exhaustion with the VPC CNI. See Incident Handling, lesson 17 and lesson 04 here (prefix delegation, secondary CIDR).

8. Upgrade blocked or broken

  • Symptom: add-on update fails; workloads break after the upgrade (removed APIs).
  • Fix/prevent: pre-checks with pluto, compatibility matrices for add-ons and controllers, staging first (lessons 09 and 14).

9. Surprise NAT gateway bill

  • Symptom: high NAT data-processing charges.
  • Confirm: VPC flow logs / Cost Explorer by usage type; image pulls and S3 traffic through NAT.
  • Fix: VPC endpoints for ECR (api + dkr), S3 (gateway endpoint), STS, CloudWatch; keep traffic in-AZ where possible.

10. Locked out of the cluster

Access entries or aws-auth mistakes. See Incident Handling, lesson 18 and the recovery playbook (lesson 17).

Try it: provoke three challenges (sandbox)

  1. Change a replacement-forcing attribute in a test cluster config and read the plan (don't apply).
  2. Create a NodePool that allows only an instance type unavailable in your region's AZs; read the NodeClaim events.
  3. Remove the kubernetes.io/role/elb tag from public subnets, create an Ingress, and read the controller logs; restore the tag.

Command summary

terraform plan | grep -E 'must be replaced|forces replacement'
terraform plan -detailed-exitcode
kubectl get nodeclaims ; kubectl describe nodeclaim <n> ; kubectl -n kube-system logs deploy/karpenter
kubectl get pv -o wide ; kubectl describe pvc <p>
kubectl -n kube-system logs deploy/aws-load-balancer-controller | tail
aws ec2 describe-subnets --filters Name=tag:kubernetes.io/role/elb,Values=1

Recap

  • Catch replacements and drift in plans; protect critical resources with prevent_destroy.
  • Karpenter: read NodeClaims and logs; fix requirements and selector tags.
  • EBS is AZ-bound: WaitForFirstConsumer and capacity per AZ.
  • No ALB: subnet tags, controller IAM, IngressClass. Cut NAT costs with VPC endpoints.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.