Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›17 · EKS: pods can't get IP addresses
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 17 of 19 · Real-world incident scenarios

EKS: pods can't get IP addresses

On EKS, new pods sit in ContainerCreating with 'failed to assign an IP address to container'. The VPC CNI gives every pod a real VPC IP, so subnets or per-instance ENI limits run out. Diagnose which limit you hit, recover quickly, and plan subnets, prefix delegation and warm-pool settings so it doesn't return.

Advanced
Key wordsEKSAmazon VPC CNIfailed to assign an IP addresssubnet exhaustionENI limitsmax podsprefix delegationWARM_IP_TARGETsecondary CIDRcustom networking

The page

Black-Friday rehearsal: the Horizontal Pod Autoscaler scales the storefront to 300 pods. About 60 stay in ContainerCreating with failed to assign an IP address to container. Karpenter adds nodes, but the new pods on them fail too. The cluster's private subnets are three /24s.

First five minutes

  • Impact: can't scale; if traffic keeps rising, users will see errors.
  • Stabilise: temporarily lower HPA max or shed non-critical workloads to free IPs for critical ones; scale down idle deployments.
  • Confirm the error text in pod events.

Every child in the school gets their own locker (a real VPC IP for each pod). The building has only so many lockers per corridor (subnet), and each classroom (node) can only use a few corridors (ENI limits). Buying more desks (nodes) doesn't help if there are no lockers left.

Diagnose

$ kubectl describe pod storefront-7f9c-abcde | grep -i 'assign an IP'
  failed to assign an IP address to container
$ aws ec2 describe-subnets --subnet-ids subnet-aaa subnet-bbb subnet-ccc \
    --query 'Subnets[].[SubnetId,CidrBlock,AvailableIpAddressCount]' --output table
$ kubectl -n kube-system logs -l k8s-app=aws-node -c aws-node --tail=50 | grep -iE 'insufficient|no available|error'
Finding Limit hit
AvailableIpAddressCount near 0 Subnet exhaustion
Subnets have space, but node's pods ≥ its IP capacity Per-instance ENI/IP limit (instance type too small for the pod count)
Many free IPs attached to idle nodes Warm pool hoarding

Recover

  • Free IPs: scale down unneeded workloads; remove idle nodes (warm IPs are released when nodes go).
  • Schedule new capacity into subnets with free space (Karpenter/nodegroup subnet selection).
  • If subnets are the limit and it's urgent, add capacity in subnets with room; plan the proper fix next.

Fix properly

  • Prefix delegation (ENABLE_PREFIX_DELEGATION=true on the aws-node DaemonSet, Nitro instances): /28 prefixes per ENI slot, much higher pod density. Update max-pods for nodes accordingly (managed node groups and Karpenter can calculate it; check your setup).
  • Warm pool tuning: WARM_IP_TARGET / MINIMUM_IP_TARGET (or WARM_PREFIX_TARGET) so nodes don't hoard.
  • Bigger or more subnets, or a secondary VPC CIDR (commonly from 100.64.0.0/10) for pods with custom networking (ENIConfig per AZ).
  • IPv6 clusters remove IPv4 exhaustion for pods (a bigger design change).

See Amazon EKS in Production with Terraform, lesson 04 for IP planning in depth.

Prevent

  • Plan pod IPs at cluster design time: peak pods × headroom per AZ.
  • Alert on subnet free IPs (CloudWatch/custom exporter) and on pods stuck in ContainerCreating.
  • Load-test scale-ups, not just steady state.

Try it: exhaust a small subnet (sandbox account)

  1. Create a sandbox EKS cluster with a deliberately small /26 private subnet for one node group.
  2. Scale a deployment until pods fail with the IP error; record the subnet's free count.
  3. Enable prefix delegation and replace the nodes; compare how many pods fit.
  4. Tune WARM_IP_TARGET and observe IPs held by idle nodes.
  5. Delete the sandbox cluster afterwards (costs!).

Going deeper: address strategy on AWS

  • Coordinate VPC ranges with the network team early; overlapping VPCs block peering and Transit Gateway routing later.
  • Security groups for pods and custom networking interact with IP planning; test them together.
  • Monitor the VPC CNI version: settings and defaults evolve between releases.

Recap

  • VPC CNI = one VPC IP per pod; you hit subnet or per-instance ENI/IP limits.
  • Diagnose with pod events, subnet free counts, and aws-node logs.
  • Fix with prefix delegation, warm-pool tuning, more subnets / secondary CIDR, or IPv6.
  • Plan and alert on IP capacity before peak events.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.