VPC CNI & IP planning

Pods are VPC citizens

The Amazon VPC CNI gives each pod an IP address from your VPC subnets, attached to the node through ENIs (elastic network interfaces). Pods are directly routable in the VPC: no overlay, security groups can apply to pods, and AWS services see real pod IPs.

The price: pods consume VPC addresses, and each instance can only hold a limited number.

How many pods fit on a node?

Without prefix delegation, the classic formula is:

max pods = ENIs × (IPv4 addresses per ENI − 1) + 2
e.g. m5.large: 3 ENIs × (10 − 1) + 2 = 29 pods

Small instances run out of pod slots long before they run out of CPU or memory.

Fix 1: prefix delegation (pod density)

With prefix delegation on Nitro-based instances, each ENI slot holds a /28 prefix (16 addresses) instead of a single IP:

$ kubectl -n kube-system set env daemonset aws-node ENABLE_PREFIX_DELEGATION=true

(Better: set it in the vpc-cni add-on's configuration through Terraform.) Then raise max-pods for new nodes; Karpenter and recent EKS node templates can compute it, and a common practical cap is 110 pods per node. Existing nodes need replacing to use prefixes.

Caveat: subnets need free, contiguous /28 blocks. Fragmented subnets can fail to allocate prefixes even when they have free IPs, so use dedicated, generously sized subnets.

Fix 2: a secondary CIDR for pods (address space)

Corporate VPC ranges are often small and precious. Add a secondary CIDR to the VPC, typically from 100.64.0.0/10 (a range reserved for carrier-grade NAT, rarely used elsewhere), create pod subnets in each AZ from it, and tell the CNI to use them with custom networking:

VPC 10.20.0.0/16            → nodes, load balancers (routable across the company)
VPC secondary 100.64.0.0/16 → pods only
   pods-a 100.64.0.0/18   pods-b 100.64.64.0/18   pods-c 100.64.128.0/18

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: eu-west-1a                  # one per AZ, matched to nodes by zone label
spec:
  subnet: subnet-0aaa1111bbb2222cc  # the 100.64.x pod subnet in eu-west-1a
  securityGroups:
    - sg-0123456789abcdef0

With AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true and ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone, pods get addresses from 100.64.x, while nodes stay in the main range. Pod traffic to other networks leaves via the node's primary interface (SNAT to the node IP), so the rest of the company never needs routes to 100.64.0.0/16.

Note: with custom networking, the node's primary ENI isn't used for pods, which slightly reduces max pods per node. Combine with prefix delegation to compensate.

Other options

• IPv6 clusters: essentially unlimited pod addresses; requires IPv6 readiness end to end.
• Security groups for pods: attach specific security groups to specific pods (via branch ENIs on supported instances), for fine-grained access to databases.

Recap

• The VPC CNI gives pods real VPC IPs via ENIs, so clusters can run out of addresses or per-node slots.
• Prefix delegation (/28 per ENI slot) raises pod density; needs contiguous free space.
• A secondary CIDR (commonly 100.64.0.0/10) + custom networking/ENIConfig gives pods their own address space.
• Plan for peak, keep pod subnets dedicated, and decide early.