Capacity with Karpenter · Prasanta Kumar Behera

Right-sized capacity on demand with Karpenter: NodePools and EC2NodeClasses, how it picks instances for pending pods, consolidation to cut cost, Spot with interruption handling, and safe disruption settings.

How Karpenter thinks

Traditional node autoscaling scales predefined groups. Karpenter starts from pending pods: it looks at what they need (CPU, memory, architecture, zone, capacity type, taints and tolerations) and launches the cheapest instance that fits, often in well under a minute. Later it consolidates, removing or replacing nodes that are empty or under-used.

Instead of owning a fixed fleet of buses and hoping they fit everyone, Karpenter is a taxi dispatcher. When passengers are waiting, it sends exactly the right car: a small one for two people, a van for a family. When cars drive around half empty, it moves passengers together and sends the extra cars home.

NodePool and EC2NodeClass

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      expireAfter: 720h                    # replace nodes after 30 days (fresh AMIs, no drift)
  limits:
    cpu: "400"                             # never exceed 400 vCPUs in this pool
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"                       # disrupt at most 10% of nodes at once
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-prod
  amiSelectorTerms:
    - alias: al2023@latest                 # pin a specific version in production
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: prod
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: prod
  metadataOptions:
    httpTokens: required
    httpPutResponseHopLimit: 1

NodePool: what capacity is allowed (requirements, limits, disruption rules). Use several for different needs (e.g. a GPU pool with a taint).
EC2NodeClass: how to launch it on AWS (AMI, subnets, security groups, IAM role, metadata options, disks).
Wide requirements (many families, both architectures, Spot + on-demand) give Karpenter room to find cheap, available capacity.

Disruption: consolidation, expiry, drift

Karpenter voluntarily replaces nodes for three reasons:

Reason	Why
Consolidation	Empty or under-used nodes; cheaper replacements
Expiration (`expireAfter`)	Keep nodes fresh (patched AMIs, rotated credentials)
Drift	The node no longer matches its NodePool/NodeClass (e.g. a new AMI)

It always respects PodDisruptionBudgets, disruption budgets (how many nodes at once, optionally per time window), and the karpenter.sh/do-not-disrupt annotation for pods that must not move (e.g. long batch jobs).

Spot done right

Allow many instance types and all AZs; Karpenter uses price-capacity-optimised allocation.
Configure the interruption queue (SQS + EventBridge rules, created by the Karpenter Terraform module), so Karpenter cordons and drains nodes on the two-minute warning.
Keep critical, stateful or singleton workloads on on-demand (a separate NodePool, or nodeSelector on capacity type).

Try it: watch Karpenter work (sandbox account)

Install Karpenter with its Terraform module and Helm chart (following the official getting-started guide for your version), with the NodePool and EC2NodeClass above.
Deploy the inflate example (pause containers requesting 1 CPU each) with 0 replicas, then scale to 20. Watch NodeClaims appear and read the controller logs.
Check which instance types, zones and capacity types it chose.
Scale back to 0 and watch consolidation remove the nodes.
Add a PDB with minAvailable equal to replicas, and see how it blocks consolidation.

Going deeper: Karpenter in production

Run Karpenter itself on a small managed node group or Fargate, never on nodes it manages.
Pin AMI versions in production and let drift roll new AMIs through the fleet on your schedule; use disruption budgets with time windows for business hours.
Set limits on every NodePool, and alerts on capacity and cost, so a bug can't scale indefinitely.
Right-size requests: Karpenter packs nodes by requests, so inflated requests mean inflated bills (see Kubernetes Administration, lesson 23).

Recap

Karpenter provisions from pending pods: the cheapest instance that fits, fast.
NodePool (requirements, limits, disruption) + EC2NodeClass (AMI, subnets, SGs, role, IMDSv2).
Consolidation, expiration, drift, bounded by disruption budgets, PDBs and do-not-disrupt.
Spot with wide instance choice and the interruption queue; critical workloads on on-demand.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.