Playbook: day-2 operations · Prasanta Kumar Behera

Repeatable procedures for running the EKS stack after day one: add capacity, grant a team access, bump an add-on, run a Kubernetes minor upgrade in the right order, check cost drivers, and tear down an environment without orphaned load balancers or network interfaces.

Rules for day 2

Every change goes through Git → plan → review → apply (lesson 10), never the console.
One change at a time, smallest layer possible.
Each procedure ends with a check.

A well-run kitchen has recipe cards for the jobs that come up every week: adding a table, a new cook starting, changing the oven, deep-cleaning at closing time. Nobody improvises them at dinner rush.

P1: add or reshape capacity

Karpenter (platform layer): add a NodePool for a new workload type.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: batch }
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
      requirements:
        - { key: karpenter.sh/capacity-type, operator: In, values: [ spot, on-demand ] }
        - { key: kubernetes.io/arch, operator: In, values: [ amd64 ] }
      taints: [ { key: workload, value: batch, effect: NoSchedule } ]
  limits: { cpu: "200" }
  disruption: { consolidationPolicy: WhenEmptyOrUnderutilized }

(Check the Karpenter API version and field names for your release; see lesson 06.)

Check: deploy a pod with the matching toleration; kubectl get nodeclaims shows a new node; it goes away after the pod is deleted.

P2: grant a team access

resource "aws_eks_access_entry" "payments" {
  cluster_name  = "prod"
  principal_arn = "arn:aws:iam::111122223333:role/payments-developers"
}

resource "aws_eks_access_policy_association" "payments_view" {
  cluster_name  = "prod"
  principal_arn = aws_eks_access_entry.payments.principal_arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  access_scope { type = "namespace", namespaces = [ "payments" ] }
}

Check: assume the role and run kubectl auth can-i list pods -n payments (yes) and -n kube-system (no).

P3: bump a managed add-on

$ aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version 1.32 \
    --query 'addons[0].addonVersions[].addonVersion' --output text

Change the version in the cluster layer, plan, apply. Check: aws eks describe-addon --cluster-name prod --addon-name vpc-cni --query 'addon.[addonVersion,status]' shows the new version ACTIVE; pods can still get IPs.

P4: Kubernetes minor upgrade (run order)

Pre-checks: read the release notes; pluto for deprecated/removed APIs in Helm releases and manifests; add-on and controller compatibility (LB controller, Karpenter, CSI drivers); PodDisruptionBudgets that could block drains.
Control plane: bump kubernetes_version one minor in the cluster layer → plan → apply. Wait for ACTIVE.
Managed add-ons: bump to versions compatible with the new minor.
Nodes: managed node groups roll to the new AMI; Karpenter replaces drifted nodes according to its disruption budgets.
Verify: all nodes on the new version, system pods healthy, smoke app OK, SLO dashboards normal.

(Details and blue/green cluster alternatives in lesson 09.)

P5: monthly cost check

NAT gateway data processing and cross-AZ traffic: add VPC endpoints (ECR, S3, STS) where heavy.
Idle capacity: requests vs usage per node pool; Karpenter consolidation working?
Load balancers nobody uses; old EBS snapshots/volumes; CloudWatch Logs retention.
Tags on everything so cost reports group by team/environment.

P6: safe teardown (non-production or retired environments)

$ kubectl get ingress,svc -A | grep -iE 'LoadBalancer|alb'     # find LB-backed objects
$ kubectl delete ingress --all -n <app-namespaces>              # controller deletes ALBs
$ kubectl delete svc <lb-services> -n <ns>                      # NLBs
$ kubectl delete nodepools --all                                 # Karpenter terminates its nodes
$ aws elbv2 describe-load-balancers --query 'LoadBalancers[].LoadBalancerName'   # nothing left?
$ cd live/dev/platform && terraform destroy -var-file=dev.tfvars
$ cd ../cluster       && terraform destroy -var-file=dev.tfvars
$ cd ../network       && terraform destroy -var-file=dev.tfvars

If the network destroy hangs on a security group or subnet, look for leftover ENIs in the VPC (lesson 17, recovery).

Try it: run three procedures

Add the batch NodePool and prove a node appears and disappears.
Create an access entry for a test role with view access to one namespace; check with kubectl auth can-i.
Tear down a dev environment with P6 and confirm no load balancers or ENIs remain.

Command summary

kubectl apply -f nodepool-batch.yaml ; kubectl get nodeclaims
# access entry + policy association via Terraform, then:
kubectl auth can-i list pods -n payments
aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version 1.32
aws eks describe-addon --cluster-name prod --addon-name vpc-cni --query 'addon.[addonVersion,status]'
pluto detect-helm -o wide ; pluto detect-files -d manifests/
# upgrade: control plane → add-ons → nodes → verify
kubectl get nodes -o wide
# teardown
kubectl delete ingress --all -n <ns> ; kubectl delete nodepools --all
terraform destroy -var-file=dev.tfvars   # platform → cluster → network

Recap

Day-2 changes go through Git and the pipeline, one at a time, each with a check.
Procedures: capacity (NodePools), access entries, add-on bumps, upgrade run order, cost checks.
Teardown: delete controller-created AWS resources first, then destroy platform → cluster → network.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.