Playbook: creating the cluster · Prasanta Kumar Behera

A run-it-in-order playbook for creating the EKS stack from this course: pre-flight checks, bootstrapping state, then the network, cluster, platform and app layers, each with a verification gate that must pass before you move on, and what to do if a layer fails.

Apply layer by layer; pass each verification gate before moving on.

How to use this playbook

This is the operational run order for the stack built in lessons 01–12: the same layers (live/prod/network, cluster, platform, plus GitOps for apps), done in order, with a check after each. Print it, tick it, and keep the output of every gate for the hand-over.

Building a house: you don't put the roof on before checking the walls are straight. Foundation, walls, roof, furniture, and a quick inspection after each stage. If the walls are crooked, you fix the walls, not the whole house.

Phase 0: pre-flight

Check	Command / action	Pass when
Identity	`aws sts get-caller-identity`	The intended account and role
Region	`aws configure get region` / `AWS_REGION`	Matches `prod.tfvars`
Quotas	`aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A` (On-Demand standard vCPUs); EIPs, VPCs, NAT gateways	Enough headroom for nodes + surge
Tools	`terraform version`, `kubectl version --client`, `helm version`	Match the repo's pinned versions
Change record	Ticket/PR with the plan	Approved

Phase 1: state (once per account)

Create the state bucket (versioning, encryption, public access blocked) with its own tiny bootstrap configuration, and enable locking (use_lockfile = true on Terraform 1.10+, or a DynamoDB table on older setups). See lesson 02.

Gate: aws s3api get-bucket-versioning --bucket <state-bucket> shows Enabled; a test terraform init in the network layer succeeds.

Phase 2: network layer

$ cd live/prod/network
$ terraform init -backend-config=backend-prod.hcl
$ terraform plan -var-file=prod.tfvars -out=prod.plan      # review: VPC, subnets per AZ, NAT, endpoints, tags
$ terraform apply prod.plan

Gate:

$ aws ec2 describe-subnets --filters Name=vpc-id,Values=<vpc> \
    --query 'Subnets[].[AvailabilityZone,CidrBlock,Tags[?Key==`kubernetes.io/role/elb`]|[0].Value]' --output table

Private subnets in 3 AZs, public subnets tagged kubernetes.io/role/elb, private ones kubernetes.io/role/internal-elb, the secondary pod CIDR (if used) attached (lesson 04).

Phase 3: cluster layer

$ cd ../cluster
$ terraform init -backend-config=backend-prod.hcl
$ terraform plan -var-file=prod.tfvars -out=prod.plan      # review: EKS version, endpoint access, node groups, access entries, add-ons
$ terraform apply prod.plan                                # EKS creation takes several minutes

Gate:

$ aws eks describe-cluster --name prod --query 'cluster.[status,version,accessConfig.authenticationMode]'
$ aws eks update-kubeconfig --name prod --region eu-west-1 --alias prod
$ kubectl get nodes -o wide                                 # system nodes Ready, spread across AZs
$ aws eks list-addons --cluster-name prod
$ kubectl -n kube-system get pods                           # vpc-cni, coredns, kube-proxy, ebs-csi, pod-identity agent Running
$ kubectl auth can-i '*' '*' --all-namespaces               # you have the access you expect (and no more for other roles)

Phase 4: platform layer

$ cd ../platform
$ terraform init -backend-config=backend-prod.hcl && terraform plan -var-file=prod.tfvars -out=prod.plan && terraform apply prod.plan

Typical contents: AWS Load Balancer Controller, Karpenter (NodePools/EC2NodeClass), metrics-server, storage classes, monitoring agents, Argo CD.

Gate:

$ kubectl -n kube-system rollout status deploy/aws-load-balancer-controller
$ kubectl get nodepools,ec2nodeclasses                      # Karpenter config accepted
$ kubectl top nodes                                         # metrics-server works
$ kubectl get storageclass                                  # expected default class

Phase 5: apps via GitOps and a smoke test

Point Argo CD at the GitOps repo (lesson 15's flow), then deploy a smoke app with an Ingress and a PVC:

$ kubectl -n smoke get ingress,pvc,pods
$ curl -s -o /dev/null -w '%{http_code}\n' http://<alb-dns-name>/     # 200

This single test proves ALB creation, target registration, EBS provisioning and node scaling (scale it up to force Karpenter to add a node).

Phase 6: hand-over checklist

[ ] Plans and gate outputs attached to the change record
[ ] Access: break-glass role tested; team access entries in place (lesson 05)
[ ] Backups/rebuild tested (lesson 11); alerts routed
[ ] Cost tags on all resources; budget alert set
[ ] Runbooks linked (this section's lessons)

If a phase fails

Phase	Typical failure	Response
Network	Quota (VPCs, EIPs, NAT)	Raise the quota, re-run the layer
Cluster	IAM permissions, unsupported version/instance type in an AZ	Fix inputs, re-plan; the network layer is untouched
Platform	Helm timeout, webhook not ready, CRD ordering	Read events/logs, fix, re-apply this layer only
Apps	Ingress without address, PVC pending	See lesson 16 (challenges)

Rolling back a new environment is terraform destroy in reverse order (platform → cluster → network), after deleting Kubernetes-created AWS resources (load balancers, Karpenter nodes); see lesson 14.

Try it: a dry run of the playbook

Run Phase 0 checks in your sandbox account and note the quota values.
Apply the network layer only; run the gate command and read the subnet tags.
Apply the cluster layer and run every Phase 3 gate; save the outputs.
Apply a minimal platform layer (LB controller only) and deploy the smoke app.
Tear it down in reverse order (lesson 14's safe teardown).

Command summary

aws sts get-caller-identity ; aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A
# per layer: network → cluster → platform
terraform init -backend-config=backend-prod.hcl ; terraform plan -var-file=prod.tfvars -out=prod.plan ; terraform apply prod.plan
aws eks describe-cluster --name prod --query 'cluster.[status,version]'
aws eks update-kubeconfig --name prod --region eu-west-1 --alias prod
aws eks list-addons --cluster-name prod ; kubectl get nodes -o wide ; kubectl -n kube-system get pods
kubectl -n kube-system rollout status deploy/aws-load-balancer-controller ; kubectl get nodepools,ec2nodeclasses
kubectl -n smoke get ingress,pvc,pods ; curl http://<alb-dns-name>/

Recap

Pre-flight (identity, region, quotas, tools), then state, network, cluster, platform, apps.
Every layer: init → plan (reviewed) → apply the saved plan → verification gate.
A smoke app with Ingress + PVC proves the platform end to end.
Failures stay inside one layer; roll back new environments in reverse order.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.