Prasanta Kumar Behera
Amazon EKS in Production with Terraform›15 · The flow: laptop → cluster → kubeconfig → app
Learning Hub / Cloud — OpenStack, AWS & EKS / Amazon EKS in Production with Terraform

Lesson 15 of 18 · Playbooks, challenges & practice

The flow: laptop → cluster → kubeconfig → app

Follow the whole path, end to end: how an engineer's laptop authenticates and gets a kubeconfig, how access entries decide what they can do, and how changes reach the cluster the CI/CD way: infra through Terraform pipelines, apps through image builds, a GitOps repo and Argo CD.

Practitioner
Key wordsaws sso loginupdate-kubeconfigexec credential pluginaws eks get-tokenaccess entriesOIDCECRGitOpsArgo CDread-only prod
Your laptop aws sso login assume a role aws eks update-kubeconfig exec: aws eks get-token kubectl (read-only in prod) Git + CI (OIDC to AWS) infra repo terraform plan / apply app repo build → ECR (by digest) GitOps repo new tag committed AWS account EKS cluster access entries ECR images Argo CD (in cluster) pulls + syncs humans read, pipelines write; every change goes through Git
Humans read through short-lived credentials; pipelines write through Git.

Two kinds of traffic to the cluster

  1. People looking and debugging (and, rarely, fixing).
  2. Changes: infrastructure and applications.

In a mature setup, the first is mostly read-only, and the second always goes through Git and pipelines.

A museum: visitors get a day ticket (short-lived credentials) that lets them look at the exhibits (read-only kubectl). Only the museum staff (pipelines) move exhibits, and every move is written in the logbook (Git). If there's a fire, a guard has the emergency key (break-glass).

Path 1: laptop → cluster

$ aws sso login --profile prod-readonly
$ aws eks update-kubeconfig --name prod --region eu-west-1 --alias prod --profile prod-readonly
$ kubectl config view --minify | grep -A8 'exec:'
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args: [ --region, eu-west-1, eks, get-token, --cluster-name, prod, --output, json ]
      env: [ { name: AWS_PROFILE, value: prod-readonly } ]
$ kubectl auth whoami
$ kubectl get pods -n payments

What happens:

  1. Your SSO session gives temporary IAM credentials for a role.
  2. kubectl runs aws eks get-token, which creates a short-lived token from those credentials.
  3. EKS checks the IAM role against its access entries and applies the associated access policies (or RBAC groups): view in payments, nothing in kube-system, for example.
  4. If the endpoint is private, you need network access to the VPC (VPN, bastion, or SSM port forwarding).

Path 2: infrastructure changes

  • Engineer opens a PR in the infra repo → pipeline assumes a read-only plan role via OIDC → posts terraform plan.
  • Merge to main → pipeline assumes the apply role (only for the approved environment) → applies the saved plan.
  • Details: lesson 10.

Path 3: application changes

  1. App repo CI (OIDC to AWS) builds the image and pushes it to ECR, recording the digest.
  2. CI commits the new digest to the GitOps repo (Kustomize/Helm values).
  3. Argo CD inside the cluster notices and syncs; pods pull from ECR (the node role / credential provider handles auth).
  4. Health and SLO checks confirm the rollout; rollback is a git revert in the GitOps repo.

(Same pattern as Hands-on Projects, project 02; Argo CD details in GitOps with Argo CD.)

Break-glass

A tightly controlled admin role with its own access entry, used only in emergencies, alerting on every use, and followed by a review. Test it every quarter.

Try it: trace every hop

  1. Configure two AWS profiles (read-only and admin) and generate kubeconfig entries for each (--alias prod-ro, --alias prod-admin).
  2. Run kubectl auth whoami and kubectl auth can-i --list -n payments with each.
  3. Look at CloudTrail for the AssumeRoleWithWebIdentity event from a pipeline run.
  4. Change an image digest in the GitOps repo and follow it: commit → Argo CD sync → new pods.
  5. Use the break-glass role once in a sandbox and check that the alert fired.

Command summary

aws sso login --profile prod-readonly
aws eks update-kubeconfig --name prod --region eu-west-1 --alias prod --profile prod-readonly
kubectl config view --minify ; kubectl auth whoami ; kubectl auth can-i --list -n payments
# pipelines: OIDC → terraform plan/apply (infra) ; docker build/push to ECR (apps) ; commit digest to GitOps repo
argocd app get <app> ; git revert <sha>    # GitOps rollback

Recap

  • Laptop: SSO → temporary credentials → update-kubeconfig → exec token → access entries.
  • Changes: infra via Terraform pipelines (OIDC), apps via ECR + GitOps + Argo CD.
  • Humans read, pipelines write; a tested break-glass role for emergencies.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.