Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›18 · EKS: locked out after editing aws-auth
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 18 of 19 · Real-world incident scenarios

EKS: locked out after editing aws-auth

After someone edited the aws-auth ConfigMap, kubectl returns 'Unauthorized' for everyone and new nodes can't join. Understand how EKS maps IAM identities, regain access with an identity that still works, repair the mapping, and move to EKS access entries so one YAML typo can't lock you out again.

Advanced
Key wordsEKSaws-auth ConfigMapaccess entriesauthentication modeUnauthorizednode role mappingcluster creatorbreak-glass accessIAM

The page

09:10 — the platform channel fills with error: You must be logged in to the server (Unauthorized). An engineer added a new team's role to aws-auth with kubectl edit and saved invalid YAML indentation. Nodes launched since then stay NotReady, and CI can't deploy. The cluster still uses the CONFIG_MAP authentication mode.

First five minutes

  • Impact: humans and CI locked out; new nodes can't join (existing kubelets may keep working for a while, but don't rely on it).
  • Stop changes: no more edits to aws-auth until one person owns recovery.
  • Find who can still get in.

The school keeps its list of who may enter on a sheet of paper taped inside the front door. Someone smudged it, and now the guard won't let anyone in, including the person who could fix the list. The better system keeps the list at the town hall (the EKS API) where one smudge can't lock everyone out.

How EKS decides who you are

  • IAM authenticates you (aws sts get-caller-identity); EKS maps that IAM identity to Kubernetes users/groups.
  • ConfigMap mode (legacy): the aws-auth ConfigMap in kube-system maps roles/users, including node roles (system:nodes). In this mode the cluster creator identity has admin access that doesn't come from aws-auth.
  • Access entries (API mode): mappings live in the EKS API (create-access-entry, access policies). API_AND_CONFIG_MAP allows both during migration.

Recover

  1. Find a working identity: the cluster creator role/user (if ConfigMap mode), or an existing access entry with cluster-admin permissions. (This is why break-glass access should be documented.)
  2. Fix the ConfigMap with that identity: restore the last good version from Git or a backup; validate YAML before applying.
$ aws sts get-caller-identity
$ kubectl -n kube-system get configmap aws-auth -o yaml > aws-auth.broken.yaml
$ kubectl apply -f aws-auth.good.yaml          # from Git / previous revision
$ kubectl get nodes                            # new nodes should turn Ready
  1. If no identity works and access entries can be enabled on your cluster version, an IAM administrator can switch to API_AND_CONFIG_MAP and create an admin access entry via the EKS API, then fix aws-auth from inside. (Check your EKS version and the current AWS documentation for the exact options.)

Harden

  • Migrate to access entries (API_AND_CONFIG_MAP, then API): manage access in Terraform/IaC with reviews, not kubectl edit.
  • Break-glass: a documented, tightly controlled admin role with an access entry, monitored for use (see Amazon EKS in Production with Terraform, lesson 05).
  • If aws-auth remains: manage it only via GitOps/IaC, validate in CI, and keep backups.
  • Alert on Unauthorized spikes in the API server audit logs.

Try it: break and fix access (sandbox)

  1. In a sandbox EKS cluster in ConfigMap mode, back up aws-auth, then add a mapping with invalid YAML using a second (non-creator) admin identity.
  2. Observe Unauthorized for that identity; confirm the creator identity still works.
  3. Restore aws-auth from the backup.
  4. Switch the cluster to API_AND_CONFIG_MAP and create an access entry for a team role with a namespaced access policy.
  5. Remove the team's aws-auth mapping and confirm access still works via the entry.

Going deeper: identity hygiene on EKS

  • Map groups/roles, never individual IAM users, and scope access policies by namespace where possible.
  • Keep node role mappings managed by EKS (managed node groups/Karpenter with access entries) rather than hand edits.
  • Review CloudTrail for CreateAccessEntry/AssociateAccessPolicy changes.

Recap

  • A bad aws-auth edit can lock out humans, CI and nodes in ConfigMap mode.
  • Recover with an identity that doesn't depend on aws-auth (cluster creator or an admin access entry), restore the mapping from Git.
  • Harden with access entries, IaC-managed access, and documented break-glass.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.