Amazon EKS in Production with Terraform›10 · CI/CD for the stack
Learning Hub / Cloud — OpenStack, AWS & EKS / Amazon EKS in Production with Terraform

Lesson 10 of 18 · Automate & Recover

CI/CD for the stack

Deliver infrastructure changes safely: Terraform plans on every pull request, applies only from the main branch after approval, AWS access through OIDC (no stored keys), separate plan and apply roles, static checks, and a nightly drift job.

Advanced
Key wordsGitHub ActionsOIDC federationplan on PRapply on mergeenvironmentsseparate rolestflintcheckovdrift job

The flow

feature branch ──PR──► CI: fmt, validate, tflint, checkov → terraform plan (read-only role) → plan posted on the PR
        review + approve ──merge to main──► CI: plan again → environment approval → apply (write role)
nightly: plan -detailed-exitcode → alert on drift

Changing the city's roads works like this: an engineer draws the plan (pull request), the computer checks it for obvious mistakes (linters), shows exactly what would change (plan), a senior engineer signs it off (approval), and only then does the road crew get the keys to start work (apply), using keys that only work for that day's job (OIDC).

OIDC from GitHub Actions to AWS

  1. Create an IAM OIDC identity provider for token.actions.githubusercontent.com in the target account.
  2. Create roles that trust it, conditioned on repository and ref/environment:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:acme/platform-infra:environment:prod"
        }
      }
    }
  ]
}
  • tf-plan role: read-only (plus state read and lock), trusted for pull requests.
  • tf-apply-prod role: write, trusted only for the prod environment (which requires approval in GitHub).

The workflow (outline)

name: infra-prod
on:
  pull_request:
    paths: ["live/prod/**", "modules/**"]
  push:
    branches: [main]
    paths: ["live/prod/**", "modules/**"]

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  plan:
    runs-on: ubuntu-latest
    defaults: { run: { working-directory: live/prod/cluster } }
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-plan
          aws-region: eu-west-1
      - run: terraform fmt -check -recursive
      - run: terraform init -input=false
      - run: terraform validate
      - run: terraform plan -input=false -var-file=prod.tfvars -out=tfplan
      # post the plan as a PR comment (e.g. with a small script or a marketplace action), upload tfplan as an artifact

  apply:
    if: github.event_name == 'push'
    needs: plan
    runs-on: ubuntu-latest
    environment: prod                 # requires approval; matches the role's trust condition
    defaults: { run: { working-directory: live/prod/cluster } }
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-apply-prod
          aws-region: eu-west-1
      - run: terraform init -input=false
      - run: terraform plan -input=false -var-file=prod.tfvars -out=tfplan
      - run: terraform apply -input=false tfplan

(Adapt it: many teams re-plan in the apply job as shown; others pass the reviewed plan artifact through. Pin actions to commit SHAs for supply-chain safety.)

Guard-rails in the pipeline

  • Static checks: tflint, and checkov or trivy config for misconfigurations (public buckets, open security groups, unencrypted volumes).
  • Policy as code: OPA/Conftest rules on the plan JSON (terraform show -json tfplan), e.g. "no deletion of EKS clusters without a label".
  • Concurrency: one apply per environment at a time (concurrency: group in GitHub Actions), plus state locking.
  • Drift job: nightly plan -detailed-exitcode; alert on exit code 2.

Try it: a safe pipeline (sandbox account + a GitHub repo)

  1. Create the OIDC provider and two roles (tf-plan, tf-apply-dev) with trust conditions for your repository.
  2. Add the workflow for a small stack (e.g. an S3 bucket); open a PR and check the plan runs with the read-only role.
  3. Try to make the PR job apply (change the workflow in the PR). It should fail on permissions.
  4. Merge, approve the environment, and watch apply run with the write role.
  5. Add checkov and make it fail on a public-read bucket, then fix it.

Going deeper: platform delivery

  • Use a per-environment workflow (or matrix) with waves: dev → staging → prod, each with its own role and approval.
  • Protect main with required reviews and status checks; restrict who can approve production environments.
  • Keep module releases versioned (Git tags); environments reference versions, so upgrades are explicit.
  • Tools such as Atlantis or Terraform Cloud/Enterprise provide PR-driven workflows with locking and policy built in.

Recap

  • Plan on PR (read-only role) → review → apply on main (write role, environment approval).
  • OIDC federation: no stored keys; trust conditioned on repo and environment.
  • Static checks and policy as code in the pipeline; one apply at a time; nightly drift detection.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.