Lesson 10 of 18 · Automate & Recover
CI/CD for the stack
Deliver infrastructure changes safely: Terraform plans on every pull request, applies only from the main branch after approval, AWS access through OIDC (no stored keys), separate plan and apply roles, static checks, and a nightly drift job.
The flow
feature branch ──PR──► CI: fmt, validate, tflint, checkov → terraform plan (read-only role) → plan posted on the PR
review + approve ──merge to main──► CI: plan again → environment approval → apply (write role)
nightly: plan -detailed-exitcode → alert on drift
Changing the city's roads works like this: an engineer draws the plan (pull request), the computer checks it for obvious mistakes (linters), shows exactly what would change (plan), a senior engineer signs it off (approval), and only then does the road crew get the keys to start work (apply), using keys that only work for that day's job (OIDC).
OIDC from GitHub Actions to AWS
- Create an IAM OIDC identity provider for
token.actions.githubusercontent.comin the target account. - Create roles that trust it, conditioned on repository and ref/environment:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/platform-infra:environment:prod"
}
}
}
]
}
tf-planrole: read-only (plus state read and lock), trusted for pull requests.tf-apply-prodrole: write, trusted only for theprodenvironment (which requires approval in GitHub).
The workflow (outline)
name: infra-prod
on:
pull_request:
paths: ["live/prod/**", "modules/**"]
push:
branches: [main]
paths: ["live/prod/**", "modules/**"]
permissions:
id-token: write
contents: read
pull-requests: write
jobs:
plan:
runs-on: ubuntu-latest
defaults: { run: { working-directory: live/prod/cluster } }
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::111122223333:role/tf-plan
aws-region: eu-west-1
- run: terraform fmt -check -recursive
- run: terraform init -input=false
- run: terraform validate
- run: terraform plan -input=false -var-file=prod.tfvars -out=tfplan
# post the plan as a PR comment (e.g. with a small script or a marketplace action), upload tfplan as an artifact
apply:
if: github.event_name == 'push'
needs: plan
runs-on: ubuntu-latest
environment: prod # requires approval; matches the role's trust condition
defaults: { run: { working-directory: live/prod/cluster } }
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::111122223333:role/tf-apply-prod
aws-region: eu-west-1
- run: terraform init -input=false
- run: terraform plan -input=false -var-file=prod.tfvars -out=tfplan
- run: terraform apply -input=false tfplan
(Adapt it: many teams re-plan in the apply job as shown; others pass the reviewed plan artifact through. Pin actions to commit SHAs for supply-chain safety.)
Guard-rails in the pipeline
- Static checks:
tflint, andcheckovortrivy configfor misconfigurations (public buckets, open security groups, unencrypted volumes). - Policy as code: OPA/Conftest rules on the plan JSON (
terraform show -json tfplan), e.g. "no deletion of EKS clusters without a label". - Concurrency: one apply per environment at a time (
concurrency:group in GitHub Actions), plus state locking. - Drift job: nightly
plan -detailed-exitcode; alert on exit code 2.
Try it: a safe pipeline (sandbox account + a GitHub repo)
- Create the OIDC provider and two roles (
tf-plan,tf-apply-dev) with trust conditions for your repository. - Add the workflow for a small stack (e.g. an S3 bucket); open a PR and check the plan runs with the read-only role.
- Try to make the PR job apply (change the workflow in the PR). It should fail on permissions.
- Merge, approve the environment, and watch apply run with the write role.
- Add
checkovand make it fail on a public-read bucket, then fix it.
Going deeper: platform delivery
- Use a per-environment workflow (or matrix) with waves: dev → staging → prod, each with its own role and approval.
- Protect
mainwith required reviews and status checks; restrict who can approve production environments. - Keep module releases versioned (Git tags); environments reference versions, so upgrades are explicit.
- Tools such as Atlantis or Terraform Cloud/Enterprise provide PR-driven workflows with locking and policy built in.
Recap
- Plan on PR (read-only role) → review → apply on main (write role, environment approval).
- OIDC federation: no stored keys; trust conditioned on repo and environment.
- Static checks and policy as code in the pipeline; one apply at a time; nightly drift detection.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.