GitOps Principles & Practice›03 · Promotion between environments

Lesson 03 of 7 · Level 1 — The model

Promotion between environments

Move a release from dev to staging to production entirely through Git: what exactly gets promoted, automated pull requests per environment, automatic updates for dev, approval and checks for production, promotion tooling, and rolling back by reverting.

Practitioner → Advanced
Key wordspromotionenvironmentsdev staging prodimage tag bumpdigestpull request promotionauto-mergeArgo CD Image UpdaterFlux image automationRenovateKargopromotion gates

Promote artifacts, not code

Build once; promote the same image digest (and chart version) from environment to environment. The only thing that changes between environments is configuration: replicas, resources, endpoints, feature flags. A rebuild for production would mean production runs an image nobody tested.

A school play: the same actors rehearse in the classroom, then in the hall, then perform on the big night. You don't swap in new actors for the final show; you move the same cast to a bigger stage.

The flow

CI: build + push digest D
 └─► PR: dev    ← D   (auto-merge)            → agent deploys to dev   → tests, smoke checks
      └─► PR: staging ← D (auto-merge or 1 approval) → agent deploys   → soak, SLO checks
           └─► PR: prod ← D (approval + checks)       → agent deploys   → monitoring, rollback ready

Each arrow is a pull request against the config repository that changes one line (the digest) in one environment's overlay or values file. The PR description links to the build, the test results and the previous environment's health.

Automating the first steps

Tool What it does Typical scope
CI job (GitHub Actions, Jenkins) Opens the promotion PR after a successful build (see "GitHub Actions — Level by Level", lesson 04) Dev (and further with gates)
Argo CD Image Updater / Flux image automation Watches the registry and writes new tags/digests back to Git Dev, preview environments
Renovate Opens PRs for new image or chart versions across repos Third-party charts and images
Kargo (and similar promotion tools) Models stages and promotes freight (image + chart + config) between them with verification Multi-stage pipelines at scale

Keep production changes as explicit, reviewed PRs even when everything before it is automated.

Gates before production

A promotion PR to prod should only be mergeable when:

  • The same digest has been healthy in staging for an agreed soak time.
  • Automated checks pass: smoke/integration tests, policy checks on the rendered manifests (Kyverno/conftest), vulnerability status.
  • SLOs in staging are green (error rate, latency).
  • The right approvers (CODEOWNERS for envs/prod/) have approved.
  • It isn't inside a change freeze window (enforce with a check, not a memo).

Progressive delivery (lesson 07) adds automated checks during the production rollout as well.

Rolling back

$ git log --oneline -- apps/orders-api/envs/prod/
7c1d2e3 orders-api: promote 1.4.0 to prod
5b4a3f2 orders-api: promote 1.3.2 to prod
$ git revert 7c1d2e3 && git push           # through a fast-track PR if prod is protected

The agent sees the previous digest in Git and deploys it. Avoid kubectl rollout undo in GitOps-managed environments: the agent would put the "bad" version straight back, because Git still says so.

Keep environments from drifting

If staging and prod overlays each accumulate their own hand edits, promotions stop meaning much. Keep overlays minimal, review diff between environments regularly, and put anything common into the base.

Try it: three environments

  1. Extend the layout from lesson 02 with dev, staging and prod overlays, each deployed by the agent to its own namespace on a kind cluster.
  2. Promote a new digest to dev via a CI-created PR (or by hand), then to staging, then to prod with a required approval.
  3. Roll back prod with git revert and watch the agent restore the previous version.
  4. Try kubectl rollout undo instead and see self-heal reapply Git's version.

Recap

  • Build once, promote the digest; environments differ only by configuration.
  • Each promotion is a PR to one environment in the config repo; automate dev, gate prod.
  • Tools: CI-created PRs, Image Updater/Flux automation, Renovate, Kargo.
  • Roll back with git revert, not kubectl.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.