Real pipelines · Prasanta Kumar Behera

Build pipelines you'd run in production: matrix builds, reusable workflows and composite actions to avoid copy-paste, container builds that output a digest, environments with approvals for deployment, and handing the result to GitOps.

Matrix builds

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false            # see all failures, not just the first
      matrix:
        os: [ubuntu-latest]
        python: ["3.11", "3.12", "3.13"]
        include:
          - os: ubuntu-latest
            python: "3.12"
            coverage: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "${{ matrix.python }}", cache: pip }
      - run: pip install -r requirements.txt && pytest -q

A matrix is baking the same cookie recipe in three ovens at once to check it works in all of them. A reusable workflow is a recipe the whole school shares: every class uses the same one instead of copying it by hand, and when it's improved, everyone gets the improvement.

Reuse instead of copy-paste

A central repo (org/ci) holds a reusable workflow:

# org/ci/.github/workflows/container.yml
on:
  workflow_call:
    inputs:
      image: { type: string, required: true }
    outputs:
      digest:
        value: ${{ jobs.build.outputs.digest }}

jobs:
  build:
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/${{ github.repository_owner }}/${{ inputs.image }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

A service repo calls it in a few lines:

jobs:
  image:
    uses: org/ci/.github/workflows/container.yml@v1
    with:
      image: shop-api
    permissions: { contents: read, packages: write }

Version the shared workflows (tags like v1) so a change doesn't break every repo at once. Use composite actions for shorter shared step sequences (setup + lint, for example).

Deploying: environments and GitOps

  deploy-prod:
    needs: image
    runs-on: ubuntu-latest
    environment: prod              # required reviewers, branch rules, env secrets
    concurrency: { group: deploy-prod, cancel-in-progress: false }
    steps:
      - uses: actions/checkout@v4
        with: { repository: org/gitops, token: "${{ secrets.GITOPS_TOKEN }}" }
      - run: |
          git config user.name "ci-bot" && git config user.email "ci-bot@example.com"
          cd shop/overlays/prod
          kustomize edit set image shop-api=ghcr.io/org/shop-api@${{ needs.image.outputs.digest }}
          git commit -am "shop-api: deploy ${{ needs.image.outputs.digest }}" && git push

With GitOps (see GitOps with Argo CD), CI's job ends at updating Git, and Argo CD applies it. (A GitHub App token is better than a personal token for pushing to the GitOps repo.)

Monorepos and speed

Path filters (on.push.paths) run a service's pipeline only when its files change.
Split slow suites into parallel jobs or matrix shards.
Cache dependencies and Docker layers (type=gha or a registry cache).

Try it: a reusable, gated pipeline

Create an org-ci repo (a personal account works) with the reusable container workflow.
In an app repo, call it and print the returned digest in a follow-up job.
Add a matrix test job for three language versions with fail-fast: false; break one version on purpose.
Create a prod environment with yourself as a required reviewer; add a deploy job that just echoes the digest, and approve it.
Add path filters so documentation-only changes skip the build.

Going deeper: pipeline design

Keep pipelines boring and fast: one clear build path, cached, parallel, under ~10 minutes for PRs.
Separate build once from deploy many: the same digest goes to dev, staging and prod.
Treat the shared CI repo as a product: versioned, tested (workflows can have their own CI), changelog.
Protect main with required checks so nothing merges without green CI.

Recap

Matrix for versions and platforms; fail-fast: false to see every failure.
Reusable workflows (whole jobs) and composite actions (steps) replace copy-paste; version them.
Build, push and capture the digest; deploy that digest.
Environments add approvals and scoped secrets; with GitOps, CI ends by committing to Git.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.