Lesson 12 of 15 · Workload & Supply Chain
Supply chain security
Know and trust what you run: minimal base images, SBOMs, vulnerability scanning, signing images and attestations with cosign, pinning by digest, and refusing unsigned images at admission.
What "supply chain" means
Your running containers are built from base images, OS packages, language libraries, build tools and CI systems you don't fully control. Attacks increasingly target these links: poisoned packages, compromised build systems, swapped images. Supply-chain security answers three questions:
- What's inside? → SBOMs and minimal images.
- Is it vulnerable? → scanning, continuously.
- Is it really ours, unchanged? → signing, provenance, and verification at deploy time.
Think of a sandwich shop. You want a list of every ingredient (SBOM), to check none are on the recall list (scanning), and a sealed sticker from your own kitchen on every box (signature), so nobody can swap in a sandwich from somewhere else. The shop door (admission) refuses any box without your sticker.
1. Start small: minimal base images
Fewer packages = fewer vulnerabilities and less for an attacker to use. Prefer distroless, Alpine or "slim" bases, multi-stage builds, and a non-root user. A Go or Java service rarely needs a shell, curl or a package manager at runtime.
2. SBOMs and scanning
$ syft registry.example.com/shop/api:1.4.2 -o spdx-json > sbom.json
$ trivy image --severity HIGH,CRITICAL registry.example.com/shop/api:1.4.2
registry.example.com/shop/api:1.4.2 (debian 12)
Total: 3 (HIGH: 2, CRITICAL: 1)
┌──────────────┬────────────────┬──────────┬───────────────────┬───────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┤
│ libexample │ CVE-2026-XXXXX │ CRITICAL │ 1.2.3-1 │ 1.2.3-2 │
└──────────────┴────────────────┴──────────┴───────────────────┴───────────────┘
(Illustrative output.) Scan in CI (fail the build on fixable criticals), in the registry (Harbor can scan on push and block pulls), and continuously for running images, because new CVEs appear after deployment.
3. Sign images and attestations
cosign (from the Sigstore project) signs images and stores signatures in the registry next to them. Always sign the digest:
$ DIGEST=$(crane digest registry.example.com/shop/api:1.4.2)
$ cosign sign --key cosign.key registry.example.com/shop/api@$DIGEST
$ cosign verify --key cosign.pub registry.example.com/shop/api@$DIGEST
Verification for registry.example.com/shop/api@sha256:… --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Key-pair signing: keys in a KMS or your CI secret store.
- Keyless signing: CI's OIDC identity is exchanged for a short-lived certificate (Sigstore Fulcio), and the signature is recorded in a transparency log (Rekor). No long-lived keys to leak.
Attestations are signed statements about an image: its SBOM, its vulnerability scan, and its build provenance (which repo, commit and pipeline produced it, the core of SLSA):
$ cosign attest --key cosign.key --type spdxjson --predicate sbom.json registry.example.com/shop/api@$DIGEST
4. Verify at admission
Signing is pointless unless the cluster refuses unsigned images. Kyverno example:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-shop-images
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 30
rules:
- name: require-signature
match:
any:
- resources:
kinds: ["Pod"]
verifyImages:
- imageReferences:
- "registry.example.com/shop/*"
mutateDigest: true # rewrite tags to the verified digest
attestors:
- entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
<your cosign.pub>
-----END PUBLIC KEY-----
Sigstore's policy-controller and Gatekeeper-based tools are alternatives.
5. Control where images come from
- Pull only from your registry (a Harbor or cloud registry that mirrors and scans upstream images). Enforce it with an admission policy on image prefixes.
- Air-gapped sites: import images with their signatures and SBOMs, and verify on import.
- Pin base images by digest in Dockerfiles, and update them deliberately (automated PRs with tools such as Renovate).
Try it: sign, verify, enforce (kind)
- Run a local registry (
docker run -d -p 5001:5000 --name kind-registry registry:2), push a small image, and get its digest. - Generate an SBOM with syft and scan the image with trivy.
cosign generate-key-pair, sign the digest, and verify it.- Install Kyverno and apply a
verifyImagespolicy for your registry prefix. - Deploy the signed image (allowed), then push an unsigned tag and try to deploy it (rejected). Read the rejection message.
(Connecting kind to a local registry needs a small containerd config patch; the kind documentation has a "local registry" guide.)
Going deeper: SLSA and beyond
- SLSA levels describe how trustworthy provenance is: from "provenance exists" up to "built on a hardened, isolated build platform". Aim for signed provenance generated by the CI system itself, not by scripts developers can change.
- Verify provenance at admission (e.g. "built from github.com/example/shop on the main branch"), not only "signed by some key".
- Track VEX statements (vulnerability exploitability) to record which CVEs don't affect you, which reduces scanner noise.
- Treat CI as production: pinned actions and runners, least-privilege tokens, OIDC to clouds (see CI/CD & Software Supply Chain).
Recap
- Minimal, non-root base images; build with multi-stage Dockerfiles.
- SBOMs (syft) and scanning (trivy) in CI, in the registry and continuously.
- Sign by digest with cosign (key-pair or keyless); add attestations (SBOM, provenance).
- Verify at admission (Kyverno, policy-controller) and restrict image sources to your registry.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.