Lesson 07 of 12 · Provisioning
Air-gapped delivery
Deliver everything a site needs without internet: a registry (Harbor) at the centre and optionally at each site, mirroring images and Helm charts as OCI artifacts, versioned and signed release bundles, and configuring nodes and distributions to pull only from your mirrors.
The problem
An air-gapped (or bandwidth-starved) site can't pull docker.io/..., registry.k8s.io/..., Helm charts, OS images or scanner databases on demand. Everything must be packaged, verified, transferred and served locally, and every component must be configured to use the local copies.
A research station in Antarctica can't pop to the shops. Before winter, a ship brings one big, sealed, labelled crate with everything for the season: food, spare parts, medicine, each item on a checklist. The station's pantry (local registry) stores it, and everyone is told: "only take from the pantry". If something's missing from the crate, you find out during the practice run at home, not in the middle of winter.
The registry: Harbor
Harbor (CNCF graduated) is a common choice:
- Projects with access control and robot accounts for automation.
- Replication rules between registries (centre → site) and proxy cache projects when upstream is reachable.
- Built-in vulnerability scanning (Trivy) and support for signatures (cosign/Notation) and other OCI artifacts.
Typical topology: a central Harbor fed by an import pipeline; a small Harbor (or plain distribution registry) at larger sites, replicated from the centre; tiny sites pull from the central one over the WAN, or get images pre-loaded.
Mirroring
$ skopeo copy --all docker://registry.k8s.io/pause:3.10 \
docker://harbor.central.example/mirror/registry.k8s.io/pause:3.10
$ helm pull oci://ghcr.io/example/charts/app --version 1.4.2
$ helm push app-1.4.2.tgz oci://harbor.central.example/charts
- Keep the upstream path in the mirror path (e.g.
mirror/registry.k8s.io/…) so mirror configuration stays simple. - Mirror by digest and with
--all(multi-arch). - Generate the image list from your rendered manifests (Helm template + Kustomize), not by hand.
- Tools such as Hauler and Zarf automate collecting images, charts and files into portable archives for air-gapped delivery.
Release bundles
Package each tested release as a bundle:
edge-release-2026.09/
├── manifest.yaml # versions, image digests, chart versions, OS image names
├── images/ # OCI archives
├── charts/ # .tgz Helm charts
├── os/ # ubuntu-2204-kube-1-31.raw.gz (+ .sha256)
├── cluster/ # cluster specs, GitOps repo snapshot
├── SHA256SUMS
└── SHA256SUMS.sig # cosign sign-blob (key held by the release team)
Inside the gap: verify the signature and checksums, import into the registry, and publish OS images to the provisioning HTTP server. Distributions provide their own helpers too (for example EKS Anywhere's download artifacts, download images and import images commands).
Configure consumers
RKE2/K3s (/etc/rancher/rke2/registries.yaml):
mirrors:
docker.io:
endpoint: [ "https://harbor.site042.example" ]
registry.k8s.io:
endpoint: [ "https://harbor.site042.example" ]
configs:
"harbor.site042.example":
tls:
ca_file: /etc/rancher/rke2/harbor-ca.crt
(Add rewrite rules if your mirror paths differ from upstream paths.)
EKS Anywhere cluster spec (excerpt):
spec:
registryMirrorConfiguration:
endpoint: harbor.site042.example
port: "443"
authenticate: true
caCertContent: |
-----BEGIN CERTIFICATE-----
…
-----END CERTIFICATE-----
Also check: Helm repositories in GitOps (point at the internal OCI registry), OS package mirrors, NTP, DNS, and the CA certificate trusted by every node.
Try it: an offline release
- Run a local registry (Harbor, or
registry:2for a quick lab) with TLS from your own CA. - Render a small app's Helm chart, extract the image list, and mirror the images with
skopeo copy --allby digest. - Build a bundle directory with a manifest,
SHA256SUMS, and a cosign signature (cosign sign-blob --key …). - On a VM with egress blocked, verify the signature, load the images into its registry, and install the app with registries configured as mirrors.
- Remove one image from the bundle and repeat: confirm the failure is loud and obvious.
Going deeper: supply chain for disconnected sites
- Keep signatures and SBOMs with the images (oras/cosign copy them as OCI artifacts), and verify at admission inside the gap.
- Size bundle transfer for your medium: delta bundles (only changed images) for upgrades.
- Track which bundle version each site runs; fleet dashboards should show it.
- Import scanner vulnerability databases on a schedule so scans inside the gap stay meaningful.
Recap
- Air-gapped = package, verify, transfer, serve locally, and configure everything to use the local copies.
- Harbor for images, charts and signatures; replicate centre → sites.
- Mirror by digest, multi-arch, from rendered manifests; automate with tools like Hauler/Zarf.
- Ship signed, versioned release bundles; configure registries.yaml, registryMirrorConfiguration or containerd mirrors.
- Test in a truly disconnected environment.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.