GitOps fleet management · Prasanta Kumar Behera

Manage hundreds of edge clusters from one Git repository: a repo layout of platform base, site profiles and per-site values, cluster labels as targeting, pull-based agents (Fleet, Argo CD, Flux), rolling changes out in waves through canary sites, and knowing what every site runs.

One repo, many sites

fleet/
├── platform/                 # base for every site: CNI settings, monitoring agents, policies
│   ├── base/
│   └── profiles/             # sizes/types: small/, compact/, factory/
├── apps/                     # workloads by product, with their own bases
├── sites/
│   ├── site-001/values.yaml  # site number, IP ranges, VIP, profile, labels
│   ├── site-002/values.yaml
│   └── …
└── releases/
    ├── canary.yaml           # which platform/app versions each wave runs
    └── stable.yaml

Imagine 500 identical lemonade stands. There's one master recipe book (platform base), a few page inserts for big and small stands (profiles), and each stand has a sticky note with only its own details: address, number of cups (site values). When the recipe changes, the five test stands try it first (canaries). Only if customers there are happy does everyone else get the new page.

Targeting with labels

Every cluster registers with labels that describe it: site, region, size, profile, wave, bundle. Fleet tools select by label:

Rancher Fleet:

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: platform
  namespace: fleet-default
spec:
  repo: https://git.example.com/edge/fleet.git
  branch: main
  paths: [ platform ]
  targets:
    - name: canary
      clusterSelector:
        matchLabels: { wave: canary }

Argo CD uses ApplicationSets with a cluster generator (see GitOps with Argo CD, lesson 05); Flux typically bootstraps each cluster from its own path (clusters/site-042/) that references shared bases.

Rolling out in waves

Wave	Sites	Gate to next wave
lab	Test sites with representative hardware	Automated tests pass
canary	~1–5% of real sites, diverse hardware/regions	Healthy for 24–72 h (SLOs, no new alerts)
early	~20%	Healthy for a few days
all	Everything else	—

Implement waves by pinning revisions per wave (a tag, branch or version value that each wave's targets point at) and promoting by moving the pointer with a reviewed commit. Rollback is moving it back.

Know what runs where

Fleet/Argo/Flux report per-cluster sync status and revision; aggregate them into a fleet dashboard.
Also track bundle version, OS image version, Kubernetes version and firmware baseline per site (lesson 12).
Alert on sites that haven't synced for a long time (they may be offline or stuck), and on drift (manual changes corrected by the agent).

Disconnected and slow sites

Agents keep the last known desired state; workloads keep running while offline.
On reconnection, a site may jump several revisions. Keep changes backwards compatible, or make sites pass through required intermediate versions.
Mirror everything the manifests reference (lesson 07); GitOps can't sync images that aren't reachable.

Try it: a mini fleet

Create four kind clusters as "sites" and a management cluster with Rancher Fleet (standalone Fleet works without Rancher) or Argo CD.
Register the sites with labels: two with wave=canary, two with wave=stable, and different site numbers.
Build the repo layout above with a base, a profile and per-site values (e.g. a ConfigMap containing the site number).
Change the platform base and deliver it only to the canary wave; verify, then promote to stable with a second commit.
Disconnect one site (stop its container network) during a change; reconnect and watch it converge.

Going deeper: fleets of hundreds or thousands

Generate cluster registration labels from the site inventory, so targeting can't drift from reality.
Shard the fleet across several management instances by region to limit blast radius and load.
Treat platform releases like software releases: changelog, version, tests, staged rollout, rollback plan.
Watch the Git server and registry load when hundreds of sites poll; tune polling intervals or use webhooks and caches.

Recap

One repo: platform base + profiles + tiny per-site values + release pointers.
Labels describe clusters; Fleet, Argo CD ApplicationSets or Flux target by label.
Waves: lab → canary → early → all, gated on health; promote and roll back by moving a pointer.
Know what runs where, alert on unsynced sites, and design for disconnection.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.