Multi-cluster & fleet management · Prasanta Kumar Behera

Why serious platforms run many clusters, how to create and upgrade them declaratively with Cluster API, and how to push the same platform configuration to every cluster with fleet GitOps.

A management cluster creates, upgrades and replaces many workload clusters.

One cluster or many?

A single cluster is simplest, until it isn't. Organisations end up with many clusters for good reasons:

Reason	Example
Blast radius	A bad upgrade or runaway controller hits one cluster, not the business
Environments	dev, staging and prod separated completely
Geography & latency	A cluster per region, or per edge site
Compliance	PCI or regulated workloads isolated with their own controls
Tenancy	Strong isolation for teams or customers (lesson 29)

The cost is N times the operations unless clusters are created, configured and upgraded the same way, automatically.

One giant classroom with 500 pupils is chaos: one sneeze and everyone's ill. A school has many classrooms with the same rules, timetable and furniture. The head office (management cluster) doesn't teach. It opens new classrooms from a standard plan and makes sure every room follows the same rules.

Cluster API: clusters as Kubernetes objects

Cluster API (CAPI) describes clusters with Kubernetes resources and reconciles them into real infrastructure through providers (AWS, Azure, vSphere, OpenStack, bare metal via Metal3, Docker for labs…):

Object	Describes
`Cluster`	The cluster as a whole, its network, and which providers to use
`KubeadmControlPlane`	Control-plane machines: count, version, kubeadm config
`MachineDeployment`	Worker pools, like a Deployment but for machines
`Machine`	One node, created and replaced by controllers

Upgrading becomes an edit: change version on the control plane, then the MachineDeployment, and controllers replace machines one by one (immutable nodes, lesson 25). Scaling is kubectl scale machinedeployment.

$ clusterctl describe cluster edge-042
NAME                                                  READY  SEVERITY  REASON
Cluster/edge-042                                      True
├─ClusterInfrastructure - DockerCluster/edge-042      True
├─ControlPlane - KubeadmControlPlane/edge-042-cp      True
│ └─3 Machines...                                     True
└─Workers
  └─MachineDeployment/edge-042-md-0                   True
    └─2 Machines...                                   True

(Output simplified.)

Other fleet managers

Rancher manages clusters it creates (RKE2/K3s) and imports others, with central RBAC and a UI. Its Fleet component does GitOps at scale.
EKS Anywhere uses Cluster API underneath, with a curated cluster spec.
Managed services (EKS, GKE, AKS) plus Terraform are the cloud equivalent.

One platform configuration, every cluster

Creating clusters is half the job. Each one also needs the same platform add-ons (CNI settings, ingress, cert-manager, monitoring, policies). Argo CD's ApplicationSet fans one template out to every matching cluster:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            env: prod              # every registered cluster with this label
  template:
    metadata:
      name: '{{name}}-monitoring'
    spec:
      project: default
      source:
        repoURL: https://github.com/example/platform.git
        targetRevision: main
        path: addons/monitoring
      destination:
        server: '{{server}}'
        namespace: monitoring
      syncPolicy:
        automated: {}

Add a new production cluster with the env: prod label and it receives monitoring automatically. More in GitOps with Argo CD.

Try it: your first management cluster

Follow the official Cluster API Quick Start with the Docker provider. It uses a kind cluster as the management cluster and creates workload clusters as containers.
Run clusterctl describe cluster and kubectl get machines to watch machines come up.
Scale workers: kubectl scale machinedeployment <name> --replicas 3, and watch a new Machine appear.
Delete a worker Machine object. The MachineDeployment replaces it: the same self-healing idea as pods, one level up.

Going deeper: fleet architecture decisions

Cluster boundaries: decide what a cluster is in your organisation (per env × region? per team? per site?) and write it down as an ADR. Changing it later is expensive.
Management cluster risk: it controls everything. Keep it small, highly available, tightly access-controlled and backed up. Workload clusters must keep running if it's down.
Cross-cluster traffic (multi-cluster Services, service mesh, global load balancing) adds real complexity. Prefer designs where most requests stay inside one cluster.
Identity and observability should be central from day one: one SSO for all clusters, and metrics and logs shipped to shared backends with a cluster label.

Recap

Many clusters limit blast radius and match environments, regions, compliance and tenants, at the cost of operations.
Cluster API makes clusters declarative objects managed from a management cluster; upgrades replace machines.
Fleet GitOps (ApplicationSets, Rancher Fleet) keeps every cluster's platform configuration identical.
Decide cluster boundaries deliberately, and centralise identity and observability.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.