Partial rollouts · Prasanta Kumar Behera

When a deploy reaches most, but not all, of the fleet: why partial rollouts happen, how mixed versions break things that assumed 'everyone upgraded', detecting incomplete rollouts with version inventory, and designing changes (expand/contract, flags, gates) that are safe while versions are mixed.

How partial rollouts happen

A rollout reaches 95% of the fleet and quietly stops for the rest:

A few clusters or edge sites are offline or their GitOps agent is stuck.
Pods on some nodes fail to start (image pull, resource limits, node selectors, admission policy differences).
A DaemonSet rollout halts on nodes where the new pod can't become ready (maxUnavailable reached).
A canary step paused and nobody resumed or aborted it.
Some workloads were pinned (a different values file, a forgotten override).

The teacher tells the class, "From tomorrow, hand in homework in the blue folder instead of the red one." Most children hear it. Two were absent. If the teacher throws away the red folder tomorrow, those two lose their homework, not because the blue folder is wrong, but because she assumed everyone had switched.

The real danger: "everyone has upgraded"

The new version isn't the problem. The problem is the next change that assumes it's everywhere:

Assumption	What breaks on the old instances
Drop the old database column	Old app version queries fail
Producers send the new message format only	Old consumers can't parse messages
Remove the deprecated API endpoint	Old clients get 404s
Rotate to a new certificate/CA only	Old instances without the new CA fail TLS

A generic scenario

A platform team rolled out a new version of a node agent to 400 clusters via GitOps. A week later, they removed the old metrics endpoint from the central collector. Twelve edge clusters, which had been offline during the rollout and then synced to an older pinned revision because of a stale override, lost all monitoring. Nobody noticed for two days because "no alerts" looked like "healthy".

What was missing: a version inventory before the contract step, an alert on missing telemetry (absence), and removal of the stale override.

Detecting incomplete rollouts

Inventory from metrics: kube_pod_container_info (kube-state-metrics) gives the image per container; aggregate by cluster/site.
GitOps status: per-cluster sync status and revision (Argo CD, Fleet, Flux).
Rollout completion SLO: "95% of clusters on the new version within 48 h, 100% within 7 days, or an explicit exception".
Absence alerts: a cluster that stops reporting is not a healthy cluster.

Designing for mixed versions

N and N-1 must interoperate, always: APIs, schemas, message formats, certificates.
Expand → migrate → contract: add new columns/fields/endpoints first; move writers and readers; only after verified completion, remove the old.
Feature flags decouple deploy from release: ship code everywhere, enable the behaviour only when the rollout is complete (and roll back instantly by flag).
Gates in automation: the contract step's pipeline checks the inventory and refuses to run if any old version remains.

Try it: find the stragglers

In a kind cluster, deploy a DaemonSet with a new image version that fails readiness on one node (e.g. a node selector or a taint mismatch); watch the rollout stall with kubectl rollout status.
Install kube-state-metrics (or use kube-prometheus-stack) and query kube_pod_container_info by image to find the straggler.
Simulate expand/contract with a tiny app and a SQLite/PostgreSQL table: add a column, deploy the new version, then drop the old column only after verifying no old pods remain.
Write an alert for "cluster hasn't reported metrics for 15 minutes".
Draft a rollout completion SLO for your platform.

Going deeper: fleet version hygiene

Publish a fleet version dashboard (component × version × cluster count) and review it weekly.
Set support windows (only N and N-1 supported) and treat older versions as incidents to resolve.
Include batch jobs, CronJobs and edge sites in inventories; they're the usual stragglers.

Recap

Partial rollouts happen through offline sites, stuck pods, halted DaemonSets, paused canaries and pinned overrides.
The danger is the next change assuming everyone upgraded.
Detect with version inventory, GitOps status, completion SLOs and absence alerts.
Design for N/N-1 compatibility: expand/migrate/contract, feature flags, and gates that check inventory.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.