Upgrades & production patterns

Upgrading Istio with revisions

            istiod 1-25-2 (old)          istiod 1-26-0 (new)
tag prod-stable ──► 1-25-2   ─────────►  tag prod-stable ──► 1-26-0
namespaces labelled istio.io/rev=prod-stable follow the tag; restart pods to switch proxies

1. Read the release notes and the supported upgrade paths (typically one minor at a time; check whether canary upgrades allow skipping).
2. istioctl x precheck, then install the new revision alongside the old one.
3. Move a canary namespace: label it with the new revision (or move a revision tag), restart its workloads, verify (golden signals, istioctl proxy-status).
4. Move the tag used by everyone to the new revision; restart namespaces in waves.
5. Upgrade gateways (they're Deployments; roll them like apps).
6. Remove the old revision when no proxies use it.

For ambient mode, ztunnel and the CNI node agent are node-level components: upgrading them affects all ambient pods on each node, so follow the ambient-specific upgrade guide and roll node by node.

Upgrading Linkerd

$ linkerd check
$ linkerd upgrade --crds | kubectl apply -f -
$ linkerd upgrade | kubectl apply -f -      # or: helm upgrade linkerd-crds … && helm upgrade linkerd-control-plane …
$ linkerd check
$ kubectl -n shop rollout restart deploy
$ linkerd check --proxy

Upgrade extensions (viz, multicluster) too, and follow the upgrade notes for your release channel.

Limiting blast radius

• Adopt incrementally: one namespace at a time; permissive mTLS first; policies after telemetry proves traffic flows are understood.
• Stage config changes: mesh-wide resources (root-namespace PeerAuthentication, mesh config) only through reviewed GitOps changes, applied to non-production clusters first.
• Separate gateways per traffic class so one bad route doesn't break all ingress.
• Keep NetworkPolicies and application-level safeguards; the mesh isn't the only layer.

Sizing and startup

• Set requests/limits for proxies based on measured usage (concurrency and config size drive memory). In sidecar mode, multiply by pod count when planning capacity.
• Handle startup/shutdown ordering: hold the app until the proxy is ready; use native sidecar support (Kubernetes sidecar containers) where your mesh version supports it, which also fixes Jobs that never complete.
• Reduce configuration pushed to each proxy (Istio Sidecar resources, discovery selectors) in large meshes.

Operate it as tier-0

• SLOs for the mesh itself: control-plane availability, config push latency, proxy error rates.
• Alerts: control plane down, xDS push errors, certificate expiry (issuer/root), proxies on unsupported versions.
• Runbooks: "traffic failing after a policy change", "proxies not receiving config", "certificate expired", and the escape hatch (take a namespace out of the mesh, or relax STRICT mTLS, with approvals).
• Game days: kill istiod/destination pods, expire a test certificate in staging, roll back a revision.

Recap

• Istio: revision-based canary upgrades with tags; move namespaces in waves; restart to update proxies; remove old revisions.
• Linkerd: linkerd upgrade (or Helm), then restart workloads and linkerd check --proxy.
• Limit blast radius: incremental adoption, staged mesh-wide config, separate gateways, NetworkPolicies.
• Size proxies, handle startup ordering (native sidecars), and run the mesh as tier-0 with SLOs, runbooks and a tested escape hatch.