Prasanta Kumar Behera
SRE & Production Incident Response›03 · Architecture antipatterns
Learning Hub / Observability & Reliability / SRE & Production Incident Response

Lesson 03 of 7 · Method

Architecture antipatterns

Recognise designs that make outages possible: hidden single points of failure (registries, DNS, identity, time), shared fate, deep synchronous chains, health checks that cascade, data paths that depend on the control plane, global config pushes, and unbounded retries and queues.

Advanced
Key wordssingle points of failureshared fatesynchronous call chainshidden dependenciesglobal config pushliveness probes on dependenciescontrol plane in data pathunbounded queuesregistry outageimagePullPolicy

Designs that make outages possible

Most large outages aren't caused by one broken part; they're caused by designs that let one broken part spread. Recognising these patterns in reviews is one of the most valuable SRE skills.

A row of dominoes is fine until one falls; then they all go. Many system designs are secret domino rows: one small thing tips, and everything follows. SRE work is spotting the dominoes and spacing them out so one falling doesn't knock over the rest.

Hidden single points of failure

Things everything depends on, often forgotten because they "always work": container registry, DNS, identity provider/SSO, NTP, certificate authority, Git server (for GitOps), secrets manager, licence servers, one load balancer pair.

For each: what happens when it's down for an hour? What can't start, deploy or log in?

Scenario: the registry went down, and so did every deploy

The central container registry's storage failed. Deploys failed immediately (expected). Then, over the next hour, a node failure and routine autoscaling needed new pods: they sat in ImagePullBackOff, and a node pool upgrade that started automatically replaced nodes whose new instances had no cached images. Capacity drained away.

What would have contained it?
  • Pull-through caches / mirrors per cluster or site (see Edge Kubernetes & Zero-Touch Provisioning, lesson 07), with the registry itself highly available.
  • Digests with imagePullPolicy: IfNotPresent, so restarts on existing nodes use cached images.
  • Pre-pulling critical images onto new nodes (node images or DaemonSets) for system components.
  • Pause automation (node upgrades, scale-downs) automatically when core dependencies are unhealthy.
  • A dependency map showing the registry as a tier-0 dependency, with its own SLO and alerts.

More antipatterns

Antipattern Why it hurts Safer pattern
Liveness probes on dependencies Dependency blip → mass restarts Liveness = process health; readiness with care
Data path depends on the control plane API server slow → app requests fail Watches/caches; apps work without kube-API access
Global config push One bad value, every cluster at once Waves with health gates (see GitOps with Argo CD, lesson 07)
Deep synchronous chains Latency and failure multiply per hop Timeouts, async messaging, graceful degradation
Unbounded retries/queues Overload amplifies; recovery takes hours Budgets, backoff, bounded queues, load shedding
Shared fate "Redundant" copies fail together Separate failure domains, staggered versions/changes
No backpressure Producers overwhelm consumers Rate limits, queue limits, 429s
Stateful everything in one place One storage problem, all apps down Separate storage per criticality; tested restores

Using this in design reviews

For each new design, ask:

  1. What are the tier-0 dependencies, and what happens when each is down for an hour?
  2. Which probes could restart everything at once?
  3. Which changes are applied globally at once?
  4. How many synchronous hops does a user request cross, and what are the timeouts?
  5. Where are retries, and could they multiply?
  6. Which "redundant" components share power, network, config or software version?

Try it: find the dominoes

  1. In a kind cluster, deploy an app whose liveness probe calls a database Service; scale the database to 0 and watch the app pods restart in a loop. Fix the probe.
  2. Deploy an app with imagePullPolicy: Always, block access to the registry (e.g. a NetworkPolicy or offline kind), delete a pod, and watch ImagePullBackOff. Repeat with a digest and IfNotPresent.
  3. List your platform's tier-0 dependencies and what breaks when each is down.
  4. Draw one user request's synchronous chain and annotate timeouts at each hop.
  5. Pick one antipattern you found and write an ADR for the fix.

Going deeper: reliability reviews

  • Keep a dependency map with tiers (power/network → DNS/NTP → storage/registry/identity → platform → apps), and check that nothing depends on a higher tier to start (lesson 06).
  • Run "dependency down" game days for each tier-0 dependency.
  • Add antipattern checks to automated policy where possible (probe lint rules, required timeouts, pull policies).

Recap

  • Outages spread through designs: hidden SPOFs, shared fate, cascading probes, control-plane-dependent data paths, global pushes, deep sync chains, unbounded retries.
  • Treat registry, DNS, identity, NTP, CA, Git, secrets as tier-0.
  • Liveness probes check the process, not dependencies.
  • Ask the six design-review questions every time.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.