Cold-start dependency loops

The problem nobody tests

Platforms usually start incrementally: first the cluster, then the registry, then SSO, then GitOps, all while the previous pieces were already running. Years later, a full outage (power loss, a network event, a bad global change) forces everything to start at once, and hidden loops appear:

• The registry runs in the cluster; pods (including the registry) can't pull images.
• The identity provider runs on the platform; nobody can log in to fix it.
• GitOps pulls from a Git server hosted on the same platform.
• DNS runs as pods whose images come from the in-cluster registry.
• The secrets manager must be unsealed by a process that itself needs secrets.

Build a dependency map in tiers

Tier	Examples	May depend on
0	Power, network, time (NTP), DNS for infrastructure	Nothing above
1	Storage, container registry/mirrors, secrets manager, identity/break-glass access	Tier 0
2	Kubernetes control planes, GitOps, observability	Tiers 0–1
3	Platform services (ingress, service mesh, databases)	Tiers 0–2
4	Applications	Tiers 0–3

Any arrow pointing upwards (a lower tier needing a higher one) is a potential cold-start loop. Break it: host the registry mirror outside the clusters it serves (or pre-load images), keep break-glass credentials that don't need SSO, keep a Git mirror outside the platform, document the manual unseal path.

Stateful clusters after unclean shutdown

What's happening, and how do you recover and prevent it?

What's happening. A RabbitMQ node that restarts waits for the cluster peers it knew when it stopped (for a limited time, with retries), because the last node to stop has the most recent state. The StatefulSet uses the default OrderedReady policy: pod-1 isn't created until pod-0 is Ready, and pod-0 isn't Ready until it sees its peers. Deadlock.

Recover.

1. Allow all members to start: set podManagementPolicy: Parallel (this field can't be changed on an existing StatefulSet; recreate it with --cascade=orphan to keep pods and PVCs, or use the RabbitMQ Cluster Operator, which uses Parallel).
2. If a node still can't find peers and you've determined which node has the most recent data, RabbitMQ's force_boot lets that node start without waiting (follow the RabbitMQ docs; forcing the wrong node can lose recent messages).
3. Check queues and consumers once the cluster forms.

Prevent. Use the vendor's operator (it encodes these rules), Parallel pod management for peer-discovering clusters, quorum queues, and a power-loss drill in staging. The same class of problem appears in other clustered systems (etcd, Galera, Kafka controllers): know each one's recovery procedure before the outage.

Break-glass and out-of-band access

• Break-glass accounts: local cluster credentials (stored securely offline or in a separate vault) that work when SSO is down; usage alerts on every login.
• Out-of-band network access to BMCs/consoles and management networks.
• Printed/offline runbooks for the cold-start order: when the wiki is on the platform, it's down too.

Cold-start drills

In a staging environment that mirrors production's dependencies:

1. Shut everything down (or cut power to a lab rack).
2. Start in tier order, timing each step and noting anything that needed manual help.
3. Fix every loop or manual step found; repeat until the cold start is boring.

Recap

• Full outages force everything to start at once, exposing hidden circular dependencies.
• Build a tiered dependency map; every upward dependency is a loop to break.
• Stateful clusters need known recovery procedures (e.g. Parallel pod management, RabbitMQ force_boot used carefully).
• Keep break-glass access, out-of-band paths and offline runbooks, and run cold-start drills.