Lesson 12 of 13 · Operations
Primary/secondary & failover
Design and rehearse ParcelPath's site failover: active-passive vs active-active across two data centres, keeping the secondary identical with GitOps, traffic switching with GSLB/DNS, promoting databases without split-brain, an executed failover runbook with timings, and failback.
Choosing the pattern
| Active-passive (ParcelPath) | Active-active | |
|---|---|---|
| Traffic | Primary site only | Both sites |
| Secondary | Platform running, apps scaled down or minimal, data replicated | Full capacity in both |
| Data | Async replicas promoted on failover | Multi-site data design (partitioning, conflict handling) |
| RTO | Minutes–an hour (promote + switch) | Seconds–minutes |
| Complexity/cost | Moderate | High |
ParcelPath's NFR-04 (orders: RPO 5 min, RTO 60 min) and ~300 km between sites point to active-passive (ADR-007). Active-active would need an application redesign for multi-site writes.
A theatre has a main stage and an understudy. The understudy knows every line (same script from GitOps), attends every rehearsal (replication), and waits backstage. If the lead is ill, the director makes sure the lead won't walk on (fencing), sends the understudy on (promotion), and tells the audience (DNS switch). You rehearse this regularly, or the understudy freezes on the night.
Keep the secondary ready
- Same platform via GitOps from the same repo, with site values (lesson 05's clusters
prod-a,prod-b). - Stateless apps deployed in both sites; in
prod-bscaled to a minimum (or zero) with capacity reserved to scale up (sizing from lesson 04 applies to DC-B too). - Data: async PostgreSQL replica cluster and Kafka mirroring in DC-B (lesson 09); monitored lag.
- Secrets, certificates, DNS records, partner allowlists all valid for DC-B. These are the usual forgotten items.
Traffic switching
- GSLB or DNS with health checks and a low TTL (e.g. 60 s) for the public hostnames. Remember that some clients cache DNS longer than the TTL.
- CDN/WAF origins configured for both sites (lesson 08), with DC-B as a standby origin.
- Automatic health-based switching is tempting, but for data-bearing systems ParcelPath chooses a human decision to fail over (avoiding flapping and split-brain), with everything after the decision automated.
The failover runbook
| Step | Action | Target time |
|---|---|---|
| 0 | Declare DR (incident commander), freeze changes | T+0 |
| 1 | Fence DC-A writers: stop app writes / isolate the old primary (if DC-A is partially alive) | T+5 min |
| 2 | Record replication lag (this is the actual RPO) | T+6 min |
| 3 | Promote the DC-B PostgreSQL replica cluster; switch Kafka consumers to the mirrored topics | T+15 min |
| 4 | Scale up stateless apps in prod-b (GitOps value change) |
T+25 min |
| 5 | Smoke tests + synthetic journeys against DC-B directly | T+35 min |
| 6 | Switch traffic (GSLB/DNS, CDN origin) | T+40 min |
| 7 | Verify SLO dashboards, partner connectivity; communicate | T+50 min |
Drill results (what to record)
| Drill (date) | RPO measured | RTO measured | Issues | Actions |
|---|---|---|---|---|
| DR-1 | 38 s | 72 min ❌ | Partner allowlist missing DC-B IPs; app config had DC-A DB hostname | Allowlist update; DB hostname via service DNS |
| DR-2 | 22 s | 47 min ✅ | Kafka consumer offsets translation manual | Automate offset sync |
Failback
After DC-A is repaired: re-establish replication from DC-B to DC-A, wait until in sync, then run a planned failover back (same runbook, calm conditions). Many teams stay on the secondary for a while rather than rushing back.
Try it: a mini DR drill
- Create two kind clusters ("site A" and "site B") and deploy the same app to both via GitOps, with B scaled to 1 replica.
- Run PostgreSQL (CloudNativePG) in A with a replica cluster in B (CloudNativePG supports replica clusters from object-storage backups/streaming).
- Write the runbook for your lab and time each step while "failing" site A (delete its cluster or block it).
- Promote B, scale the app up, point a local DNS name (e.g.
/etc/hostsor CoreDNS) at B, and run smoke tests. - Record measured RPO and RTO and two improvements.
Going deeper: DR maturity
- Drill regularly (e.g. twice a year), including at least one surprise tabletop exercise.
- Automate steps into a single pipeline with manual approval at the decision point.
- Watch for shared dependencies that break "independent sites": identity providers, Git, registries, DNS providers, certificate authorities.
Recap
- ParcelPath: active-passive across two sites (ADR-007); active-active needs multi-site data design.
- Keep the secondary ready with GitOps, reserved capacity, monitored replication, and valid secrets, certs, DNS, allowlists.
- Failover order: decide → fence → promote → scale → test → switch traffic → verify.
- Measure RPO/RTO in drills, fix findings, and plan failback as a second, planned failover.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.