Capstone: cost, HA & the platform view

Capstone: design a metrics platform for about 500,000 active series across several clusters. Estimate ingestion, storage and memory from first principles, choose the architecture, design HA and meta-monitoring, and present the cost and trade-offs as a design review.

The brief

Design the metrics platform for:

5 Kubernetes clusters (3 data-centre, 2 edge sites with occasionally unstable links)
about 500,000 active series in total today, expected to double in 2 years
30 s scrape interval
Retention: 15 days raw for everything, 13 months for gold aggregates (lesson 06)
3 teams who must not affect each other; on-call via PagerDuty or similar
On-premises object storage available (lesson 03)

This is the final school project: design the whole hospital monitoring system, not just one nurse's notebook. You have to work out how many notebooks per day, how big the archive must be, what happens when a nurse is sick, and how you'd know if the alarm system itself broke.

Step 1: estimate

Quantity	Calculation	Result
Samples/s	500,000 ÷ 30	≈ 16.7k/s (33k/s in 2 years)
Samples/day	16.7k × 86,400	≈ 1.44 billion
Storage/day (compressed)	× ~1.5 bytes	≈ 2.2 GB/day per copy
Raw 15 days	× 15	≈ 33 GB per copy
Gold aggregates 13 months	e.g. 20k series → 4% of raw	small (≈ tens of GB)

These are assumptions. Measure your own bytes per sample (prometheus_tsdb_compaction_chunk_size_bytes, block sizes on disk) and memory per series from a real cluster, then plan 2× headroom for spikes, queries and growth.

Step 2: choose the architecture

A defensible design (one of several):

DC clusters (3):   Prometheus HA pair per cluster ──remote_write──┐
Edge sites (2):    Prometheus (local alerting, 2d) ──remote_write─┤   (buffers on outage)
                                                                  ▼
                        Mimir (or Thanos Receive) with tenants per team, limits
                                   │ blocks
                                   ▼
                        S3-compatible object storage (erasure coded)
                                   ▲
Grafana ──► query-frontend ──► queriers / store-gateways
Alertmanager cluster (3) ◄── rulers / Prometheus rules
Meta-monitoring Prometheus (separate, small) ──► watches everything above

Justify each choice: why push (edge sites, tenants, limits), why keep full Prometheus at the edge (local alerting during outages), why these retention tiers, why this object store. A Thanos-based design (sidecars in DCs, Receive for edge, downsampling for 13-month dashboards) is equally valid if the reasoning holds (lesson 08).

Step 3: HA and failure modes

Failure	Mitigation
One Prometheus replica dies	HA pair; dedup (HA tracker or querier)
Ingester/receiver dies	Replication factor 3, zone-aware
Object store node dies	Erasure coding across nodes
Edge uplink down	Local Prometheus + local alerting; WAL buffering; gap accepted beyond it
Alertmanager pod dies	3-replica cluster (gossip)
Whole monitoring stack down	Meta-monitoring + Watchdog to an external dead man's switch
Cardinality explosion	Per-tenant limits, sample limits, growth alerts (lesson 02)

Step 4: operate it

SLOs for the platform: ingestion delay < 1 min, query success > 99.5%, alert delivery within 2 min.
Runbooks for top alerts (ingester OOM, compactor halted, remote-write failing, disk/object store full).
Cost visibility: series and retention per team, reviewed monthly.
Upgrades: staging platform first; the mixin dashboards before and after.

Try it: the design review

Measure a real (or lab) cluster: active series, bytes per sample, memory per series. Replace the assumptions in Step 1 with your numbers.
Draw your architecture and write a one-page justification per major choice.
Fill in the failure-mode table for your design, adding at least two failures of your own.
Estimate hardware/cloud cost for 1 year and for 2× growth.
Present it to a colleague (or write it as a decision record) and list the three weakest assumptions.

Going deeper: beyond the capstone

Model query load, not just ingestion: a few heavy dashboards over 13 months can dominate cost; recording rules and caching fix most of it.
Consider native histograms and OTLP ingestion in your roadmap; both change series counts and pipelines.
Rehearse disaster recovery: rebuild the platform from Git and object storage in a new location.

Recap

Estimate from first principles: series ÷ interval, bytes per sample, retention, replication, plus headroom, then measure.
Choose push or pull with reasons (edge, tenants, retention, skills).
Design for failure: HA pairs, RF 3, erasure coding, Alertmanager cluster, meta-monitoring with a dead man's switch.
Run it as a product: SLOs, runbooks, cost visibility, careful upgrades.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.