Prasanta Kumar Behera
Scaling Prometheus to Production›02 · Why Prometheus doesn't scale
Learning Hub / Observability & Reliability / Scaling Prometheus to Production

Lesson 02 of 10 · Modules

Why Prometheus doesn't scale

Why one Prometheus hits a wall: how the TSDB stores data (head block, WAL, 2-hour blocks, compaction), why memory follows active series, how cardinality multiplies, the limits of local retention and single-cluster views, and how to find and stop a cardinality explosion.

Advanced
Key wordsTSDBhead blockWALblocks and compactionactive seriescardinalitymemory per seriesretentionsample_limitmetric_relabel_configspromtool tsdb analyze

Inside the TSDB

scrape ──► head block (memory, last ~2h) ──► WAL on disk (crash recovery)
                  │ every 2h
                  ▼
           immutable 2h block on disk (chunks + index + tombstones)
                  │ compaction
                  ▼
           larger blocks (up to 10% of retention, max 31 days) ─► deleted after retention
  • New samples go to the in-memory head block, protected by a write-ahead log (WAL).
  • Every 2 hours, the head is cut into an immutable block; compaction merges blocks into larger ones.
  • Data is compressed very well (on the order of 1–2 bytes per sample on disk).
  • Memory is dominated by the head: every active series costs index entries and recent chunks (roughly a few KB each, varying by version and label sizes). Heavy queries add more.

Prometheus keeps a notebook per patient per measurement. One notebook per patient's temperature is fine. But if you start a new notebook for every visitor who ever came to see the patient (a label with visitor names), the nurse's desk is buried in notebooks and she can't work. That's a cardinality explosion.

Cardinality multiplies

A series is one unique combination of metric name + label values. Series count = product of each label's distinct values:

Labels Series
method (5) × status (10) 50
+ endpoint (50) 2,500
+ pod (40) 100,000
+ user_id (unbounded) 💥

Common culprits: user or session IDs, full URL paths with IDs (/orders/12345), pod IPs, error messages as labels, and histogram buckets multiplied by many labels.

The walls of a single Prometheus

  1. Memory and CPU: one process, vertical scaling only. Beyond a few million active series, operations get painful.
  2. Retention: local disk; long retention = big disks, slow long-range queries, and data lost with the volume.
  3. Scope: one Prometheus per cluster sees only that cluster; there's no global query.
  4. HA: two replicas scrape the same targets and store two slightly different copies; nothing deduplicates them.

Scenario: the cardinality explosion

Scenario: one label that took down monitoring

A team added a path label with the full request URL to their HTTP metrics. Active series went from 800k to 6 million in an hour. Prometheus was OOM-killed, restarted, replayed its WAL (slowly, and OOM again), and alerting was down for the whole cluster.

How do you recover and prevent it?

Recover

  1. Stop the source: drop the label or metric at scrape time with metric_relabel_configs (in the ServiceMonitor's metricRelabelings), or scale the offending app's scraping down temporarily.
  2. Give Prometheus enough memory to replay the WAL, or (if acceptable) remove the WAL to restart fresh, losing the last couple of hours.
  endpoints:
    - port: metrics
      metricRelabelings:
        - action: labeldrop
          regex: path

Prevent

  • sampleLimit per ServiceMonitor (the scrape fails, rather than Prometheus) and enforcedSampleLimit globally; labelLimit, labelValueLengthLimit too.
  • Alert on growth: prometheus_tsdb_head_series rising fast, per-job scrape_samples_scraped.
  • Instrumentation review: use route templates (/orders/{id}), never raw paths or IDs.
  • Split big tenants into separate Prometheus instances or move to a horizontally scalable backend (lesson 07).

Try it: measure and break (lab only)

  1. In your kube-prometheus-stack lab, open Status → TSDB Status and note the top metrics by series.
  2. Run topk(10, count by (__name__) ({__name__=~".+"})) and compare.
  3. Deploy a small app that exposes a counter with a random user_id label on each request (a few lines of Python with prometheus_client), and generate traffic; watch prometheus_tsdb_head_series climb.
  4. Add a labeldrop metric relabeling and watch series stop growing (old series age out of the head).
  5. Set sampleLimit: 1000 on the ServiceMonitor and read the scrape error on the Targets page.

Going deeper: sizing a single Prometheus

  • Budget memory from measured series counts on your version; test with realistic label sizes.
  • Use recording rules to precompute expensive aggregations used by dashboards and alerts.
  • Native histograms (a newer feature) can reduce series counts for latency metrics compared to classic bucket histograms; check support in your Prometheus version and clients.
  • Scrape interval matters less than series count: 15s vs 30s doubles samples, but not series memory.

Recap

  • TSDB: head block in memory + WAL, 2h blocks, compaction; memory tracks active series.
  • Cardinality multiplies across labels; unbounded labels explode it.
  • Single Prometheus limits: vertical scaling, local retention, no global view, no HA dedup.
  • Defend with relabeling, sample limits, growth alerts and instrumentation standards.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.