Retention by data priority · Prasanta Kumar Behera

Not all metrics deserve the same lifetime. Classify metrics by priority, keep aggregated SLO and capacity data for years and raw debug data for days, pre-aggregate with recording rules, drop unused series at scrape time, and implement tiers in Thanos or Mimir.

Why priority matters

A typical platform stores millions of series. Some are critical for years (SLOs, capacity planning, billing); many are only useful during an incident (per-pod, per-container details); some are never queried at all. Treating them all the same means paying long-term prices for short-term data.

You don't keep every school worksheet forever. You keep report cards (summaries) for years, this term's worksheets until the exams (short retention), and you throw away the scribbles right away (dropped metrics). The report card is small, but tells the long story.

Classify

Tier	Examples	Retention
Gold	SLI/SLO recording rules, capacity aggregates, business KPIs	1–2+ years (downsampled)
Silver	Service RED metrics, node USE metrics	30–90 days
Bronze	Per-pod/per-container detail, debug metrics	7–15 days
Drop	Unused metrics, unbounded labels	Never stored

Agree on the tiers with teams, and make the default bronze: long retention should be a conscious choice.

Pre-aggregate with recording rules

groups:
  - name: gold-aggregates
    interval: 1m
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (cluster, namespace, service) (rate(http_requests_total[5m]))
      - record: service:http_request_errors:rate5m
        expr: sum by (cluster, namespace, service) (rate(http_requests_total{code=~"5.."}[5m]))
      - record: namespace:container_cpu_usage:sum_rate5m
        expr: sum by (cluster, namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))

The recorded series are a tiny fraction of the raw ones and answer the long-term questions (trends, capacity, SLO history).

Drop what nobody uses

  endpoints:
    - port: metrics
      metricRelabelings:
        - sourceLabels: [ __name__ ]
          regex: "go_gc_.*|process_.*_bucket|apiserver_request_sli_duration_seconds_bucket"
          action: drop

(The regex is only an example; check which metrics your dashboards and rules use before dropping anything.)

Find candidates by comparing stored metric names with those referenced in dashboards, rules and query logs. mimirtool analyze does this for Grafana dashboards and rule files, even if you don't run Mimir.

Implementing tiers

Thanos: retention is per bucket (and per resolution), so use separate Prometheus instances or Receive tenants writing to different buckets with different compactor retention, or send gold recording rules to a long-retention tier via remote write.
Mimir: per-tenant retention overrides (compactor_blocks_retention_period); route tiers to different tenants.
Prometheus agents at the edge can remote-write selected series (with writeRelabelConfigs) to the long-term tier and everything else to a short-term one.

Try it: a retention plan for your lab

List the top 20 metrics by series count in your lab (lesson 02) and classify each as gold/silver/bronze/drop.
Write the gold recording rules above (adapt to your metrics) and verify they produce far fewer series (count({__name__=~"service:.*"})).
Add a drop relabeling for two unused metrics; confirm they disappear from new scrapes.
Run mimirtool analyze grafana against your Grafana (API key) and mimirtool analyze prometheus to find unused metrics.
Write a one-page retention policy: tiers, defaults, who approves exceptions.

Going deeper: making it stick

Show cost per team (series × retention) on a dashboard; people optimise what they can see.
Add CI checks for new metrics: naming, allowed labels, and a declared tier.
Revisit the tiers yearly; business needs change (a metric becomes an SLO, or a product is retired).

Recap

Classify metrics into gold/silver/bronze/drop; default to short retention.
Recording rules keep small aggregates for years; raw detail stays short.
Drop unused metrics at scrape time; find them with usage analysis (mimirtool).
Implement tiers with separate buckets/instances (Thanos) or per-tenant retention (Mimir).

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.