Thanos: compactor, downsampling, hot/cold

Make long-term metrics affordable and fast with Thanos: the compactor that merges and deduplicates blocks, downsampling to 5-minute and 1-hour resolution (and what it doesn't save), retention per resolution, and splitting hot and cold data across store gateways and caches.

The compactor's jobs

Compaction: merge small 2h blocks into bigger ones (better compression, fewer objects, faster queries).
Vertical compaction / dedup (optional): merge blocks from HA replicas into one, halving storage (--deduplication.replica-label).
Downsampling: create 5-minute and 1-hour resolution copies of older blocks.
Retention: delete blocks older than the retention for each resolution.

It's a singleton: one compactor per bucket (or per disjoint set of blocks if you shard by external labels). It needs a local disk big enough to download and rewrite the largest blocks it compacts.

The archive gets thousands of tiny two-hour notebooks. The compactor is the archivist who binds them into thick monthly books, throws away duplicate copies from the two nurses, writes summary books ("average per hour") for the old years, and recycles books older than the rules allow. There must be only one archivist, or two of them rebind the same books and make a mess.

Downsampling

Resolution	Created for blocks older than	Good for
Raw	—	Recent detail (hours to weeks)
5m	~40 hours	Weeks to months
1h	~10 days	Months to years

Downsampled blocks store aggregates (min, max, sum, count, counter) per interval, so rate(), max_over_time() and friends still work on long ranges. Queriers pick a resolution automatically for long ranges (with --query.auto-downsampling).

Downsampling alone doesn't save space: it adds data. Savings come from retention per resolution:

--retention.resolution-raw=30d
--retention.resolution-5m=180d
--retention.resolution-1h=2y

Keep raw long enough that the downsampled versions exist before raw data is deleted, and remember that some queries (percentiles over short windows) lose precision on downsampled data.

Hot and cold

Data	Where it's served from	Cost
Last hours–days	Prometheus + sidecar (or Receive)	Local SSD
Recent weeks	Store gateway, with index/chunk caches	Object storage + cache memory
Months–years	Store gateway shard for old time ranges (downsampled)	Object storage

Shard store gateways by time (--min-time=-30d for the hot shard, --max-time=-30d for the cold shard) so the hot shard has more cache and replicas. Add a query frontend with result caching (lesson 04) for repeated dashboards.

When the compactor halts

The compactor halts (and sets thanos_compact_halted to 1) on problems it won't fix automatically, most commonly overlapping blocks from two sources with the same external labels. Fix the source (unique external labels!), then remove or mark the bad blocks with the bucket tools, and restart. Alert on the halted metric and on "no compaction progress": an unnoticed halted compactor means no downsampling and no retention, so costs grow.

Try it: compaction and retention (using lesson 04's bucket)

Deploy thanos compact --wait as a single-replica StatefulSet with a PVC and your objstore Secret.
Run thanos tools bucket inspect and look at the blocks' levels and resolutions over time.
Set short lab retentions (e.g. raw 2d) and watch old raw blocks disappear after the compactor runs.
Enable --deduplication.replica-label=prometheus_replica and compare block counts before and after.
Open thanos tools bucket web to see the block timeline.

Going deeper: cost engineering

Measure cost per retained series-month for each resolution; share it with teams.
Combine with retention by priority (lesson 06): not every metric deserves two years.
Monitor compactor disk usage and duration; very large blocks need big volumes. Limit maximum block size or duration where supported.

Recap

The compactor (one per bucket) compacts, deduplicates, downsamples and applies retention.
5m after ~40h, 1h after ~10d; savings come from shorter raw retention, not downsampling itself.
Serve hot and cold data differently: sharded store gateways, caches, query frontend.
Alert on compactor halted; unique external labels prevent overlaps.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.