Thanos: sidecar, store, querier · Prasanta Kumar Behera

Give Prometheus a global view and long-term storage with Thanos: the sidecar that uploads blocks and serves recent data, the store gateway that serves the bucket, the querier that fans out and deduplicates HA pairs, the query frontend, and when to use Receive instead.

Thanos around existing Prometheus servers: sidecars, object storage, store gateway, querier.

The Thanos idea

Keep Prometheus as it is, and add components around it.

All components speak the StoreAPI (gRPC, port 10901 by default), so the querier can fan out to any mix of sidecars, store gateways and receivers.

Each hospital ward keeps its own recent notebooks (Prometheus + sidecar), and every two hours photocopies the finished pages to the central archive (object storage). A librarian at the archive (store gateway) finds old pages quickly. When a doctor asks a question, a helpful receptionist (querier) asks every ward and the librarian at once, removes duplicate pages written by two nurses (dedup), and hands back one clean answer.

Sidecar

Runs in the Prometheus pod. Every 2 hours it uploads the newly cut block to the bucket.
Serves recent data (not yet uploaded) to the querier through the StoreAPI.
Requires Prometheus's local compaction to be off (min = max block duration = 2h). The Prometheus Operator configures this when you enable the sidecar with object storage.

With kube-prometheus-stack values:

prometheus:
  prometheusSpec:
    replicas: 2
    externalLabels:
      cluster: prod-eu-1
    retention: 2d                   # local retention can now be short
    thanos:
      objectStorageConfig:
        existingSecret:
          name: thanos-objstore     # key must match below
          key: objstore.yml
  thanosService:
    enabled: true                   # exposes the sidecar's gRPC port

(Value names have changed across chart versions; check the chart's values.yaml for yours.)

Store gateway and querier

$ thanos store --objstore.config-file=objstore.yml --data-dir=/var/thanos/store
$ thanos query \
    --endpoint=dnssrv+_grpc._tcp.kps-kube-prometheus-stack-thanos-discovery.monitoring.svc \
    --endpoint=thanos-store.monitoring.svc:10901 \
    --query.replica-label=prometheus_replica \
    --query.replica-label=replica

Store gateway: needs a local disk for index headers and caches; scale by sharding blocks (time ranges or hash) across replicas.
Querier: stateless; add replicas behind a Service. (The discovery Service name above depends on your Helm release name.) --endpoint replaced the older --store flag.
Deduplication merges series that differ only by replica labels. Grafana points at the querier (or query-frontend) as a Prometheus data source.

Query frontend

thanos query-frontend sits in front of queriers: it splits long-range queries by day, caches results (memcached/Redis/in-memory), and retries. This makes long dashboards much faster.

Receive: the push alternative

Instead of sidecars, Prometheus instances (or agents) remote-write to Thanos Receive, which stores data in a local TSDB, uploads blocks, and serves recent data. Use it when sidecars aren't reachable from the querier (edge, other networks, many small clusters), or when you want multi-tenancy by tenant header. It's more to operate (a hashring of receivers with replication).

Try it: global view across two clusters

Create two kind clusters, each with kube-prometheus-stack (2 replicas, distinct cluster external labels) and the sidecar uploading to your lesson 03 bucket.
In one cluster, run a store gateway and a querier with endpoints for both sidecars (expose the sidecar gRPC services between clusters on the Docker network for the lab).
In the querier UI, run count by (cluster) (up) and see both clusters.
Toggle deduplication in the UI and compare up series counts.
Wait for the first block upload (up to 2h, or check mc ls on the bucket), then query a time range older than local retention.

Going deeper: running Thanos well

Enable index and chunk caching (memcached) for store gateways and results caching for the query frontend.
Watch sidecar upload failures (thanos_objstore_bucket_operation_failures_total, thanos_shipper_upload_failures_total) and alert on them: missing uploads mean silent data loss when local retention expires.
Community Helm charts and the kube-thanos manifests are common deployment paths; check a chart's maintenance status and image sources before adopting it.
Use Thanos Ruler only when rules must span clusters; keep most rules in each Prometheus.

Recap

Sidecar: uploads 2h blocks and serves recent data; store gateway: serves the bucket; querier: global PromQL with dedup; query frontend: split + cache.
Unique external labels per Prometheus; replica labels for HA dedup.
Receive is the push-based option for unreachable or multi-tenant sources.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.