Grafana Mimir · Prasanta Kumar Behera

The push-based model: Prometheus (or agents) remote-write into Grafana Mimir, a horizontally scalable, multi-tenant metrics store. The write and read paths, the hash ring and replication, tenants and limits, deployment modes, and what to monitor.

Mimir write and read paths, all backed by object storage.

The push model

Mimir (Grafana Labs, AGPLv3, descended from Cortex) is a horizontally scalable, multi-tenant Prometheus-compatible backend. Every component scales independently.

Instead of the archive visiting every ward (Thanos querier asking sidecars), every ward posts its notes to a big central post office (Mimir). The post office has sorting desks (distributors), several clerks who each keep three copies of new letters (ingesters), and a warehouse for old ones (object storage). Each ward has its own PO box (tenant), with a limit on how much mail it can send.

Components

Component	Role
Distributor	Validates samples, applies limits, shards series to ingesters via the hash ring
Ingester	Holds recent data in memory + WAL, cuts 2h blocks, uploads to object storage
Querier	Executes PromQL over ingesters (recent) and store-gateways (older)
Query-frontend / query-scheduler	Split, queue, cache and retry queries
Store-gateway	Serves blocks from the bucket (like Thanos store)
Compactor	Compacts and deduplicates blocks per tenant, applies retention
Ruler / Alertmanager	Multi-tenant rules and alerting

Recent Mimir versions also offer an alternative ingest storage architecture that places a Kafka-compatible log between distributors and ingesters to decouple writes from reads; check which architecture your version recommends.

Sending data

# kube-prometheus-stack values
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: prod-eu-1
    remoteWrite:
      - url: http://mimir-gateway.mimir.svc/api/v1/push
        headers:
          X-Scope-OrgID: platform
        queueConfig:
          maxSamplesPerSend: 2000

(The gateway Service name depends on the chart version and values.) HA Prometheus pairs can both write; Mimir's HA tracker accepts samples from one replica at a time using cluster and __replica__ labels, which avoids storing duplicates.

Tenants and limits

Multi-tenancy is on by default: every request needs X-Scope-OrgID (disable it for a single-tenant setup).
Put an authenticating gateway in front to map credentials to tenants.
Set limits per tenant in the runtime config: ingestion rate and burst, max_global_series_per_user, per-metric series limits, query limits, retention (compactor_blocks_retention_period).

overrides:
  team-a:
    ingestion_rate: 50000
    max_global_series_per_user: 1500000
    compactor_blocks_retention_period: 90d

Deployment

Monolithic (-target=all): one binary with everything; good for labs and small setups, and can be scaled horizontally.
Microservices (the mimir-distributed Helm chart): each component as its own Deployment/StatefulSet; production default for larger scale.
Needs object storage (lesson 03), and caches (memcached) for queries, index and chunks at scale.

Monitor Mimir with its mixin dashboards and alerts (Mimir publishes them): ingestion errors, ring health, ingester memory, compactor progress, query latency, and per-tenant limits being hit.

Try it: Mimir in the lab

Install grafana/mimir-distributed on kind with the small/test values and your lesson 03 buckets (or run Mimir monolithic with Docker for a lighter lab).
Configure your kube-prometheus-stack to remote-write to Mimir with tenant lab.
Add Mimir as a Prometheus data source in Grafana (URL of the gateway's /prometheus path, plus the X-Scope-OrgID header) and query up.
Set a low max_global_series_per_user for the tenant and watch writes get rejected in Prometheus's remote-write metrics and Mimir's logs.
Kill one ingester pod and confirm queries still work (replication factor 3).

Going deeper: Mimir at scale

Ingesters are the most memory-hungry component; size them from series per ingester (series × replication factor / ingester count).
Use zone-aware replication so the 3 copies live in different zones.
Enable shuffle sharding to limit how many ingesters and queriers one tenant can affect.
Watch remote-write health on the senders (prometheus_remote_storage_samples_failed_total, pending samples, and shard counts).

Recap

Push model: Prometheus/agents remote-write to Mimir; Mimir stores and serves at scale.
Write path: distributor → ingesters (RF 3) → blocks in object storage; read path: query-frontend → queriers → ingesters + store-gateways.
Tenants via X-Scope-OrgID, protected by per-tenant limits and retention.
Monolithic for small setups, mimir-distributed microservices for scale; monitor with the mixin.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.