MinIO object storage · Prasanta Kumar Behera

Why long-term metrics live in object storage, what Thanos and Mimir need from it, running an S3-compatible store on-premises (MinIO, with a note on its changed open-source status, and alternatives like Ceph RGW), and configuring buckets, credentials and durability.

Why object storage?

Prometheus TSDB blocks are immutable files. Object storage is:

cheap and durable (erasure coding or replication across drives and nodes),
effectively unlimited (add capacity without resizing volumes),
shared: queriers, store gateways and compactors all read the same bucket over HTTP.

Both Thanos and Mimir keep recent data locally (Prometheus or ingesters) and everything older in a bucket. In the cloud, use S3/GCS/Azure Blob. On-premises, you need an S3-compatible store.

Object storage is a giant self-storage warehouse. You don't care which shelf your boxes are on; you get a receipt (the object key) and can fetch them any time. It's slower than your desk drawer (local disk), but it never runs out of room, and it keeps extra pieces of every box in different rooms so a fire in one room doesn't lose anything.

On-premises options

Option	Notes
MinIO	Popular, simple, S3-compatible. Licensed AGPLv3. Its open-source edition changed significantly in 2025 (reduced admin console features, changes to how community builds are published and maintained), so check the current status before adopting it, and consider its commercial edition or alternatives.
Ceph RGW	The RADOS Gateway in a Ceph cluster (see Kubernetes Storage & Data Protection, lesson 03); good if you run Ceph already
Others	SeaweedFS, Garage and other S3-compatible stores; vendor appliances

Whatever you choose, test it with your Thanos/Mimir version: they use a subset of the S3 API, but behaviour under load (listing many objects, multipart uploads) matters.

Durability: erasure coding

Distributed MinIO and Ceph split objects into data + parity shards across drives and nodes:

e.g. 8 data + 4 parity: any 4 drives can fail; storage overhead 1.5×.
Spread across failure domains (nodes, racks) so one server loss doesn't take out too many shards.
Plan healing time: rebuilding a big drive takes hours and loads the cluster.

Buckets and credentials

$ mc alias set lab https://s3.internal.example "$ACCESS_KEY" "$SECRET_KEY"
$ mc mb lab/thanos
$ mc mb lab/mimir-blocks lab/mimir-ruler lab/mimir-alertmanager

One bucket per system (Mimir uses separate buckets or prefixes for blocks, ruler and Alertmanager state).
A dedicated access key per system, with a policy limited to its buckets.
Leave lifecycle deletion to the compactor (Thanos/Mimir retention). Bucket lifecycle rules that delete blocks behind the compactor's back cause errors.
Enable versioning or replication to another site only if you understand the cost; the compactor rewrites and deletes many objects.

Thanos bucket configuration (a Secret mounted by each component):

type: S3
config:
  bucket: thanos
  endpoint: s3.internal.example:443
  access_key: ${ACCESS_KEY}
  secret_key: ${SECRET_KEY}
  insecure: false                 # true only for plain-HTTP labs

Mimir equivalent:

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.internal.example:443
      access_key_id: ${ACCESS_KEY}
      secret_access_key: ${SECRET_KEY}
blocks_storage:
  s3:
    bucket_name: mimir-blocks
ruler_storage:
  s3:
    bucket_name: mimir-ruler
alertmanager_storage:
  s3:
    bucket_name: mimir-alertmanager

(Mimir expands ${VAR} only when started with -config.expand-env=true; otherwise use its secret mechanisms.)

Try it: an S3-compatible store for the next lessons

Run an S3-compatible store in your kind cluster, e.g. a single-node MinIO for the lab (check the current image/licence status first) or a small Ceph RGW if you have Rook from the storage track.
Install the mc client (or aws CLI with --endpoint-url) and create thanos and mimir-* buckets.
Create a dedicated user/key with access only to the thanos bucket; verify it can't list mimir-blocks.
Upload and list a test object; measure a large upload's throughput.
Save the Thanos objstore YAML above as a Secret in the monitoring namespace for lesson 04.

Going deeper: object storage for metrics

The compactor and store gateways do many LIST and GET range requests; slow listing on huge buckets hurts. Monitor request latency and errors on the storage side.
Put the object store on a fast network close to queriers; long-range queries pull a lot of data.
Back up, or replicate, only what you can't regenerate: metrics are often considered acceptable to lose beyond a point, but decide explicitly.

Recap

Long-term metrics live in object storage: cheap, durable, shared by all components.
On-prem: MinIO (check its current open-source status), Ceph RGW, or other S3-compatible stores.
Erasure coding across failure domains; plan healing.
Separate buckets and keys per system; let the compactor handle retention.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.