Federation & multi-cluster · Prasanta Kumar Behera

Three ways to see many clusters: Prometheus federation for small aggregated slices, remote-write (often from agent-mode Prometheus) into a central store, and global query across per-cluster stores. When each fits, how disconnections behave, and the labels that keep it all consistent.

Three patterns

Pattern	How	Good for	Weak for
Federation	Central Prometheus scrapes `/federate` from others	Small sets of aggregated series	Raw data, scale
Remote write	Each cluster pushes to a central store (Mimir, Thanos Receive, others)	Central storage, edge/disconnected sites, agents	Bandwidth, central dependency
Global query	Querier fans out to per-cluster stores (Thanos sidecars)	Keeping data local, incremental adoption	Needs reachability at query time

Imagine many schools. Federation: each school sends a monthly summary to the district office. Remote write: each school mails every page to the district archive as it's written. Global query: the district office phones every school when it has a question. Summaries are small, mailing everything is complete, and phoning works only if every school answers.

Federation for aggregates

scrape_configs:
  - job_name: federate-clusters
    honor_labels: true
    metrics_path: /federate
    params:
      "match[]":
        - '{__name__=~"job:.*|namespace:.*|cluster:.*"}'   # recording-rule outputs only
    static_configs:
      - targets: [ "prom-eu.example.com:9090", "prom-us.example.com:9090" ]

Use it for a global overview built from recording rules (lesson 06). Federating raw series across many clusters does not scale.

Remote write, often from agents

apiVersion: monitoring.coreos.com/v1alpha1
kind: PrometheusAgent
metadata:
  name: edge
  namespace: monitoring
spec:
  replicas: 1
  externalLabels:
    cluster: edge-store-042
  serviceMonitorSelector: {}
  remoteWrite:
    - url: https://metrics.example.com/api/v1/push
      headers:
        X-Scope-OrgID: edge
      writeRelabelConfigs:
        - sourceLabels: [ __name__ ]
          regex: "go_.*"
          action: drop

(The PrometheusAgent API version may differ in your Operator version.)

Agent mode keeps a WAL but no queryable TSDB, rules or alerting: lighter for edge nodes.
Remote write retries and reads from the WAL, so short outages are absorbed. Longer outages than the WAL's lifetime (a few hours by default) create gaps. For sites with long disconnections, keep a full local Prometheus (for local alerting and local retention) and accept that the central view catches up later, or use a store-and-forward design.
Watch pending, failed and dropped sample counters on every sender, and alert when a site stops sending (absence of its up series centrally).

Global query

Thanos querier across sidecars (lesson 04): data stays in each cluster until blocks upload; the central querier needs network access to every sidecar. Good in data centres with stable links; weaker for edge.

Labels make it work

Every source needs unique external labels (cluster, region, env), or series collide.
Standardise label names across clusters (same cluster label key everywhere) so dashboards work fleet-wide.
Alert centrally on fleet conditions (a site missing, many clusters degraded) and locally on anything that must work without the uplink.

Try it: an edge fleet (kind)

Create two "edge" kind clusters with the Prometheus Operator and a PrometheusAgent each, remote-writing to your lab Mimir (lesson 07) or Thanos Receive.
Query count by (cluster) (up) centrally.
Block one edge cluster's egress (NetworkPolicy or disconnect its Docker network) for 10 minutes; watch prometheus_remote_storage_samples_pending grow; reconnect and check for gaps.
Add a federation job on a central Prometheus for recording-rule series only; compare series counts with remote write.
Write an alert that fires when a cluster's series stop arriving centrally.

Going deeper: fleet-scale metrics

Size remote-write queues (queueConfig: shards, capacity, batch size) for bursts after reconnection; the central store must absorb catch-up traffic too.
Use per-site tenants or labels plus per-tenant limits, so one misbehaving site can't flood the platform.
Keep local alerting for site-critical conditions; central alerting can't help a site that's offline.

Recap

Federation: aggregated slices only. Remote write: push everything (agents at the edge). Global query: fan out to per-cluster stores.
Agents are light but depend on the WAL for buffering; plan for real outage lengths.
Unique, consistent external labels are mandatory.
Alert on missing sources centrally and keep local alerting where sites must survive alone.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.