Prasanta Kumar Behera
Observability with OpenTelemetry›07 · Production observability patterns
Learning Hub / Observability & Reliability / Observability with OpenTelemetry

Lesson 07 of 7 · Modules

Production observability patterns

Run observability as a platform: control cost with sampling, cardinality budgets and retention tiers, isolate teams with multi-tenancy, redact sensitive data in the pipeline, scale and monitor the Collectors and backends, and give teams a clear contract.

Advanced
Key wordstelemetry costsamplingcardinality budgetsretention tiersmulti-tenancyX-Scope-OrgIDPII redactionCollector scalingself-monitoringplatform contract

Observability is a platform

Once many teams send telemetry, the observability stack becomes a shared platform with its own customers, SLOs, budget and roadmap. The three hard problems are cost, isolation and reliability of the telemetry pipeline itself.

A city's water system serves every house. It needs meters so each street pays its share (tenants and budgets), filters so dirty water never reaches the tap (redaction), and engineers watching the pipes themselves (self-monitoring). If it's free and unlimited, one leaking house can drain the reservoir for everyone.

Cost control per signal

Signal Levers
Traces Tail sampling (all errors, slow, small baseline); drop health-check spans; limit attribute sizes
Metrics Cardinality budgets per team; drop unused metrics and labels (relabeling, filter processor); longer scrape intervals where fine; downsampling for long retention (see Scaling Prometheus to Production)
Logs Drop debug/noise at the source; structured logs instead of verbose text; shorter retention for high-volume streams

Show teams their own usage (series, GB/day, spans/s per tenant), because visibility changes behaviour faster than rules.

Multi-tenancy

  • Loki, Tempo and Mimir are multi-tenant: every request carries a tenant ID in the X-Scope-OrgID header.
  • Put an authenticating gateway in front (or the Collector gateway) that maps callers to tenants; never let clients choose their tenant freely.
  • Set per-tenant limits (ingestion rate, active series, max streams, query limits) and retention.
  • In Grafana, give each team data sources scoped to their tenant (or use one data source with multiple tenants where appropriate, for platform teams).

Sensitive data

  • Redact at the source (don't log tokens, mask emails) and add a safety net in the Collector (attributes/transform processors, or the redaction processor that allow-lists attributes).
  • Control who can query what (tenants, data-source permissions), and keep retention as short as your use cases allow.
  • Treat telemetry backends as systems containing personal data under your privacy obligations.

Reliability of the pipeline

  • Scale gateway Collectors horizontally (HPA on CPU/memory), with the loadbalancing exporter for tail sampling.
  • Use persistent queues on exporters for backend outages; decide what to drop first when full (debug logs before traces before metrics).
  • Monitor the pipeline: Collector refused/failed/queue metrics, backend ingestion errors and rejected samples, and end-to-end canaries (a synthetic span/log/metric sent every minute and checked on the other side).
  • Give the observability platform its own SLOs (ingestion delay, query success).

The platform contract

Publish what teams get and what they must do:

  • Required attributes: service.name, service.version, deployment.environment, team/owner.
  • Endpoints: the node-local agent ($(NODE_IP):4317/4318) and how to enable auto-instrumentation (annotation).
  • Limits and defaults: sampling, retention, cardinality budget, log fields.
  • Golden dashboards and SLO templates (lesson 06) available on day one.

Try it: platform guard rails

  1. In your lab Collector, add a filter processor that drops spans for /healthz and a metric with an unbounded label; measure the volume before and after.
  2. Add a transform or attributes rule that deletes an http.request.header.authorization attribute if present.
  3. Configure Loki (or Tempo) with multi-tenancy enabled and send data for two tenants with different X-Scope-OrgID headers; confirm isolation in Grafana.
  4. Set a per-tenant ingestion limit low and watch the rejections in the backend's metrics.
  5. Build a small "pipeline health" dashboard from Collector self-metrics.

Going deeper: operating at scale

  • Charge back or show back telemetry costs per team; tie budgets to cardinality and GB/day.
  • Keep vendor neutrality with OTLP and the Collector, so backends can change without re-instrumenting services.
  • Review alert and dashboard hygiene regularly: unused dashboards and never-firing or always-firing alerts are cost and noise.
  • Plan disaster recovery for the observability stack itself: during a major outage it's the tool you need most.

Recap

  • Observability at scale is a platform: cost, isolation and pipeline reliability.
  • Cut cost with tail sampling, cardinality budgets, source-side log reduction and retention tiers.
  • Multi-tenancy via X-Scope-OrgID behind an authenticating gateway, with per-tenant limits.
  • Redact sensitive data at source and in the Collector.
  • Scale and monitor the pipeline, with canaries and its own SLOs, and publish a clear platform contract.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.