Signal correlation · Prasanta Kumar Behera

Make metrics, traces and logs one investigation instead of three tabs: consistent resource attributes, exemplars that link metric spikes to traces, trace IDs in logs, and Grafana data-source links for metric → trace → logs in a few clicks.

From three tabs to one flow

Without correlation, an incident looks like: alert fires → open a dashboard → guess a time → open the log tool → guess a query → open the tracing tool → search by service and time. With correlation it's click, click, click:

latency panel spike ──(exemplar)──► the slow trace ──(trace to logs)──► that pod's log lines
          ▲                                  │
          └──────────(trace to metrics)──────┘

Imagine a treasure map, a diary and a photo album of the same trip. Without links, you flip through all three guessing which page matches. With correlation, every map point has a sticker number, and the same sticker is on the right diary page and the right photo. You jump straight between them.

Step 1: agree on names

All signals need the same resource attributes:

service.name (set with OTEL_SERVICE_NAME or the Operator)
k8s.namespace.name, k8s.pod.name, k8s.cluster.name (the Collector's k8sattributes and resource processors)
Metrics from Prometheus scraping use job/namespace/pod labels, so map them consistently (or send OTel metrics that carry the same resource attributes).

Step 2: trace IDs in logs

Use your language's OTel logging integration so each log record carries trace_id and span_id automatically. In Python, the logging instrumentation can inject them into log records. Java agents add them to MDC for Logback/Log4j. Sending logs via OTLP keeps trace context as first-class fields.

Step 3: exemplars

Exemplars attach a trace ID to histogram samples:

Instrumentation libraries (OTel SDKs, Prometheus clients) record exemplars when a span is active.
Prometheus stores them with --enable-feature=exemplar-storage (and they must be exposed in a format that carries exemplars, e.g. OpenMetrics, or sent via remote write/OTLP).
Grafana shows exemplars as dots on time-series panels; clicking one opens the trace.

Step 4: wire Grafana's data sources

From	Link	Configure in
Metrics panel (Prometheus/Mimir)	Exemplar → trace	Prometheus data source: exemplar trace ID destination = Tempo
Trace (Tempo)	Span → logs	Tempo data source: trace to logs (Loki; filter by service/pod, time shift)
Trace (Tempo)	Span → metrics	Tempo data source: trace to metrics (queries using span attributes)
Log line (Loki)	trace_id → trace	Loki data source: derived field (regex or structured metadata → Tempo)

(Kibana and APM products offer similar links when logs and traces share IDs.)

Scenario: the dashboard is green while users see errors

Customer support reports failed checkouts. Every dashboard is green: pods healthy, CPU fine, the service's error rate under 0.1%.

What might the health checks not be checking?

Where you measure: the service's own metrics can't see requests that never reach it (DNS, ingress, TLS, a network policy, a misrouted canary). Measure at the edge (ingress/load balancer) and with synthetic checks that run the real user journey.
What counts as an error: the app returns 200 with an error body, or 4xx for what is really a server problem; partial failures (the payment succeeded, the confirmation email didn't).
Averages hiding a subset: only one region, one app version, or one payment provider fails. Break down by those dimensions, and use traces filtered by status = error.
Liveness vs usefulness: /healthz returns OK while a dependency is down.

Fix: SLIs from the user's point of view (lesson 06), edge and synthetic monitoring, and traces/exemplars to find the failing subset.

Try it: click through an incident

Use the OpenTelemetry Demo (lesson 01) or your lab stack with Prometheus, Tempo, Loki and Grafana.
Configure the three data-source links (exemplars, trace to logs, derived field on trace_id).
Inject latency or an error (the Demo's feature flags) and start from a latency panel: exemplar → trace → logs.
Go the other way: search Loki for an error line and jump to its trace.
Deliberately rename service.name in one service and see which links break.

Going deeper: correlation at scale

Standardise attributes in a platform contract (required resource attributes, log fields), and check it in CI or at the Collector.
Keep clocks in sync on all nodes; time-shifted links miss otherwise.
Add deployment markers (annotations) on dashboards; "what changed?" is the first question in most incidents (see SRE & Production Incident Response).

Recap

Correlation = consistent resource attributes + trace IDs in logs + exemplars + data-source links.
Flow: metric spike → exemplar → trace → logs, and back.
Measure from the user's side too; green internal dashboards can hide real failures.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.