Distributed tracing · Prasanta Kumar Behera

How distributed tracing works: traces and spans, context propagation with W3C traceparent, auto- and manual instrumentation, head vs tail sampling and what each costs, and storing and querying traces in Grafana Tempo or Jaeger.

Traces and spans

A trace is the story of one request. It's made of spans: units of work with a name, start/end time, attributes, events and a status. Each span has a span ID and a parent span ID; all share the trace ID.

trace 4bf92f35…
└─ GET /checkout           frontend    1,920 ms
   ├─ GET /cart            cart           40 ms
   ├─ POST /charge         payments    1,780 ms   ← the slow part
   │  └─ SELECT … FROM cards   db       1,700 ms   ← missing index?
   └─ POST /email          email          60 ms

A trace is like a relay race baton with a notebook attached. Each runner (service) writes when they got the baton and when they passed it on. At the end, you read the notebook and see exactly which runner was slow. If one runner forgets to pass the notebook along (drops the header), the story breaks in two.

Context propagation

When service A calls B, it adds the trace context to the request, by default in the W3C Trace Context header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ver  trace-id (16 bytes)             parent span-id   flags (01 = sampled)

Instrumented HTTP/gRPC clients and servers do this automatically. Watch for gaps: message queues, custom proxies and async jobs need the context carried in message headers.

Instrumentation

Auto-instrumentation covers common frameworks and clients without code changes:

$ pip install opentelemetry-distro opentelemetry-exporter-otlp
$ opentelemetry-bootstrap -a install          # installs instrumentations for detected libraries
$ OTEL_SERVICE_NAME=cart OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
  OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf opentelemetry-instrument python app.py

On Kubernetes, the OpenTelemetry Operator injects it with an annotation (lesson 02). Add manual spans for business steps auto-instrumentation can't see:

from opentelemetry import trace
tracer = trace.get_tracer("cart")

def apply_discount(cart, code):
    with tracer.start_as_current_span("apply_discount") as span:
        span.set_attribute("discount.code", code)
        ...

Sampling

	Head sampling	Tail sampling
Decided	At the first span (SDK)	After the trace completes (Collector)
Knows about errors/latency	❌	✅
Cost	Cheap	Memory for buffered traces; all spans of a trace must reach the same Collector
Config	`parentbased_traceidratio` in SDKs	`tail_sampling` processor with policies

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

A trace is kept if any policy matches (errors, slow, or the 5% baseline). Use parentbased samplers so services agree on each trace's decision.

Storing and querying traces

Grafana Tempo: stores traces in object storage; queried by trace ID and with TraceQL, e.g. { resource.service.name = "payments" && status = error } or { span.http.route = "/checkout" && duration > 2s }.
Jaeger: the long-standing CNCF tracing backend with its own UI; recent versions are built on the OpenTelemetry Collector and support several storage backends.
Both accept OTLP, so the Collector config (lesson 02) is the same except for the endpoint.

Try it: trace a small system

Write (or reuse) two tiny Python/Flask services where frontend calls backend over HTTP; auto-instrument both.
Run Jaeger all-in-one or Tempo + Grafana locally, and send traces via a Collector.
Find a trace spanning both services; check that backend's span has frontend's span as parent.
Add a manual span with an attribute; make backend sleep randomly and find slow traces with a latency query.
Add tail sampling (errors + slow + 5%) and compare stored trace counts before and after.

Going deeper: tracing in production

Put span metrics (spanmetrics connector) or service metrics next to traces: alert on metrics, investigate with traces.
Keep attribute values bounded for fields you search often; very large attributes (payloads) increase cost and risk leaking data.
Propagate context through queues (Kafka headers) and batch jobs (span links) for end-to-end views.
Watch for clock skew between nodes when spans look out of order.

Recap

A trace = spans sharing a trace ID, linked by parent IDs.
Context propagation (W3C traceparent) connects services; gaps break traces.
Auto-instrumentation first, manual spans for business steps.
Head sampling is cheap but blind; tail sampling keeps errors and slow traces but needs care.
Tempo (object storage, TraceQL) or Jaeger, both via OTLP.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.