Mesh observability · Prasanta Kumar Behera

Use the mesh as a free source of golden signals: the standard Istio and Linkerd metrics and their labels, PromQL for success rate and latency per service pair, topology views with Kiali and Linkerd viz, access logs, and why tracing still needs applications to propagate headers.

Golden signals for free

Every meshed request passes through proxies, which record requests, errors and latency with source and destination labels. That's RED metrics (see Observability with OpenTelemetry, lesson 01) for every service without code changes.

The mesh proxies are like toll booths on every road between towns. Without asking the drivers anything, they count cars, note which ones broke down, and time each journey. You instantly know which road is jammed, even for towns that never send you reports.

Istio

# success rate per destination workload (server-side view)
sum by (destination_workload) (rate(istio_requests_total{reporter="destination", response_code!~"5.."}[5m]))
/
sum by (destination_workload) (rate(istio_requests_total{reporter="destination"}[5m]))

# p99 latency per caller → callee pair
histogram_quantile(0.99,
  sum by (le, source_workload, destination_workload)
    (rate(istio_request_duration_milliseconds_bucket{reporter="source"}[5m])))

reporter="destination": the server's view. reporter="source": the caller's view (includes timeouts and connection failures the server never saw).
Tune which labels are recorded with the Telemetry API to control cardinality in big meshes.

Linkerd

$ linkerd viz stat deploy -n shop           # success rate, RPS, p50/p95/p99 per deployment
$ linkerd viz routes deploy/cart -n shop    # per-route metrics (with route definitions)
$ linkerd viz edges deploy -n shop          # who talks to whom, and with which identity

The proxies expose Prometheus metrics (request_total, response_total with a classification label, response_latency_ms_bucket), which the viz extension's Prometheus scrapes. For production, scrape them with your own Prometheus rather than the bundled one.

Topology and config views

Kiali (for Istio): live service graph with traffic rates and error highlights, plus Istio configuration validation.
Linkerd viz dashboard: per-namespace golden metrics and live tap.
Use them for exploration; alert from Prometheus rules (SLOs, burn rates).

Access logs

Envoy access logs (enable via the Telemetry API or mesh config) record each request with response flags such as UH (no healthy upstream), UF (upstream connection failure), UO (upstream overflow, circuit breaker) and URX (retry limit exceeded). These flags explain why the proxy failed a request and are invaluable in incidents.

Tracing needs the application

Proxies can create spans, but context propagation across a service's internals is the application's job: the app must copy trace headers (traceparent for W3C, or B3 headers) from the incoming request to its outgoing requests. OpenTelemetry instrumentation does this automatically. Without it, you get disconnected single-hop traces.

Try it: mesh golden signals

With Istio and Bookinfo (lesson 02), install Prometheus and Kiali from the Istio samples addons.
Generate traffic, then write the success-rate and p99 queries above; compare reporter="source" and reporter="destination" for the same pair.
Inject a fault (lesson 05) and watch it in Kiali's graph and the metrics.
Enable Envoy access logs and find the response flags for failed requests.
With Linkerd and emojivoto (lesson 03), compare linkerd viz stat with the underlying Prometheus metrics.

Going deeper: observability cost in meshes

Mesh metrics can be high cardinality (workload × workload × code × …); drop labels you don't use and consider recording rules for dashboards.
Keep proxy access logs sampled or error-only in busy meshes.
Correlate mesh metrics with app traces via exemplars and shared attributes (see Observability with OpenTelemetry, lesson 05).

Recap

Meshes give uniform golden signals for every service and caller→callee pair.
Istio: istio_requests_total, istio_request_duration_milliseconds_bucket; pick a reporter to avoid double counting.
Linkerd: viz stat/routes/edges and proxy metrics.
Kiali/viz for topology; access log response flags for why requests failed.
Tracing still needs apps to propagate headers.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.