Modules · wrap-up
Cheat sheet & self-check
Every command from this section on one page.
Signals
Metrics | Cheap numbers over time: rates, errors, latency percentiles, saturation |
Logs | Detailed events: what exactly happened, with context |
Traces | One request's path across services, with timing per step (spans) |
Methods & standards
RED: Rate, Errors, Duration | For request-driven services |
USE: Utilization, Saturation, Errors | For resources (CPU, disk, queues) |
OTLP: gRPC :4317, HTTP :4318 | OpenTelemetry's wire protocol |
service.name, k8s.namespace.name, k8s.pod.name | Resource attributes that tie signals together |
Config building blocks
receivers: otlp, prometheus, filelog, hostmetrics, kubeletstats | How data gets in |
processors: memory_limiter → k8sattributes → … → batch | What happens to it (order matters) |
exporters: otlp, otlphttp, prometheusremotewrite, debug | Where it goes |
connectors: spanmetrics, count | Turn one pipeline's output into another's input |
service.pipelines.{traces,metrics,logs} | Wire it together per signal |
Operate
otelcol-contrib validate --config config.yaml | Check a config before deploying |
exporters: debug: { verbosity: detailed } | Print what flows through (temporarily) |
:8888/metrics (service.telemetry) | The Collector's own metrics |
kubectl get opentelemetrycollectors,instrumentations -A | Operator-managed resources |
Instrumentation
OTEL_SERVICE_NAME=cart | Name the service (the most important attribute) |
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-agent:4318 | Where to send telemetry |
OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 | Head sampling at 10%, respecting the parent's decision |
opentelemetry-instrument python app.py | Python auto-instrumentation (opentelemetry-distro) |
java -javaagent:opentelemetry-javaagent.jar -jar app.jar | Java auto-instrumentation |
Querying
traceparent: 00-<trace-id 32 hex>-<span-id 16 hex>-01 | W3C propagation header |
{ resource.service.name = "cart" && status = error } | TraceQL: failed spans in cart |
{ span.http.route = "/checkout" && duration > 2s } | TraceQL: slow checkout spans |
LogQL
{namespace="shop", app="cart"} | Stream selector (labels only) |
{namespace="shop"} |= "timeout" != "healthz" | Line filters (contains / not contains) |
{app="cart"} | json | status >= 500 | Parse JSON, filter on a field |
{app="cart"} | logfmt | line_format "{{.level}} {{.msg}}" | Parse logfmt, reformat |
sum by (app) (rate({namespace="shop"} |= "error" [5m])) | Error lines per second, per app |
Label rules
Good labels: cluster, namespace, app, container, level | Few values, used to select streams |
Bad labels: user_id, trace_id, request_id, pod IP | Unbounded → too many streams |
Structured metadata (Loki 3) | High-cardinality fields attached without being labels |
Links to set up
Exemplars on histograms (Prometheus: --enable-feature=exemplar-storage) | Metric spike → example trace |
Tempo data source: trace to logs (Loki) | Span → that pod's logs around that time |
Tempo data source: trace to metrics | Span → the service's RED metrics |
Loki data source: derived field on trace_id | Log line → trace |
Prerequisites
Same service.name / namespace / pod names across signals | The join keys |
trace_id + span_id in every log line | Via OTel log bridges or logging instrumentation |
Synchronized clocks (NTP/chrony) | Time-window jumps line up |
Definitions
SLI = good events / valid events | e.g. requests < 300 ms and not 5xx, divided by all requests |
SLO = target for an SLI over a window | e.g. 99.9% over 30 days |
Error budget = 1 − SLO | 0.1% → 43.2 minutes of total outage per 30 days |
Burn rate = error ratio / (1 − SLO) | 1 = using the budget exactly over the window |
Standard alert pairs (30-day SLO)
14.4× over 1h AND 5m → page | 2% of budget in 1 hour |
6× over 6h AND 30m → page | 5% of budget in 6 hours |
1× over 3d AND 6h → ticket | 10% of budget in 3 days |
Cost levers
Traces: tail sampling (errors + slow + small %) | Keep what matters |
Metrics: drop unused series/labels (relabeling, filter processor) | Cardinality is cost |
Logs: drop noise at the source; shorter retention for debug data | Largest volume |
Retention tiers per signal and tenant | Pay for history only where it's used |
Platform
X-Scope-OrgID: <tenant> | Tenant header for Loki, Tempo and Mimir |
transform / attributes / redaction processors | Remove PII and secrets before storage |
otelcol_exporter_send_failed_*, otelcol_processor_refused_* (names vary by version) | Collector health signals |
