Prasanta Kumar Behera
Observability with OpenTelemetry›01 · Observability fundamentals
Learning Hub / Observability & Reliability / Observability with OpenTelemetry

Lesson 01 of 7 · Modules

Observability fundamentals

What observability adds to monitoring, what metrics, logs and traces are each good for, the RED and USE methods, and how OpenTelemetry gives you one standard way to produce and ship all three signals with consistent names.

Beginner → Practitioner
Key wordsmonitoring vs observabilitymetricslogstracesREDUSEOpenTelemetryOTLPresource attributessemantic conventions

Monitoring vs observability

Monitoring answers questions you knew to ask in advance: is CPU high? is the error rate above 1%? Observability is being able to answer new questions about a system from its outputs, without shipping new code: why are only Android users in one region seeing timeouts since the 14:05 deploy?

Monitoring is the smoke alarm: it beeps when there's smoke. Observability is being a detective with good clues: footprints (logs), a timeline of who went where (traces), and the thermometer readings every minute (metrics). With good clues, you can solve mysteries nobody predicted.

Three signals, three jobs

Signal Good at Cost Example
Metrics Trends, alerting, "how much/how often" Low (aggregated) http_requests_total{code="500"}
Logs Detail of individual events High (per event) payment declined: card expired
Traces Where time goes across services Medium (sampled) checkout → cart → payments (1.8 s in payments)

A typical investigation: a metric alert fires (errors up) → traces show which hop fails → logs of that pod explain why. Lesson 05 makes those jumps one click.

Two methods that tell you what to measure

  • RED for services: Rate (requests/s), Errors (failed/s), Duration (latency distribution).
  • USE for resources: Utilization, Saturation (queueing), Errors.

Start every service dashboard with RED, and every node/database/queue dashboard with USE.

OpenTelemetry in one picture

app code ──(OTel API/SDK or auto-instrumentation)──► OTLP ──► OTel Collector
                                                            │  receive, process, export
                                  ┌─────────────────────────┼──────────────────────┐
                                  ▼                         ▼                      ▼
                         metrics: Prometheus/Mimir   traces: Tempo/Jaeger   logs: Loki/Elasticsearch
                                  └──────────── Grafana (or a vendor) ─────────────┘
  • API and SDKs for many languages, plus auto-instrumentation (Java agent, Python, Node.js, .NET; eBPF-based options for others).
  • OTLP: one protocol for all signals (gRPC port 4317, HTTP 4318).
  • The Collector: a pipeline that receives, processes and exports telemetry (lesson 02).
  • Semantic conventions: standard attribute names (service.name, http.response.status_code, k8s.pod.name), so tools and dashboards work across services.

OpenTelemetry is a CNCF project; traces, metrics and logs are stable in the specification, while support maturity varies by language SDK. Check your language's status page.

Try it: see all three signals (Docker)

  1. Run the OpenTelemetry Demo (the official "Astronomy Shop") with Docker Compose, following its docs; it includes a Collector, Jaeger, Prometheus and Grafana.
  2. Open Grafana and find a RED dashboard for one service.
  3. Open Jaeger, pick the frontend service, and open a trace: count the services involved and find the slowest span.
  4. Use the demo's feature flags to inject a failure, then find it in metrics, then in a trace, then in logs.
  5. Write down which signal was fastest for each question you asked.

Going deeper: instrument with intent

  • Instrument boundaries first: inbound requests, outbound calls, queues, database clients. Auto-instrumentation covers most of it.
  • Put business context (tenant, plan, feature flag) on spans and logs, but keep it out of metric labels (cardinality; see Scaling Prometheus to Production).
  • Decide ownership: the platform team runs the Collector and backends; service teams own instrumentation and dashboards for their service.

Recap

  • Observability = answering new questions from telemetry; monitoring = known questions.
  • Metrics (cheap trends), logs (detail), traces (where time goes); use them together.
  • RED for services, USE for resources.
  • OpenTelemetry: APIs/SDKs, auto-instrumentation, OTLP, the Collector and semantic conventions, with backends of your choice.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.