Structured logging & Loki · Prasanta Kumar Behera

Logs the observability way: structured logs with trace context, Grafana Loki's label-indexed design, choosing labels without blowing up cardinality, LogQL for filtering, parsing and metrics from logs, and collecting logs with Alloy or the OTel Collector.

Structured logs with context

A useful log line is structured and carries context:

{"ts":"2026-09-27T10:15:02.123Z","level":"error","service":"cart","msg":"payment declined",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","order_id":"A-1042"}

Structured (JSON or logfmt): fields are parseable without fragile regex.
Trace context (trace_id, span_id): lets you jump from a log line to the trace (lesson 05). OTel logging integrations and log bridges add these automatically in many languages.
Consistent names across services (the OTel semantic conventions or your own standard).

Loki is a library that only catalogues the shelf labels: "Shop department, cart team, errors". It doesn't index every word in every book, which makes it cheap. To find something, you first pick the right shelves using labels, then flip through those books quickly. If you invent a new shelf for every single visitor (a label per user), the library drowns in shelves.

Loki's design

A stream = a unique set of labels, e.g. {cluster="prod-eu-1", namespace="shop", app="cart", container="cart"}.
Loki indexes labels only, compresses log lines into chunks, and stores them in object storage (S3, GCS, MinIO).
Queries select streams by labels, then scan the matching chunks for filters.

This makes Loki cheap to run, as long as labels stay low-cardinality.

Choosing labels

Label	Good?	Why
`cluster`, `namespace`, `app`, `container`	✅	Few values, always used to narrow queries
`level`	✅ (often)	Few values
`pod`	⚠️	Churns with every rollout; often fine, but consider leaving it out
`user_id`, `trace_id`, `request_id`, `ip`	❌	Unbounded: explodes streams

Loki 3 adds structured metadata: key-value pairs attached to log lines (e.g. trace_id) that aren't labels but can be filtered on. OTLP ingestion places many resource attributes there automatically.

LogQL

{namespace="shop", app="cart"} |= "timeout"                    # line contains
{namespace="shop"} != "healthz" |~ "5\\d\\d"                   # not contains, regex
{app="cart"} | json | status >= 500 | line_format "{{.msg}}"   # parse + field filter
sum by (app) (rate({namespace="shop"} |= "error" [5m]))         # metric from logs
topk(5, sum by (app) (count_over_time({cluster="prod-eu-1"}[1h])))  # noisiest apps

Order matters for speed: label selector → line filters → parser → field filters.

Collecting logs

Grafana Alloy (Grafana's OpenTelemetry-compatible collector) is the recommended agent for Loki; Promtail is deprecated in its favour.
The OpenTelemetry Collector with the filelog receiver and k8sattributes can send logs to Loki via OTLP (lesson 02).
Either way: a DaemonSet tails /var/log/pods, adds Kubernetes metadata, and pushes to Loki (the same pattern as Fluent Bit in Centralized Logging with EFK).

Loki or Elasticsearch?

	Loki	Elasticsearch
Index	Labels only	Full-text and fields
Storage cost	Low (object storage)	Higher (indexed data on disks)
Ad-hoc full-text search across everything	Slower (scans)	Fast
Operational model	Grafana stack (with Mimir, Tempo)	Elastic stack (Kibana)

Many platforms pick Loki for Kubernetes application logs (cheap, Grafana-native), and Elasticsearch/OpenSearch where heavy full-text search, security analytics or existing Elastic skills justify it.

Try it: Loki and LogQL

Install Loki (single-binary / monolithic mode for a lab) and Grafana with their Helm charts, plus Alloy or an OTel Collector DaemonSet.
Deploy an app that logs JSON with levels and random 5xx statuses.
In Grafana Explore, query {namespace="default"}, then add | json | status >= 500.
Build a panel with sum by (app) (rate({namespace="default"} | json | level="error" [5m])).
Add a label with pod IPs to your collector config on purpose, watch the active stream count grow (Loki's metrics), then revert.

Going deeper: Loki in production

Set retention per tenant or stream (compactor-based retention), and limits (ingestion rate, streams per user) to protect the cluster.
Use recording rules in Loki's ruler for expensive LogQL metric queries you run constantly.
For multi-tenancy, every request carries an X-Scope-OrgID tenant header (lesson 07).
Don't log secrets or personal data; mask them in the collector before they're stored.

Recap

Logs should be structured and carry trace context.
Loki indexes labels only and stores chunks in object storage: cheap, if labels stay low-cardinality.
High-cardinality values go in the line or structured metadata, never labels.
LogQL: selector → filters → parser → field filters; plus metric queries.
Collect with Alloy or the OTel Collector; choose Loki vs Elasticsearch by search needs and cost.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.