Parsing & routing · Prasanta Kumar Behera

Turn raw lines into useful fields: prefer JSON logs, parse the rest with regex parsers, join multi-line stack traces, use pod annotations to choose parsers or exclude noise, enrich and drop fields, and route different logs to different data streams.

Structure first

The best parser is the one you don't need. Ask teams to log JSON with consistent fields:

{"@timestamp":"2026-09-27T10:15:02.123Z","log.level":"error","service.name":"cart",
 "message":"payment declined","trace.id":"4bf92f3577b34da6a3ce929d0e0e4736","http.status":502}

With merge_log on, the kubernetes filter parses this into fields automatically. Aligning names with a common schema (Elastic Common Schema, ECS, or OpenTelemetry semantic conventions) makes cross-service dashboards possible.

Unstructured logs are like letters written in everyone's own handwriting: someone has to read and decode each one. Structured JSON logs are like filled-in forms: name here, date here, problem here. The library can file forms instantly; letters need a patient decoder (a regex parser).

Parsing the rest

For apps you can't change (nginx, legacy software), define parsers:

parsers:
  - name: nginx_access
    format: regex
    regex: '^(?<remote>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+) (?<path>[^"]*) (?<proto>[^"]*)" (?<status>\d{3}) (?<size>\d+)'
    time_key: time
    time_format: '%d/%b/%Y:%H:%M:%S %z'
    types: status:integer size:integer

Then choose it per pod with an annotation (with k8s-logging.parser: on):

metadata:
  annotations:
    fluentbit.io/parser: nginx_access

Test regexes against real lines before deploying (Fluent Bit's stdout output or a local fluent-bit run), because one bad parser silently leaves logs unparsed.

Multiline: one exception, one document

  filters:
    - name: multiline
      match: kube.*
      multiline.key_content: log
      multiline.parser: java, python, go

Built-in multiline parsers exist for docker, cri, java, python, go and ruby; custom ones use start/continuation regex rules. Note the two stages: the tail input's multiline.parser cri, docker re-joins lines split by the runtime; the multiline filter joins application stack traces.

Enrich, drop, rename

    - name: grep
      match: kube.*
      exclude: log ^\s*$                   # drop empty lines
    - name: modify
      match: kube.*
      add: cluster prod-eu-1               # which cluster sent it (top-level keys only)

Add cluster name and environment, since many clusters share one Elasticsearch.
The modify filter works on top-level keys. To drop noisy Kubernetes metadata, use the kubernetes filter's own options (annotations: off, labels: off), or a nest/lua filter for nested fields.
Drop noise (health-check access logs, debug logs in prod) at the source: it's the cheapest place.
Remove or mask sensitive fields (tokens, emails) before they're stored. The lua filter can mask patterns.

Routing

pipeline:
  filters:
    - name: rewrite_tag
      match: kube.*
      rule: $kubernetes['namespace_name'] ^(payments)$ payments.$TAG false

  outputs:
    - name: es
      match: payments.*
      index: logs-payments-default          # longer-retention ILM via its template
      # … same connection settings …
    - name: es
      match: kube.*
      index: logs-k8s-default

The false at the end of the rule means the original record isn't kept, so payments logs go only to their own data stream.

Scenario: logs from one namespace silently stop

The orders team reports no logs in Kibana since 03:00, but other namespaces are fine. Their pods are running and logging.

Where do you look?

Annotations: did a deploy add fluentbit.io/exclude: "true" or a parser annotation that doesn't match their format (records fail parsing or land in an unexpected field)?
Mapping conflicts: a new field type (e.g. status became a string) makes Elasticsearch reject documents. Fluent Bit logs the bulk errors (Trace_Error on in the es output shows details); Elasticsearch shows nothing in Kibana because nothing was indexed.
Routing: a new rewrite_tag rule or output match sends them elsewhere.
Back-pressure on the nodes where their pods run (lesson 04): paused inputs, dropped chunks.

Fix the cause, then decide whether missing logs can be recovered (only if the files still exist on the nodes).

Try it: shape your logs

Deploy a pod that prints JSON lines and one that prints nginx-style access lines; confirm the JSON becomes fields, and the nginx lines don't (yet).
Add the nginx_access parser and the annotation; check fields in Kibana.
Deploy a Python app that raises an exception every few seconds; add the multiline filter and verify one document per traceback.
Route a namespace to its own data stream with rewrite_tag.
Create a mapping conflict on purpose (a field as number, then string) and find the rejection in Fluent Bit's logs.

Going deeper: parsing at scale

Parse at the edge (Fluent Bit) for cheap operations; use Elasticsearch ingest pipelines for enrichment that needs lookups (geoip, enrich processors).
Version your parsers and test them in CI with sample log lines.
Put an explicit mapping (index template) on important fields instead of relying on dynamic mapping; add a catch-all for unknown fields (flattened type, or dynamic: false for unexpected subtrees).

Recap

JSON logs + merge_log beat any regex; align field names (ECS/OTel).
Regex parsers for the rest, chosen per pod with annotations; test them.
Multiline: runtime re-join in tail, stack traces with the multiline filter.
Enrich (cluster name), drop noise, mask secrets, and route with tags to different data streams.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.