Prasanta Kumar Behera
Centralized Logging with EFK›01 · EFK architecture
Learning Hub / Observability & Reliability / Centralized Logging with EFK

Lesson 01 of 8 · Modules

EFK architecture

How a Kubernetes logging platform fits together: where container logs are written on each node, how a Fluent Bit DaemonSet collects and enriches them, how Elasticsearch stores and indexes them, and how Kibana searches them, plus what to decide before building one.

Beginner → Practitioner
Key wordscentralized loggingEFKELKFluent Bit DaemonSetcontainer logs/var/log/containersElasticsearchKibanadata flowstructured logs
every node apps → stdout/stderr /var/log/containers/*.log Fluent Bit (DaemonSet) parse · enrich · buffer Elasticsearch HTTPS bulk in data streams, ILM Kibana Discover, KQL dashboards, alerts ECK operator manages Elasticsearch + Kibana
Fluent Bit on every node, Elasticsearch to store and search, Kibana to explore.

Why centralize logs?

On Kubernetes, logs are short-lived: a pod is deleted and its logs are gone; the kubelet rotates files; a node is replaced. During an incident you need to search all pods, across days, by namespace, app or error text, and to alert on patterns. That's what a central logging platform provides.

Every classroom (node) has a notebook where each student (container) writes what they're doing. The notebooks get torn out and thrown away when they're full. A messenger (Fluent Bit) visits every classroom, copies each new line, adds a sticker saying which class and which student, and takes it to the school library (Elasticsearch). Anyone can then search the library's catalogue (Kibana), even for pages that were thrown away in the classroom long ago.

Where container logs come from

  1. Applications write to stdout/stderr (the Kubernetes convention).
  2. The container runtime (containerd/CRI-O) writes each line to a file on the node: /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log, in the CRI log format (<timestamp> <stream> <P|F> <message>).
  3. /var/log/containers/ holds symlinks named <pod>_<namespace>_<container>-<id>.log, which is how collectors learn the pod and namespace.
  4. The kubelet rotates these files (containerLogMaxSize, containerLogMaxFiles).

The EFK pipeline

Component Role Runs as
Fluent Bit Collect, parse, enrich, buffer, ship DaemonSet (one per node)
Elasticsearch Store, index, search, aggregate StatefulSets managed by the ECK operator
Kibana Search UI, dashboards, alerting Deployment

ELK replaces the collector/processor with Logstash (a JVM service, very flexible but heavier). Lesson 04 compares the options.

Decisions before you build

  • Volume: GB per day, and peak lines per second. This sizes everything else.
  • Retention: how many days are hot and searchable fast, how long in cheaper tiers, when to delete (lesson 08).
  • Structure: JSON logs from applications make parsing trivial and fields searchable (lesson 05).
  • Access: who can see which namespaces' logs (logs contain personal data and secrets more often than people think).
  • Licensing: Elasticsearch and Kibana are source-available under Elastic's licences (with an AGPL option since 2024); OpenSearch is the Apache-2.0 fork. The concepts in this track apply to both, but features differ.

Try it: find the logs on a node

  1. In a kind cluster, run kubectl run logger --image=busybox -- sh -c 'while true; do echo "hello $(date)"; sleep 2; done'.
  2. kubectl logs logger, then find the same lines on the node: docker exec -it kind-control-plane ls -l /var/log/containers/ | grep logger and tail the file.
  3. Look at the CRI format of a line (timestamp, stream, tag, message).
  4. Delete the pod; confirm kubectl logs can no longer show its lines. That gap is what central logging fills.
  5. Estimate your cluster's log volume: sum the growth of /var/log/pods over an hour on one node.

Going deeper: logging design

  • Logs are the most expensive telemetry per useful bit. Prefer metrics for counting and traces for latency, and keep logs for detail (see Observability with OpenTelemetry).
  • Plan for back-pressure: when Elasticsearch is slow, where do logs wait, and what gets dropped first? (lessons 04 and 05)
  • Separate audit logs (Kubernetes API audit, security events) from application logs, with stricter retention and access.

Recap

  • Apps log to stdout/stderr; the runtime writes files on each node; the kubelet rotates them.
  • Fluent Bit (DaemonSet) tails, parses, enriches and ships; Elasticsearch stores and searches; Kibana is the UI.
  • Decide volume, retention, structure, access and licensing before building.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.