Audit logging & runtime detection · Prasanta Kumar Behera

See what happens in your cluster: design an API audit policy that records what matters without drowning you, and add runtime detection with Falco to catch suspicious behaviour inside containers and nodes.

Two lenses: the API and the runtime

Lens	Sees	Tool
API audit	Every request to the API server: who, what, when, result	Kubernetes audit logging
Runtime	What processes do inside containers and on nodes	Falco (or Tetragon and similar eBPF tools)

You need both. Someone who steals a ServiceAccount token shows up in audit logs; someone who exploits your app and spawns a shell shows up at runtime.

The school has a visitors' book at the front desk: who came in, when, which room they asked for (API audit). It also has hallway cameras that notice if someone climbs through a window or opens the headteacher's drawer (runtime detection). The book can't see the window, and the camera can't read the book.

API audit policy

Rules are evaluated top to bottom; first match wins. Levels: None, Metadata, Request, RequestResponse.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]              # log once the response is known
rules:
  # Drop high-volume, low-value noise
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version"]

  # Secrets, ConfigMaps and tokens: who touched them, never the content
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]

  # Access changes and interactive access: full detail
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  - level: Request
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # Everything else: metadata
  - level: Metadata

Enable it with the flags in the cheat sheet. On kubeadm, mount the policy and log directories into the API server static pod (extraVolumes in the ClusterConfiguration). Ship the log to your log platform (see Centralized Logging with EFK) and keep it long enough for investigations.

Questions audit logs answer:

$ jq -r 'select(.objectRef.resource=="secrets" and .verb=="get") | [.requestReceivedTimestamp, .user.username, .objectRef.namespace, .objectRef.name] | @tsv' audit.log
$ jq -r 'select(.objectRef.subresource=="exec") | [.user.username, .objectRef.namespace, .objectRef.name] | @tsv' audit.log

Runtime detection with Falco

Falco watches system calls (via eBPF) and matches them against rules. Its default rules already catch many attack patterns: a shell spawned in a container, writes below /etc, reading sensitive files, unexpected outbound connections, privileged containers.

$ helm repo add falcosecurity https://falcosecurity.github.io/charts && helm repo update
$ helm install falco falcosecurity/falco -n falco --create-namespace
$ kubectl exec -it deploy/web -- sh -c 'cat /etc/shadow'
$ kubectl -n falco logs -l app.kubernetes.io/name=falco | grep -i -E 'shell|sensitive'
… Notice A shell was spawned in a container with an attached terminal … container_name=web …
… Warning Sensitive file opened for reading by non-trusted program … file=/etc/shadow …

(Rule names and message wording vary by Falco version.)

A custom rule, using macros from the default rule set:

- rule: Package manager run in container
  desc: Package managers should not run in immutable production containers
  condition: spawned_process and container and proc.name in (apt, apt-get, dnf, yum, apk, pip)
  output: "Package manager in container (user=%user.name cmd=%proc.cmdline container=%container.name image=%container.image.repository)"
  priority: WARNING
  tags: [container, drift]

Route alerts with Falcosidekick to Slack, your SIEM or an incident tool, and tune rules for your workloads. A detection nobody reads is noise.

Try it: audit + runtime on kind

Create a kind cluster whose API server mounts an audit policy (kind's documentation has an "auditing" guide using extraMounts and kubeadmConfigPatches). Use the policy above.
Read a Secret and kubectl exec into a pod; find both events in the audit log on the control-plane node with jq.
Confirm the Secret's value doesn't appear in the log.
Install Falco, exec into a pod and run cat /etc/shadow; find the alert.
Add the "Package manager run in container" rule (Falco chart customRules), then run apt-get update inside a Debian-based pod.

Going deeper: detection engineering

Map detections to attacker techniques (e.g. the MITRE ATT&CK containers matrix) to find gaps.
Alert on rare, high-value audit events: new ClusterRoleBindings, break-glass user activity, Secrets read by unexpected identities, exec into production.
Tetragon (Cilium) can also enforce (kill a process on a policy match), not just detect. Use enforcement carefully and progressively.
Keep audit logs tamper-resistant: shipped off-node quickly, write-once storage, restricted access.

Recap

Audit policy: first match wins; None for noise, Metadata for Secrets, detail for RBAC changes and exec.
Enable with API server flags; ship logs centrally; answer "who did what" with jq.
Falco detects runtime behaviour (shells, sensitive files, drift) with eBPF; tune rules and route alerts.
Use both lenses: API and runtime.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.