Kibana on-call dashboard · Prasanta Kumar Behera

Go from an empty Kibana to a dashboard on-call engineers actually use, plus alerts on log patterns: which panels answer 'what's broken right now', building them with Lens, alert rules and their connectors (and licensing limits), and keeping alerts few and actionable.

What on-call needs from logs

At 3 a.m., the questions are: Is something broken? Where? Since when? What do the errors say? A good logging dashboard answers these top to bottom in under a minute.

An on-call dashboard is the car's dashboard, not the engine manual. It shows a few big, clear things: speed, fuel, warning lights. When a light turns red, you open the manual (the saved search with the actual log lines).

A dashboard, top to bottom

Errors over time (stacked by service.name or namespace): the "warning light".
5xx responses from ingress or access logs: user impact.
Top 10 pods/services by error count (a table): where to look.
Log volume per namespace: spikes (log storms) and gaps (pipeline problems).
Latest errors (a saved search panel with namespace, pod, message): the lines themselves.

Build panels with Lens (drag fields; choose a date histogram and a breakdown), add dashboard-level controls (namespace, cluster dropdowns), and save the dashboard in the team's space. Export dashboards (saved objects) to Git so they can be recreated.

Alerting on logs

Kibana rules check conditions on a schedule and trigger actions through connectors:

Rule type	Example
Elasticsearch query	`service.name : "payments" and log.level : "error"` matches more than 20 docs in 5 minutes
Log threshold	Ratio of `http.status >= 500` to all requests above 5%
Absence (query count below 1)	No logs from `ingress-nginx` in 10 minutes → pipeline or ingress down

Connectors deliver the alert: Index and Server log are available on the free Basic license; Slack, email, PagerDuty, webhooks and others depend on your subscription (check Elastic's subscription page for your version). On self-managed clusters without those, teams often alert from Grafana (Elasticsearch data source) or ElastAlert 2, routing to the same place as metric alerts (e.g. Alertmanager).

Keep alerts few and good

Prefer metric-based alerts for rates and latency (cheaper, faster; see Scaling Prometheus to Production). Use log alerts for specific patterns metrics don't capture (a particular error message, security events, pipeline silence).
Every alert needs an owner, a runbook link and a saved search link.
Use windows and thresholds that ignore one-off blips, and review noisy alerts weekly.

Try it: from empty Kibana to an alert

Deploy a small app that logs JSON with random errors and 5xx statuses (or use the loggers from earlier lessons).
Build the five panels above with Lens and add a namespace control.
Create an Elasticsearch query rule for "more than 10 errors in 5 minutes" with the Server log or Index connector; trigger it by increasing the error rate.
Create an "absence" rule for one namespace; delete that app's pods and wait for it to fire.
Export the dashboard as NDJSON and import it into another space.

Going deeper: operational maturity

Put dashboards and rules in code (saved objects export, the Kibana API, or Terraform's Elastic provider) with review, like any other config.
Tie log alerts to SLOs rather than raw counts where possible (see Observability with OpenTelemetry).
Add a "logging pipeline health" dashboard for the platform team: Fluent Bit retries/drops, Elasticsearch indexing rate, rejected writes, disk usage.

Recap

On-call dashboard: errors over time, 5xx, top offenders, volume per source (gaps!), latest errors.
Build with Lens, add controls, keep dashboards in Git.
Rules + connectors alert on log patterns and absence; check what your license includes, or alert via Grafana/ElastAlert 2.
Few, owned, runbook-linked alerts; metrics for rates, logs for patterns.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.