Prasanta Kumar Behera
Centralized Logging with EFK›06 · Kibana & KQL
Learning Hub / Observability & Reliability / Centralized Logging with EFK

Lesson 06 of 8 · Modules

Kibana & KQL

Search logs quickly in Kibana: data views, Discover, KQL syntax for fields, wildcards, ranges and boolean logic, when to use Lucene or ES|QL, saving searches, and giving teams access to only their logs with spaces and roles.

Beginner → Practitioner
Key wordsKibanaDiscoverdata viewsKQLLuceneES|QLsaved searchesfield filterstime rangespacesroles

Data views and Discover

A data view tells Kibana what to search (e.g. logs-*) and which field is the time (@timestamp). Discover is where you search and read logs: choose the data view, set the time range, type a query, add filters, and pick columns (namespace, pod, level, message) so each line is readable.

Kibana is the library's search computer. A data view chooses which shelves to search. KQL is how you type your question: "books by this author, from this year, about dragons, but not the scary ones." The time picker is choosing which years to look in: a smaller range is much quicker to search.

KQL essentials

Want KQL
Exact field value kubernetes.namespace_name : "shop"
Any of several log.level : (error or fatal)
Phrase in message message : "connection refused"
Wildcard kubernetes.pod_name : cart-*
Range http.status >= 500
Exists trace.id : *
Negation not kubernetes.container_name : "istio-proxy"
Nested logic service.name : "cart" and (log.level : "error" or http.status >= 500)

Tips:

  • Filter first, then search: namespace/service/level as filters (pills), then the message text.
  • Keep the time range tight: last 15 minutes during incidents, widen only when needed.
  • Wildcards work best on keyword fields; leading wildcards (*error) are expensive.
  • The field list on the left shows top values: a quick way to see which pods or levels dominate.

Lucene and ES|QL

  • Lucene syntax (toggle in the query bar) adds regex and fuzzy queries: message:/timeout.*/.
  • ES|QL (a piped query language in recent Elasticsearch versions) is useful for ad-hoc analytics on logs:
FROM logs-*
| WHERE @timestamp > NOW() - 1 hour AND log.level == "error"
| STATS errors = COUNT(*) BY service.name
| SORT errors DESC
| LIMIT 10

Saved searches and sharing

  • Save useful searches (with columns and filters) as building blocks for dashboards and alerts (lesson 07).
  • Share links with the query and time range embedded, in incident channels, so everyone looks at the same thing.
  • Use absolute time ranges in links for incident timelines (relative links change meaning over time).

Access: spaces and roles

  • Spaces separate saved objects (data views, dashboards) per team.
  • Roles grant index privileges: e.g. read on logs-payments-* only for the payments team; features per space.
  • Document- and field-level security (restricting documents or hiding fields within an index) is a paid feature in Elastic's default distribution; routing teams to separate data streams (lesson 05) achieves most of the same with basic features.
  • Authenticate people via SSO (SAML/OIDC; check which realms your subscription level includes).

Try it: incident-style searching

  1. Create a data view logs-* with @timestamp.
  2. In Discover, add columns kubernetes.namespace_name, kubernetes.pod_name, log.level, message.
  3. Write KQL for: all errors in one namespace in the last 15 minutes, excluding one container; then all 5xx responses from pods starting with cart-.
  4. Run the ES|QL query above (if your version supports it) to find the noisiest services.
  5. Save the search as "shop errors" and share a link with an absolute time range.

Going deeper: fast, safe searching

  • Big wildcard data views over months of data are slow: create narrower data views (per team or type) and use runtime fields sparingly.
  • Correlate with traces: if logs carry trace.id, link from Kibana to your tracing UI (see Observability with OpenTelemetry).
  • Train on-call engineers with saved searches for common incidents rather than expecting perfect KQL at 3 a.m.

Recap

  • Data views define what to query; Discover is for reading and filtering logs.
  • KQL: field : value, and/or/not, wildcards, ranges, field : * for exists.
  • Lucene for regex/fuzzy; ES|QL for piped analytics.
  • Save and share searches; control access with spaces, roles and separate data streams.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.