Elasticsearch fundamentals

Documents, indices and shards

• A document is one JSON record: here, one log line with its fields.
• An index is a collection of documents with the same mapping (schema).
• An index is split into primary shards; each primary has replica copies on other nodes.
• Shards are the unit of distribution: they move between nodes, recover, and are searched in parallel.

Cluster health

GET _cluster/allocation/explain tells you why a shard isn't assigned (disk watermark, no eligible node, allocation rules…).

Mappings: text vs keyword

{
  "mappings": {
    "properties": {
      "@timestamp":  { "type": "date" },
      "message":     { "type": "text" },
      "log.level":   { "type": "keyword" },
      "kubernetes":  { "properties": {
        "namespace_name": { "type": "keyword" },
        "pod_name":       { "type": "keyword" } } },
      "http.status": { "type": "short" }
    }
  }
}

• text: analyzed for full-text search (message).
• keyword: exact values for filters, sorting, aggregations (namespace, level, service).
• Dynamic mapping creates fields automatically, which is convenient but risky: a field that's a number in one log and a string in another causes mapping conflicts, and free-form keys cause mapping explosion (thousands of fields). Control it with index templates and limits (index.mapping.total_fields.limit).

Data streams for logs

Logs are append-only and time-based. A data stream (e.g. logs-shop-prod) is written to by name, while Elasticsearch creates backing indices (.ds-logs-shop-prod-2026.09.27-000001) and rolls over to a new one by size or age (lesson 08). An index template with "data_stream": {} defines settings and mappings for them. Documents need an @timestamp field, and writes use the create operation.

Nodes, heap and memory

• Recent Elasticsearch versions size the JVM heap automatically from the container's memory limit (about half). Keep the heap below ~31 GB so the JVM can use compressed object pointers.
• The rest of the memory is filesystem cache, which Elasticsearch relies on heavily for search speed.
• Set memory requests = limits for Elasticsearch pods and use fast disks (SSD) for hot data.

Shard sizing rules of thumb

• Aim for shards of roughly 10–50 GB; fewer, larger shards beat many small ones.
• Use 1 primary shard for small daily log volumes; increase only when a single shard would exceed the target size.
• Watch the total: there's a default limit of 1,000 shards per non-frozen data node (cluster.max_shards_per_node), and you should stay far below it.

Recap

• Documents in indices, split into primary and replica shards.
• green/yellow/red is about shard assignment; allocation/explain tells you why.
• text for full-text, keyword for filters and aggregations; control dynamic mapping with templates.
• Logs go into data streams with rollover.
• Dedicated masters, heap ≤ ~31 GB (about half of memory), 10–50 GB shards, and not too many of them.