Lifecycle & backup · Prasanta Kumar Behera

Keep a logging cluster affordable and recoverable: index lifecycle management with rollover and tiers, deleting on schedule, disk watermarks and the red-cluster recovery, snapshot repositories on S3-compatible storage, SLM policies, and restore drills you've actually run.

Index lifecycle management (ILM)

Logs lose value with age. ILM moves each backing index through phases and deletes it on schedule:

PUT _ilm/policy/logs-30d
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
      "warm":   { "min_age": "3d",  "actions": { "forcemerge": { "max_num_segments": 1 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

Attach the policy in the data stream's index template ("index.lifecycle.name": "logs-30d"). min_age counts from rollover. Tiers: hot (fast SSD, writes), warm/cold (cheaper nodes, reads), frozen (searchable snapshots, a paid feature), delete. Recent versions also offer a simpler data stream lifecycle (just retention) as an alternative to ILM.

A newspaper library keeps today's papers on the front desk (hot), last week's on shelves in the back (warm), last year's in the basement (cold), and recycles very old ones (delete). Every night, a photographer takes pictures of everything and stores them in another building (snapshots), and once in a while the librarian practises rebuilding a shelf from the photos (restore drill).

Disk watermarks

Watermark (default)	Effect
low 85%	No new shards allocated to the node
high 90%	Shards relocated away from the node
flood_stage 95%	`read_only_allow_delete` block on affected indices: writes fail

Scenario: the cluster is red and disk is at 95%

A log storm filled the hot nodes. Writes are failing (Fluent Bit shows cluster_block_exception), and health is red because a new backing index couldn't allocate its primary.

Recovery steps

Stop the bleeding: find the storm source (volume per namespace), drop or rate-limit it at Fluent Bit.
Free space: delete the oldest backing indices you can afford to lose (DELETE .ds-logs-…-000123, which you can do on a data stream's non-write indices), or add data nodes/disk.
Once usage is below the high watermark, the read-only block is released automatically (recent versions). Otherwise remove it: PUT logs-*/_settings {"index.blocks.read_only_allow_delete": null}.
Check GET _cluster/allocation/explain for the red index; after space is freed it should allocate. Then roll the data stream over if needed.
Afterwards: ILM delete ages that fit the disk, alerts at 75–80% disk, and a cap on per-namespace volume.

Snapshots and SLM

Register a repository on S3 or S3-compatible storage (MinIO, Ceph RGW). The s3 repository type is built into Elasticsearch 8.x; credentials go into the keystore (with ECK, via spec.secureSettings referencing a Secret with s3.client.default.access_key and s3.client.default.secret_key). For MinIO, set the client endpoint and path-style access in the Elasticsearch config.

PUT _snapshot/s3-backups
{ "type": "s3", "settings": { "bucket": "es-snapshots", "base_path": "logs-prod" } }

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "s3-backups",
  "config": { "indices": [ "logs-*" ], "include_global_state": false },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}

Snapshots are incremental (only new segments are copied). Restore to renamed indices to inspect without disturbing live data:

POST _snapshot/s3-backups/nightly-2026.09.26/_restore
{
  "indices": "logs-payments-*",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_aliases": false
}

(Restoring data stream backing indices has specific rules; see the snapshot docs for your version.) Snapshots are not a copy of the running cluster in the same failure domain: keep the bucket in another location or account, with versioning or object lock against deletion.

Try it: lifecycle and a real restore

Create an ILM policy with rollover at max_docs: 1000 (lab only) and delete after 10m; attach it to a lab data stream, send logs, and watch _ilm/explain.
Run MinIO in the cluster, create a bucket, add the S3 credentials via ECK secureSettings, and register the repository.
Create an SLM policy, execute it, and list snapshots.
Delete a backing index, then restore it from the snapshot (renamed) and search it in Kibana.
Time the restore and write down the runbook steps you needed.

Going deeper: cost and durability

Size hot storage from daily volume × hot days × (1 + replicas) × overhead; alert well before 85%.
Lower replicas on warm/cold if snapshots cover the risk; never zero replicas on hot data you can't lose.
Snapshot cluster state (templates, ILM policies, Kibana saved objects) separately with include_global_state or keep it in Git.
Test major upgrades by restoring a snapshot into a staging cluster first.

Recap

ILM: rollover in hot, age through warm/cold (frozen is paid), delete on schedule; or a simple data stream lifecycle.
Watermarks 85/90/95%; flood stage blocks writes. Fix the source, free space, then check allocation.
Snapshots to S3-compatible storage in another failure domain; SLM schedules and retains them.
Restore drills prove the backup works and measure recovery time.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.