ECK operator on Kubernetes

Why an operator?

Elasticsearch is stateful and cluster-aware: node discovery, TLS certificates, bootstrapping masters, rolling restarts that don't lose shards, upgrades in the right order. ECK (Elastic Cloud on Kubernetes) encodes this in an operator: you declare the cluster, and ECK creates StatefulSets, Services, certificates and credentials, and handles changes safely.

Install ECK

$ helm repo add elastic https://helm.elastic.co && helm repo update
$ helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace

(Or apply the crds.yaml and operator.yaml manifests for your ECK version from Elastic's docs. Pin the version.)

A logging cluster

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logs
  namespace: logging
spec:
  version: 8.15.3                 # use a current supported version
  nodeSets:
    - name: masters
      count: 3
      config:
        node.roles: [ "master" ]
      volumeClaimTemplates:
        - metadata: { name: elasticsearch-data }
          spec:
            accessModes: [ "ReadWriteOnce" ]
            resources: { requests: { storage: 10Gi } }
            storageClassName: fast-ssd
    - name: hot
      count: 3
      config:
        node.roles: [ "data_hot", "data_content", "ingest" ]
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests: { memory: 8Gi, cpu: "2" }
                limits: { memory: 8Gi }
      volumeClaimTemplates:
        - metadata: { name: elasticsearch-data }
          spec:
            accessModes: [ "ReadWriteOnce" ]
            resources: { requests: { storage: 500Gi } }
            storageClassName: fast-ssd
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: logs
  namespace: logging
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: logs

The volume claim must be named elasticsearch-data. Use local or fast block storage for data nodes; Elasticsearch replicates shards itself, so replicated storage underneath is often unnecessary (see Kubernetes Storage & Data Protection).

What ECK generates

Create dedicated users or API keys with minimal roles for Fluent Bit (write to logs-*) and for people (read specific data streams).

Kernel settings

Elasticsearch needs vm.max_map_count of at least 262144 on each node. Options: set it in the node image or OS config, a privileged init container (sysctl -w vm.max_map_count=262144), or a DaemonSet. For small test clusters only, set node.store.allow_mmap: false in the nodeSet config instead.

Scaling and upgrades

• Scale by changing count; ECK adds nodes, and when scaling down it migrates shards off the nodes before removing them.
• Upgrade by changing spec.version (and Kibana's). ECK performs a rolling upgrade. First read the release notes, check the supported upgrade path (a major upgrade must start from a specific recent minor of the previous major; the docs name it), and take a snapshot (lesson 08).
• Changing a nodeSet's name or storage class creates a new nodeSet and migrates data, which is powerful but slow for big clusters.

Recap

• ECK manages Elasticsearch and Kibana as custom resources: StatefulSets, TLS, credentials, safe changes.
• Define nodeSets by role, with elasticsearch-data volume claims and requests = limits.
• ECK creates the HTTPS service, CA and elastic user secret; create least-privilege users for clients.
• Raise vm.max_map_count; scale via count, upgrade via version, with snapshots and release notes first.