Rook-Ceph · Prasanta Kumar Behera

Run Ceph inside Kubernetes with the Rook operator: what MONs, OSDs and MGRs do, block (RBD), file (CephFS) and object (RGW) storage, failure domains with CRUSH, and the health commands every operator needs.

Ceph and Rook in one picture

Ceph is a mature, distributed storage system providing block, file and object storage from the same cluster of disks. It's used widely in private clouds, including OpenStack (see OpenStack Private Cloud). Rook is a Kubernetes operator that deploys and manages Ceph as pods, driven by custom resources.

Ceph is a giant library spread over many buildings. The librarians' council (MONs) keeps the master catalogue and must agree on it. The shelves (OSDs, one per disk) hold the books, with every book stored in three different buildings. A clever filing rule (CRUSH) decides exactly which shelves each book goes on, so anyone can find it without asking a central desk.

The daemons

Daemon	Role	How many
MON	Cluster maps and quorum	3 (or 5), on different nodes
MGR	Metrics, dashboard, orchestration modules	2 (active + standby)
OSD	Stores data on one disk; replicates and recovers	One per disk
MDS	Metadata for CephFS	2+ if you use CephFS
RGW	S3/Swift object gateway	2+ if you use object storage

Three interfaces

Interface	Kubernetes use	Access mode
RBD (RADOS Block Device)	Databases, general PVCs	RWO (block)
CephFS	Shared file storage	RWX
RGW	S3-compatible buckets (backups, artifacts, logs)	ObjectBucketClaims, or plain S3

A minimal Rook deployment (outline)

Nodes with raw, empty disks (no partitions or filesystems). Rook won't touch disks with data on them.
Install the Rook operator (Helm chart rook-release/rook-ceph).
Create a CephCluster resource. The main knobs:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:<version>     # pick a supported release
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
  storage:
    useAllNodes: true
    useAllDevices: true                    # or list specific nodes and devices

Create a CephBlockPool (replicated, size: 3, failureDomain: host) and a StorageClass using the RBD CSI driver. Rook ships example manifests for all of these in its repository.

Health: learn to read ceph status

$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     4b1c…
    health: HEALTH_WARN
            1 osds down
            Degraded data redundancy: 1204/3612 objects degraded (33.333%)
  services:
    mon: 3 daemons, quorum a,b,c
    mgr: a(active), standbys: b
    osd: 6 osds: 5 up, 6 in
  data:
    pools:   2 pools, 64 pgs
    usage:   210 GiB used, 1.6 TiB / 1.8 TiB avail
    pgs:     40 active+clean
             24 active+undersized+degraded

Read it top-down: health (OK/WARN/ERR), quorum, OSDs up/in, and PG states (active+clean is the goal). Here one OSD is down; data is still available but degraded, and Ceph will re-replicate once the OSD is marked out.

Failure domains with CRUSH

CRUSH places replicas according to rules and a hierarchy (root → rack → host → OSD). failureDomain: host spreads copies over hosts; rack or zone spreads them wider, if your topology labels support it. Match the failure domain to how things actually fail in your data centre.

Try it: a small Rook-Ceph cluster

Use three VMs, each with an extra empty disk (e.g. 20 GiB), in a kubeadm cluster.

Install the Rook operator and apply a CephCluster like the one above (Rook's cluster-test.yaml example is fine for a lab).
Deploy the toolbox and run ceph status until it shows HEALTH_OK and all PGs active+clean.
Create an RBD pool and StorageClass, then a PVC and a pod that writes data.
Stop one OSD's node. Watch ceph status show degraded PGs, then recovery when it returns.
Run ceph osd tree and ceph df and explain every line.

Going deeper: Ceph in production

Ceph wants dedicated disks, fast networks (a separate cluster/replication network helps), and enough RAM per OSD. Undersized hardware is the most common cause of pain.
Keep capacity below ~75–80%: recovery needs free space, and Ceph stops writes when OSDs hit their full ratios.
Change one thing at a time and let the cluster return to HEALTH_OK between steps. Recovery traffic competes with client I/O.
For many organisations, an external Ceph cluster consumed by Kubernetes (Rook in external mode, or plain ceph-csi) is simpler to operate than Ceph inside every cluster.

Recap

Ceph = MONs (quorum, maps) + OSDs (data, one per disk) + MGRs, plus MDS (CephFS) and RGW (S3) as needed.
RBD for block (RWO), CephFS for shared files (RWX), RGW for objects.
Rook runs it all as Kubernetes resources; the toolbox gives you the ceph CLI.
CRUSH failure domains decide where replicas go; aim for HEALTH_OK and active+clean, and keep free space.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.