Storage: Glance, Cinder & Ceph · Prasanta Kumar Behera

Images and volumes in OpenStack: Glance images and formats, Cinder volumes, types and backends, boot-from-volume, snapshots vs backups, and why Ceph RBD (with raw images and copy-on-write clones) is the most common storage backend.

Three kinds of storage in OpenStack

Kind	Service	Lifecycle
Images	Glance	Templates to boot from
Ephemeral disks	Nova (local disk or Ceph)	Live and die with the instance
Volumes	Cinder	Independent block devices: attach, detach, snapshot, back up

Glance is the recipe book for building rooms. Ephemeral disks are the room's whiteboard: wiped when the guest leaves. Cinder volumes are suitcases: you can carry them to another room, make a quick copy on the same shelf (snapshot), or send a copy to a warehouse across town (backup).

Glance images

$ openstack image create --disk-format qcow2 --container-format bare \
    --file noble-server-cloudimg-amd64.img --property os_distro=ubuntu ubuntu-24.04

Formats: qcow2 (compact, common for downloads) and raw (required for efficient Ceph cloning). Convert with qemu-img convert.
Properties tune instances: disk bus, firmware (hw_firmware_type=uefi), hw_qemu_guest_agent=yes, OS type.
Visibility: private, shared (to specific projects), community, public.
Maintain a small golden image set built by a pipeline, not ad-hoc uploads.

Cinder volumes

Cinder schedules volumes onto backends (LVM, Ceph RBD, SAN arrays via drivers) using volume types:

$ openstack volume type create ssd --property volume_backend_name=ceph-ssd
$ openstack volume create --size 50 --type ssd data-01
$ openstack server add volume vm1 data-01

Inside the instance, the volume appears as a new disk (e.g. /dev/vdb or /dev/sdb): partition, format and mount it.

Boot from volume (--boot-from-volume 20 on server create, or creating a volume from an image first) puts the root disk on Cinder, so it can outlive the instance and be snapshotted and backed up like any volume.

Snapshots vs backups

	Snapshot	Backup
Stored	Same backend	Separate target (object storage, NFS, another Ceph cluster)
Speed	Fast	Slower (copies data)
Survives backend loss	❌	✅
Service	cinder-volume	cinder-backup

Ceph: the usual backend

Most production OpenStack clouds use Ceph (see Kubernetes Storage, lesson 03) for everything, in separate pools:

Pool	Used by
`images`	Glance
`vms`	Nova ephemeral disks (`images_type = rbd`)
`volumes`	Cinder
`backups`	cinder-backup (often a different cluster or site)

Benefits: copy-on-write clones from raw images (instant boots, little space), live migration without copying disks, and evacuation after host failure with disks intact.

Try it: images and volumes (lab cloud)

Upload a qcow2 cloud image, then convert it to raw and upload that too. If your lab uses Ceph, compare boot times.
Create a 5 GiB volume, attach it, format it (mkfs.ext4 /dev/vdb), mount it and write a file.
Snapshot the volume, write a second file, then create a new volume from the snapshot and check which files it has.
If cinder-backup is enabled, back up the volume and restore it into a new volume.
Boot an instance from a volume, delete the instance (keeping the volume), and boot a new instance from the same volume.

Going deeper: storage operations

Watch Ceph capacity and health as closely as compute capacity; a full Ceph cluster stops the whole cloud (see Kubernetes Storage, lesson 03).
Use multiple volume types for performance tiers and QoS specs (IOPS limits) to tame noisy neighbours.
Stuck volumes (attaching/detaching/deleting) usually mean a mismatch between Cinder's database and reality. Lesson 08 covers reconciling safely.
Encrypt volumes where needed (volume types with encryption, keys in Barbican).

Recap

Glance images (qcow2 vs raw for Ceph), ephemeral disks, Cinder volumes with types and backends.
Boot from volume decouples the root disk from the instance.
Snapshots (same backend) vs backups (separate target via cinder-backup).
Ceph RBD for images, VMs, volumes and backups: copy-on-write clones, easy migration.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.