Prasanta Kumar Behera
OpenStack Private Cloud›03 · Nova: compute & scheduling
Learning Hub / Cloud — OpenStack, AWS & EKS / OpenStack Private Cloud

Lesson 03 of 9 · Modules

Nova: compute & scheduling

How OpenStack places and runs instances: flavors and extra specs, Placement's resource tracking, scheduler filters and weighers, host aggregates and availability zones, overcommit, and moving instances with live migration and evacuation.

Practitioner
Key wordsNovaflavorsextra specsPlacementscheduler filtershost aggregatesavailability zonesovercommitlive migrationevacuate

Flavors: the shape of an instance

A flavor defines vCPUs, RAM, root disk and optional extra specs that request special placement or hardware:

Extra spec (examples) Effect
hw:cpu_policy=dedicated Pin vCPUs to dedicated host cores
hw:mem_page_size=large Back memory with huge pages
hw:numa_nodes=1 Keep the instance on one NUMA node
aggregate_instance_extra_specs:ssd=true Only hosts in aggregates tagged ssd=true
trait:HW_CPU_X86_AVX512F=required Only hosts with a CPU trait

These are the same NUMA/pinning/huge-page ideas as in Linux, lesson 15, applied at cloud scale, and common for telecom workloads.

Booking a hotel room: the flavor is the room type (single, double, suite). Extra specs are special requests: "a quiet room near the lift", "a room with a bath". Placement is the booking system that knows exactly how many beds are free in each building, and the scheduler picks the best room that meets every request.

Placement and the scheduler

  1. Placement tracks resource providers (compute hosts, and nested providers like NUMA nodes or GPUs) with inventories (VCPU, MEMORY_MB, DISK_GB…), allocation ratios and traits.
  2. The scheduler asks Placement for allocation candidates that fit the flavor.
  3. Filters remove unsuitable hosts (availability zone, aggregate metadata, affinity policies, image properties…).
  4. Weighers rank the rest (e.g. spread by free RAM, or pack).
  5. The chosen allocation is claimed in Placement, and the build proceeds.
$ openstack server show vm9 -c fault -f value
{'code': 500, 'message': 'No valid host was found. ', …}

Look in the nova-scheduler logs for lines showing how many hosts each filter left. The filter that drops the count to zero is your answer.

Overcommit

CPU and RAM can be overcommitted (allocation ratios > 1), because most VMs don't use all their vCPUs all the time. Defaults have changed between releases, so set ratios explicitly per workload class: generous CPU overcommit for dev, none for pinned or latency-sensitive aggregates, and usually no RAM overcommit in production (memory contention causes swapping or OOM kills on the host).

Aggregates and availability zones

  • Host aggregates group hosts with shared properties (hardware generation, SSD, GPUs, licensing) and carry metadata the scheduler matches against.
  • An aggregate can be exposed as an availability zone that users choose (--availability-zone az1), typically mapped to racks or power domains.
  • Server groups with anti-affinity spread related instances (e.g. three database VMs) across hosts.

Moving instances

Operation When Downtime
Live migration Host maintenance, rebalancing Near-zero (memory copied while running)
Cold migration / resize Change host or flavor Instance restarts
Evacuate Host failed; rebuild instances elsewhere (needs shared storage to keep disks) Restart on the new host

With Ceph-backed ephemeral disks, live migration moves only memory. With local disks, block migration copies the disk too (slower, more network). Drain a host for maintenance with openstack compute service set --disable <host> nova-compute and then live-migrate its instances.

Try it: shape and place (lab cloud)

  1. Create two flavors, one normal and one with hw:cpu_policy=dedicated (the latter may fail to schedule in a lab without configured pinned CPUs; read the fault and scheduler logs).
  2. Create an aggregate fast with property ssd=true, add one host, and a flavor that requires it. Boot and check which host was chosen.
  3. Inspect Placement: resource providers, inventories and allocation ratios.
  4. Create a server group with anti-affinity and boot three instances into it. With one compute host, what happens to the second?
  5. On a multi-node lab, live-migrate an instance and watch openstack server show -c OS-EXT-SRV-ATTR:host.

Going deeper: compute at scale

  • Separate aggregates for different overcommit and performance classes; tie them to flavors so users can't accidentally mix them.
  • Keep Placement and reality in sync: stale allocations (from failed operations) block capacity. Tools and nova-manage commands exist to heal them.
  • Plan host maintenance as rolling live migrations, with capacity headroom (N+1 hosts) reserved.
  • Big clouds shard with cells v2; each cell has its own database and message queue.

Recap

  • Flavors (+ extra specs) describe the instance; Placement tracks capacity and traits; the scheduler filters and weighs.
  • "No valid host" → scheduler logs (which filter emptied the list) + Placement capacity.
  • Aggregates/AZs and server groups control placement; set allocation ratios deliberately.
  • Live migrate for maintenance, evacuate after host failure; shared storage (Ceph) makes both easier.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.