Operations & troubleshooting · Prasanta Kumar Behera

Operate an OpenStack cloud day to day: check service health, follow one request through every log with its request ID, keep RabbitMQ and Galera healthy, and fix instances and volumes stuck in transitional states without making things worse.

Daily health

A healthy cloud has:

every compute/network/volume service up (openstack compute service list, network agent list, volume service list)
the Galera cluster at full size and synced
the RabbitMQ cluster with all members, no partitions, and no queues growing without consumers
free capacity (Placement, Ceph) above your headroom threshold
the APIs responding (synthetic checks: issue a token, list servers, boot a tiny instance every few minutes)

Put all of that into monitoring (see Scaling Prometheus to Production) rather than checking by hand.

Running a cloud is like running a post office. The services are clerks, RabbitMQ is the conveyor belt carrying parcels between them, and the database is the ledger. If a clerk falls asleep, parcels pile up on the belt. If the ledger says a parcel was delivered but it's still on the shelf, you check the shelf before correcting the ledger.

Following one request: request IDs

Every API call gets a request ID (req-…), returned to the client and logged by every service that touches it:

$ openstack --debug server create … 2>&1 | grep -i 'x-openstack-request-id'
$ grep -r 'req-3f1c…' /var/log/kolla/nova/ /var/log/kolla/neutron/

Follow it from nova-api → conductor → scheduler → the chosen nova-compute → Neutron/Cinder calls. The first ERROR in that chain is where to dig. (Log locations shown are Kolla-Ansible's; package-based installs use /var/log/<service>/ or the journal.)

Common failures and where to look

Symptom	First checks
No valid host	nova-scheduler logs (which filter emptied the list), Placement capacity, AZ/aggregate/flavor extra specs (lesson 03)
Boots but no IP	Port status, DHCP (agent or OVN), security groups (lesson 04)
Stuck in BUILD	The compute node's nova-compute log, image download/conversion, Neutron port binding (`vif plugging timed out`)
Volume stuck attaching/detaching/deleting	cinder-volume and nova-compute logs, the hypervisor's view, the storage backend
Random 401s	Fernet key mismatch between controllers, clock skew, memcached
Everything slow or hanging	RabbitMQ queues and partitions, database connections, controller load

Stuck resources: reality first

Instances and volumes can get stuck in transitional states (BUILD, deleting, detaching) when a service crashed mid-operation. The database says one thing; the hypervisor or storage says another.

Find out what actually happened: is the VM running on the host (virsh list --all inside the nova_libvirt container)? Is the disk still attached (virsh domblklist)? Does the RBD image or iSCSI session still exist?
Make reality consistent (finish or undo the operation on the host or backend).
Then reset the recorded state with the admin commands (openstack server set --state …, openstack volume set --state …, or openstack volume set --detached in some cases), and retry the operation.

Resetting state first is how you end up with a disk attached to two VMs, or a volume deleted while in use.

RabbitMQ and Galera

RabbitMQ: watch for network partitions (rabbitmqctl cluster_status) and queues with messages but no consumers. After a partition, follow your release's recovery procedure. Restarting the affected consumer services often clears stuck RPC after the cluster is healthy.
Galera: wsrep_cluster_size should equal the number of controllers and wsrep_local_state_comment should be Synced. After a full outage, the cluster must be bootstrapped from the most advanced node. Kolla-Ansible has kolla-ansible mariadb_recovery for this. Never bootstrap from a random node.

Scenario: a volume stuck in 'detaching'

A user detached a data volume; it has shown detaching for an hour. The cinder-volume log shows a timeout talking to the backend during the call. On the compute node, virsh domblklist shows the disk no longer attached to the VM, and the backend shows no active connection.

What do you do?

Reality says the detach completed: the VM no longer has the disk and the backend has no connection. Only the database is behind. Reset the volume to available and mark it detached (openstack volume set --state available --detached vol1 or your release's equivalent, admin only), then confirm the user can attach it again.

If domblklist had still shown the disk, you'd complete the detach on the hypervisor side first (or retry the detach), not reset the state.

Try it: break and repair (all-in-one lab)

Stop the nova_compute container; watch openstack compute service list show it down after a short while; try to boot an instance and read the fault. Start it again.
Boot an instance with --debug, capture the request ID, and follow it through the nova logs.
Stop the Neutron DHCP/metadata component (or OVN controller) and boot an instance; see it get no IP. Restore it.
Put a test volume into a wrong state with openstack volume set --state error, then practise verifying reality and resetting it to available.
Check RabbitMQ queues and Galera status with the commands in the cheat sheet.

Going deeper: operating at scale

Runbooks for each alert (see SRE & Production Incident Response); most OpenStack incidents recur with the same few patterns.
Tune RPC timeouts and connection pools carefully; raising timeouts blindly hides problems.
Clean up orphans regularly: ports without instances, allocations without consumers, RBD images without volumes. Use each project's audit tooling where available (e.g. nova-manage placement audit).
Track capacity trends per aggregate and AZ, and plan hardware months ahead.

Recap

Daily health: services up, Galera synced, RabbitMQ clean, capacity headroom, synthetic API checks.
Follow a problem with its request ID across service logs.
Stuck resources: verify and fix reality first, then reset state, then retry.
RabbitMQ partitions/consumers and Galera bootstrap from the most advanced node are the classic control-plane issues.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.