VM creation fails platform-wide while instances keep running. Check the shared dependencies first¶
When every provisioning request fails or hangs at once, the cause is rarely in Nova. The control plane (APIs, schedulers, conductors, message bus, databases) fails separately from the data plane (hypervisors, instance disks, network paths). Running instances survive a control-plane failure; what is lost is the ability to create, change, migrate or delete resources. This page is the read-only triage of the shared dependencies that every capability rests on.
Symptom¶
- Users cannot create instances; some requests time out, some go to
ERROR, some stay inBUILD/scheduling. - Existing instances run and are reachable.
- The failures are not confined to one host, project or network.
- Often: other operations (volume attach, floating IP associate, port create) also hang or fail.
Impact¶
Loss of one capability (provisioning and changes), platform-wide. Workloads are not at risk from the condition itself. They are at risk from an uncontrolled attempt to fix it: an unstable control plane that keeps accepting requests creates orphaned allocations, half-built instances and records that disagree between Nova, Placement and Neutron, and every such record makes recovery slower.
Quick checks¶
# 1. Service liveness as the services report it
openstack compute service list # any 'down'? all of them down = bus or database, not hosts
openstack network agent list # same question (agent-based backends)
# 2. Transitional-state instances piling up
openstack server list --all-projects --long -c ID -c Status -c "Task State" | grep -E 'BUILD|scheduling|networking|spawning'
# 3. API health at the load balancer: are the APIs answering, and how slowly?
time openstack token issue -f value -c id >/dev/null
time openstack server list --limit 1 >/dev/null
# 4. What changed? Deployments, config changes, certificate rotations, host maintenance, network changes in the
# last hours. This question resolves more incidents than any metric.
Diagnosis, in dependency order¶
Follow the chain a VM creation traverses: Keystone and memcached (authentication), the API (HAProxy and the VIP), the database (Galera), the message bus (RabbitMQ), then Placement, Glance, Neutron, Cinder.
Keystone and memcached¶
Slow or failing token validation makes every API call slow or failing. Check memcached is reachable from the API hosts and not full; check Keystone logs for Fernet key errors (keys must be identical on every Keystone node; a rotation that removed a key still validating tokens breaks the whole platform at once). Check TLS certificate expiry on every endpoint: expired certificates are one of the most common causes of a complete control-plane outage on long-lived platforms.
openssl s_client -connect <internal-api-vip>:5000 </dev/null 2>/dev/null | openssl x509 -noout -enddate
Load balancer and VIP¶
Is the VIP on exactly one node (ip addr on the controllers)? Are the backends up in HAProxy's stats? A flapping VIP
or backends marked down explains intermittent API failures.
Galera (MariaDB)¶
SHOW STATUS LIKE 'wsrep_cluster_size'; -- expected: number of nodes
SHOW STATUS LIKE 'wsrep_cluster_status'; -- anything but Primary: the node refuses writes
SHOW STATUS LIKE 'wsrep_local_state_comment'; -- Synced expected; Donor/Desynced, Joining = recovering
SHOW STATUS LIKE 'wsrep_flow_control_paused'; -- fraction of time replication stalled the node; near 1 = stuck
Kolla-Ansible: sudo docker exec -it mariadb mariadb -uroot -p -e "SHOW STATUS LIKE 'wsrep%'" on a controller
(the 2026.1 image has no mysql command; -p alone prompts for the password, which is database_password in
/etc/kolla/passwords.yml on the deploy host). Without a terminal -it fails: drop it and pass -p<password>
only from a root-only script, since a password on the command line lands in the process list and the shell
history. Run it
on every node: a node in non-Primary while the others are Primary is a node that lost quorum.
RabbitMQ¶
rabbitmqctl cluster_status # every node running? any "partitions" listed? alarms?
rabbitmq-diagnostics check_alarms # memory or disk alarm blocks publishers
rabbitmqctl list_queues name messages consumers | awk '$2>0 && $3==0' # queues piling up with no consumer
rabbitmqctl list_unresponsive_queues
Kolla-Ansible: prefix with sudo docker exec rabbitmq. A partition means the bus is split into sides the services
cannot agree on; services connected to different sides exchange nothing. Two facts decide what you may do next:
whether the cluster uses the Mnesia metadata store (every version before 3.13, and clusters upgraded in place),
where cluster_partition_handling (pause_minority is the usual OpenStack choice) governs behaviour, or the Khepri
store (default for new clusters in recent 4.x), where that setting no longer applies; and whether the queues are
quorum queues (recommended by oslo.messaging, rabbit_quorum_queue), which need a majority of their members and
become unavailable when two nodes of three are down (their data is kept on the surviving member and they come back
once a majority returns).
A queue whose messages accumulate while its consumers vanish is a service that has stopped processing, whatever
its process list says. A nova-compute that reports up but has a stale bus connection does nothing.
The OpenStack services¶
Only now read nova-api.log, nova-conductor.log, nova-scheduler.log, neutron-server.log, for the request id
of a failed boot. Typical confirmations: AMQP server on ... is unreachable, MessagingTimeout,
Lost connection to MySQL server, WSREP has not yet prepared node for application use.
Possible causes¶
| Cause | How to tell |
|---|---|
| RabbitMQ network partition | cluster_status lists partitions (Mnesia), or, with Khepri, the nodes disagree on the running nodes while Network Partitions stays (none); RPC timeouts in every service; agents report up but nothing moves |
| RabbitMQ memory or disk alarm | check_alarms; publishers block: reads answer, but every compute can go down (both did on that lab) and every agent dead, and a create hangs until the load balancer gives up (502 after 60 s, seen in the lab run of RabbitMQ partition or alarm on a two-compute Kolla-Ansible lab) |
| Galera node out of Primary, or whole cluster non-Primary | wsrep_cluster_status; services log WSREP errors |
| Galera flow control stalled | wsrep_flow_control_paused near 1; everything slow, nothing failing outright |
| Certificate expired, Fernet keys inconsistent | Every API call fails authentication; dated exactly |
| memcached down or full | Token validation slow; Keystone logs cache errors |
| VIP or HAProxy problem | Intermittent failures depending on which backend answers |
What to do next¶
This page is read-only on purpose. Two situations lead to different runbooks, and they start from the same alert:
- A RabbitMQ node has left the cluster while the others still form a majority: rejoin or reset that one node, watch consumers reconnect. Bounded and routine.
- The RabbitMQ cluster has partitioned: do not touch the nodes yet. Establish which side the services talk to, decide whether to stop the APIs first, and escalate before any node is reset, because the wrong reset discards the queues the services are still using. See Reopening provisioning after a control-plane incident for the controlled-degradation path: stop the APIs, recover the bus and the database on a platform whose state is no longer changing, verify, reopen.
For Galera recovery after a full outage (safe_to_bootstrap, --wsrep-recover, bootstrapping from the most
advanced node), follow the Galera documentation and your deployment tool's procedure (Kolla-Ansible:
kolla-ansible mariadb-recovery). Bootstrapping the wrong node discards committed transactions; it is the step to
rehearse before you need it.
Validity and limits¶
Derived from an operator's incident (users could not create VMs, instances ran, cause later established as a
RabbitMQ partition) and from the shared-component signals they recommend watching; release not recorded.
Lab (Kolla-Ansible 22.2.0, 2026.1, 3 controllers, RabbitMQ 4.2.9 with Khepri, Galera with ProxySQL, 2026-10-27):
every check ran on a healthy lab and during a memory alarm injected on one RabbitMQ node: 2 computes down, 14/14
agents dead, a new instance in BUILD/scheduling, token and list still 1.4-1.6 s, check_alarms and
cluster_status showed the alarm, no queue with messages and no consumer, list_unresponsive_queues empty; the
instance went ACTIVE once the alarm cleared. A partition with Khepri was observed in the run of
RPC timeouts. Not run on the lab: the certificate check
(no TLS), Galera out of Primary or flow control, memcached, Fernet keys.
RabbitMQ facts (Khepri, quorum queues, removal of classic mirroring in 4.0) are version-dependent: check which
store and queue types your cluster uses before applying partition guidance. OVN deployments do not carry Neutron
agent RPC on RabbitMQ (OVN uses its own databases), so the "agents stale" symptom does not apply to them.
Related¶
- What a VM creation traverses, and tracing it by request id.
- Reopening provisioning after a control-plane incident.
- Source: OpenStack, the Day After Tomorrow (S. Zaouam), chapters 1, 8, 10, 11, 17 and 19.
Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.