RPC timeouts in every service, computes reported down. RabbitMQ is partitioned, blocked by an alarm, or a node is missing¶
Symptom¶
Several services at once log RPC timeouts:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <id>
and some of these follow:
- new instances stay in
BUILD(scheduling), volumes stay increating, actions on instances never finish; - computes go
downinopenstack compute service listalthoughnova-computeruns, and its log saysLost connection to nova-conductor for reporting service status.; - Neutron agents flap between alive and dead;
- the APIs answer, slowly, because reads go to the database; actions that need a service behind the bus fail.
Impact¶
The whole control plane, for every action that goes through the message bus: create, delete, migrate, attach. Running instances keep running and keep their network (see control plane down). Requests sent while the bus was broken may be lost or delivered twice, which leaves instances, volumes or ports in transitional states to clean up afterwards.
Quick checks¶
Read-only. On every RabbitMQ node.
Kolla-Ansible: prefix each command with sudo docker exec rabbitmq.
# 1. Which nodes this node sees, partitions, alarms
rabbitmq-diagnostics cluster_status
# 2. Resource alarms on this node (memory or disk)
rabbitmq-diagnostics check_local_alarms
# 3. Queues with messages waiting, and their consumers
rabbitmqctl list_queues name type messages consumers --silent | sort -k3 -n -r | head -20
# 4. Metadata store: is Khepri in use? (enabled = Khepri, otherwise Mnesia)
rabbitmqctl list_feature_flags name state --silent | grep khepri
On a controller, the container state and log:
sudo docker ps -a --filter name=rabbitmq --format '{{.Names}} {{.Status}}'
sudo ls /var/log/kolla/rabbitmq/ # the node log is rabbit@<hostname>.log
sudo tail -n 100 /var/log/kolla/rabbitmq/rabbit@<hostname>.log
Any: the same commands without the prefix, on each RabbitMQ node, and the RabbitMQ log.
| What you see | Case |
|---|---|
| Every node lists all others as running, no partition, no alarm, no queue growing | RabbitMQ is healthy: look at the services and the database (shared dependencies) |
| One node stopped or missing from the running nodes, the others agree | Cause 1: a node is down |
Nodes disagree on the running nodes (each side lists only its own), or Network Partitions lists nodes; the log says node '<node>' down: net_tick_timeout |
Cause 2: a network partition |
An alarm in cluster_status or check_local_alarms; rabbitmqctl list_connections name state shows blocked / blocking connections |
Cause 3: a memory or disk alarm |
RPC queues (conductor, scheduler, compute.<host>, q-plugin…) growing, with 0 consumers |
Cause 4: consumers gone |
Diagnosis¶
What Kolla-Ansible 2026.1 deploys¶
- RabbitMQ 4.2 (the 2026.1 release notes), with
cluster_partition_handling = pause_minority(rabbitmq_cluster_partition_handling). - OpenStack services use quorum queues (
rabbit_quorum_queue = true) and streams for fanout (rabbit_stream_fanout = true,om_enable_rabbitmq_stream_fanout), withuse_queue_manager = true(nova.conf.j2and the other service templates). - Since RabbitMQ 4.2, Khepri is the default metadata store for a new cluster; a cluster upgraded from an older RabbitMQ keeps Mnesia until Khepri is enabled (RabbitMQ metadata store guide, 4.2).
What each failure does¶
- Quorum queues and Khepri need a majority of their members. On the majority side, a new leader is elected and the queues keep working; on the minority side, they make no progress (RabbitMQ partitions guide). With three nodes, one node lost is absorbed; two lost stops the queues.
pause_minority(relevant with Mnesia): nodes that are not in a strict majority pause, stop listening, check every second, and resume when they see the majority again. Clients connected to them reconnect elsewhere. The guide warns that it does not protect against a node that was suspended (a frozen VM, a long pause) and then resumed. With Khepri (lab, RabbitMQ 4.2.9), the minority node did not pause: it kept accepting AMQP connections (anova-schedulerreconnected to it),cluster_statusshowedNetwork Partitions (none)on both sides, and a build sent during the partition failed withMessagingTimeoutafter 60 s.- Alarms: when a node's memory goes above its watermark, or its free disk below its limit, RabbitMQ blocks every
publishing connection on every node of the cluster until the alarm clears (RabbitMQ alarms guide). Connections
that only consume keep working, but a connection that also publishes is blocked: RPC callers, and RPC servers when
they publish their replies. In practice, RPC stops. On the lab, a memory alarm on one node: every service logged
The broker has blocked the connection: connection blocked, see broker logs, both computes wentdownand every Neutron agent dead, withoutMessagingTimeoutorLost connection to nova-conductoron the computes; aserver creategot502 Bad Gatewayfrom HAProxy after 60 s, whilenova-apifinished the request (202) once the alarm cleared and the instance then wentACTIVE: a client that retries can create a second instance. - Streams for fanout keep messages until the retention policy deletes them (Kolla-Ansible's defaults: 500 MB
segments, kept 1800 s once full). Many ready messages in a
streamqueue are expected, not a backlog (Kolla-Ansible RabbitMQ guide). - nova-compute reports its state through
nova-conductorover RPC. When the bus fails, its reports stop, and Nova marks itdownafterservice_down_time(60 s by default), while the compute and its instances are fine.nova/servicegroup/drivers/db.pylogsLost connection to nova-conductor for reporting service status.once when the reports start failing, andRecovered from being unable to report status.when they work again: the second line on every compute is a good sign that the bus is back.
Possible causes¶
Cause 1: one node down¶
A controller rebooted, the container crashed, the host ran out of memory. With three or more nodes the cluster keeps working; clients connected to that node reconnect. Timeouts are short-lived.
Cause 2: a network partition¶
The controllers lost contact with each other. cluster_status lists the partition. On the minority side, the nodes
pause (Mnesia with pause_minority) or their queue replicas stop making progress (quorum queues, Khepri).
Cause 3: a memory or disk alarm¶
A node's disk filled up (often its logs, or the volume behind /var/lib/rabbitmq), or a backlog of messages used
its memory. One node's alarm blocks publishers on the whole cluster.
Cause 4: consumers gone¶
The service that consumes a queue stopped, or cannot connect: every RPC to it waits until the timeout, and its queue grows. The bus itself is fine; the problem is the service (container stopped, crash loop, database errors).
Resolution¶
Fix the cause first, then check that the services recovered (see Verification): oslo.messaging reconnects on its own once RabbitMQ is reachable again.
For cause 1: start the node again¶
⚠️ State-changing. Starts RabbitMQ on that node; it rejoins the cluster, and its quorum queue replicas catch up from the leaders. Undo:
sudo systemctl stop kolla-rabbitmq-container.service.
sudo systemctl start kolla-rabbitmq-container.service # Kolla-Ansible 2026.1; other methods: start the service
sudo docker exec rabbitmq rabbitmq-diagnostics cluster_status # the node is back in the running nodes
The node takes some time to boot (40 s on the lab): until then cluster_status on it fails with epmd reports: node
'rabbit' not running at all. Wait for it:
sudo timeout 300 sh -c 'until docker exec rabbitmq rabbitmq-diagnostics -q check_running; do sleep 2; done'
Streams do not get a replica back on a node that was down when they were declared. Once the cluster is healthy,
Kolla-Ansible's guide points to contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh to add the missing
replicas, and warns that stream membership changes are not entirely safe: run it only on a healthy cluster.
For cause 2: repair the network, then let the cluster heal¶
With quorum queues, Khepri, or pause_minority, the minority side catches up or resumes on its own once the network
is back (lab, Khepri: 4 s after the network came back, detected a new leader in the minority node's log, and all
three nodes running). Check cluster_status on every node.
If a partition is still listed after the network is fixed (Mnesia, or a node that was suspended) (not run on the lab: its partition healed by itself), the RabbitMQ partitions guide says to choose the side you trust, "stop all nodes in the other partitions, then start them all up again", so that they restore their state from the trusted side, then restart the trusted side's nodes to clear the warning.
🛑 Destructive. Changes made on the untrusted side during the partition (queues, bindings, messages there) are lost: those nodes take their state from the trusted side. Clients on those nodes are disconnected and reconnect. Precondition: the network between all RabbitMQ nodes works again; you know which side the services used (the one where their queues have consumers); a copy of the definitions taken from a trusted node (
sudo docker exec rabbitmq rabbitmqctl export_definitions /var/lib/rabbitmq/definitions-backup.json). Stop every untrusted node before starting any of them: restarting them one at a time can bring a node back next to a peer of its own side instead of the trusted side. Cannot be undone.
# 1. on every node of the side you do not trust: stop (all of them first)
sudo systemctl stop kolla-rabbitmq-container.service
# 2. then on every node of that side: start
sudo systemctl start kolla-rabbitmq-container.service
# 3. then on each node of the trusted side, one at a time
sudo systemctl restart kolla-rabbitmq-container.service
For cause 3: free the resource¶
(Not run on the lab: no disk alarm was raised, and the memory alarm was cleared by setting the watermark back.) Free disk space or memory on the node that raised the alarm: remove what filled the disk (see compute disk full for the method, the same applies to a controller), or find the queue whose backlog grew (quick check 3) and fix its consumer (cause 4). The alarm clears by itself when usage falls below the limit, and publishers are unblocked.
For cause 4: bring the consumer back¶
Find the service named by the queue (conductor for nova-conductor, scheduler, compute.<host> for that
compute, cinder-volume…, q-plugin and the agent queues for Neutron) and fix it, from its own log. A restarted
service drains its queue.
⚠️ State-changing. Restarts one service container; its in-flight requests fail and are not retried. Undo: none needed.
sudo systemctl restart kolla-<service-container>-container.service # for example kolla-nova_conductor-container.service
Last resort: reset RabbitMQ's state¶
Kolla-Ansible's rabbitmq-reset-state stops the RabbitMQ application on each node, resets it (force_reset, or
reset with Khepri), starts it and waits for every node to join. The RabbitMQ guide of Kolla-Ansible uses it with
every OpenStack service stopped first, then redeployed.
🛑 Destructive. Every queue, exchange and message of the cluster is deleted, and
rabbitmqctl resetalso wipes users, vhosts, permissions and policies; the services recreate their queues when they start. Requests in flight are lost, and the resources they concerned stay in transitional states. Every service in<service-tags>is stopped from the stop to the deploy, on every host where it runs: APIs refuse requests, and with the Neutron agents stopped, instances get no DHCP answers or metadata until they are back. Thenovatag also covers the computes'nova_compute,nova_sshandnova_libvirt: keepnova_libvirtrunning withskip_stop_containers(below); running instances keep running in any case (restarting nova_libvirt). Precondition: causes 1 to 4 ruled out; a maintenance window;globals.ymland/etc/kolla/configunchanged since the last deploy, or reviewed (genconfigpicks up any drift, anddeployapplies it). Cannot be undone.
🛑 Known failure with Khepri. Kolla-Ansible 2026.1 deploys RabbitMQ 4.2 with Khepri as its metadata store, where
force_resetis refused. On the lab (2026.1, three controllers, Khepri),kolla-ansible rabbitmq-reset-statefailed: the fallbackrabbitmqctl resettimed out on one node, which was left with its application stopped, and the two others waited for three nodes and failed (Not enough nodes joined ...'s cluster). The details and the manual recovery are in reopening provisioning after a control-plane incident. With Khepri, do not rely on this command until a run succeeds on your release. First check whether the partition healed by itself (rabbitmqctl cluster_statuson every node), and prefer re-joining only the affected node (not run on the lab): on that noderabbitmqctl stop_app,rabbitmqctl reset,rabbitmqctl join_cluster rabbit@<healthy-node>,rabbitmqctl start_app, then check that every quorum queue and stream has a member on it (rabbitmqctl list_queues name type members). The commands below are for a cluster that still uses Mnesia, or once the command has been shown to work with Khepri on your release.
# not run from this page; on the reopening-provisioning lab (Khepri) rabbitmq-reset-state failed, see above
kolla-ansible genconfig -i <inventory>
kolla-ansible stop -i <inventory> --tags <service-tags> --yes-i-really-really-mean-it \
-e '{"skip_stop_containers": ["nova_libvirt"]}'
kolla-ansible rabbitmq-reset-state -i <inventory>
kolla-ansible deploy -i <inventory> --tags <service-tags>
<service-tags>: the services that use RabbitMQ, for example nova,neutron,cinder,glance,heat. After the deploy,
check that the users and vhosts are back (rabbitmqctl list_users, rabbitmqctl list_vhosts).
Verification¶
# on every node: all nodes running, no partition, no alarm
sudo docker exec rabbitmq rabbitmq-diagnostics cluster_status
# no RPC queue growing, and each one has consumers
sudo docker exec rabbitmq rabbitmqctl list_queues name type messages consumers --silent | sort -k3 -n -r | head
# from a host with admin credentials
openstack compute service list # every compute up again within about a minute
openstack network agent list # every agent alive
openstack volume service list # only with Cinder
Then a test build, and the clean-up of what stayed stuck: reopening provisioning after a control-plane incident and stale records.
Validity and limits¶
- Written from Kolla-Ansible 2026.1 (RabbitMQ guide,
rabbitmq.conf.j2,reset-state.yml, role defaults, release notes), the Nova 2026.1 source (servicegroup/drivers/db.py,service_down_time) and the RabbitMQ 4.2 partitions, alarms, metadata store and quorum queue guides. - Lab observations (Kolla-Ansible 22.2.0, 2026.1, 3 controllers, RabbitMQ 4.2.9 on Erlang/OTP 27.3.4 (patch 18), Khepri
enabled,
pause_minority, 2026-10-27): quick checks ran as written on the three nodes. Cause 1: one node stopped 29 s, a build wentACTIVEin 25 s meanwhile, no compute or agent down; restarted node running 40 s aftersystemctl start. Cause 2: one node cut from the two others on the clustering port 25672:down: net_tick_timeout74 s after the cut on both sides,Network Partitions (none)everywhere, computes and agents stayed up, one build failed (MessagingTimeout, 60 s); healed by itself 4 s after the cut was removed. Cause 3: memory alarm set withrabbitmqctl set_vm_memory_high_watermark absolute 10MBon one node for 4.5 min: as in What each failure does; every service up and every agent alive within 16 s ofset_vm_memory_high_watermark 0.6, the stuck buildACTIVEwithin 26 s. Cause 4:nova_conductorornova_schedulerstopped on every controller, its queue at 0 consumers (see nova-conductor or nova-scheduler down). - Not run on the lab: the partition recovery (stop and start of the untrusted side), a disk alarm, the last resort
(
rabbitmq-reset-state), Mnesia and itspause_minoritypause, a partition of more than one node. - RabbitMQ 4.3 removes Mnesia and partition handling strategies; on a later Kolla-Ansible release, read that release's RabbitMQ version first.
- Not covered: TLS problems between services and RabbitMQ, credentials (see rotating the secrets in passwords.yml), RabbitMQ upgrades, and notification queues filled by a missing consumer (Ceilometer, Designate).
Related¶
- VM creation fails platform-wide. Check the shared dependencies first
- Reopening provisioning after a control-plane incident
- Volume stuck in creating: a request lost on the bus
- Kolla-Ansible 2026.1: RabbitMQ
- RabbitMQ 4.2: partitions, alarms, metadata store, quorum queues
Provenance: written by agent:ozarland-author, from the Kolla-Ansible and Nova 2026.1 documentation and source and the RabbitMQ 4.2 documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.