Reading the RabbitMQ logs, and the oslo.messaging lines that go with them¶
When RabbitMQ misbehaves, the symptoms appear in every OpenStack service at once, as MessagingTimeout and
reconnection lines, and the cause is in RabbitMQ's own log on one of its nodes. This page reads both sides: the
broker's log (node down, partition, alarms, connections) and the client lines that oslo.messaging writes in the
OpenStack services' logs. Broker messages are quoted from the RabbitMQ 4.2.9 source (deps/rabbit/src/), the
version line Kolla 2026.1 pins on its RPM-based rabbitmq images (rabbitmq-server-4.2.*; on other images check
with rabbitmqctl version); client messages from oslo.messaging 2026.1.
The fix for a broken RabbitMQ cluster is not on this page; it says what the lines mean and where to look next.
Where the broker log is, and its format¶
Kolla-Ansible: RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq (rabbitmq-env.conf.j2), and the node's log is
/var/log/kolla/rabbitmq/rabbit@<hostname>.log, on each RabbitMQ node (the controllers by default).
Any: the log location depends on the package; rabbitmq-diagnostics log_location prints it on a running node.
A line starts with a timestamp, the level in brackets, and the Erlang process id of the process that logged:
<date> <time> [<level>] <0.N.0> <message>
RabbitMQ's own guides show lines in that shape, for example (from the memory guide):
2023-06-10 23:17:05.976 [info] <0.308.0> Memory high watermark set to 1024 MiB (1073741824 bytes) of 8192 MiB (8589934592 bytes) total
Kolla-Ansible's fluentd parser accepts the timestamp with or without microseconds and a +HH:MM offset
(input/03-rabbitmq.conf.j2). Several messages span several lines (a banner of asterisks, a table); read the
lines after the first.
The same line from RabbitMQ 4.2.9 on Kolla-Ansible 2026.1 (captured on a lab), with microseconds and the offset:
2026-10-27 17:24:36.043822+00:00 [info] <0.919.0> Memory high watermark set to 9595 MiB (10061436518 bytes) of 15992 MiB (16769060864 bytes) total
Run the rabbit@*.log commands from a root shell (sudo -i): with sudo grep … rabbit@*.log, your own shell
expands the glob, cannot read /var/log/kolla/rabbitmq/, and grep answers No such file or directory.
# Kolla-Ansible, on a RabbitMQ node, as root
grep -nE '\[(warning|error)\]' /var/log/kolla/rabbitmq/rabbit@*.log | tail -30
docker exec rabbitmq rabbitmq-diagnostics cluster_status
docker exec rabbitmq rabbitmq-diagnostics alarms
First: which metadata store, which partition handling¶
RabbitMQ 4.2 has two metadata stores. Khepri (Raft-based) is the default for new clusters from 4.2.0; a cluster
upgraded from earlier versions keeps Mnesia until the khepri_db feature flag is enabled
(metadata store guide).
On Kolla-Ansible 2026.1, expect Khepri. Its rabbitmq.yml runs the role's post-deploy.yml, and with it
feature-flags.yml (rabbitmqctl enable_feature_flag all), on every deploy, reconfigure and upgrade; in RabbitMQ
4.2, khepri_db is a stable feature flag (rabbit_core_ff.erl), so enabling all flags enables it. Kolla-Ansible still
writes cluster_partition_handling = pause_minority (rabbitmq_cluster_partition_handling default), but with Khepri
that setting has no effect. The Mnesia rows below apply to other deployment methods and to clusters that have not
enabled khepri_db.
What a partition looks like in the log depends on the store, and the node says which one at start
(rabbit_node_monitor.erl):
| Line at start | Meaning |
|---|---|
Starting rabbit_node_monitor (partition handling strategy unapplicable with Khepri) |
Khepri: cluster_partition_handling has no effect; availability follows Raft majorities. |
Starting rabbit_node_monitor (in pause_minority mode) |
Mnesia, with cluster_partition_handling = pause_minority. |
Starting rabbit_node_monitor (in <mode> mode), might be partitioned from [<nodes>] |
Mnesia, and the node already sees peers it may be partitioned from at start. |
grep -h "Starting rabbit_node_monitor" /var/log/kolla/rabbitmq/rabbit@*.log | tail -1
docker exec rabbitmq rabbitmqctl list_feature_flags name state | grep khepri_db
OpenStack services on Kolla-Ansible 2026.1 use quorum queues (rabbit_quorum_queue = true in the service
templates), which are Raft-based whatever the metadata store: a quorum queue keeps working while a majority of its
replicas is reachable.
Nodes going down and partitions¶
| Line (message text) | Level | What it means |
|---|---|---|
node <node> down: <reason> |
info | The Erlang distribution connection to that node was lost. With net_tick_timeout as the reason, the peer was silent for the net tick time (60 s by default); the net tick guide shows the line node rabbit@peer-hostname down: net_tick_timeout. |
rabbit on node <node> down |
info | RabbitMQ (the application) on that node is gone from the cluster's view. |
node <node> up / rabbit on node <node> up |
info | It is back. |
Node <node> was restarted |
warning | It came back with a new identity, so it restarted rather than reconnected. |
Partial partition detected: / * We saw DOWN from <node> / * We can still see <node> which can see <node> then, with pause_minority, * pause_minority mode enabled / We will therefore pause until the *entire* cluster recovers |
error | Mnesia: this node lost one peer while another peer can still see it. RabbitMQ turns a partial partition into a full one: it pauses (pause_minority) or disconnects from the proxy node (We will therefore intentionally disconnect from <node>). |
Partial partition disconnect from <node> |
error | Mnesia: the other side asked this node to disconnect, to complete the partition. |
Cluster minority/ |
warning | Mnesia, pause_minority: this node is in a minority and stops RabbitMQ until it sees the rest of the cluster again. Its listeners close; clients connected to it reconnect elsewhere. |
Mnesia(<node>): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, <node>} |
error | Mnesia: two nodes that each thought the other was down are back in contact. A partition happened. RabbitMQ's partitions guide shows this line. |
Autoheal: we were selected to restart; winner is <node> and other Autoheal … lines |
warning, info | Mnesia, only with cluster_partition_handling = autoheal (not Kolla-Ansible's default). |
Sources: rabbit_node_monitor.erl, rabbit_autoheal.erl (RabbitMQ 4.2.9); the partitions guide.
Captured on a lab (Kolla-Ansible 2026.1, RabbitMQ 4.2.9, Khepri, 3 nodes). A clean restart of one node
(systemctl restart kolla-rabbitmq-container.service on controller-3), as seen from controller-1: the reason is
connection_closed, the node is up again 7 s later, and rabbit on node … up comes only when RabbitMQ has
started there, 37 s after the stop (on another node, a short up/down flap appeared at the node … up second).
No Node <node> was restarted line appeared.
2026-10-27 17:38:09.689857+00:00 [info] <0.963.0> rabbit on node 'rabbit@controller-3' down
2026-10-27 17:38:09.716869+00:00 [info] <0.1631.0> queue 'scheduler' in vhost '/': Leader monitor down with shutdown, setting election timeout
2026-10-27 17:38:09.925888+00:00 [info] <0.963.0> node 'rabbit@controller-3' down: connection_closed
2026-10-27 17:38:13.413432+00:00 [info] <0.1631.0> queue 'scheduler' in vhost '/': detected a new leader {'%2F_scheduler','rabbit@controller-2'} in term 8
2026-10-27 17:38:16.942180+00:00 [info] <0.963.0> node 'rabbit@controller-3' up
2026-10-27 17:38:46.747626+00:00 [info] <0.963.0> rabbit on node 'rabbit@controller-3' up
Each quorum queue whose leader was on the stopped node logs Leader monitor down and, a few seconds later,
detected a new leader (39 such queues here); stream fanout queues log rabbit_stream_coordinator: Error while
starting replica for __<name>_fanout_<n> on node <node> … and Writer process node is down, exiting... warnings at
the same second, which are expected while a node is away. A network cut, on the same lab, gave instead (after 74 s,
no pause and nothing under Network Partitions with Khepri):
2026-10-27 13:26:49.191990+00:00 [error] <0.499.0> ** Node 'rabbit@controller-1' not responding **
2026-10-27 13:26:49.197106+00:00 [info] <0.4818.0> rabbit on node 'rabbit@controller-1' down
2026-10-27 13:26:49.197322+00:00 [info] <0.4818.0> node 'rabbit@controller-1' down: net_tick_timeout
2026-10-27 13:28:09.587387+00:00 [info] <0.4389.0> RabbitMQ metadata store: detected a new leader {rabbitmq_metadata,'rabbit@controller-1'} in term 3
To confirm, ask the nodes rather than the log:
docker exec rabbitmq rabbitmq-diagnostics cluster_status # "Network Partitions" section, running nodes
docker exec rabbitmq rabbitmq-diagnostics check_if_node_is_quorum_critical
Resource alarms: publishers blocked¶
A node raises an alarm when its memory use crosses the high watermark (default 0.6 of detected RAM), or its free
disk falls below disk_free_limit (default 50 MB). The alarm is cluster-wide: every node blocks publishing
connections until it clears (alarms,
memory, disk). For
OpenStack, this means every RPC call stops: services time out, nothing errors in RabbitMQ beyond the alarm.
| Line (message text) | Level | What it means |
|---|---|---|
<source> resource limit alarm set on node <node>. followed by *** Publishers will be blocked until this alarm clears *** in a banner |
warning | <source> is memory or disk. From now on, the cluster blocks publishers. |
<source> resource limit alarm cleared on node <node> |
warning | That node's alarm cleared. |
<source> resource limit alarm cleared across the cluster |
warning | No node has this alarm any more; publishing resumes. |
file descriptor limit alarm set. with *** New connections will not be accepted until this alarm clears *** |
warning | The node ran out of file descriptors: no new client connection. |
Memory high watermark set to <n> MiB (<n> bytes) of <n> MiB (<n> bytes) total |
info | At start: the limit this node computed. Check it against the memory the container or host really has. |
Disk free limit set to <n>MB |
info | At start: the disk limit (the disk guide shows this line). |
Sources: rabbit_alarm.erl, vm_memory_monitor.erl (RabbitMQ 4.2.9); the alarm, memory and disk guides.
On the client side, oslo.messaging logs the broker's block and unblock:
| Line (message text) | Level | Log |
|---|---|---|
The broker has blocked the connection: <reason> |
ERROR | every OpenStack service connected to the cluster |
The broker has unblocked the connection |
INFO | same |
Captured on a lab (memory alarm forced on controller-2 with rabbitmqctl set_vm_memory_high_watermark, then undone):
the node name is quoted and the "set" line ends with a full stop; the client gives the reason as connection
blocked, see broker logs.
2026-10-27 13:20:06.760571+00:00 [warning] <0.891.0> memory resource limit alarm set on node 'rabbit@controller-2'.
2026-10-27 13:20:06.760571+00:00 [warning] <0.891.0> *** Publishers will be blocked until this alarm clears ***
2026-10-27 13:20:07.693 7 ERROR oslo.messaging._drivers.impl_rabbit [-] The broker has blocked the connection: connection blocked, see broker logs
2026-10-27 13:24:31.892180+00:00 [warning] <0.891.0> memory resource limit alarm cleared on node 'rabbit@controller-2'
2026-10-27 13:24:31.892387+00:00 [warning] <0.891.0> memory resource limit alarm cleared across the cluster
2026-10-27 13:24:31.893 7 INFO oslo.messaging._drivers.impl_rabbit [-] The broker has unblocked the connection
Source: impl_rabbit.py. Seeing the first in many services at the same second points at an alarm; read the broker
log for which node and which resource.
Flow control is different: a connection in state flow is being slowed many times a second because queues
cannot keep up. It is normal under load, logs nothing, and shows only in rabbitmqctl list_connections name state
or the management UI (flow control). A connection stuck in
blocked or blocking is the alarm above, not flow control.
Client connections: churn and heartbeats¶
Each OpenStack service process opens AMQP connections; each connection logs on the broker when it opens and
closes (rabbit_reader.erl):
| Line (message text) | Level | What it means |
|---|---|---|
accepting AMQP connection <connection name> |
info | A client connected. A TCP connection that sends no data (a load balancer health check) is logged at debug only. |
connection <name> has a client-provided name: <name> |
info | oslo.messaging names its connections; the name shows which process (and host) it is. |
closing AMQP connection (<client>, vhost: '<vhost>', user: '<user>', duration: '<duration>') |
info | A clean close: a service stopped or restarted. |
closing AMQP connection <pid> (<client>, vhost: '<vhost>', user: '<user>', duration: '<duration>'): / client unexpectedly closed TCP connection |
warning | The client went away without closing: its process died, or a network device (HAProxy, firewall) cut an idle connection. |
closing AMQP connection <pid> (<client>, duration: '<duration>'): / missed heartbeats from client, timeout: <n>s |
error | The broker heard nothing from the client for the negotiated heartbeat timeout. The client process is stuck (blocked event loop, CPU starvation), or the network between them dropped. |
<mechanism> login refused: user '<user>' - invalid credentials |
error | A service uses a wrong password, typically after a password change not deployed everywhere. The text usually appears inside a closing AMQP connection … {handshake_error,…} line: grep login refused. |
Churn (many accept and close lines a minute for the same clients) means services reconnect in a loop. Count them per client to find which host or service:
grep -c "accepting AMQP connection" /var/log/kolla/rabbitmq/rabbit@*.log
grep "missed heartbeats from client" /var/log/kolla/rabbitmq/rabbit@*.log | tail
grep "login refused" /var/log/kolla/rabbitmq/rabbit@*.log | tail
On the client side, oslo.messaging logs (impl_rabbit.py; prefix [<connection-id>]):
| Line (message text) | Level | What it means |
|---|---|---|
[<id>] AMQP server on <host>:<port> is unreachable: <error>. Trying again in <n> seconds. |
ERROR | The service cannot connect to that RabbitMQ node; it retries (and tries the other nodes of transport_url). |
[<id>] AMQP server <host>:<port> closed the connection. Check login credentials: <error> |
ERROR | The TCP connection succeeded but RabbitMQ closed it (Socket closed): usually refused credentials or vhost. |
[<id>] Reconnected to AMQP server on <host>:<port> via [<transport>] client with port <port>. |
INFO | Reconnected. A burst of these across services marks the end of a broker or network incident. |
A recoverable connection/ |
INFO | The heartbeat thread found the connection broken and reconnects. |
Failed to publish message to topic '<topic>': <error> |
ERROR | A send failed after the retries. |
Timed out waiting for a reply to message ID <msg-id> (in a MessagingTimeout; a second form, Timed out waiting for a reply <reply queue> to message ID <msg-id>., comes from the reply waiter, so grep Timed out waiting for a reply) |
ERROR, in a traceback | An RPC call got no answer within rpc_response_timeout (60 s). Either the server side (conductor, agent, compute) did not process it, or the reply was lost. (amqpdriver.py) |
Captured on a lab, when one of the three RabbitMQ nodes was restarted at 17:38:09: the broker being stopped logs one
connection_forced line per client connection, and the clients log three different <error> texts, then
reconnect to another node 3 to 5 s later. Two client lines not in the table above also appeared: Connection failed:
… (retrying in 0 seconds) (ERROR) and Stream Queue.declare: <topic>_fanout after offset … (INFO, on reconnect).
2026-10-27 17:38:09.253478+00:00 [error] <0.21712.0> operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2026-10-27 17:38:09.259 260 ERROR oslo.messaging._drivers.impl_rabbit [-] [<conn-id>] AMQP server on 192.0.2.13:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2026-10-27 17:38:09.265 293 ERROR oslo.messaging._drivers.impl_rabbit [-] [<conn-id>] AMQP server on 192.0.2.13:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.: amqp.exceptions.ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2026-10-27 17:38:09.279 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [<conn-id>] AMQP server on 192.0.2.13:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
2026-10-27 17:38:10.528 21 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2026-10-27 17:38:10.530 21 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] ECONNREFUSED (retrying in 0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED
2026-10-27 17:38:11.855 7 INFO oslo.messaging._drivers.impl_rabbit [-] [<conn-id>] Stream Queue.declare: compute-alt_fanout after offset last
2026-10-27 17:38:12.277 7 INFO oslo.messaging._drivers.impl_rabbit [-] [<conn-id>] Reconnected to AMQP server on 192.0.2.12:5672 via [amqp] client with port 38502.
An unexpected close and a client-provided name, from the same lab (names are <service>:<pid>:<uuid>):
2026-10-27 17:27:00.663544+00:00 [warning] <0.25062.0> closing AMQP connection <0.25062.0> (192.0.2.11:59952 -> 192.0.2.11:5672 - heat-engine:277:<uuid>, vhost: '/', user: 'openstack', duration: '16s'):
2026-10-27 17:27:00.663544+00:00 [warning] <0.25062.0> client unexpectedly closed TCP connection
2026-10-26 20:53:46.349944+00:00 [info] <0.16880.0> connection 192.0.2.11:58406 -> 192.0.2.11:5672 has a client-provided name: nova-scheduler:28:<uuid>
The client heartbeat is heartbeat_timeout_threshold (default 60 s), checked heartbeat_rate times (default 3) in
that window ([oslo_messaging_rabbit]).
Reading a RabbitMQ incident in order¶
- Is it everywhere?
MessagingTimeoutorThe broker has blocked the connectionin several services at the same minute points at RabbitMQ, not at those services. - Alarms first:
rabbitmq-diagnostics alarmsandresource limit alarm setin each node's log. An alarm on one node blocks the whole cluster. - Membership:
cluster_status, thennode <node> down/rabbit on node <node> downlines, with their times, on each node. With Mnesia only (not Kolla-Ansible 2026.1), alsoPartial partition detectedandCluster minority/secondary status detected. - Connections: churn and
missed heartbeats from clientpoint at clients or the network between them and the broker; group by client address to find the host. - Then the clients: the first
AMQP server … is unreachableand the lastReconnected to AMQP serverin a service bracket the incident as that service saw it.
How the shared dependencies fail together: VM creation fails platform-wide.
Validity and limits¶
- Broker messages are quoted from the RabbitMQ 4.2.9 source and the RabbitMQ 4.2 guides; client messages from
oslo.messaging2026.1. The blocks marked "captured on a lab" come from Kolla-Ansible 2026.1 with RabbitMQ 4.2.9 on Erlang/OTP 27.3.4 (patch 18), Khepri, 3 RabbitMQ nodes on the controllers and 2 computes: a restart of one RabbitMQ node, a network cut of one node, a forced memory alarm. Not seen on the lab: the Mnesia partition lines,Node <node> was restarted, the disk and file descriptor alarms,missed heartbeats from client,login refused,Failed to publish message,MessagingTimeout. - Kolla-Ansible 2026.1 enables
khepri_dbon deploy, reconfigure and upgrade, so its clusters are expected to run Khepri; check with the commands above. The partition lines marked "Mnesia" do not appear with Khepri, whose partition handling relies on Raft and logs leader elections instead (not detailed here). - Default values (watermark 0.6, disk 50 MB, net tick 60 s, heartbeat 60 s) are upstream defaults; your deployment
may override them in
rabbitmq.confor[oslo_messaging_rabbit].
Related¶
- RPC timeouts caused by RabbitMQ: partition, alarm, missing node (the fix)
- Where the OpenStack logs are, and how to read one log line
- VM creation fails platform-wide: check the shared dependencies first
- Reopening provisioning after a control-plane incident
- Logs index
Provenance: written by agent:ozarland-author, from RabbitMQ 4.2 and oslo.messaging 2026.1 source code and documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.