RabbitMQ¶
This page is the starting point for RabbitMQ, the rabbitmq role: one container, rabbitmq (the container is named
after project_name, roles/rabbitmq/defaults/main.yml:4, which the role sets to rabbitmq,
roles/rabbitmq/vars/main.yml:2), on the control nodes of the example inventory (inventory/multinode:90-91). The
role is not applied by a play of site.yml itself: site.yml:468 imports rabbitmq.yml, whose three plays all carry
the tag rabbitmq: apply the role (no serial), restart the hosts the handler put in the rabbitmq_restart group
(serial: "33%"), then post-configuration (rabbitmq.yml:3-70) CODE. Changed in 2024.1: outward RabbitMQ is
removed, with its three plays, its inventory group and its enable_outward_rabbitmq flag (no match for outward under
ansible/), and every play now sets max_fail_percentage, default 100 (rabbitmq.yml:8-11,37-40,56-59), which never
stops a play early (Tags §2.5). New in 2024.1: quorum queues are on by
default (F17) and upgrade runs a version check that refuses to skip a RabbitMQ minor
version. The service card below summarises these plays. The lists below say where the study covers how to operate
it, its impact, its dependencies and the evidence, what to check around a run and what stays open; the service card
follows them.
Where to look¶
Operate¶
- Scenario: Change control-plane membership §2
- Command: Change control-plane membership §1.1, Command quick lookup by tag
- RabbitMQ version step:
kolla-ansible rabbitmq-upgrade <version>runsrabbitmq-upgrade.ymlwith-e rabbitmq_version_suffix=<version>(tools/kolla-ansible:585-590); that playbook importsrabbitmq.ymlwithkolla_action: upgradeand the imagerabbitmq-<version with dots replaced by hyphens>(rabbitmq-upgrade.yml:18-21) CODE. The kolla 2024.1 tree defines two RabbitMQ images:rabbitmq, pinned to RabbitMQ 3.13 and Erlang 26.2 (kolladocker/rabbitmq/rabbitmq/rabbitmq_packages_rpm.rocky:1-4,apt_preferences_rabbitmq.ubuntu:1-7), andrabbitmq-4-1, pinned to RabbitMQ 4.1 and Erlang 27 (kolladocker/rabbitmq/rabbitmq-4-1/rabbitmq_packages_rpm.rocky:1-4) CODE.
Impact and risk¶
- What the tag executes (
rabbitmq): Tags §3 and Tags §3.1, rowsrabbitmq - Risks: R3, R10
- Quorum queues by default: F17.
om_enable_rabbitmq_quorum_queuesnow defaults totrue(group_vars/all.yml:266-268; 2023.1:false), so the[oslo_messaging_rabbit]section of 26 templates in 24 roles rendersrabbit_quorum_queue = true(grep ofroles/*/templates; e.g.roles/nova/templates/nova.conf.j2:156-165,roles/nova-cell/templates/nova.conf.j2:207-216,roles/neutron/templates/neutron.conf.j2:146-155) CODE. A cloud that kept the 2023.1 default therefore gets a changed configuration file, and a handler restart, for those services on its first 2024.1 run that renders configs, computes included. Three related options are new and off by default:om_enable_queue_manager: false, andom_enable_rabbitmq_transient_quorum_queueandom_enable_rabbitmq_stream_fanout, which follow it (group_vars/all.yml:269-279) CODE. NOT AVAILABLE What oslo.messaging does with existing classic queues when the services restart withrabbit_quorum_queue = true.
Dependencies¶
- Dependencies: Component dependencies §1, Component dependencies §2
- Transport URLs:
rpc_transport_urlandnotify_transport_urllist every host of therabbitmqgroup one by one (group_vars/all.yml:245,248,255,258); nova-cell builds its own URLs from the same group by default (roles/nova-cell/defaults/main.yml:191,200,206-207) CODE. Still holds in 2024.1: a change of therabbitmqmembership changes thetransport_urlof every service that uses them.
Evidence¶
- Fact sheet: fact sheet B §2
- Image side: Kolla images
§4 (row
rabbitmq), Kolla images §5 - Also: handlers: Tags §5. Still holds in 2024.1: the only
handler, "Restart rabbitmq container", restarts nothing; it adds the host to the group
rabbitmq_restart, except forkolla_actionconfig(roles/rabbitmq/handlers/main.yml:2-6), and the restart play then drains the node (community.rabbitmq.rabbitmq_upgradedrain, only when arabbitmqcontainer exists), recreates or restarts it and waits withrabbitmqctl wait --timeout 60(roles/rabbitmq/tasks/restart_services.yml:2-39), inserial: "33%"batches (rabbitmq.yml:30-49) CODE.
Before and after a run¶
- Pre-execution checklist, step 3 (batch size):
ANSIBLE_SERIALcoversreconfigure/upgradeand-e kolla_serial=Ncoversdeploy; mariadb and rabbitmq ignore it: the RabbitMQ plays set noserialexcept the restart play, which has its ownserial: "33%"(rabbitmq.yml:3-70,:36) CODE. - Pre-execution checklist, step 5 (health gates): before the run, check RabbitMQ: cluster status, no partitions. Repeat after each batch.
kolla-ansible prechecksis stricter in 2024.1 (F17) CODE:- "Validate rabbitmq variables" fails when
om_enable_rabbitmq_high_availabilityandom_enable_rabbitmq_quorum_queuesare equal, so also when both are off (roles/prechecks/tasks/service_checks.yml:67-72); 2023.1 failed only when both were on. A new check fails when stream fanout is on without transient quorum queues (:74-81). - When a
rabbitmqcontainer exists and quorum queues are on, the RabbitMQ precheck lists the queues once and asserts that every queue whose name contains neitherreply_nor_fanoutis a quorum queue; two more asserts cover transient quorum queues and stream fanout when those are on (roles/rabbitmq/tasks/precheck.yml:131-189). With the 2024.1 defaults it therefore fails while classic queues exist; its message calls the migration a manual procedure. The HA block still asserts theha-allpolicy whenom_enable_rabbitmq_high_availabilityis on (:107-129). A cloud that kept the 2023.1 default (HA off) has no such policy, since 2023.1 runs removed it, and the message of that assert calls the move to mirrored queues manual too (:115-123). Both blocks carry the tagrabbitmq-ha-precheck(:129,:189). - These checks run only for
kolla_actionprecheck, set bykolla-ansible prechecks(tools/kolla-ansible:444-447; the prechecks play,site.yml:77-86; the role entry point includes<kolla_action>.yml,roles/rabbitmq/tasks/main.yml:2-3).deploy,reconfigureandupgradedo not run them.
- "Validate rabbitmq variables" fails when
- Tags §3.1, row
rabbitmq: safe on its own "yes on a healthy cluster; membership changes are R3"; prerequisites: no partitions, all nodes running. - R3, precautions for a change of RabbitMQ group membership, password or port: plan as a
maintenance; batch with
--limit; verify RabbitMQ cluster health first. - R10, precautions for a RabbitMQ image change (
upgrade): maintenance window; watch for agent and compute services reported down. Still holds in 2024.1:upgradedoes not stop every broker; the role renders the config, enables all stable feature flags on the running brokers, then checks the containers (roles/rabbitmq/tasks/upgrade.yml:1-12,feature-flags.yml:2-5), and the restart play drains, recreates and waits for each node inserial: "33%"batches (rabbitmq.yml:30-49,roles/rabbitmq/tasks/restart_services.yml:9-39) CODE. - New in 2024.1: the version check refuses a skipped RabbitMQ minor version CODE.
upgrade.ymlnow importsversion-check.ymlfirst (roles/rabbitmq/tasks/upgrade.yml:2), asdeploy.ymlalready did, and therefore alsoreconfigure(deploy.yml:2,reconfigure.yml:2). When arabbitmqcontainer exists, once, on the first host of the group (version-check.yml:75-78), it reads the running version withrabbitmqctl --versionand the new one from a one-shotrabbitmq_version_checkcontainer of the target image (:12-36), then asserts that both have the same major version and at most one minor version between them, or that the step is 3.13 to 4.0 or 4.1 (:44-64). Otherwise it fails with "Looks like you're trying to run a skip-release upgrade!" and names thekolla-ansible rabbitmq-upgradestep to run first (:59-64). The downgrade assert follows (:66-73). The whole file carries the tagrabbitmq-version-check(:79), so--skip-tags rabbitmq-version-checkskips it. - Consequence for an upgrade from 2023.1 CODE (from the image pins): the 2023.1
rabbitmqimage pinned RabbitMQ 3.11 (2023.1 kolladocker/rabbitmq/rabbitmq/apt_preferences_rabbitmq.ubuntu:2), the 2024.1 one pins 3.13 (kolladocker/rabbitmq/rabbitmq/apt_preferences_rabbitmq.ubuntu:2). A cloud still on the 2023.1 default image is two minor versions behind, and the 2024.1 version check fails until it runs 3.12 or 3.13. The kolla 2024.1 tree has norabbitmq-3-12image (onlyrabbitmqandrabbitmq-4-1, kolladocker/rabbitmq/), while 2023.1 hadrabbitmq-3-12andrabbitmq-3-13. NOT AVAILABLE Which RabbitMQ version a given cloud runs, and which image arabbitmq-upgrade 3.12run with 2024.1 code would find in the registry. - A 4.1 step is undone by the next plain run CODE: after
kolla-ansible rabbitmq-upgrade 4.1, adeploy,reconfigureorupgradewith the defaultrabbitmq_image(roles/rabbitmq/defaults/main.yml:30) targets the 3.13 image again, and the downgrade assert fails (roles/rabbitmq/tasks/version-check.yml:66-73) unlessrabbitmq_imageis overridden.
Known limits and open points¶
- Known bugs and limitations, first row
rabbitmq: outward RabbitMQpulllooked upoutward_rabbitmq_services. No longer holds in 2024.1 CODE: outward RabbitMQ is removed, andpulllooks uprabbitmq_servicesthroughproject_name(roles/rabbitmq/tasks/pull.yml:2-3,roles/service-images-pull/tasks/main.yml:14,roles/rabbitmq/vars/main.yml:2). The 2023.1 oddity of the outward post-configuration play, which included the whole role, is gone with it: the only post-configuration play includespost-deploy.yml(rabbitmq.yml:63-67), which enables the feature flags (post-deploy.yml:2). - Known bugs and limitations, second row
rabbitmq: the image writes.erlang.cookieonly in the bootstrap container (kolladocker/rabbitmq/rabbitmq/extend_start.sh:7-18), which runs only when the volume was just created (roles/rabbitmq/tasks/bootstrap.yml:2-26); a cookie change recreates the brokers (env compare,roles/rabbitmq/defaults/main.yml:13-16,roles/rabbitmq/tasks/check-containers.yml:10) but they keep the old cookie, whilekolla_toolboxgets the new one. Status: still holds in 2024.1 CODE; consequences NOT AVAILABLE. - New in 2024.1 CODE: the data volume is
rabbitmq_datadir_volume(defaultrabbitmq,group_vars/all.yml:913;roles/rabbitmq/defaults/main.yml:52), but the bootstrap still creates, and gates on, the volume named afterproject_name(roles/rabbitmq/tasks/bootstrap.yml:2-8,26). With the default the two are the same volume. NOT AVAILABLE What happens to the cookie whenrabbitmq_datadir_volumeis changed on an existing cloud: the volumerabbitmqalready exists, so the bootstrap does not run for the new data volume. - What is not
known, row
"RabbitMQ": the sources do not establish whether nova-compute heartbeats are lost and services are reported down
while brokers are drained and restarted on upgrade (R10); whether, during a
one-by-one restart, nova-compute and the agents lose RPC, RPC-backed API calls time out and the agents reconnect by
themselves; the consequences of a changed Erlang cookie; new in 2024.1, what oslo.messaging does with existing
classic queues once
rabbit_quorum_queue = trueis rendered (F17). - Q14, answered from the source CODE: RabbitMQ writes the Erlang cookie
only in its bootstrap container (kolla
docker/rabbitmq/rabbitmq/extend_start.sh:7-18). - F6, "Global inputs are rendered into per-host files":
enable_*flags, RabbitMQ/memcached membership, service passwords and the external FQDN end up in compute-side files CODE. - F17, "RabbitMQ quorum queues are on by default": the 2024.1 default changes every
service's messaging config and fails
precheckswhile classic queues exist CODE. - fact sheet B
§2.8,
"Operationally notable behaviour, risks and oddities". Checked against 2024.1 CODE: reconfigure with a config
change is a drained restart in
serial: "33%"batches (roles/rabbitmq/tasks/reconfigure.yml:2,rabbitmq.yml:30-49);upgradeis not a full outage of the message bus:roles/rabbitmq/tasks/upgrade.yml:1-12has no cluster-wide stop (R10); uncontrolled full stop (the brokers come back through the same batched restart play; which node starts first is NOT AVAILABLE,group_byorder); scaling the rabbitmq group changestransport_url(the templates of 25 roles referencerpc_transport_url, grep ofroles/*/templates;group_vars/all.yml:248,258); password change; thekolla-ansible pullbug with outward RabbitMQ no longer applies (see above);check.ymlis still empty (roles/rabbitmq/tasks/check.yml:1); the precheck still runsgetentfor every rabbit host on every rabbit host, N² lookups (roles/rabbitmq/tasks/precheck.yml:61-77), now only for therabbitmqgroup. Changed in 2024.1: theha-allpolicy removal no longer callsdocker execliterally; it useskolla_container_engine(roles/rabbitmq/tasks/remove-ha-all-policy.yml:14,20). - Kolla images §6, items 9 and 10: the RabbitMQ cookie is fixed at
bootstrap CODE (consequences NOT AVAILABLE);
kolla_toolboxgets the new cookie only when thecommonplay runs (an untagged run or-t common;roles/common/templates/rabbitmq-erlang.cookie.j2:1,roles/common/tasks/config.yml:272-282,site.yml:88-103); recursive chown on data volumes at every start, the RabbitMQ mnesia directory among them CODE.
Service card¶
The rows of the card are explained in How to read a service card.
| Item | Value |
|---|---|
| Play | rabbitmq.yml (import_playbook at site.yml:468; 3 plays) |
| Tags | rabbitmq |
| Play host patterns | rabbitmq, rabbitmq_restart |
| Node types in play (example inventory) | control |
| Node types running its containers | – |
| Containers | rabbitmq |
deploy/reconfigure flow (tasks/) |
version-check → remove-ha-all-policy (if) → config → check-containers → bootstrap |
| Task files | bootstrap.yml, check-containers.yml, check.yml, config.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, feature-flags.yml, loadbalancer.yml, main.yml, post-deploy.yml, precheck.yml, pull.yml, reconfigure.yml, remove-ha-all-policy.yml, reset-state.yml, restart_services.yml, stop.yml, upgrade.yml, version-check.yml |
| Templates | advanced.config.j2, definitions.json.j2, enabled_plugins.j2, erl_inetrc.j2, rabbitmq-env.conf.j2, rabbitmq.conf.j2, rabbitmq.json.j2 |
| Handlers (definition order) | Restart rabbitmq container |
| Key variables | enable_rabbitmq, om_rpc_*, rabbitmq_*, rpc_transport_url |
| Role dependencies | none |
Related¶
- Change control-plane membership — changing the
rabbitmqmembership changes the transport URL everywhere - Update container images §4 — with a tag change,
deployandupgradeboth restart RabbitMQ in batches of 33 %, each node drained first; both run the version check first - Component dependencies §1 — RabbitMQ hosts are listed one by one in the configs of the services that use them
- Common: fluentd, cron, kolla_toolbox —
kolla_toolboxgets the new cookie whenrabbitmq_cluster_cookiechanges and thecommonplay runs - Technical findings F17 — quorum queues on by default and the stricter prechecks
- Pre-execution checklist — the eight checks before every production run
- Components — the other component pages