RabbitMQ¶
This page is the starting point for RabbitMQ, the rabbitmq role: the containers rabbitmq and outward_rabbitmq
(the container is named after project_name, roles/rabbitmq/defaults/main.yml:4; project_name: outward_rabbitmq
for the outward instance, rabbitmq.yml:71), on the control nodes of the example inventory
(inventory/multinode:92-96). The role is not applied by a play of site.yml itself: site.yml:438 imports
rabbitmq.yml, whose six plays all carry the tag rabbitmq: for rabbitmq and then for outward-rabbitmq
(when enable_outward_rabbitmq), apply the role (no serial), restart the hosts the handler put in the
<project>_restart group (serial: "33%"), then post-configuration (rabbitmq.yml:3-115) CODE. The service
card below summarises these plays. The lists below say where the study covers how to operate it, its impact, its
dependencies and the evidence, what to check around a run and what stays open; the service card follows them.
Where to look¶
Operate¶
- Scenario: Change control-plane membership §2
- Command: Change control-plane membership §1.1, Command quick lookup by tag
Impact and risk¶
Dependencies¶
- Dependencies: Component dependencies §1, Component dependencies §2
Evidence¶
- Fact sheet: fact sheet B §2
- Image side: Kolla images
§4 (row
rabbitmq), Kolla images §5 - Also: handlers: Tags §5
Before and after a run¶
- Pre-execution checklist, step 3 (batch size):
ANSIBLE_SERIALcoversreconfigure/upgradeand-e kolla_serial=Ncoversdeploy; mariadb and rabbitmq ignore it. - Pre-execution checklist, step 5 (health gates): before the run, check RabbitMQ: cluster status, no partitions. Repeat after each batch.
- Tags §3.1, row
rabbitmq: safe on its own "yes on a healthy cluster; membership changes are R3"; prerequisites: no partitions, all nodes running. - R3, precautions for a change of RabbitMQ group membership, password or port: plan as a
maintenance; batch with
--limit; verify RabbitMQ cluster health first. - R10, precautions for a RabbitMQ image change (
upgrade): maintenance window; watch for agent and compute services reported down. In 2023.1upgradeno longer stops every broker: the restart play drains each node (rabbitmq_upgradedrain), recreates it and waits withrabbitmqctl wait --timeout 60, inserial: "33%"batches (rabbitmq.yml:26-41,roles/rabbitmq/tasks/restart_services.yml:9-39) CODE.
Known limits and open points¶
- Known bugs and limitations, first row
rabbitmq: outward RabbitMQpulllooks upoutward_rabbitmq_services. Status in 2023.1: CODE: with outward RabbitMQ enabled, the outward plays setproject_name: outward_rabbitmq(rabbitmq.yml:71) and nokolla_role_name, soservice-images-pulllooks upoutward_rabbitmq_services(roles/rabbitmq/tasks/pull.yml:2-3,roles/service-images-pull/tasks/main.yml:14); that variable is not defined anywhere, thevarslookup raises an undefined-variable error (ansible-core 2.14.18lib/ansible/plugins/lookup/vars.py:94-104) and "Pull images" fails on the outward-rabbitmq hosts. - New in 2023.1 CODE: the outward post-configuration play includes the whole role, not
post-deploy.yml(rabbitmq.yml:100-115, notasks_from, unlike:51-55forrabbitmq). Ondeployandreconfigureit therefore runs the deploy flow of the role again on the outward-rabbitmq hosts (roles/rabbitmq/tasks/main.yml:2-3) instead offeature-flags.yml(post-deploy.yml:2). NOT AVAILABLE Whether the outward brokers then run without their feature flags enabled, and with what consequence. - Known bugs and limitations, second row
rabbitmq: the image writes.erlang.cookieonly in the bootstrap container (new volume); a cookie change recreates the brokers (env compare) but they keep the old cookie, whilekolla_toolboxgets the new one. Status: CODE; consequences NOT AVAILABLE. - What is not known, row "RabbitMQ": the sources do not establish whether nova-compute heartbeats are lost and services are reported down while brokers are drained and restarted on upgrade (R10); whether, during a one-by-one restart, nova-compute and the agents lose RPC, RPC-backed API calls time out and the agents reconnect by themselves; the consequences of a changed Erlang cookie.
- Q14, answered from the source CODE: RabbitMQ writes the Erlang cookie only in its bootstrap container.
- F6, "Global inputs are rendered into per-host files":
enable_*flags, RabbitMQ/memcached membership, service passwords and the external FQDN end up in compute-side files CODE. - fact sheet B
§2.8,
"Operationally notable behaviour, risks and oddities", eight items. Checked against 2023.1 CODE: reconfigure with a
config change (now a drained restart in
serial: "33%"batches, see R10 above); "upgrade = full outage of the message bus" no longer holds:roles/rabbitmq/tasks/upgrade.yml:2-10has no cluster-wide stop (R10); uncontrolled full stop (the brokers come back through the same batched restart play; which node starts first is NOT AVAILABLE,group_byorder); scaling the rabbitmq group changestransport_url(the templates of 30 roles referencerpc_transport_url, grep ofroles/*/templates;group_vars/all.yml:212,222); password change; thekolla-ansible pullbug with outward rabbitmq enabled (still present, see above);check.ymlis empty (roles/rabbitmq/tasks/check.yml), whilerabbitmq_hostnameno longer exists in 2023.1 (no match inansible/); the precheck still runsgetentfor every rabbit host on every rabbit host, N² lookups (roles/rabbitmq/tasks/precheck.yml:65-69,155-159). - Kolla images §6, items 9 and 10: the RabbitMQ cookie is fixed at
bootstrap CODE (consequences NOT AVAILABLE);
kolla_toolboxgets the new cookie only when thecommonplay runs (an untagged run or-t common;roles/common/templates/rabbitmq-erlang.cookie.j2:1,site.yml:86-97); recursive chown on data volumes at every start, the RabbitMQ mnesia directory among them CODE.
Service card¶
The rows of the card are explained in How to read a service card.
| Item | Value |
|---|---|
| Play | rabbitmq.yml (import_playbook at site.yml:438; 6 plays) |
| Tags | rabbitmq |
| Play host patterns | rabbitmq, rabbitmq_restart, outward-rabbitmq, outward_rabbitmq_restart |
| Node types in play (example inventory) | control |
| Node types running its containers | – |
| Containers | rabbitmq |
deploy/reconfigure flow (tasks/) |
version-check → remove-ha-all-policy (if) → config → check-containers → bootstrap |
| Task files | bootstrap.yml, check-containers.yml, check.yml, config.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, feature-flags.yml, loadbalancer.yml, main.yml, post-deploy.yml, precheck.yml, pull.yml, reconfigure.yml, remove-ha-all-policy.yml, reset-state.yml, restart_services.yml, stop.yml, upgrade.yml, version-check.yml |
| Templates | advanced.config.j2, definitions.json.j2, enabled_plugins.j2, erl_inetrc.j2, rabbitmq-env.conf.j2, rabbitmq.conf.j2, rabbitmq.json.j2 |
| Handlers (definition order) | Restart rabbitmq container |
| Key variables | enable_rabbitmq, om_rpc_*, rabbitmq_*, rpc_transport_url |
| Role dependencies | none |
Related¶
- Change control-plane membership — changing the
rabbitmqmembership changes the transport URL everywhere - Update container images §4 — with a tag change,
deployandupgradeboth restart RabbitMQ in batches of 33 %, each node drained first - Component dependencies §1 — RabbitMQ hosts are listed one by one in the configs of the services that use them
- Common: fluentd, cron, kolla_toolbox —
kolla_toolboxgets the new cookie whenrabbitmq_cluster_cookiechanges and thecommonplay runs - Pre-execution checklist — the eight checks before every production run
- Components — the other component pages