Change control-plane membership¶
Impact Restarts on drift Maintenance window
What adding, removing or replacing a controller triggers on the rest of the platform through the rabbitmq,
memcached, keystone and etcd groups, which are children of control in the default inventory. The page gives no
command line and no verification steps: it states what restarts, with the risks R3 and R4, lists what restarts when
control04 is added and -t nova,neutron reconfigure is run, and says that Galera and RabbitMQ membership changes need
their own procedures (2024.1 still has no automated member removal for them, while the rewritten etcd role can remove a
deleted member when asked to).
The scenario section is written for the assumed deployment.
Warning
R3 and R4: in the default inventory a controller change restarts all nova and neutron containers on all computes, and most control-plane services — see §1.1.
1. Commands¶
1.1 Remove or replace a controller¶
- Changing the
rabbitmqmembership changes the transport URL everywhere, and changingmemcachedchanges the memcache server lists. In the default inventory both are children ofcontrol(inventory/multinode:90-91,inventory/multinode:116-117), so a controller change does both. That restarts all nova and neutron containers on all computes, and most control-plane services (R3 and R4 in Risk register). Still holds in 2024.1 CODE:rpc_transport_urlandnotify_transport_urlloop over therabbitmqgroup (group_vars/all.yml:245-248,group_vars/all.yml:255-258). Amemcached-only change does not touch nova-compute (the cellnova.confhas no memcache setting; no match inroles/nova-cell/templates/nova.conf.j2) but still restarts the neutron agents on computes, whoseneutron.conflists every memcached host (roles/neutron/templates/neutron.conf.j2:123). - Galera and RabbitMQ membership changes need their own procedures. Still holds in 2024.1: the rabbitmq and mariadb
roles never remove a node from the cluster (no member-removal command in
roles/rabbitmqorroles/mariadb), they only render the current group into the configuration (roles/mariadb/templates/galera.cnf.j2:29,roles/rabbitmq/templates/rabbitmq.conf.j2:18-21) CODE (fact sheets A, B). With quorum queues on by default in 2024.1 (F17), the rabbitmq role runs no command that adds or removes quorum-queue replicas when a node joins or leaves (no such task inroles/rabbitmq/tasks) CODE; where RabbitMQ then places the replicas is NOT AVAILABLE (RabbitMQ semantics). - Changed in 2024.1: etcd membership is handled by the role (when
enable_etcd, defaultno,group_vars/all.yml:764;etcdis a child ofcontrol,inventory/multinode:62-63) CODE. A host whosekolla_etcdvolume is new is added withetcdctl member add, run on the leader, and started withETCD_INITIAL_CLUSTER_STATE: "existing"(roles/etcd/tasks/bootstrap_services.yml:2-25,roles/etcd/defaults/main.yml:44-49). A member that is no longer in the inventory is removed withetcdctl member removeonly whenetcd_remove_deleted_membersis set (default"no",roles/etcd/defaults/main.yml:82;roles/etcd/tasks/lookup_cluster.yml:20-26,roles/etcd/tasks/remove_deleted_members.yml:15-39). Members are matched byansible_facts.hostname(roles/etcd/tasks/remove_deleted_members.yml:19-27). 2023.1 had oneRestart etcd containerhandler and no member add or remove. - Keystone's Fernet schedule depends on the group CODE. Each keystone host's rotation cron is computed from its
index in
groups['keystone']and the size of the group (roles/keystone/tasks/config.yml:195-200), andfernet-push.shlists every other keystone host as an rsync target (roles/keystone/templates/fernet-push.sh.j2:17-22). Adding or removing a keystone host changes both files on every keystone host, which notifies thekeystone_fernetrestart there (roles/keystone/tasks/config.yml:212-233). A new host gets the existing keys: when a runningkeystone_fernetwas found, no new keys are bootstrapped (roles/keystone/tasks/bootstrap_service.yml:52-75) and the push runs from the first running host (roles/keystone/tasks/distribute_fernet.yml:12-18).
2. Change control-plane membership (add or remove a controller)¶
CODE Adding control04 to control (and therefore to rabbitmq, memcached, mariadb, keystone, etcd,
…) and running -t nova,neutron reconfigure, read from the templates:
- Restarted on every compute: nova_compute, nova_libvirt, nova_ssh, neutron_openvswitch_agent.
- Restarted on every controller: all nova containers and neutron_server (
neutron-serveris a child ofcontrol,inventory/multinode:252-253). - Restarted on both network nodes: all neutron agents; haproxy (backend lists changed;
loadbalanceris a child ofnetwork,inventory/multinode:84-85;roles/haproxy-config/tasks/main.yml:22-23). - Why:
rpc_transport_urllists every RabbitMQ host (group_vars/all.yml:248,roles/nova-cell/defaults/main.yml:206; used atroles/nova-cell/templates/nova.conf.j2:37androles/neutron/templates/neutron.conf.j2:58), and the haproxy backends list every API host (roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:101-110). Loop-notify (Tags §2.4) restarts the sibling containers. The neutron play targets every compute (site.yml:671-685). - Plus: RabbitMQ and Galera cluster joins, which need their own plays (
-t rabbitmq,-t mariadb), in the separate playbooksrabbitmq.ymlandmariadb.yml(site.yml:400,site.yml:468). Galera membership changes restart all DB nodes (galera.cnflists all nodes,roles/mariadb/templates/galera.cnf.j2:29); existing members restart in a play withserial: "33%"(roles/mariadb/handlers/main.yml:56-73,mariadb.yml:18-35) (fact sheet B §1.4 and §1.8 item 11). RabbitMQ members restart the same way (rabbitmq.yml:30-49), sincerabbitmq.conflists every node (roles/rabbitmq/templates/rabbitmq.conf.j2:18-21). - Plus, under
-t keystone:keystoneandkeystone_fernetrestart on every keystone host, since theirkeystone.confcarries the transport URL and the memcache server list (roles/keystone/templates/keystone.conf.j2:4,roles/keystone/templates/keystone.conf.j2:51);keystone_fernetalso gets a new crontab and push script (see §1.1). With the default serial 0 these restarts happen on all keystone hosts at once (site.yml:488,roles/keystone/handlers/main.yml:45-75). - Plus, under
-t etcd(when enabled): the etcd play has noserial(site.yml:470-481), soANSIBLE_SERIALdoes not batch it. The new member is added and started; then the existing members are restarted by the handlers, the non-leaders in% 4waves and the leader last (roles/etcd/handlers/main.yml:40-65). The bootstrap fact that notifies these handlers is reported changed on every etcd host of the play (roles/etcd/tasks/bootstrap.yml:16-25), andrecreate_or_restart_containerrestarts a container even when nothing in it differs with the defaultCOPY_ALWAYSstrategy (group_vars/all.yml:43;module_utils/kolla_docker_worker.py:341-363,module_utils/kolla_podman_worker.py:507-524), so every existing member restarts although its own configuration does not list the members CODE. Clients reach etcd through the internal VIP (e.g.roles/cinder/templates/cinder.conf.j2:280), so their configuration does not change; the haproxyetcdbackend does (roles/etcd/defaults/main.yml:11-17). Whether clients fail while the members restart is NOT AVAILABLE.
Takeaway: controller membership changes are fleet-wide restart events for every messaging consumer. Schedule them as such, and batch the compute side.
Related¶
- Risk register — the risks this page names: R3 (changing RabbitMQ group membership) and R4 (changing memcached membership)
- RabbitMQ — service card and where to look
- Memcached — service card and where to look
- MariaDB/Galera — service card and where to look; Galera membership changes restart all DB nodes
- Keystone — Fernet keys, rotation schedule and key distribution
- Tags §2.5 — why the etcd play is not batched by
ANSIBLE_SERIAL - Pre-execution checklist — the checks before every production run
- Write config first, apply in controlled batches — how to apply in batches (
genconfig, thendeploy-containers --limit;ANSIBLE_SERIAL); its examples use the nova tags - Why a small change creates a large workload — one extra controller changes the transport URL in every messaging service everywhere