Skip to content

Change control-plane membership

Impact Restarts on drift Maintenance window

What adding, removing or replacing a controller triggers on the rest of the platform through the rabbitmq, memcached, keystone and etcd groups, which are children of control in the default inventory. The page gives no command line and no verification steps: it states what restarts, with the risks R3 and R4, lists what restarts when control04 is added and -t nova,neutron reconfigure is run, and says that Galera and RabbitMQ membership changes need their own procedures (2024.1 still has no automated member removal for them, while the rewritten etcd role can remove a deleted member when asked to).

The scenario section is written for the assumed deployment.

Warning

R3 and R4: in the default inventory a controller change restarts all nova and neutron containers on all computes, and most control-plane services — see §1.1.

1. Commands

1.1 Remove or replace a controller

  • Changing the rabbitmq membership changes the transport URL everywhere, and changing memcached changes the memcache server lists. In the default inventory both are children of control (inventory/multinode:90-91, inventory/multinode:116-117), so a controller change does both. That restarts all nova and neutron containers on all computes, and most control-plane services (R3 and R4 in Risk register). Still holds in 2024.1 CODE: rpc_transport_url and notify_transport_url loop over the rabbitmq group (group_vars/all.yml:245-248, group_vars/all.yml:255-258). A memcached-only change does not touch nova-compute (the cell nova.conf has no memcache setting; no match in roles/nova-cell/templates/nova.conf.j2) but still restarts the neutron agents on computes, whose neutron.conf lists every memcached host (roles/neutron/templates/neutron.conf.j2:123).
  • Galera and RabbitMQ membership changes need their own procedures. Still holds in 2024.1: the rabbitmq and mariadb roles never remove a node from the cluster (no member-removal command in roles/rabbitmq or roles/mariadb), they only render the current group into the configuration (roles/mariadb/templates/galera.cnf.j2:29, roles/rabbitmq/templates/rabbitmq.conf.j2:18-21) CODE (fact sheets A, B). With quorum queues on by default in 2024.1 (F17), the rabbitmq role runs no command that adds or removes quorum-queue replicas when a node joins or leaves (no such task in roles/rabbitmq/tasks) CODE; where RabbitMQ then places the replicas is NOT AVAILABLE (RabbitMQ semantics).
  • Changed in 2024.1: etcd membership is handled by the role (when enable_etcd, default no, group_vars/all.yml:764; etcd is a child of control, inventory/multinode:62-63) CODE. A host whose kolla_etcd volume is new is added with etcdctl member add, run on the leader, and started with ETCD_INITIAL_CLUSTER_STATE: "existing" (roles/etcd/tasks/bootstrap_services.yml:2-25, roles/etcd/defaults/main.yml:44-49). A member that is no longer in the inventory is removed with etcdctl member remove only when etcd_remove_deleted_members is set (default "no", roles/etcd/defaults/main.yml:82; roles/etcd/tasks/lookup_cluster.yml:20-26, roles/etcd/tasks/remove_deleted_members.yml:15-39). Members are matched by ansible_facts.hostname (roles/etcd/tasks/remove_deleted_members.yml:19-27). 2023.1 had one Restart etcd container handler and no member add or remove.
  • Keystone's Fernet schedule depends on the group CODE. Each keystone host's rotation cron is computed from its index in groups['keystone'] and the size of the group (roles/keystone/tasks/config.yml:195-200), and fernet-push.sh lists every other keystone host as an rsync target (roles/keystone/templates/fernet-push.sh.j2:17-22). Adding or removing a keystone host changes both files on every keystone host, which notifies the keystone_fernet restart there (roles/keystone/tasks/config.yml:212-233). A new host gets the existing keys: when a running keystone_fernet was found, no new keys are bootstrapped (roles/keystone/tasks/bootstrap_service.yml:52-75) and the push runs from the first running host (roles/keystone/tasks/distribute_fernet.yml:12-18).

2. Change control-plane membership (add or remove a controller)

CODE Adding control04 to control (and therefore to rabbitmq, memcached, mariadb, keystone, etcd, …) and running -t nova,neutron reconfigure, read from the templates:

  • Restarted on every compute: nova_compute, nova_libvirt, nova_ssh, neutron_openvswitch_agent.
  • Restarted on every controller: all nova containers and neutron_server (neutron-server is a child of control, inventory/multinode:252-253).
  • Restarted on both network nodes: all neutron agents; haproxy (backend lists changed; loadbalancer is a child of network, inventory/multinode:84-85; roles/haproxy-config/tasks/main.yml:22-23).
  • Why: rpc_transport_url lists every RabbitMQ host (group_vars/all.yml:248, roles/nova-cell/defaults/main.yml:206; used at roles/nova-cell/templates/nova.conf.j2:37 and roles/neutron/templates/neutron.conf.j2:58), and the haproxy backends list every API host (roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:101-110). Loop-notify (Tags §2.4) restarts the sibling containers. The neutron play targets every compute (site.yml:671-685).
  • Plus: RabbitMQ and Galera cluster joins, which need their own plays (-t rabbitmq, -t mariadb), in the separate playbooks rabbitmq.yml and mariadb.yml (site.yml:400, site.yml:468). Galera membership changes restart all DB nodes (galera.cnf lists all nodes, roles/mariadb/templates/galera.cnf.j2:29); existing members restart in a play with serial: "33%" (roles/mariadb/handlers/main.yml:56-73, mariadb.yml:18-35) (fact sheet B §1.4 and §1.8 item 11). RabbitMQ members restart the same way (rabbitmq.yml:30-49), since rabbitmq.conf lists every node (roles/rabbitmq/templates/rabbitmq.conf.j2:18-21).
  • Plus, under -t keystone: keystone and keystone_fernet restart on every keystone host, since their keystone.conf carries the transport URL and the memcache server list (roles/keystone/templates/keystone.conf.j2:4, roles/keystone/templates/keystone.conf.j2:51); keystone_fernet also gets a new crontab and push script (see §1.1). With the default serial 0 these restarts happen on all keystone hosts at once (site.yml:488, roles/keystone/handlers/main.yml:45-75).
  • Plus, under -t etcd (when enabled): the etcd play has no serial (site.yml:470-481), so ANSIBLE_SERIAL does not batch it. The new member is added and started; then the existing members are restarted by the handlers, the non-leaders in % 4 waves and the leader last (roles/etcd/handlers/main.yml:40-65). The bootstrap fact that notifies these handlers is reported changed on every etcd host of the play (roles/etcd/tasks/bootstrap.yml:16-25), and recreate_or_restart_container restarts a container even when nothing in it differs with the default COPY_ALWAYS strategy (group_vars/all.yml:43; module_utils/kolla_docker_worker.py:341-363, module_utils/kolla_podman_worker.py:507-524), so every existing member restarts although its own configuration does not list the members CODE. Clients reach etcd through the internal VIP (e.g. roles/cinder/templates/cinder.conf.j2:280), so their configuration does not change; the haproxy etcd backend does (roles/etcd/defaults/main.yml:11-17). Whether clients fail while the members restart is NOT AVAILABLE.

Takeaway: controller membership changes are fleet-wide restart events for every messaging consumer. Schedule them as such, and batch the compute side.