Skip to content

Change control-plane membership

Impact Restarts on drift Maintenance window

What adding, removing or replacing a controller triggers on the rest of the platform through the rabbitmq and memcached groups, which are children of control in the default inventory. The page gives no command line and no verification steps: it states what restarts, with the risks R3 and R4, lists what restarts when control04 is added and -t nova,neutron reconfigure is run, and says that Galera and RabbitMQ membership changes need their own procedures (Ussuri has no automated member removal).

The scenario section is written for the assumed deployment.

Warning

R3 and R4: in the default inventory a controller change restarts all nova and neutron containers on all computes, and most control-plane services — see §1.1.

1. Commands

1.1 Remove or replace a controller

  • Changing the rabbitmq membership changes the transport URL everywhere, and changing memcached changes the memcache server lists. In the default inventory both are children of control, so a controller change does both. That restarts all nova and neutron containers on all computes, and most control-plane services (R3 and R4 in Risk register). A memcached-only change does not touch nova-compute (the compute nova.conf has no memcache list) but still restarts the neutron agents on computes (neutron.conf).
  • Galera and RabbitMQ membership changes need their own procedures. Ussuri has no automated member removal (fact sheets A, B).

2. Change control-plane membership (add or remove a controller)

CODE Adding control04 to control (and therefore to rabbitmq, memcached, mariadb, …) and running -t nova,neutron reconfigure, read from the templates:

  • Restarted on every compute: nova_compute, nova_libvirt, nova_ssh, neutron_openvswitch_agent.
  • Restarted on every controller: all nova and neutron containers.
  • Restarted on both network nodes: all neutron agents; haproxy (backend lists changed).
  • Why: rpc_transport_url lists every RabbitMQ host (group_vars/all.yml:206, roles/nova-cell/defaults/main.yml:174), and the haproxy backends list every API host (roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:93). Loop-notify (Tags §2.4) restarts the sibling containers.
  • Plus: RabbitMQ and Galera cluster joins, which need their own plays (-t rabbitmq, -t mariadb). Galera membership changes restart all DB nodes (galera.cnf lists all nodes, fact sheet B §1.4 and §1.8 item 11).

Takeaway: controller membership changes are fleet-wide restart events for every messaging consumer. Schedule them as such, and batch the compute side.