Skip to content

Exercise 10. Lose one controller: what keeps working

Multinode variant, 3 controllers only (CONTROLLERS="3", 5 VMs). Run on a lab on 2026-10-25 (multinode variant, 3 controllers + 2 computes): see the lab review.

Goal: make one of the three controllers fail, see that the APIs keep answering through the virtual IP, that keepalived moves the VIP if that controller held it, and that the Galera and RabbitMQ clusters keep their quorum; then bring the controller back and watch it rejoin. Time: about 40 minutes, most of it waiting for the agents and the clock. Pattern: Three controllers and two computes, nested inside an existing OpenStack cloud (how the VIP, keepalived and the clusters are set up in this lab)

Before you start

Read before every exercise and the multinode variant's guide. This exercise uses two terminals:

  • Lab terminal: logged in to controller 1 (<prefix><lab>-ctl-1), with the lab's admin credentials loaded. Controller 1 is also the deploy host: this exercise never stops it, so that its tools stay available.
  • Hosting terminal: on your own computer, in the directory of the scripts, with the OS_CLOUD of your config.env (export OS_CLOUD=<your-cloud>): it talks to the cloud that hosts the lab VMs, not to the lab.

Note the VIP, written <vip> below. It is kolla_internal_vip_address in /etc/kolla/globals.yml on controller 1. In the hosting terminal, the port that reserves it is <prefix><lab>-vip (openstack port show <prefix><lab>-vip -c fixed_ips), and openstack server list --name '^<prefix><lab>-' lists the lab VMs with their addresses.

Find which controller holds the VIP. From controller 1, for each controller address in ~/multinode:

ssh -i ~/.ssh/quicklab_ed25519 ubuntu@<controller-address> "ip -br addr | grep -F '<vip>'"

The controller that prints a line holds it. Pick the controller you will fail, written <ctl-x>: the VIP holder if it is controller 2 or 3, otherwise controller 2. Its VM is <ctl-x-vm> in the hosting terminal.

Check the clusters before you break anything. On any controller:

sudo docker exec -it mariadb mariadb -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_%'"
sudo docker exec rabbitmq rabbitmqctl cluster_status

mariadb asks for the root password: it is database_password in /etc/kolla/passwords.yml on controller 1 (sudo grep '^database_password' /etc/kolla/passwords.yml). Type it at the prompt rather than after -p, so that it stays out of your shell history and the process list.

wsrep_cluster_size is 3 and wsrep_cluster_status is Primary; RabbitMQ lists three running nodes. (The 2026.1 MariaDB image has no mysql client: use mariadb.)

In a second lab terminal, start a loop that calls an API through the VIP every 2 seconds, and leave it running:

while true; do date +%T; openstack token issue -f value -c expires >/dev/null && echo ok || echo FAILED; sleep 2; done

1. Break it

🛑 Destructive. Pauses <ctl-x-vm>: it stops answering at once, without leaving its clusters cleanly, the way a controller that hangs or loses its network would. Disposable lab only.

In the hosting terminal:

openstack server pause <ctl-x-vm>

A clean openstack server stop would let MariaDB and RabbitMQ leave their clusters in order; a pause does not, which is the failure worth practising.

2. What you should see

  • In the loop, few or no FAILED lines, but some slow calls: on the lab, none failed, one took 17 s just after the pause and several took 6 to 8 s over the next two minutes. If <ctl-x> held the VIP, keepalived moves it to another controller within a few seconds: check with the ip -br addr command above on the two others.
  • Galera, on a remaining controller: wsrep_cluster_size 2, wsrep_cluster_status still Primary. Two of three is a majority: the cluster keeps its quorum and accepts writes.
  • RabbitMQ: rabbitmqctl cluster_status lists <ctl-x> among the nodes but not among the running ones.
  • HAProxy stops sending requests to <ctl-x>'s APIs. Now create an instance, within the first few minutes after the pause:

    openstack server create --image cirros --flavor m1.tiny --network demo-net --key-name mykey --wait ex10-test
    

    Expect it to fail. On the lab it ended in ERROR after about 7 minutes, with Build of instance … aborted: Failed to allocate the network(s). The Open vSwitch agents of the computes were connected to RabbitMQ on <ctl-x>. A paused VM does not close its TCP connections, so the agents waited for answers until they gave up on it, about 7.5 minutes after the pause (AMQP server on <address>:5672 is unreachable in their logs), reconnected to another controller, and only then wired ports again. A controller that hangs costs more than one that dies.

    About 8 minutes after the pause, delete it and try again: this time it becomes ACTIVE.

    openstack server delete --wait ex10-test
    openstack server create --image cirros --flavor m1.tiny --network demo-net --key-name mykey --wait ex10-test
    

Before reading on: the cluster survived the loss of one controller. What would happen with two? Exercise 11 answers.

3. Bring it back

⚠️ State-changing. Resumes <ctl-x-vm>.

openstack server unpause <ctl-x-vm>     # hosting terminal

4. Check

Check the clock of the controller you paused, before you judge the services. A paused VM comes back with its clock behind by about the time it was paused, and timedatectl may still say it is synchronised. Nova and Neutron then read its heartbeats as old, and services flap between up and down from one call to the next. From controller 1, compare <ctl-x>'s clock with controller 1's:

date -u +%T; ssh -i ~/.ssh/quicklab_ed25519 ubuntu@<ctl-x-address> 'date -u +%T'

If it is behind, step it:

⚠️ State-changing. Restarts the time service of the controller, which steps its clock.

ssh -i ~/.ssh/quicklab_ed25519 ubuntu@<ctl-x-address> 'sudo systemctl restart systemd-timesyncd'

On the lab, the clock was 9 minutes behind after a 9-minute pause; once it was stepped, every service was up within 40 s and stayed so. The runbook clock skew explains the symptom on a compute; it is the same here.

Then, on any controller:

  • wsrep_cluster_size is 3 again, Primary: <ctl-x> rejoined Galera (on the lab, within about 30 s of the unpause).
  • rabbitmqctl cluster_status lists three running nodes (on the lab, within about 5 s).
  • The loop shows ok, and openstack compute service list and openstack network agent list show every service up again, and stay so.

If <ctl-x> does not rejoin after about 10 minutes, look at the logs of its mariadb and rabbitmq containers (sudo docker logs --tail 50 mariadb) before you do anything else.

5. Clean up

If you stopped half-way, unpause <ctl-x-vm> first (the command above). Stop the loop with Ctrl-C, then delete the test instance:

openstack server delete --wait ex10-test

Done with the lab for today? Delete everything: a running lab keeps costing, and this one runs 5 VMs.