Skip to content

Exercise 8. Drain a compute for planned maintenance

Multinode variant only: it needs two computes. Run on a lab on 2026-10-25 (multinode variant, 1 controller + 2 computes): see the lab review.

Goal: take a compute out of service without stopping anyone's instances: stop new instances landing on it, move its instances to the other compute, work on it, and put it back. Time: about 30 minutes (about 10 minutes of commands on the lab, plus the reboot wait and reading). Runbook: Moving instances off a host for maintenance (status: lab-verified; it also covers live migration)

Before you start

Read before every exercise and the multinode variant's guide. Work on controller 1, with the lab's admin credentials loaded.

List the two computes and note their names, written <compute-1> (the one you will drain) and <compute-2> below:

openstack compute service list --service nova-compute

Create two test instances on <compute-1>:

for n in a b; do
  openstack server create --image cirros --flavor m1.tiny --network demo-net --key-name mykey \
    --availability-zone nova:<compute-1> --wait ex8-test-$n
done
openstack server list --host <compute-1> -c Name -c Status

1. Stop new instances landing on it

⚠️ State-changing. Disables the nova-compute service of <compute-1>: the scheduler no longer places new instances there. Instances already on it keep running.

openstack compute service set --disable --disable-reason "planned maintenance" <compute-1> nova-compute
openstack compute service list --service nova-compute

2. What you should see

  • <compute-1> shows Status disabled and State up: disabled is a scheduling decision, not a failure.
  • A new instance created without a host goes to <compute-2>:

    openstack server create --image cirros --flavor m1.tiny --network demo-net --key-name mykey --wait ex8-new
    openstack server show ex8-new -c OS-EXT-SRV-ATTR:host
    

Before reading on: disabling a compute moved nothing. What will you do with ex8-test-a and ex8-test-b?

3. Drain it with the runbook

Follow the runbook from its Quick checks:

  1. Its checks before moving anything: the other compute is up and has room, and the instances to move are listed. openstack hypervisor list --long no longer shows vCPU and memory with this client; use the runbook's capacity check through Placement instead: the multinode scripts install the osc-placement client plugin, and openstack allocation candidate list --resource VCPU=1 lists one candidate per compute (run on the lab on 2026-10-28, osc-placement 4.9.1).
  2. Move each instance off <compute-1>, live, as the runbook's Resolution shows. On this lab the instances have no shared storage, so a live migration also copies their disk (block migration); the runbook says which option that takes.

    ⚠️ State-changing. Live-migrates ex8-test-a and ex8-test-b to <compute-2>. They keep running, but their network stops for a while at the switch-over: exercise 9 measures how long.

  3. Check that nothing is left on the host before you work on it:

    openstack server list --all-projects --host <compute-1>    # empty
    
  4. The maintenance itself. Here, reboot the compute VM, as you would for a kernel update. From your own computer, in the directory of the scripts, with the OS_CLOUD of your config.env:

    ⚠️ State-changing. Reboots the lab VM that is <compute-1>. Nothing runs on it any more.

    openstack server list --name '^<prefix><lab>-'     # the VM named like <compute-1>
    openstack server reboot <compute-1-vm>
    

4. Check, and put it back

Follow the runbook's Verification: after the reboot, <compute-1> is up again, the moved instances are ACTIVE on <compute-2>, and nothing was lost.

Wait for the compute before you enable it. Just after the reboot, <compute-1> still shows up: that is its last heartbeat, not a new one. On the lab it then showed down about a minute later, and up again about 4 minutes after the reboot. Wait until it is up with a new Updated At, later than the reboot:

openstack compute service list --service nova-compute -c Host -c Status -c State -c "Updated At"
openstack server list --host <compute-2> -c Name -c Status

If you enable it before that, the client may print Failed to set service status to enabled: the API timed out reaching the compute, which was still rebooting. On the lab the change took effect anyway; check the Status column, and run the command again if it still says disabled.

Then put the compute back in service:

⚠️ State-changing. Enables <compute-1> again: the scheduler places new instances on it.

openstack compute service set --enable <compute-1> nova-compute

Instances do not move back on their own. Moving them back is the same live migration, in the other direction.

5. Clean up

If you stopped half-way, enable <compute-1> again first (the command above). Then delete the test instances:

openstack server delete --wait ex8-test-a ex8-test-b ex8-new

Done with the lab for today? Delete everything: a running lab keeps costing, and this one runs several VMs.