Skip to content

Compute host unreachable, instances down. Fence before you evacuate

Evacuation (openstack server evacuate) rebuilds an instance on another host and reuses its volumes. Nova requires the source compute service to be reported down first, but Nova cannot know whether the host is really dead or merely unreachable from the control plane. Evacuating from a host that is still running the instances produces two copies of each server writing to the same volume. The precondition is fencing, not the missing heartbeat.

Symptom

$ openstack compute service list --service nova-compute -c Host -c State
| compute-07 | down |

$ openstack server list --all-projects --host compute-07 -c ID -c Status
| ...        | ACTIVE |      <- Nova still believes they run; users say they are unreachable

Monitoring shows the host unreachable (no ping, no SSH, no agent). Users report their instances on that host are not responding.

Nova keeps those instances ACTIVE, power state Running, on the dead host: it cannot know they stopped. The host's Neutron agent is usually reported dead too, but not necessarily at the same moment: Neutron's agent_down_time (75 s by default) can lag Nova's service_down_time (60 s), so for a while the agent may still show as alive.

Impact

Every instance on the host is down or unreachable. Nothing else is affected unless the host also ran a network node role. The risk during recovery is worse than the outage: a wrong evacuation corrupts volumes.

Quick checks

# 1. Nova's view of the host and when it last reported
openstack compute service list --service nova-compute --host <host> --long

# 2. Which instances and which volumes are involved
openstack server list --all-projects --host <host> --long -c ID -c Name -c Status -c Networks
openstack volume list --all-projects --status in-use -c ID -c "Attached to" | grep -F "<instance-id>"

# 3. Can the control plane reach the host at all?
ping -c 3 <host>             # management network
ssh <host> uptime            # if it answers, the host is NOT dead; stop here and diagnose the service instead

If the host answers on the management network, this is a nova-compute or message-bus problem, not a dead host. Do not evacuate. Restart nova-compute and investigate its connection to RabbitMQ: see nova-compute down, instances still running. If the host answers on TCP but refuses your SSH login, it is alive too: see nova-compute down, host alive, SSH refused. Only a failure at the TCP level, confirmed out of band, points to a dead host.

Diagnosis

The question to answer is "is the host powered off or isolated?", and the platform cannot answer it. Use a channel that does not depend on the host's network stack:

  • Out-of-band management (IPMI, iDRAC, iLO, BMC): power state, console. This is the authoritative source.
  • Switch side: is the host's port up, is traffic flowing? A host that is up with a dead management NIC may still be running instances and serving them on the data-plane network.
  • Storage side: with Ceph, ceph osd blocklist ls and client sessions can show whether the host's libvirt clients still hold RBD images open (rbd status <pool>/<image> lists watchers).

A host that is "up" by any of these channels is isolated, not dead. Instances may still be running on it.

Possible causes

Cause How to tell
Hardware failure, host powered off Out-of-band interface shows powered off or a hardware fault
Kernel panic or hang Console shows a panic; power state is on
Management network isolation Host reachable from out-of-band console; instances still running; data plane may be fine
nova-compute crashed or lost its bus connection Host reachable over SSH; only the service is down

Resolution

Step 1: fence

🛑 Destructive. Powering off the host ends every instance still running on it, if any. It is the only way to make evacuation safe. Affects all instances on this host. Precondition: you have decided the host must be taken out; the instances are already unavailable to users or the risk of split-brain outweighs the remaining service. Cannot be undone, but the instances will be rebuilt on other hosts in step 3.

Power off through the out-of-band interface and confirm the power state (IPMI not run on the lab: the lab host was powered off through its hosting cloud's API instead):

# -E reads the password from the IPMI_PASSWORD environment variable, so it stays out of the process list and history
ipmitool -I lanplus -H <bmc-address> -U <user> -E chassis power off
ipmitool -I lanplus -H <bmc-address> -U <user> -E chassis power status

If out-of-band management is unavailable, isolate the host at the switch (shut its ports) so it cannot write to storage or serve traffic (not run on the lab). Isolation at the switch is weaker than power-off (the host may still write to local disks) but prevents the double-writer problem on shared storage, provided it covers every port the host reaches storage through: the storage network ports as well as the management and data-plane ones.

With Ceph RBD-backed instance disks or volumes, also fence the host at the storage layer, so that a host that is isolated rather than powered off cannot keep writing to the RBD images (not run on the lab: the lab had no Ceph):

🛑 Destructive. Blocklists the dead host's Ceph client: every RBD image it still has open, for every instance on that host, can no longer be written by it, and writes in flight are lost. Affects that client address only. Precondition: you have identified the client address of that host (for example from rbd status <pool>/<image> watchers of one of its instances' images) and decided the host is out. Undo: ceph osd blocklist rm <client-addr>, which does not give the instances on the host their lost writes back; remove the entry only once the host has been rebooted.

# Older Ceph releases (before Pacific) call it "blacklist": ceph osd blacklist add <client-addr>
# The entry expires after one hour unless you pass a longer expiry in seconds as a last argument
ceph osd blocklist add <client-addr>
ceph osd blocklist ls

Step 2: keep the scheduler away from the host

⚠️ State-changing. Prevents new instances from being placed on the host when it returns.

openstack compute service set --disable --disable-reason "fenced <date>, hardware failure" <host> nova-compute

Optionally, mark the service down so that Nova accepts the evacuation without waiting for service_down_time. Do it only after step 1: it tells Nova the host is fenced.

⚠️ State-changing. Nova treats the compute service as down at once, whatever its heartbeat says. Affects this host only. Run it only after the host is fenced (step 1). Needs compute API microversion 2.11 or later. Undo: openstack compute service set --up <host> nova-compute (step 4).

openstack compute service set --down <host> nova-compute

Step 3: evacuate

🛑 Destructive. Evacuation rebuilds the instance on another host. With shared instance storage or boot-from-volume, the disk is preserved. With local ephemeral storage, the instance is rebuilt from its image and the ephemeral disk content is lost. Check each instance's storage before evacuating. Precondition: step 1 done; Nova reports the compute service down.

# One instance; omit --host to let the scheduler choose (--host needs microversion 2.29 or later)
openstack --os-compute-api-version 2.29 server evacuate --host <destination-host> <server-id>

Nova detects shared instance storage itself: the legacy --on-shared-storage flag has had no effect since microversion 2.14.

From compute API microversion 2.95, an evacuated instance stays stopped on its new host and must be started (openstack server start <server-id>); before 2.95 it comes back ACTIVE. Your client's default microversion may be 2.95 or later (python-openstackclient 9.0 sent 2.103 on the lab), so pass it explicitly with --os-compute-api-version to know which behaviour you get.

To evacuate every instance of the host in one go, script a loop over the list from the quick checks (not run on the lab). Evacuate in batches on a large host: each rebuild loads the destination's storage and network.

Automation that evacuates on a lost heartbeat (Masakari, Pacemaker fence agents, home-made scripts) must perform step 1 itself, or it will one day destroy the workload it was meant to save.

Step 4: when the host comes back

Nova handles a returning evacuated host: on startup, nova-compute finds the evacuation migration records and destroys the local copies of the evacuated instances. Do not start nova-compute on the host before confirming the evacuations completed, and keep the service disabled until the host is diagnosed.

If you marked the service down in step 2, mark it up once the host is back and nova-compute reports again: wait for State up while Forced Down is still True in openstack compute service list --long, or for nova-compute to log its start on the host. Do not wait for Updated At to change: forcing the service down already updates it, so a script watching it can mark the service up too early. Sent while the host is still booting, --up can fail with Failed to set service state to up; it worked a minute later on the lab.

⚠️ State-changing. Nova reads the service state from its heartbeat again. Precondition: the evacuations are complete and the host's nova-compute has started and cleaned up. Undo: --down again.

openstack compute service set --up <host> nova-compute

Then check on the host that libvirt no longer holds the evacuated instances:

# On the host. Kolla-Ansible:
docker exec nova_libvirt virsh list --all
# Any:
virsh list --all

Verification

# Every instance has a new host and is ACTIVE (SHUTOFF until started, from microversion 2.95)
openstack server list --all-projects --long -c ID -c Status -c Host | grep -F "<previous-host>"   # should be empty
openstack server show <server-id> -c status -c OS-EXT-SRV-ATTR:host

# Volumes are attached to the rebuilt instances, once
openstack volume show <volume-id> -c status -c attachments

# Migration records
openstack server migration list --type evacuation --status done

# The source host is still disabled
openstack compute service list --host <previous-host>

Reach a sample of the evacuated instances from the network and check their application state; a rebuilt instance boots from scratch and may need its services started.

Lab observations

The page was run on two different labs. Each one observed part of the procedure.

Lab builds up to 2026-10-13: 3 controllers, 2 computes, local disks, microversion 2.29

On these builds, compute-2, a virtual machine, was powered off at once with no clean shutdown, as a power failure would, and stayed off until the recovery powered it on again through its hosting cloud: it was fenced by construction.

Before During (120 s) After evacuation and return of the host
Instance on compute-2, seen from the network answering down (from 2 s after the power-off) answering again after a gap of 181.5 s
Instance on compute-1, seen from the network answering 291 requests, 0 lost 1155 requests, 0 lost
Instance of compute-2 in Nova ACTIVE on compute-2 ACTIVE, running, on compute-2 (host off) evacuated: ACTIVE on compute-1 in about 15 s
Services 22 up nova-compute and Open vSwitch agent of compute-2 down 22 up
libvirt on compute-2 after its return – – empty: the evacuated instance was cleaned up
  • Recovery as run: openstack compute service set --down (the service was not disabled), then openstack server evacuate <server-id> with --os-compute-api-version 2.29 and no destination (the scheduler chose compute-1), then power-on of the host, wait until its nova-compute reported again (a new Updated At), openstack compute service set --up, and virsh list --all on the host.
  • The 181.5 s include the 120 s the lab waited before recovering (in production: detection and decision), the evacuation (about 15 s) and the boot of the instance on compute-1 with its floating IP (about 45 s).
  • The instance had a local disk: it was rebuilt from its image on compute-1, and kept its port and floating IP. At microversion 2.29 it came back ACTIVE.
  • On the host's return, nova-compute destroyed its copy of the evacuated instance: libvirt on compute-2 was empty.
  • Instances on the other compute were not affected.
  • Same result on three lab builds; last observed run 2026-10-13. No Cinder on these builds.

Lab review on 2026-10-23: 1 controller, 2 computes, Cinder, microversion 2.103

The lab reviewer ran the whole procedure twice on a disposable multinode lab (Cinder LVM over iSCSI on the controller, local instance disks, CirrOS 0.6.3, python-openstackclient 9.0). compute-2 was a virtual machine of a hosting cloud, and its out-of-band channel was the hosting cloud's API (server stop to power it off, server show to confirm the power state), standing in for IPMI. Each run had a local-disk instance and a boot-from-volume instance; the first run also had a data volume on the local-disk instance, and in the second run each root disk held a marker file.

  • Symptom and quick checks as written: Nova reported the service down within 2 minutes, the Open vSwitch agent dead, no ping, SSH timed out.
  • Step 2 as written: --disable --disable-reason …, then --down (Forced Down True).
  • Step 3: openstack server evacuate --host <destination> <server-id> with the client's default microversion, 2.103. Both instances came back SHUTOFF on the destination, as written for 2.95 and later; openstack server start brought them ACTIVE.
  • What the disks kept: the local disk was rebuilt from the image and its marker was gone; the boot-from-volume disk still had its marker.
  • Verification as written: no instance left on the old host, one attachment per volume, to the instance on its new host, and both evacuations listed by server migration list --type evacuation --status done.
  • Step 4: on the host's return, nova-compute logged Destroying instance for the evacuated instances and cleaned their allocations; virsh list --all was empty. --up worked once nova-compute was running. Once, sent while the host was still booting, it failed with Failed to set service state to up, and worked a minute later.
  • Updated At also changed right after --down, so waiting for it to change is not a reliable sign that nova-compute is back (step 4 now says what to wait for instead).
  • Timings, the data-plane impact on other instances and the instances' network reachability after evacuation are not recorded for this run.

Validity and limits

Derived from a real incident described by an operator: the team refused to evacuate on the strength of a missing heartbeat, confirmed the host's state through its out-of-band interface, shut it down, and only then evacuated. The commands assume the libvirt driver. Evacuation semantics for local storage, and the behaviour of a returning host, should be checked against the Nova documentation of your release.

Observed on OpenStack 2026.1, Kolla-Ansible 22.2.0, ml2/OVS, Docker, Ubuntu 24.04, on two labs (see Lab observations): lab builds with 3 controllers and 2 computes, local instance disks and no Cinder (up to 2026-10-13); and a lab with 1 controller and 2 computes and Cinder LVM (2026-10-23). Observed between them: the symptom, power-off through the hosting cloud's API as out-of-band channel, disabling the service and marking it down, evacuation of local-disk and boot-from-volume instances at microversions 2.29 and 2.103, what each disk kept, the volume attachments after evacuation, and the cleanup on the host's return with the service marked up. Not observed on the lab: fencing through IPMI or the switch, the Ceph blocklist, evacuation with shared instance storage, Ceph, the Ceph checks of the diagnosis, evacuation in batches, OVN, other releases and deployment tools, production scale.


Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow and a lab reproduction by @soufian-zaouam on 2026.1. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.

Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.