A compute shows disabled or down. Why, what still works on it, and how to bring it back¶
openstack compute service list gives each nova-compute two independent fields. Status is
enabled or disabled: whether the scheduler may place new instances there. State is up or down: whether
Nova believes the service is alive, from its heartbeat or because an operator forced it down. Each combination
means something different for the instances already there, and each is set and cleared differently. Mixing them up
is how instances get lost: a compute forced down while it still runs its instances lets Nova evacuate or
locally delete them.
Symptom¶
$ openstack compute service list --service nova-compute --long
+------+--------------+--------+------+----------+-------+----------------------------+-----------------------------------------------+-------------+
| ID | Binary | Host | Zone | Status | State | Updated At | Disabled Reason | Forced Down |
+------+--------------+--------+------+----------+-------+----------------------------+-----------------------------------------------+-------------+
| ... | nova-compute | <host> | nova | disabled | up | 2026-10-26T10:00:05.000000 | AUTO: Connection to libvirt lost: <reason> | False |
| ... | nova-compute | <host> | nova | disabled | down | 2026-10-26T09:12:44.000000 | <operator reason> | True |
| ... | nova-compute | <host> | nova | enabled | down | 2026-10-26T08:58:31.000000 | None | False |
+------+--------------+--------+------+----------+-------+----------------------------+-----------------------------------------------+-------------+
Forced Down appears with compute API 2.11 or later, which recent clients negotiate.
What the operator notices first is usually one of:
- builds never land on that compute. The scheduler excludes it through the
COMPUTE_STATUS_DISABLEDtrait before the filters run; the ComputeFilter debug line<host-state> is disabled, reason: <reason>shows only when the trait is missing or out of sync; - the scheduler log has, at warning level,
<host-state> has not been heard from in a while(ComputeFilter, fromnova/scheduler/filters/compute_filter.py); - a resize, cold migration or volume detach of one of its instances fails with
Service is unavailable at this time.(HTTP 409), and other actions (reboot, stop) are accepted but never done, leaving a task state set; openstack server evacuateis refused withCompute service of <host> is still in use., because the service is stillup.
Impact¶
| Status / State | New instances | Actions on its instances | Delete of its instances | Evacuate |
|---|---|---|---|---|
| enabled / up | yes | yes | normal | refused |
| disabled / up | no | yes | normal | refused |
| any / down, heartbeat lost | no | resize, cold migration, volume detach refused (Service is unavailable at this time.); others sent to a service that does not process them |
local delete: the database record goes, the guest on the hypervisor is not touched | allowed |
| any / down, forced down | no | same refusals; if nova-compute still runs, it processes the others |
local delete | allowed |
Running instances are not stopped by any of these states. The danger is in what Nova allows once the service is
down: a local delete leaves the guest running on the hypervisor, with its disk and its network, while Nova
forgets it; an evacuate starts a second copy of an instance that may still run.
Quick checks¶
Read-only.
# 1. Status, state, reason, forced down, and the last heartbeat
openstack compute service list --service nova-compute --host <host> --long
# 2. Is nova-compute alive on the compute? (Kolla-Ansible)
sudo docker ps --filter name=nova_compute --filter name=nova_libvirt --format '{{.Names}} {{.Status}}'
sudo tail -n 50 /var/log/kolla/nova/nova-compute.log
# 3. Is the disabled trait set on its resource provider? Nova excludes it from scheduling through this trait.
# <node-uuid> is the provider's UUID: openstack resource provider list --name <hypervisor-hostname>
openstack resource provider trait list <node-uuid> | grep COMPUTE_STATUS_DISABLED
# 4. Did libvirt drop? The libvirt driver logs the event it reacts to
sudo grep -e "Connection event" -e "Connection to libvirt lost" /var/log/kolla/nova/nova-compute.log | tail -5
# 5. Instances on it, and evacuations that left from it
openstack server list --all-projects --host <host> -c ID -c Status
openstack server migration list --host <host> --type evacuation
Diagnosis¶
Status: who disabled it, from the reason¶
- A reason starting with
AUTO:was set by the libvirt driver itself (DISABLE_PREFIXinnova/virt/libvirt/driver.py). It disables the service when its connection to libvirt drops (Connection to libvirt lost: <reason>, or justConnection to libvirt lost, fromnova/virt/libvirt/host.py), and enables it again by itself when the connection comes back. It never overrides a reason that does not start withAUTO:. - Any other reason, or none, was set through the API: an operator, a script, a maintenance tool.
- Disabling also adds the
COMPUTE_STATUS_DISABLEDtrait to the compute's resource provider, so the scheduler's request filter (compute_status_filter) excludes it before the filters run. If the trait cannot be updated,nova-computelogsAn error occurred while updating COMPUTE_STATUS_DISABLED trait on compute node resource provider <uuid>, and the periodic task fixes it later.
State: heartbeat or forced¶
downwithForced DownFalse: the last heartbeat (Updated At) is older than[DEFAULT] service_down_time(60 s by default;nova-computereports everyreport_interval, 10 s).nova-computeis stopped, hung, cannot reach the database through the conductor, or the compute is gone. See nova-compute down, instances still running.downwithForced DownTrue: someone ranopenstack compute service set --down. Nova treats the service as down whatever its heartbeat says (nova/servicegroup/api.py), even ifnova-computeruns fine. This is meant for a host that has been fenced, so that evacuation can start without waiting forservice_down_time.
What happens to an instance deleted while its compute is down¶
nova/compute/api.py does a local delete when the instance's compute service is not up: it removes the instance
from the database, its ports and volume attachments, and its allocations, but nothing on the compute. When
nova-compute runs again there, its periodic task _cleanup_running_deleted_instances finds the guest it no longer
knows and applies [DEFAULT] running_deleted_instance_action (default reap): Destroying instance with name label
'<instance-name>' which is marked as DELETED but still present on host. The task runs at once when nova-compute
starts (run_immediately=True), then every running_deleted_instance_poll_interval (1800 s by default): a
returning host destroys such a guest within about a minute of starting. While nova-compute stays down, the guest
runs, without Nova knowing it.
Possible causes¶
| What you see | Cause |
|---|---|
disabled, reason AUTO: Connection to libvirt lost or AUTO: Connection to libvirt lost: ..., state up |
Cause 1: libvirt dropped; the service re-enables itself when it comes back |
disabled, another reason, state up |
Cause 2: disabled on purpose (maintenance, fencing) and never re-enabled |
down, Forced Down False |
Cause 3: no heartbeat |
down, Forced Down True |
Cause 4: forced down, usually after a host failure, and never set back up |
Resolution¶
For cause 1: let libvirt come back¶
Fix nova_libvirt (see restarting nova_libvirt).
When the connection is back, the driver logs a connection event and re-enables the service; nothing to do by hand.
Do not run --enable while libvirt is still down: the service would be scheduled to and fail every build.
For cause 2: enable it when the reason no longer holds¶
Read the reason first, and ask whoever set it.
⚠️ State-changing. Makes the compute schedulable again: new instances can land on it at once. Precondition: the reason for disabling is resolved, the service is
up. Undo:openstack compute service set --disable --disable-reason "<why>" <host> nova-compute.
openstack compute service set --enable <host> nova-compute
For cause 3: get the heartbeat back¶
Start or repair nova-compute on the compute
(nova-compute down, instances still running). Until it
reports again, do not delete its instances: each delete would be a local delete.
For cause 4: set it back up only after its evacuations are cleaned up¶
Nova refuses --up while evacuations from that host are still done (nova/api/openstack/compute/services.py):
Unable to force up host <host> as `done` evacuation migration records remain associated with the host. Ensure the compute service has been restarted, allowing these records to move to `completed` before retrying this request.
nova-compute moves them to completed when it starts on the repaired host, after destroying the local copies
of the evacuated instances (Destroying instance as it has been evacuated from this host but still exists in the
hypervisor). A nova-compute that kept running while forced down does not do it: it must be restarted. The client
shows only Compute service nova-compute of host <host> failed to set.; the reason above is in the API response
(openstack --debug compute service set --up <host> nova-compute).
🛑 Destructive. At start-up
nova-computedestroys the local guest of every instance evacuated from this host and, unless the storage is shared, its local disks (_destroy_evacuated_instances,nova/compute/manager.py; the lab saw the source copy destroyed). Precondition: every evacuated instance isACTIVEon its destination, and nothing is needed from the source copies. Cannot be undone.
Kolla-Ansible: sudo systemctl restart kolla-nova_compute-container.service
Any: restart the nova-compute service.
⚠️ State-changing. Lets Nova consider the service alive again from its heartbeat. Precondition: the host is repaired,
nova-computeruns, has been restarted since the evacuation, and has reported (Updated Atrecent), andopenstack server migration list --host <host> --type evacuationshows nodonerecord. Undo: none. Forcing the service down again is step 2 of fence before you evacuate, with its precondition: on a host that is not fenced, it allows local deletes and evacuations of instances that are alive.
openstack compute service set --up <host> nova-compute
# then, when the host is ready for new instances (not run on the lab: the service was still enabled):
openstack compute service set --enable <host> nova-compute
Never force down a compute whose instances still run there and that is not fenced: Nova then allows local deletes and evacuations of instances that are alive. See fence before you evacuate.
Verification¶
openstack compute service list --service nova-compute --host <host> --long
# expected: enabled, up, Forced Down False, Updated At within the last few seconds
openstack resource provider trait list <node-uuid> | grep COMPUTE_STATUS_DISABLED
# expected: nothing
openstack allocation candidate list --resource VCPU=1 --resource MEMORY_MB=512 --forbidden COMPUTE_STATUS_DISABLED | grep <node-uuid>
# expected: the compute is a candidate
Lab observations¶
OpenStack 2026.1, Kolla-Ansible 22.2.0, 3 controllers and 2 computes, Ubuntu 24.04, nested KVM, ML2/OVS, local disks, no Cinder, 2026-10-27. All four causes produced on purpose on one compute.
- Cause 1,
systemctl stop kolla-nova_libvirt-container.service:nova-computeloggedConnection event '0' reason 'Connection to libvirt lost: 0'at once; the service showeddisabled,up, reasonAUTO: Connection to libvirt lost: 0within 3 s, with theCOMPUTE_STATUS_DISABLEDtrait. A build forced to that host failed withNo valid host was found.(scheduler:Got no allocation candidates from the Placement API). The two guests kept running. Aftersystemctl start,Connection event '1' reason 'None'came 8 s later and the service wasenabled, trait gone, at the next check 15 s after the start. - Cause 2:
--disable --disable-reasonset the trait and removed the compute from the allocation candidates;--enablecleared both. - Cause 3,
systemctl stop kolla-nova_compute-container.service:down63 s after the last heartbeat. Before that,openstack server evacuatewas refused with HTTP 400Compute service of <host> is still in use.. Whiledown: resize refused with HTTP 409Service is unavailable at this time.; the scheduler logged... has not been heard from in a while; a reboot was accepted (statusREBOOT); a delete returned at once, the port was deleted and the guest kept running. Whennova-computestarted again, it cleared the reboot's task state at start-up (Instance in transitional state (rebooting) at start-up ... clearing task state), then received the queued reboot and dropped it (Task possibly preempted, fromreverts_task_stateonUnexpectedTaskStateError), and destroyed the deleted instance's guest 34 s after the start (Destroying instance with name label '<instance-name>' which is marked as DELETED but still present on host.). - Cause 4,
--downwhilenova-computeran:down,Forced DownTrueat once,Statusstayedenabled. An evacuation was accepted and left adonerecord;--upwas refused (message above, through--debug). The source copy of the evacuated instance kept running on the compute untilnova-computewas restarted; it was then destroyed, the record went tocompleted, and--upworked. Verification as written. - Not run:
--enablewhile libvirt was down, anAUTO:disable over an operator reason, the trait update error, volume detach (no Cinder), stop of an instance, the--enableafter--up(the service was still enabled), and forcing down a fenced host (the lab host was not fenced).
Validity and limits¶
- Written from Nova 2026.1:
nova/api/openstack/compute/services.py(enable, disable, forced down,_check_for_evacuations),nova/servicegroup/api.pyanddrivers/db.py,nova/virt/libvirt/driver.py(_set_host_enabled,DISABLE_PREFIX),nova/virt/libvirt/host.py,nova/compute/manager.py(_update_compute_provider_status,_cleanup_running_deleted_instances,_destroy_evacuated_instances),nova/compute/api.py(check_instance_host, local delete, evacuate),nova/scheduler/filters/compute_filter.py,nova/scheduler/request_filter.py, andnova/conf/service.pyandcompute.py. Reproduced on a lab (Kolla-Ansible, 2026.1, ML2/OVS, no Cinder), see Lab observations. - Only
resize(which includes cold migration) and volume detach check that the compute is up (check_instance_host(check_is_up=True)innova/compute/api.py); delete and evacuate check it on their own. Other actions are cast over RPC without that check. - Kolla-Ansible paths for the logs; the
openstackcommands apply to any deployment. - Not covered: Ironic, the
[compute] consecutive_build_service_disable_thresholdweigher (see the max retries page), and service groups other than the database driver.
Related¶
- Compute host unreachable, instances down. Fence before you evacuate
- nova-compute is down on a compute, but its instances are still running
- No valid host was found, although capacity exists: a disabled service is one of its causes.
- Instance stuck in deleting
Provenance: written by agent:ozarland-author, from the Nova 2026.1 documentation and source. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.