nova-conductor or nova-scheduler is down. What still works, what waits, what fails, and where failed builds go (cell0)¶
nova-conductor and nova-scheduler run on the controllers and talk to everything else over RabbitMQ. When every
copy of one of them is down, Nova does not fail at once and does not fail everywhere: some requests wait in a queue
and finish later, some wait for a long timeout and then fail, and running instances do not notice. This page says
which is which, what the logs show, and where the instances that failed to schedule end up: in the cell0
database.
Symptom¶
nova-conductor down on every controller¶
- New instances stay in
BUILDwith task statescheduling, without error. -
Within a minute, every compute is reported
downbyopenstack compute service list, while its instances keep running. Eachnova-computelog says once (nova/servicegroup/drivers/db.py):WARNING nova.servicegroup.drivers.db [-] Lost connection to nova-conductor for reporting service status.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <message-id> -
A
nova-computethat starts while the conductors are down repeats (nova/conductor/api.py):WARNING nova.conductor.api [...] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <message-id>about every 60 s, until
INFO nova.conductor.api [...] nova-conductor connection established successfully. -
In RabbitMQ, the
conductorqueue grows and has no consumer (on the lab: 3, then 6 messages, 0 consumers).
nova-scheduler down on every controller¶
-
New instances stay in
BUILD/schedulingfor about a minute (rpc_response_timeout), then go toERROR. Their fault is the exception's class name, notNo valid host(exception_to_dictinnova/compute/utils.pykeeps only the class name of a non-Nova exception):$ openstack server show <server-id> -c status -c fault | fault | {'code': 500, 'message': 'MessagingTimeout', ...} | | status | ERROR | -
The conductor log has the failure and its traceback (
nova/conductor/manager.py):ERROR nova.conductor.manager [...] Failed to schedule instances: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <message-id> ERROR nova.conductor.manager File ".../nova/conductor/manager.py", line 1677, in schedule_and_build_instances ... -
In RabbitMQ, the
schedulerqueue has no consumer.openstack compute service liststill shows the schedulersupuntil their heartbeat is older thanservice_down_time.
Impact¶
| conductor down | scheduler down | |
|---|---|---|
| Running instances | unaffected | unaffected |
openstack server list, show, API reads |
work (nova-api reads the databases itself) | work |
| New instances | wait in BUILD, finish when conductor is back |
fail after about rpc_response_timeout (60 s), ERROR in cell0 |
| Resize, cold migration, live migration without a host, evacuate | wait in the queue | wait, then fail |
| Start, stop, reboot of existing instances | accepted by the API and sent to the compute, which needs the conductor for its database updates: they time out (a start: MessagingTimeout in nova-compute.log about 2 min later on the lab) and can leave a task_state set (see Resolution, A task state left set) |
work |
| Compute services | reported down after service_down_time (60 s) |
stay up |
| Danger | deleting an instance while its compute is reported down is a local delete (see Diagnosis) |
none beyond failed requests |
Quick checks¶
Read-only. On a controller.
# 1. Are the containers running? (Kolla-Ansible; nova_super_conductor exists only with enable_cells)
sudo docker ps -a --filter name=nova_conductor --filter name=nova_scheduler --filter name=nova_super_conductor \
--format '{{.Names}} {{.Status}}'
# 2. Do their queues have consumers? (<vhost> is the one in transport_url, "/" by default on Kolla-Ansible)
sudo docker exec rabbitmq rabbitmqctl list_queues -p <vhost> name messages consumers | grep -E '^(conductor|scheduler)\s'
# expected: consumers > 0 on both (\s skips the per-host conductor.<host> and scheduler.<host> queues).
# 0 consumers and messages growing on conductor: no conductor listening.
# 3. Their logs (Kolla-Ansible)
sudo tail -n 50 /var/log/kolla/nova/nova-conductor.log
sudo tail -n 50 /var/log/kolla/nova/nova-scheduler.log
# 4. With admin credentials: services as Nova sees them
openstack compute service list
openstack compute service list shows nova-conductor and nova-scheduler rows too, with their own heartbeat:
down there is the quickest confirmation, but it says nothing about why.
Diagnosis¶
What the conductor does, so what stops without it¶
nova-apicasts new builds to the conductor (schedule_and_build_instances). A cast does not wait: the API returns202and the request sits in theconductorqueue until a conductor consumes it.nova-computehas no database access of its own; its database reads and writes, including the service heartbeat, go through the conductor. Without a conductor, computes stop reporting and are seendownafter[DEFAULT] service_down_time(60 s), and any compute operation that saves the instance stalls.- Resize, migrations, evacuate and rebuild are also cast to the conductor (
ComputeTaskAPI).
When a conductor comes back, it consumes the queued requests, and the computes log Recovered from being unable to
report status. (on the lab: computes up 11 s and the queued build ACTIVE 32 s after the restart).
What the scheduler does, so what fails without it¶
The conductor calls the scheduler (select_destinations) and waits for the answer. nova/scheduler/rpcapi.py
prepares this call with timeout=CONF.long_rpc_timeout (1800 s) and call_monitor_timeout=CONF.rpc_response_timeout
(60 s). With a call monitor, oslo.messaging raises MessagingTimeout as soon as no heartbeat has come for
call_monitor_timeout, and only a scheduler that picked up the message sends heartbeats. long_rpc_timeout bounds a
call that a live scheduler is working on. So with no scheduler consuming, a build fails after about 60 s: the
conductor logs Failed to schedule instances and buries the instance in cell0 (_bury_in_cell0 in
nova/conductor/manager.py): it creates the instance record
in the cell0 database with vm_state error and the exception as fault.
cell0, where failed builds live¶
Every instance that failed before it reached a compute (no valid host, scheduler timeout, a quota or image error at
scheduling) lives in the cell0 database, not in the cell database of the computes. It has no host and nothing to
clean up on any compute. openstack server list shows it like any other instance. nova-manage cell_v2 list_cells
--verbose shows cell0's database connection.
If the cell0 database itself is unreachable, nova-api treats it as a down cell: with the default
[api] list_records_by_skipping_down_cells = True, listings skip it (and from compute API 2.69 return partial
records for its instances) instead of failing.
A local delete while computes are reported down¶
With the conductor down, every compute is down for Nova. Deleting an instance then is a local delete: Nova
removes the record, its ports, attachments and allocations, but the guest keeps running on its compute until
nova-compute finds it and applies [DEFAULT] running_deleted_instance_action (reap by default). With the
conductor down, the computes cannot run that check either; it runs again at the next nova-compute start and every
running_deleted_instance_poll_interval (1800 s) after. Avoid deletes until the conductor is back and the computes
are up.
Possible causes¶
| What you see | Cause |
|---|---|
| Containers stopped or restarting; their log shows the start-up error | Cause 1: the service itself (configuration, database unreachable at start-up) |
Containers running, queues without consumers, MessagingTimeout or connection errors in their log |
Cause 2: RabbitMQ (see RPC timeouts caused by RabbitMQ) |
| Containers running and consuming, but every request is slow | Cause 3: a dependency they wait on (database, Placement for the scheduler): see VM creation fails platform-wide |
Resolution¶
For cause 1: start the service again¶
Read the last lines of its log first: a container that exits at start-up will exit again.
⚠️ State-changing. Restarts the conductor or scheduler on one controller. Requests that this copy was processing fail and are retried or buried; queued requests are then consumed. Precondition: the cause of the exit is fixed. Undo: none needed.
sudo systemctl restart kolla-nova_conductor-container.service # or kolla-nova_scheduler-container.service
If the configuration is at fault, fix it on the deploy host and run kolla-ansible reconfigure -i <inventory> -t
nova (not run on the lab).
For cause 2: fix RabbitMQ first¶
Restarting Nova services while RabbitMQ is partitioned or blocked does not help: they reconnect and wait again.
After the service is back¶
- Builds queued while the conductor was down finish by themselves; check that none stays in
BUILDafter a few minutes. - Builds that timed out while the scheduler was down are in
ERRORin cell0. They are not retried: their owners build them again, and the failed records can be deleted.--status ERRORlistsERRORinstances of every cell, including real instances on computes with disks and volumes: select only the ones of the outage.
openstack server list --all-projects --status ERROR --long -c ID -c Name -c Host -c "Project ID"
openstack server show <server-id> -c OS-EXT-SRV-ATTR:host -c fault -c created
🛑 Destructive. Deletes the instance record and its fault, which is the only trace of why it failed. Who is affected: the project that owns it, which must build again. Precondition:
OS-EXT-SRV-ATTR:hostis empty, the fault isMessagingTimeoutorNo valid hostandcreatedfalls in the outage window, and you have recorded the ID, project and fault. Undo: none.
openstack server delete <server-id>
A task state left set¶
A start, stop or reboot sent while the conductor was down can leave the instance with a task_state (powering-off,
rebooting...) that nothing clears. Once the conductor is back, check what the guest really does:
openstack server show <server-id> -c status -c OS-EXT-STS:power_state -c OS-EXT-STS:task_state
Choose the state from the power state: active if the guest runs (Running), otherwise error, then hard-reboot
it (a plain start is refused from error: start accepts only a stopped instance). When a stop timed out, the
compute has often powered the guest off already: setting active would leave a record that says it runs. Resetting
the state is an admin action (os_compute_api:os-admin-actions:reset_state is admin-only by default).
On the lab, a hard reboot was refused with the task_state at powering-on (409 ... Cannot 'reboot' instance
<server-id> while it is in task_state powering-on): reset the state first. A hard reboot is accepted only with no
task_state or one of task_states.ALLOW_REBOOT (rebooting, reboot_pending, reboot_started, resuming,
rebooting_hard, unpausing, pausing, suspending).
python-openstackclient 10.5.0 asks Do you wish to continue? [y/n] before server set --state and --auto-approve
skips the question; without a terminal it exits without changing the state (seen on two labs, once with exit code
1). Whether earlier clients (9.0.0 on other labs) prompt or accept --auto-approve is not recorded here: check
openstack server set --help.
⚠️ State-changing. Changes the instance record only (its
vm_state, and clearstask_state); nothing happens to the guest. Precondition: the conductor is back and the power state has been checked as above. Undo: set the other state with the same command; the oldtask_statecannot be restored, which is the point.
openstack server set --auto-approve --state active <server-id> # the guest runs (not run on the lab)
openstack server set --auto-approve --state error <server-id> # the guest is off
If the guest is off and was set to error, bring it back with a hard reboot:
⚠️ State-changing. Powers the guest on and recreates its libvirt domain from Nova's records. Precondition: the power state shows the guest off (
ShutdownorNOSTATE), and its owner expects it to run. Undo:openstack server stop <server-id>.
openstack server reboot --hard <server-id>
Verification¶
⚠️ State-changing. The last two commands create and delete a test instance. Needs quota, an image, a flavor and a network.
sudo docker exec rabbitmq rabbitmqctl list_queues -p <vhost> name messages consumers | grep -E '^(conductor|scheduler)\s'
# expected: consumers > 0, messages near 0
openstack compute service list
# expected: every nova-conductor, nova-scheduler and nova-compute up
openstack server create --image <image> --flavor <flavor> --network <network> --wait <test-name>
openstack server delete --wait <test-name>
Validity and limits¶
- Written from Nova 2026.1:
nova/conductor/api.py(wait_until_ready),nova/conductor/manager.py(schedule_and_build_instances,_bury_in_cell0),nova/compute/utils.py(exception_to_dict),nova/scheduler/rpcapi.py(select_destinationswithlong_rpc_timeoutandcall_monitor_timeout),nova/servicegroup/drivers/db.py,nova/conf/rpc.py,nova/conf/api.py(list_records_by_skipping_down_cells),nova/compute/api.py(local delete), and Kolla-Ansible 2026.1 (novaandnova-cellrole defaults:nova_scheduler,nova_conductor,nova_super_conductoronly withenable_cells). The MessagingTimeout text is oslo.messaging's. - Lab observations (Kolla-Ansible 22.2.0, 2026.1, 3 controllers + 2 computes, RabbitMQ quorum queues, defaults
rpc_response_timeout60 andservice_down_time60, 2026-10-27): with everynova_conductorstopped, a new build stayedBUILD/scheduling, both computes weredown51-58 s after the last conductor stopped, theconductorqueue had 0 consumers and 3, then 6 messages; aserver startof a stopped instance was accepted,nova-computeloggedMessagingTimeout120-122 s later (two runs) and the instance kepttask_statepowering-onafter the conductors were back;systemctl restartof the conductors brought the computesupin 11 s and the queued buildACTIVEin 32 s, not resubmitted;--auto-approve --state errorthenreboot --hardbrought the instanceACTIVE. With everynova_schedulerstopped, a build wentERROR61-63 s after creation (two runs), no host, faultMessagingTimeout; the cleanup commands ran as written.Timed out waiting for nova-conductorwas seen when the computes started before the conductors at lab start-up, not injected. - Not run: resize, migrations, evacuate, a stop or reboot of a running instance, a delete while the computes were
down(local delete), the cell0 database unreachable,--state active,kolla-ansible reconfigure. - The impact table follows the code paths; actions not listed (console, attach) behave like the closest one but were not traced.
- Whether a queued cast survives depends on the queue type and RabbitMQ staying up: with Kolla-Ansible 2026.1's quorum queues it does; a RabbitMQ reset drops it.
- Not covered: multi-cell deployments (a super conductor and cell conductors), and
nova-apiornova-metadatadown.
Related¶
- RPC timeouts caused by RabbitMQ
- Control plane down: what still works for running instances
- No valid host was found, although capacity exists: the other common way into cell0.
Provenance: written by agent:ozarland-author, from the Nova and Kolla-Ansible 2026.1 documentation and source. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.