id: PG-42
category: control-plane
title: Conductor or scheduler down?
situation: >-
  Nova does not fail at once, and not everywhere. Some requests wait in a queue and finish later, some time
  out. Running instances do not notice.

diagram:
  type: compare
  caption: The same outage looks different for each
  columns:
    - kicker: nova-conductor down
      title: Requests wait
      icon: hourglass
      tone: warn
      points:
        - "New builds stay in `BUILD`, and finish when it is back"
        - "Every compute is reported `down`"
        - Start, stop and reboot time out
    - kicker: nova-scheduler down
      title: Builds fail
      icon: octagon-x
      tone: stop
      points:
        - New builds fail after about 60 s
        - "They end in `ERROR`, with no host, in cell0"
        - Start, stop and reboot still work
  note: API reads work in both cases. `nova-api` reads the databases itself.

checks:
  - label: Services, as Nova sees them
    hint: openstack compute service list
  - label: Do their queues have consumers?
    hint: rabbitmqctl list_queues -p <vhost> name messages consumers
  - label: Their own logs
    hint: nova-conductor.log, nova-scheduler.log
  - label: Builds still waiting
    hint: openstack server list --all-projects --status BUILD

caution:
  - level: avoid
    text: >-
      Deleting instances while the conductor is down. Computes are reported down, so each delete is a local
      delete.
  - level: avoid
    text: >-
      Restarting Nova services while RabbitMQ is partitioned or blocked. They reconnect, and wait again.

lesson: With the conductor gone, every compute looks down. Nothing is wrong on the hosts.

runbook:
  path: recovery/nova-conductor-or-scheduler-down-what-still-works.md
  status: lab-verified
  release: "2026.1"
  also:
    - recovery/rabbitmq-partition-or-alarm-rpc-timeouts.md

scope: >-
  Written from the Nova 2026.1 source and lab-verified with Kolla-Ansible. The 60 s is rpc_response_timeout,
  Nova's default. Builds that timed out are not retried: their owners build again. A start, stop or reboot
  sent while the conductor was down can leave a task state that nothing clears. Applies when every copy of
  the service is down, in a single cell. Queued builds survive with quorum queues and RabbitMQ staying up; a
  RabbitMQ reset drops them.

linkedin:
  title: "nova-conductor down: every compute looks dead, and none of them is"
  caption: |
    nova-conductor and nova-scheduler live on the controllers. When every copy of one of them is down, Nova does not fail at once, and not everywhere.

    With the conductor down:
    → new builds stay in BUILD, and finish by themselves when it is back
    → within a minute every compute is reported "down", while its instances keep running
    → start, stop and reboot time out

    With the scheduler down:
    → new builds fail after about a minute, and land in ERROR in cell0, with no host
    → start, stop and reboot still work

    In both cases API reads work, and running instances notice nothing.

    The trap is the wall of "down" computes. nova-compute has no database access of its own: its heartbeat goes through the conductor. Nothing is wrong on the hosts. And deleting an instance at that moment is a local delete.
  hashtags: [OpenStack, Nova, ControlPlane, IncidentResponse, CloudOperations, OzarLand]
