id: PG-06
category: compute
title: Compute “down”. Is the host dead?
situation: >-
  `down` means Nova heard nothing from `nova-compute` for 60 s (by default), or that someone forced it
  down. It says nothing about the host.

diagram:
  type: compare
  caption: Same line in Nova, two different realities
  columns:
    - kicker: Only the agent stopped
      title: Host alive
      icon: heart-pulse
      tone: ok
      points:
        - The host answers ping and SSH
        - "`virsh list`: domains running, or paused"
        - Users notice nothing
      verdict: Fix the agent. Never evacuate
    - kicker: The host is gone
      title: Host dead
      icon: power
      tone: stop
      points:
        - No answer at TCP level
        - Confirmed out of band (BMC, switch port)
        - Instances unreachable for users
      verdict: Fence, then evacuate
  note: A domain `paused` on an I/O error counts as running. It keeps its disk locks.

checks:
  - label: Which compute, since when
    hint: openstack compute service list --service nova-compute --long
  - label: Is the host alive?
    hint: ping -c 3 <host> ; ssh <host> uptime
  - label: Do its instances run?
    hint: virsh list --all (on the compute)
  - label: Why did the agent stop?
    hint: "container exit code and OOM flag ; AMQP errors in nova-compute.log"

caution:
  - level: destructive
    text: >-
      Evacuating a host that is alive starts a second copy of each instance, both writing to the same
      volumes. That corrupts data.
  - level: avoid
    text: >-
      Deleting its instances now. Each is a local delete: the guest runs on, then is destroyed when the
      agent returns.

lesson: "“Down” is a missing heartbeat, not a dead host. Prove it before you act."

runbook:
  path: recovery/nova-compute-down-instances-still-running.md
  status: lab-verified
  release: "2026.1"
  also:
    - recovery/nova-compute-down-host-alive-ssh-refused.md
    - recovery/compute-host-failure-fence-before-evacuate.md

scope: >-
  Lab-verified on OpenStack 2026.1 with Kolla-Ansible and ML2/OVS, and observed in production on an earlier
  release by the runbook's maintainer. The lab stopped the agent on purpose: the real causes (crash, OOM kill,
  lost RabbitMQ connection) were not reproduced. A host that answers on TCP but refuses your SSH login is
  alive too. Applies to the libvirt driver.

linkedin:
  title: "\"Compute down\" does not mean the host is down"
  caption: |
    openstack compute service list shows a compute "down". What does Nova actually know?

    Only this: it has heard nothing from the nova-compute agent of that host for service_down_time, 60 seconds by default, unless someone forced the service down. Nothing about the host. Nothing about the instances.

    When only the agent stopped, the hypervisor and the instances run on, and users notice nothing. The danger is the operator, or the automation, that treats it as a dead host: evacuating now starts a second copy of every instance, writing to the same volumes.

    Before anything else:
    → does the host answer ping and SSH?
    → does virsh list show the domains running?

    If yes: fix the agent. Never evacuate.
  hashtags: [OpenStack, Nova, HighAvailability, CloudOperations, SRE, OzarLand]
