Reading the Nova logs. What normal looks like, and what the common lines mean¶
Nova writes four logs that matter for an instance: nova-api, nova-conductor, nova-scheduler (on the
controllers) and nova-compute (on each compute). Each answers a different question. This page says what each
one logs when things are fine, which lines signal a problem, what they mean, and where to look next. Every message
quoted here is the message text from Nova's 2026.1 source (file named in each table); the values in angle brackets
are placeholders.
File locations and the line format: where the logs are. Following one request through these logs: what a VM creation traverses.
Which log answers which question¶
| Question | Log | Where |
|---|---|---|
| Did the API accept the request, and what did it answer? | nova-api.log, and the uWSGI request line in nova-api-uwsgi.log on Kolla-Ansible |
controllers |
| Was a host found? Why not? | nova-scheduler.log |
controllers |
| What happened between the API and the compute, and after a failure (retries, cell0)? | nova-conductor.log |
controllers |
| What did the compute do with the instance: claim, network, disk, hypervisor? | nova-compute.log |
the compute in OS-EXT-SRV-ATTR:host, or every compute that tried |
Nova's own record of what happened to an instance is often faster than the logs, and tells you which compute to read:
openstack server show <server-id> -c status -c fault -c OS-EXT-SRV-ATTR:host
openstack server event list <server-id>
openstack server event show <server-id> <request-id> # per-step result; tracebacks for admins
When a service starts¶
Every Nova service built on nova.service logs, at start:
Starting <topic> node (version <version>)
(nova/service.py, INFO; <topic> is compute, conductor or scheduler). Use it to find restarts:
sudo grep -n "Starting compute node" /var/log/kolla/nova/nova-compute.log | tail -5
A service that started before nova-conductor was reachable logs, until the conductor answers
(nova/conductor/api.py):
| Line | Level | Meaning |
|---|---|---|
Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection... |
WARNING | The service sent an RPC ping to the conductor and got no reply. It retries; it does not stop. Repeated: the conductor is down, or RabbitMQ does not deliver. |
nova-conductor connection established successfully |
INFO | The wait is over. Logged only after at least one timeout. |
nova-compute¶
What normal looks like¶
- At start, the libvirt driver logs the host's capabilities (
Libvirt host capabilities <xml>,INFO,nova/virt/libvirt/host.py), and on the first start of a new compute the resource tracker logsCompute node record created for <host>:<node> with uuid: <uuid>(INFO,resource_tracker.py). - Every periodic run of the resource tracker (every 60 s by default), at
DEBUGonly:Final resource view: name=<node> phys_ram=<n>MB used_ram=<n>MB phys_disk=<n>GB used_disk=<n>GB total_vcpus=<n> used_vcpus=<n> pci_stats=[...] stats={...}. With debug on, this is the compute's own view of its capacity, to compare with Placement. - For each instance it builds:
Claim successful on node <node>(claims.py,INFO), thenTook <n> seconds to spawn the instance on the hypervisor.(INFO), then libvirt lifecycle events asVM <state> (Lifecycle Event)(INFO; for exampleStarted,Resumed,Paused,Stopped), all with[instance: <uuid>].
At the default INFO level, a healthy compute is quiet between builds. A periodic [-] line repeating every minute
at WARNING or above is worth reading.
Lines that signal a problem¶
| Line (message text) | Level | What it means | Next |
|---|---|---|---|
Failed to build and run instance followed by a traceback |
ERROR | The build failed on this compute for a reason not handled specifically. The traceback's last line is the cause. Nova may retry on another compute: the fault then reads Build of instance <uuid> was re-scheduled: <reason> here, and the next compute logs its own attempt. |
Exceeded maximum number of retries |
Instance failed to spawn followed by a traceback |
ERROR | The failure happened inside the build's resource setup or the hypervisor spawn. Read the traceback: a libvirt error, a disk or image problem, a timeout. | the traceback's last line; libvirt's log on the compute |
Failed to allocate network(s) with a traceback, then the fault Build of instance <uuid> aborted: Failed to allocate the network(s), not rescheduling. |
ERROR | The network part of the build failed, often the wait for network-vif-plugged. Aborted, not retried on another compute. |
Waiting for vif_plugged |
Timeout waiting for ['network-vif-plugged-<port-id>'] for instance with vm_state <state> and task_state <state>. Event states are: <...> |
WARNING | Nova waited vif_plugging_timeout seconds for Neutron's event and none came. In 2026.1 the list holds the event keys built by InstanceExternalEvent.make_key (<name>-<tag>); the vif_plugged runbook quotes an older tuple form. grep "Timeout waiting for" matches both. |
Waiting for vif_plugged; follow the port id in neutron-server.log and the agent log of the compute |
Build of instance <uuid> aborted: Failure prepping block device. |
ERROR (fault) | The boot volume or block device mapping could not be prepared. Not retried. | Boot from volume fails |
Build of instance <uuid> aborted: <reason> with a flavor or image reason (for example FlavorDiskTooSmall, ImageNotActive) |
ERROR (fault) | The request cannot work on any compute; Nova does not retry it. | fix the flavor or image |
Insufficient compute resources: <reason>. |
(exception text) | The claim failed on this compute: its own view of capacity disagreed with the scheduler's. Retried on an alternate. | Placement and the resource tracker; No valid host |
Connection event '0' reason 'Connection to libvirt lost: <reason>' (or … 'Failed to connect to libvirt: <error>' at start) |
INFO | nova-compute lost its connection to libvirtd. The libvirt driver disables the compute service with a reason starting with AUTO: (_set_host_enabled in driver.py), and re-enables it when the connection comes back. |
openstack compute service list --long (Disabled Reason); libvirt's log; restarting nova_libvirt |
Connection to libvirt failed: <error> with a traceback |
ERROR | nova-compute could not open a connection to libvirtd at all. | Is libvirtd (nova_libvirt on Kolla-Ansible) running? |
Error updating resources for node <node>. with a traceback |
ERROR | The periodic resource update failed: Placement unreachable, a database or RPC error, or a driver error. The compute's inventory in Placement is not refreshed while this lasts. | the traceback; Placement's log |
Lost connection to nova-conductor for reporting service status. |
WARNING | The service heartbeat (sent through the conductor over RabbitMQ) timed out. Logged once; Recovered from being unable to report status. (INFO) when it works again. Long enough, and the service shows down. |
RabbitMQ; nova-conductor; nova-compute down |
Instance shutdown by itself. Calling the stop API. Current vm_state: <state>, current task_state: <state>, original DB power_state: <n>, current VM power_state: <n> |
WARNING | The power state sync found the domain shut down while Nova thought it ran. Nova records the stop. The cause is inside the guest or on the host (OOM killer, QEMU crash). | QEMU's log of the instance; the host's kernel log |
Sources: nova/compute/manager.py, nova/compute/claims.py, nova/compute/resource_tracker.py,
nova/servicegroup/drivers/db.py, nova/virt/libvirt/host.py, nova/virt/libvirt/driver.py,
nova/exception.py (BuildAbortException, RescheduledException, ComputeResourcesUnavailable).
Kolla-Ansible: on the compute:
sudo grep -nE ' [0-9]+ (ERROR|CRITICAL) ' /var/log/kolla/nova/nova-compute.log | tail -20
sudo grep -n "<server-id>" /var/log/kolla/nova/nova-compute.log
sudo grep -nE "Connection event|Connection to libvirt|Lost connection to nova-conductor|Error updating resources" /var/log/kolla/nova/nova-compute.log | tail
Any: the same grep on /var/log/nova/nova-compute.log, or journalctl -u <unit> piped to grep.
nova-scheduler¶
What normal looks like¶
At INFO, the scheduler logs little for a successful request. With debug on, it logs each filter's result and the
weighing; that is the level to use when you need to know why a host was not chosen.
Lines that signal a problem¶
| Line (message text) | Level | What it means | Next |
|---|---|---|---|
Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up. |
INFO | Placement returned no resource provider that has the requested resources and traits. The scheduler raises NoValidHost without running any filter. |
Placement: inventories, allocations, traits; No valid host |
Filtering removed all hosts for the request with instance ID '<uuid>'. Filter results: [...] |
INFO (short form), DEBUG (full form) | Placement found candidates, then the filters removed every one. The Filter results list shows how many hosts each filter kept, in order: the filter after which the count drops to 0 is the one to look at. |
the filter's configuration; No valid host |
Sources: nova/scheduler/manager.py, nova/filters.py.
sudo grep -n "Got no allocation candidates\|Filtering removed all hosts" /var/log/kolla/nova/nova-scheduler.log | tail
A request that fails here never reaches a compute: there is nothing in any nova-compute.log for it.
nova-conductor¶
Lines that signal a problem¶
| Line (message text) | Level | What it means | Next |
|---|---|---|---|
Failed to schedule instances with a traceback ending in NoValidHost: No valid host was found. <reason> |
ERROR | The scheduler found no host. The conductor puts the instance in ERROR in cell0 (the instance never reaches a cell database), so server show works but there is nothing on any compute. |
the scheduler's log for the same request id |
Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance <uuid>. |
(fault, MaxRetriesExceeded) |
Computes were chosen and each failed; no alternate was left. | each compute that tried: max retries runbook |
Host '<host>' is not mapped to any cell |
(exception text) | A compute exists that no cell knows about, usually a new compute before host discovery. | nova-manage cell_v2 list_hosts; VM creation traverses |
Sources: nova/conductor/manager.py, nova/exception.py (NoValidHost, MaxRetriesExceeded,
HostMappingNotFound).
nova-api¶
What normal looks like¶
The application log (nova-api.log) records errors and some actions; on Kolla-Ansible 2026.1 the per-request line
(method, path, status, duration) is in uWSGI's log, nova-api-uwsgi.log.
Lines that signal a problem¶
| Line (message text) | Level | What it means | Next |
|---|---|---|---|
HTTP exception thrown: <message> |
INFO | The API answered a 4xx on purpose (not found, conflict, bad request). Usually a client error, not a platform one. | the message itself |
Unexpected exception in API method with a traceback; the client gets Unexpected API Error. and HTTP 500 |
ERROR | An unhandled exception inside the API: often a dependency (database, RabbitMQ, Keystone, Placement) failing underneath. | the traceback; shared dependencies |
MessagingTimeout: Timed out waiting for a reply to message ID <msg-id> in a traceback |
ERROR | A synchronous RPC call (to the conductor, or to a compute) got no reply within rpc_response_timeout (60 s by default in oslo.messaging). The other side is down, overloaded, or RabbitMQ did not deliver. |
RabbitMQ; the service that should have answered |
Sources: nova/api/openstack/wsgi.py; oslo_messaging/_drivers/amqpdriver.py for the MessagingTimeout text.
Reading a failure in order¶
server showfault andserver event list: which action, which request id, and whether a host was assigned.- No host assigned: read
nova-scheduler.logandnova-conductor.logfor the request id. Placement or a filter decided. - A host assigned: read that compute's
nova-compute.logfor the instance UUID. With retries, read every compute listed inserver event show. - The last line of the traceback names the exception; the lines just above it name the call that raised it.
- Dependencies:
MessagingTimeout, database errors and 401s in several services at once point below Nova.
Captured on a lab¶
Kolla-Ansible 22.2.0 (2026.1, Nova 33.0.3) all-in-one, 2026-10-23 to 2026-10-27, redacted. The ERROR lines are
from other lab tests (Glance unreachable, an image larger than the flavor, RabbitMQ down). The exception is on the
same line as the message, after ::
INFO nova.service [-] Starting compute node (version 33.0.3)
INFO nova.virt.libvirt.driver [None req-<id> - - - - - -] Connection event '1' reason 'None'
INFO nova.compute.claims [None req-<id> <user-id> <project-id> - - default default] [instance: <server-id>] Claim successful on node compute-1
INFO nova.compute.manager [...] [instance: <server-id>] Took 5.49 seconds to spawn the instance on the hypervisor.
ERROR nova.compute.manager [...] [instance: <server-id>] Failed to build and run instance: nova.exception.GlanceConnectionFailed: Connection to glance host http://192.0.2.10:9292 failed: ...
ERROR nova.compute.manager [...] [instance: <server-id>] Instance failed to spawn: nova.exception.FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 1073741824 bytes, image is 2147483648 bytes.
WARNING nova.servicegroup.drivers.db [-] Lost connection to nova-conductor for reporting service status.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <id>
INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
INFO nova.filters [...] Filtering removed all hosts for the request with instance ID '<server-id>'. Filter results: ['ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', ...
INFO nova.scheduler.manager [...] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
ERROR nova.conductor.manager [...] Failed to schedule instances: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <id>
INFO nova.api.openstack.wsgi [...] HTTP exception thrown: Instance <name> could not be found.
Connection event '1' reason 'None' is the connection to libvirt being established, at every start. With debug
on, the configuration dump at start contains option values such as CRITICAL: the error grep above matches the
level field only (' [0-9]+ (ERROR|CRITICAL) '), not those lines. The files are readable by root only, hence
sudo.
Validity and limits¶
- Every message on this page is the message text from Nova 2026.1 (and
oslo.messaging2026.1 forMessagingTimeout), read in the source files listed above; Captured on a lab has the real lines seen. The lab-verified vif_plugged runbook shows theBuild of instance … abortedfault; itsTimeout waiting for …line uses a tuple form that differs from the 2026.1 source. A lab request (#59) asks for real samples. - Level and exact wording can change between releases; a message text quoted here is for 2026.1.
- Only the libvirt driver is covered.
- "What normal looks like" lists the lines that the source logs at
INFOorDEBUG; how many of them appear on your platform depends on itsdebugsetting and itsdefault_log_levels.
Related¶
- Where the OpenStack logs are, and how to read one log line
- What a VM creation traverses
- No valid host with free capacity
- Exceeded maximum number of retries
- Waiting for vif_plugged
- Logs index
Provenance: written by agent:ozarland-author, from Nova 2026.1 source code and documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.