What a VM creation traverses, and how to trace one failed boot by request id¶
A failed VM creation says what the user experienced, not which service caused it. A scheduling problem looks like a Nova problem, a network timeout like a Neutron problem, a slow operation like a capacity problem, and the architecture makes each plausible without making any true. Knowing the path, and following one request id through the logs, replaces guessing by timestamp.
The path¶
user ──► Keystone (token) ──► nova-api ─────────────────────────────► API database (request recorded)
│ RPC (RabbitMQ)
▼
nova-conductor ──► nova-scheduler ──► Placement (candidates, claim allocation)
│ cell mapping: which cell DB holds the instance
│ RPC
▼
nova-compute (selected host)
├──► Glance: image (or Cinder: boot volume)
├──► Neutron: create + bind port ──► agent / ovn-controller wires it
│ ◄── network-vif-plugged event (Neutron ─► nova-api)
└──► libvirt/QEMU: start the domain
Where each step fails, and what it looks like:
| Step | Failure shows as | First place to look |
|---|---|---|
| Keystone / memcached | 401, 503, or everything slow | Keystone logs, memcached, certificate expiry |
| nova-api → conductor (RPC) | Request accepted, instance never leaves scheduling; MessagingTimeout |
RabbitMQ health, nova-api.log |
| Scheduler / Placement | ERROR, fault No valid host was found; no host assigned |
nova-scheduler.log, Placement candidates (runbook) |
| Cell mapping | Host '<host>' is not mapped to any cell |
nova-conductor.log, nova-manage cell_v2 list_hosts |
| Glance | BUILD/spawning then image download error |
nova-compute.log, glance-api.log, image store backend |
| Cinder (boot from volume) | BUILD/block_device_mapping stuck; volume stays creating |
cinder-volume.log, storage backend |
| Neutron port create/bind | binding_failed; Failed to bind port (the user may only see Exceeded maximum number of retries) |
neutron-server.log |
| vif plug on the host | BUILD/spawning for vif_plugging_timeout then ERROR |
Agent / ovn-controller on the host (runbook) |
| libvirt / QEMU | ERROR with a libvirt error in fault |
nova-compute.log, libvirtd log on the host |
Inside Nova the API, conductor, scheduler and compute talk over RabbitMQ. Calls to Keystone, Placement, Glance,
Neutron and Cinder are REST calls through the service catalogue. With agent-based networking, neutron-server
talks to its agents over RabbitMQ as well; OVN uses its own databases instead. Every service reads and writes its
own database. A user-visible failure can therefore originate in any service, in the bus, in the database cluster,
or in the network and storage underneath.
Tracing one request¶
Every API request carries a request id (req-<uuid>) that the services propagate in their logs. Get it, then
follow it.
# From the failed instance's actions (the request id of the create action)
openstack server event list <server-id>
openstack server event show <server-id> <request-id> # per-service events and tracebacks (admin)
# Or from the client that made the call: the create's id is on the 202 response; the other ids are lookups
openstack --debug server create ... 2>&1 | grep -E "^RESP: \[202\]" | grep -ioE "x-openstack-request-id: req-[0-9a-f-]+"
server event show can be thin: after a build failure that was rescheduled and then gave up, it may list only
compute__do_build_and_run_instance with result Success, and the instance fault says Exceeded maximum number of
retries. The cause is then only in the logs.
Then search every service's logs for that id, in path order:
Kolla-Ansible: logs are under /var/log/kolla/<service>/ on each node.
sudo grep -h "<request-id>" /var/log/kolla/nova/nova-api.log /var/log/kolla/nova/nova-conductor.log \
/var/log/kolla/nova/nova-scheduler.log /var/log/kolla/placement/placement-api.log \
/var/log/kolla/neutron/neutron-server.log /var/log/kolla/glance/glance-api.log | sort
# and on the selected compute host
sudo grep "<request-id>" /var/log/kolla/nova/nova-compute.log
Any: journalctl -u <service> | grep <request-id>, or the configured log files. A platform whose logs are
centralised and searchable by request id follows one boot across Nova, Placement, Neutron and Glance in minutes.
Note that Neutron and Placement log Nova's request id (the global request id) next to their own for the calls
Nova makes to them ([req-<nova-id> req-<own-id> ...]), so the one grep above finds their lines too. Glance, at
the default log level (observed on Kolla-Ansible with debug = False), logged no line for a successful call, so finding nothing there does not mean Glance was not
called. A few unrelated lines can carry the id as well (on a lab, a later GET /v2.1 version request in
nova-api.log); check that each line belongs to the request.
Reading what you find¶
- The last service that logged the request id is where it stopped; the first error after that point is the cause, or the symptom of a dependency (bus, database) failing underneath.
- Separate facts (observed in logs), hypotheses (explain the facts) and assumptions (not yet shown). Familiar fixes are attractive; a known fix applied to a different failure mode adds risk.
- Try to disprove your hypothesis as well as confirm it: can the behaviour be reproduced on one host, with one flavor, on one network? Reducing the surface makes every observation worth more.
- If one request fails and others succeed, compare them: same host? same network type? same image? The difference is the lead.
Lab observations¶
Kolla-Ansible 2026.1 all-in-one, ML2/OVS. Two failed boots traced with the commands above:
- No valid host (a flavor larger than the host): Nova's id found in
nova-api,placement-api(GET /allocation_candidates),nova-scheduler(Got no allocation candidates from the Placement API) andnova-conductor(NoValidHost), in about 2 s;server event showgave the conductor event with the error. - Port binding failed (Open vSwitch agent stopped until reported dead): the same grep found the
neutron-serverlinesRefusing to bind port ... to dead agentandFailed to bind port ... on host, with Nova's id;nova-compute.loghadPortBindingFailed. The instance went toERROR15 s after the request with the faultExceeded maximum number of retries, andserver event showreported the compute event asSuccess.
Validity and limits¶
Describes the Nova cells v2 architecture (mandatory since Ocata) with the libvirt driver and the Neutron ML2 or OVN
backends. Log paths are given for Kolla-Ansible; other deployments differ. Request-id pairing in logs depends on
release and on oslo.log configuration. The lab confirmation (OpenStack 2026.1, Kolla-Ansible all-in-one, ML2/OVS)
traced two failures, no valid host and a port binding failure; it did not reach the Cinder, Glance download,
libvirt and cell-mapping rows of the table.
Related¶
- VM creation fails platform-wide: check the shared dependencies first.
- Source: OpenStack, the Day After Tomorrow (S. Zaouam), chapters 5, 8 and 9.
Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.