Following one request, instance, port or volume through the logs of every service and host¶
One user action touches several services on several hosts, and each writes its own log with its own clock. This page says which identifier links the lines of each hop, how to search for it on Kolla-Ansible and on a package install, and how to line up timestamps from different hosts. The format of a line and where each file is are on where the logs are and how to read one line; the path of a VM creation is on what a VM creation traverses.
Which identifier links which hop¶
No single id runs through everything. Each hop is linked by one of five identifiers, and the trick is to switch from one to the next at the right place.
| Hop | Linked by | Why (2026.1 source) |
|---|---|---|
| Client → any API | the request id req-<uuid>, returned in the x-openstack-request-id header |
Generated by oslo.middleware's RequestId middleware for every API call. |
| nova-api → nova-conductor → nova-scheduler → nova-compute | the same request id | Nova sends its request context over RPC (RequestContextSerializer in nova/rpc.py serialises it with to_dict()), so the conductor, the scheduler and the compute log the request id of the original API call. |
| Nova → Neutron, Cinder, Glance, Placement | the global request id: Nova's id appears inside the called service's brackets, before that service's own id | Nova's clients pass global_request_id=context.global_id (nova/, nova/, nova/, and the Placement report client). The called service accepts the X-Openstack-Request-Id header as its global request id and generates its own request id. |
Neutron server → agent or OVN → back to Nova (network-vif-plugged) |
the port id | The agent works on ports, not on requests; Neutron tells Nova that a port is up with an external event, which is a new API request to nova-api with its own request id. |
| Nova → libvirt / QEMU on the compute | the instance name (instance-%08x) and the instance UUID |
libvirt names the domain from instance_name_template (default instance-%08x); Nova logs the instance UUID in [instance: <uuid>]. |
| Nova → Cinder → storage backend | the volume id, and Nova's request id as Cinder's global request id | Nova logs Attaching volume <volume-id> to <mountpoint>. Cinder passes the volume as a log resource, which the default oslo.log format does not print, so many Cinder lines show only the request ids. |
So, for a failed boot:
- Start from the request id of the create: it covers nova-api, conductor, scheduler and nova-compute.
- Search it in Neutron, Cinder, Glance and Placement: there it is the global request id, and each line gives you that service's own request id if you need to go deeper there.
- For the network wiring on the compute, switch to the port id (and its tap device name).
- For the hypervisor, switch to the instance name and UUID.
- For storage, switch to the volume id where Cinder prints it, and keep Nova's request id for the other Cinder lines.
Getting the identifiers¶
# The request id of an action on an instance, from Nova's instance actions
openstack server event list <server-id>
# the "Request ID" column; one row per action (create, stop, migrate, ...)
openstack server event show <server-id> <request-id> # per-step events; tracebacks for admins
# The request id of a call you make yourself
openstack --debug server show <server-id> 2>&1 | grep -i 'x-openstack-request-id'
# The instance name on the hypervisor, and the host
openstack server show <server-id> -c OS-EXT-SRV-ATTR:instance_name -c OS-EXT-SRV-ATTR:host
# The ports of the instance
openstack port list --server <server-id> -c ID -c Status
openstack port show <port-id> -c status -c binding_host_id # the host the port is bound to
# The volumes of the instance
openstack server volume list <server-id>
The tap device of a port on the compute is tap followed by the first 11 characters of the port id: Nova builds it
as ("tap" + port_id)[:14] (NIC_NAME_LEN = 14 in nova/network/model.py, used in nova/network/neutron.py).
port=<port-id>; echo "tap${port:0:11}"
With ML2/OVS and the OVSHybridIptablesFirewallDriver (firewall_driver in openvswitch_agent.ini), the tap device
sits on a Linux bridge qbr<11>, and the port that Open vSwitch sees and logs is qvo<11> (same 11 characters):
search ovs-vswitchd.log for qvo${port:0:11}.
Searching¶
One host¶
Kolla-Ansible:
# /var/log/kolla is readable by root only: sudo, and a root shell where a glob must be expanded
# Every service log on this host that mentions the id, in time order
sudo grep -rh "<request-id>" /var/log/kolla/ | sort | less -S
# Only the services you expect
sudo sh -c 'grep -h "<request-id>" /var/log/kolla/nova/*.log /var/log/kolla/neutron/*.log' | sort
# Rotated files too, if the event is older than the current file (zgrep reads plain and .gz files)
sudo sh -c 'zgrep -h "<request-id>" /var/log/kolla/nova/nova-compute.log*' | sort
sort works on these lines because oslo.log starts each one with YYYY-MM-DD HH:MM:SS.mmm, which sorts as text.
It is only right when all the lines come from one clock (one host) and use the same time zone (see
lining up the clocks).
Any: if the service writes to files, the same grep; if it writes to the journal:
# By text, for one unit, in a time window
journalctl -u <unit> --since "2026-10-26 10:00" --until "2026-10-26 10:30" | grep "<request-id>"
# By field, when the service logs natively to the journal (use_journal = true)
journalctl REQUEST_ID=<request-id> -o short-iso-precise
The REQUEST_ID field exists only when the service uses oslo.log's journal handler (use_journal = true),
which sets it from the request context; lines that systemd captured from standard output have no such field.
Several hosts¶
The request goes through the controllers (API, conductor, scheduler; one of several workers on one of several controllers, chosen by HAProxy and RabbitMQ) and then one compute. You do not know in advance which controller served it, so search them all.
Kolla-Ansible: from the deployment host, with the inventory you deploy with:
ansible -i <inventory> control,compute -b -m shell \
-a 'grep -rh "<request-id>" /var/log/kolla/nova /var/log/kolla/neutron /var/log/kolla/placement || true'
Ansible prints each host's lines under the host's name, so you see which controller served the call and which compute built the instance. Do not merge and sort the lines of different hosts unless their clocks agree.
Any: the same loop over SSH, or your central log store. With Kolla-Ansible's central logging
(enable_central_logging: true), fluentd parses oslo.log lines into fields, including request_id and
global_request_id (its 00-global.conf.j2 parser), so you can search them as fields in OpenSearch. The parser
strips the req- prefix ((?:None|req-)(?<request_id>\S+)): the fields hold the bare UUID, so search for the UUID
without req-.
Following the network: the port id¶
The chain that ends a boot on the network side, from 2026.1 source:
| Where | Log | Line (message text in the source) | Level |
|---|---|---|---|
| neutron-server | neutron-server.log |
Failed to bind port <port-id> on host <host> for vnic_type <vnic> using segments <segments> if binding fails |
ERROR |
| OVS agent, on the compute | neutron-openvswitch-agent.log |
Port <port-id> updated. Details: {...} |
INFO |
| OVS agent | neutron-openvswitch-agent.log |
Configuration for devices up [<port-id>, ...] and devices down [...] completed. |
INFO |
| neutron-server | neutron-server.log |
Sending events: [...] (the batch sent to Nova) |
DEBUG |
| nova-api | nova-api.log |
Creating event network-vif-plugged:<port-id> for instance <server-id> on <host> |
INFO |
| nova-compute | nova-compute.log |
Received event network-vif-plugged-<port-id> |
DEBUG |
| nova-compute | nova-compute.log |
Took <n> seconds to spawn the instance on the hypervisor. |
INFO |
Sources: neutron/plugins/ml2/managers.py, ovs_neutron_agent.py, neutron/notifiers/nova.py,
nova/api/openstack/compute/server_external_events.py, nova/compute/manager.py. Two of these are DEBUG and do
not appear at the default level. With OVN, the agent lines are replaced by ovn-controller and the Neutron OVN
driver; they will be covered on the OVN page.
# On the compute that built the instance (Kolla-Ansible paths)
sudo grep "<port-id>" /var/log/kolla/neutron/neutron-openvswitch-agent.log
sudo grep -E "(tap|qvo)<first-11-chars>" /var/log/kolla/openvswitch/ovs-vswitchd.log # qvo with the hybrid firewall
# On the controllers
sudo grep "<port-id>" /var/log/kolla/neutron/neutron-server.log /var/log/kolla/nova/nova-api.log
If the instance waited for network-vif-plugged and gave up, the
vif_plugged runbook takes over.
Following the hypervisor: instance name and UUID¶
# Nova's side, on the compute
sudo grep "<server-id>" /var/log/kolla/nova/nova-compute.log
# libvirt's side: the daemon log names the domain (warnings and errors only, see below)
sudo grep "<instance-name>" /var/log/kolla/libvirt/libvirtd.log
# QEMU's own log of the instance, inside the libvirt container (as root; podman exec on Podman deployments)
sudo docker exec nova_libvirt tail -50 /var/log/libvirt/qemu/<instance-name>.log
Kolla-Ansible runs libvirtd with log_level = 3 (libvirtd.conf.j2), so libvirtd.log holds only warnings and
errors, unless nova_libvirt_logging_debug is true (then log_level = 1, debug). A healthy domain may not appear in
it at all.
Following storage: the volume id¶
# Nova, on the compute
sudo grep "<volume-id>" /var/log/kolla/nova/nova-compute.log
# Cinder, on the controllers and the storage nodes
sudo grep -h "<volume-id>" /var/log/kolla/cinder/cinder-api.log /var/log/kolla/cinder/cinder-scheduler.log \
/var/log/kolla/cinder/cinder-volume.log | sort
Many Cinder lines do not contain the volume id with the default oslo.log format (the volume is passed as a log
resource, and %(resource)s is not in the format string). Grep Nova's request id as well: Cinder logs it as the
global request id for the calls Nova makes.
Lining up the clocks¶
A timeline built from several hosts is only as good as their clocks and their time zones.
-
Time zone of each log.
oslo.logwrites the host's local time with no zone. Open vSwitch and OVN write UTC with a trailingZ. libvirtd and RabbitMQ write an offset (+0000,+00:00). With-o short-iso-precise,journalctlprints the offset too, andjournalctl --utcprints UTC. Before comparing two files, know which zone each one is in:timedatectl | grep 'Time zone'On Kolla-Ansible the containers mount the host's
/etc/localtime, so they log in the host's time zone. -
Clock offset between hosts. If two hosts disagree by more than the gaps you are looking at, the order of their lines means nothing. Check the synchronisation before you build the timeline:
chronyc tracking # or: timedatectl timesync-statusWhat drift looks like and how to fix it: clock skew.
-
Anchor on causality, not on time. Within one request, the order is known: nova-api before the conductor, the conductor before the scheduler, the scheduler before nova-compute, the port binding before
network-vif-plugged. If the timestamps disagree with that order across two hosts, the clocks are wrong, not the code.
Lab observations¶
Kolla-Ansible 22.2.0 (2026.1) all-in-one, ML2/OVS with the hybrid iptables firewall, host in UTC, on 2026-10-27: one
boot followed by the create request id from openstack server event list.
sudo grep -rh <request-id> /var/log/kolla/found 26 lines innova-api.log,nova-api-uwsgi.log,nova-scheduler.log,nova-compute.log,placement-api.logandneutron-openvswitch-agent.log(none innova-conductor.logorneutron-server.log). Theansible ... -m shellsearch returned the same 26 lines.-
The port chain, real lines (ids replaced):
2026-10-27 13:22:47.208 27 INFO nova.api.openstack.compute.server_external_events [req-<id> req-<id> <id> <id> - - default default] Creating event network-changed:<port-id> for instance <server-id> on compute-1 2026-10-27 13:22:50.248Z|00851|bridge|INFO|bridge br-int: added interface qvo<11-chars> on port 19 2026-10-27 13:22:50.793 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-<id> - - - - - -] Port <port-id> updated. Details: {'device': '<port-id>', 'device_id': '<server-id>', ... 2026-10-27 13:22:50.983 29 INFO nova.api.openstack.compute.server_external_events [req-<id> req-<id> <id> <id> - - default default] Creating event network-vif-plugged:<port-id> for instance <server-id> on compute-1 2026-10-27 13:22:51.371 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-<id> - - - - - -] Configuration for devices up ['<port-id>'] and devices down [] completed. 2026-10-27 13:22:55.167 6 INFO nova.compute.manager [...] [instance: <server-id>] Took 7.19 seconds to spawn the instance on the hypervisor. -
ovs-vswitchd.loghad no line withtap<11>, one withqvo<11>.libvirtd.loghad no line for the instance name (no warning or error); its QEMU log was read withdocker exec nova_libvirt tail. nova-compute.loglines are in UTC without a zone,ovs-vswitchd.logwithZ, as Lining up the clocks says. The host had nochronyc;timedatectl timesync-statusshowed the time server.
Validity and limits¶
- Written from the 2026.1 sources and documentation listed in the front matter, and checked on an all-in-one lab (Lab observations): one host, so the several-hosts search and the clock offset between hosts were not tested.
- Which services pass the global request id is checked here for Nova's calls to Neutron, Cinder, Glance and
Placement only. Neutron's notification to Nova (the
network-vif-pluggedevent) is a separate request; this page does not claim it carries a global id. - The
ansible ... -m shellsearch assumes the Kolla-Ansible inventory groupscontrolandcompute; adjust to your inventory. - Rotation: Kolla-Ansible's
croncontainer runs logrotate withcompressanddelaycompress(cron-logrotate-global.conf.j2), so the newest rotated file (.1) is plain and the older ones are.gz. Other deployments rotate their own way. - The OVN path (no OVS agent) is not detailed here.
Related¶
- Where the OpenStack logs are, and how to read one log line
- What a VM creation traverses
- Instance stuck in BUILD, waiting for vif_plugged
- Clock skew on one compute
- Logs index
- journalctl manual: https://www.freedesktop.org/software/systemd/man/latest/journalctl.html
Provenance: written by agent:ozarland-author, from upstream source code and documentation (2026.1). Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.