Instance stuck in BUILD, then ERROR after vif_plugging_timeout¶
A networking problem wearing a compute symptom. The instance sits in BUILD with task state spawning for about
five minutes, then goes to ERROR. Nova is waiting for Neutron to tell it the port is wired, and the message never
comes.
Symptom¶
$ openstack server show <server-id> -c status -c OS-EXT-STS:task_state -c OS-EXT-STS:vm_state
status | BUILD
task_state | spawning
vm_state | building
After vif_plugging_timeout seconds (default 300), the instance goes to ERROR. With vif_plugging_is_fatal
(default true) the fault message is:
Build of instance <server-id> aborted: Failed to allocate the network(s), not rescheduling.
Its details, and nova-compute.log on the selected host, show the chain behind it:
Timeout waiting for [('network-vif-plugged', '<port-id>')] for instance with vm_state building and task_state spawning
... InstanceEventTimeout
... VirtualInterfaceCreateException: Virtual Interface creation failed
... BuildAbortException ... not rescheduling
Scheduling itself succeeded: while the instance is in BUILD it has a host (OS-EXT-SRV-ATTR:host) and an
allocation in Placement. Once the build is aborted, Nova clears the host field: an ERROR instance of this kind shows
no host (observed on a lab). The build is not retried on another host.
Impact¶
New instances fail on one host, a set of hosts, or everywhere, depending on the cause. Running instances are not
affected. Each failed build leaves an ERROR instance that counts in the project's quota until deleted. Its
Placement allocation is held while it waits, then removed when the build is aborted (observed on a lab).
Quick checks¶
Run on a controller with admin credentials, unless stated otherwise.
# 1. Which host was selected, and the fault
openstack server show <server-id> -c OS-EXT-SRV-ATTR:host -c fault
# Empty host on an ERROR instance: the build was aborted and the host cleared. Find the host in
# nova-compute.log on the computes (grep <server-id>) or from the other failed builds.
# 2. Is the port bound, and to what?
openstack port list --server <server-id> -c ID -c Status
openstack port show <port-id> -c binding_vif_type -c binding_host_id -c status
# port list has no "Binding Host" column and silently drops it: read the host from port show
# binding_vif_type = binding_failed -> Neutron could not bind the port on that host
# status = DOWN with a valid vif_type -> bound, but never came up: agent/controller side
# 3. Is the networking agent or OVN controller alive on that host?
openstack network agent list --host <compute-host>
# 4. Is it one host or all of them?
openstack server list --all-projects --status ERROR --long -c ID -c Host -c Created | head
Diagnosis¶
Nova (compute host)¶
nova-compute.log for the instance UUID shows the wait: Preparing to wait for external event network-vif-plugged,
then the timeout. If instead you see network-vif-plugged received and the boot still fails, this is a different
runbook.
Kolla-Ansible: docker logs nova_compute 2>&1 | grep <server-id> or /var/log/kolla/nova/nova-compute.log.
Any: journalctl -u nova-compute | grep <server-id> or the configured log file.
Neutron server¶
Search neutron-server.log for the port UUID. Look for the binding attempt and its result:
Failed to bind port <port-id> on host <host>: no mechanism driver accepted the binding. Usual causes: the agent on that host is dead or not reporting, the network's segmentation type (VLAN, VXLAN, Geneve) is not configured on that host's bridge mappings, or the physical network name does not match.- Binding succeeded but no
network-vif-pluggedwas sent: the agent or OVN controller never reported the port up, or Neutron could not reach the Nova API to deliver the event (check[nova]section credentials and endpoint inneutron.conf, andneutron-server.logforFailed to notify nova on events).
Agent or OVN controller (compute host)¶
ML2/OVS: openstack network agent list --host <host> must show Open vSwitch agent alive. An agent whose
process is frozen stops reporting and shows not alive, while its container still looks up.
When the agent stopped reporting matters. Neutron binds the port when the build starts. If the agent was alive then,
the binding succeeds (binding_vif_type ovs), the port stays DOWN, and the build waits for
vif_plugging_timeout: this runbook. If the agent was already reported dead, the binding fails at once:
Failed to bind port in neutron-server.log, PortBindingFailed in nova-compute.log. Nova then tries another
host, and when none is left the fault is Exceeded maximum number of retries. Exhausted all hosts available for
retrying build failures, within seconds, not after five minutes. On a lab, an agent paused 70 s before the request
gave the second case; one paused right before it gave the first.
# On the compute (Kolla-Ansible)
sudo docker ps -a --filter name=neutron_openvswitch_agent --format '{{.Names}} {{.Status}}'
# "Up ... (Paused)" means the container is paused; "Exited" means it stopped
On the host,
ovs-vsctl show should list the port under br-int: as qvo<port-id-prefix> with the iptables hybrid firewall
driver (the one in the lab's openvswitch_agent.ini; the tap device then sits on a qbr Linux bridge), as
tap<port-id-prefix> with the openvswitch firewall driver; neutron-openvswitch-agent.log shows the
port being processed. An agent that is alive but has a stale or stuck RPC connection (after a RabbitMQ incident, for
example) processes nothing: look for AMQP server unreachable or long gaps in the log.
OVN (not run on the lab): on the host, docker logs ovn_controller (Kolla-Ansible) or journalctl -u ovn-controller. Check that
the chassis is registered and claims the port:
ovn-sbctl show # run where ovn-sbctl can reach the southbound DB
ovn-sbctl --columns=logical_port,chassis find Port_Binding logical_port=<port-id>
ovn-nbctl lsp-get-up <port-id> # true once the port is up
An empty chassis means the controller on that host never claimed the port: the host's ovn-bridge-mappings or
ovn-encap-ip external_ids may be wrong, or ovn-controller cannot reach the southbound database (check the
ovn-remote setting and TLS).
Message bus (agent-based backends only)¶
If every host is affected at once after a RabbitMQ or network event, the agents may be connected to a partitioned or stale bus. See VM creation fails platform-wide: check the shared dependencies.
Possible causes¶
| Cause | How to tell |
|---|---|
| Agent or OVN controller stopped processing after the port was bound | network agent list shows it dead now, but the port is bound (binding_vif_type not binding_failed) and DOWN; or ovn-sbctl show has no chassis for the host. An agent already dead when the build started gives binding_failed instead (see Diagnosis) |
| Bridge mappings or physical network mismatch on the host | binding_vif_type = binding_failed; neutron-server.log says no mechanism driver bound the port; only networks of one type fail |
| Agents alive but RPC stale after a bus incident | Agents report alive (heartbeat is on a separate connection) but process nothing; recovers on restart |
| Neutron cannot notify Nova | Port is ACTIVE, neutron-server.log shows notification errors; every host affected |
| Host simply slow (storage, many ports at once) | Event arrives just after the timeout; raising vif_plugging_timeout is a workaround, not a fix |
Resolution¶
Agent or controller down, or RPC stale¶
⚠️ State-changing. Restarting
neutron-openvswitch-agentcan briefly disrupt data-plane changes on the host and, on some versions and configurations, existing flows; restartingovn-controlleris usually transparent but re-syncs every port on the host. Do it on one host first.
Kolla-Ansible: restart the agent through the systemd unit Kolla-Ansible creates for the container (since 2026.1;
a plain docker restart bypasses the unit), on the host:
sudo systemctl restart kolla-neutron_openvswitch_agent-container.service or
sudo systemctl restart kolla-ovn_controller-container.service (OVN not run on the lab). On a paused agent, the unit restart
worked on a lab: it took about 60 s, because the paused container ignores the stop and is killed (Exited (137)),
then the agent was alive again.
Any: systemctl restart neutron-openvswitch-agent or systemctl restart ovn-controller.
If docker ps shows the agent container (Paused), resume it instead of restarting it:
⚠️ State-changing. Resumes the frozen agent process; it then processes the ports that changed meanwhile. Undo:
docker pause neutron_openvswitch_agent(which reproduces the failure). Find out what paused it.
# On the compute (Kolla-Ansible)
sudo docker unpause neutron_openvswitch_agent
Builds that already failed are not retried: clean them up as below.
Bridge mappings or physical network mismatch¶
(Not run on the lab, nor is disabling the compute service below.)
Fix the host's configuration (bridge_mappings in the OVS agent config, or ovn-bridge-mappings in
ovs-vsctl get open . external_ids) so that it matches the other hosts and the network's provider:physical_network,
then restart the agent or controller as above. Until it is fixed, keep the scheduler away from the host:
⚠️ State-changing. Disabling the compute service stops new instances landing on the host. Running instances are not affected.
openstack compute service set --disable --disable-reason "binding_failed on <network>" <compute-host> nova-compute
Neutron cannot notify Nova¶
(Not run on the lab.)
Fix the [nova] section of neutron.conf (auth URL, credentials, region, endpoint type) and restart
neutron-server. Verify with a test boot.
Clean up the failed instances¶
⚠️ State-changing. Deleting the
ERRORinstances frees quota and Placement allocations. Confirm with the owner first if the instance may hold data on an attached volume (the volume itself is not deleted unless it was created withdelete_on_termination).
openstack server delete <server-id>
Verification¶
# A test boot on the affected host reaches ACTIVE within the normal time
openstack server create --flavor <flavor> --image <image> --network <network> \
--availability-zone nova:<compute-host> vif-test
openstack server show vif-test -c status -c OS-EXT-SRV-ATTR:host
openstack port list --server vif-test -c Status # ACTIVE
openstack server delete vif-test
# No agent or chassis missing
openstack network agent list
Lab observations¶
Reproduced on a lab: OpenStack 2026.1, Kolla-Ansible 22.2.0, ML2/OVS with VXLAN, Docker, Ubuntu Noble, 3 controllers
and 2 computes. Kolla-Ansible 22.2.0 sets neither vif_plugging_timeout nor vif_plugging_is_fatal, so Nova's
defaults (300 s, true) applied. The failure was produced by freezing (pausing) the Open vSwitch agent container of
compute-1, then requesting a test instance on that compute.
| Before | During (360 s) | After (agent resumed) | |
|---|---|---|---|
| Open vSwitch agent compute-1 | alive | not alive (container Up (Paused)) |
alive |
| Test instance | none | fault after 317 s: Failed to allocate the network(s), not rescheduling (InstanceEventTimeout → VirtualInterfaceCreateException), no host |
deleted |
- The fault arrived about 300 s after the request, as
vif_plugging_timeoutpredicts, and the build was not tried on the other compute. - The fault message is the
Build of instance … abortedone;Virtual Interface creation failedappears in its details and innova-compute.log, not as the message itself. The earlier version of this page said otherwise. - The host field of the
ERRORinstance was empty. - An earlier attempt, on a lab that had run many other failures overnight, gave a fault after 134 s without this signature. Its message was not read; it is not counted.
Run again on an all-in-one lab (one VM: controller, network and compute), OpenStack 2026.1, Kolla-Ansible 22.2.0, ML2/OVS, iptables hybrid firewall:
| Agent paused | Port | Result |
|---|---|---|
70 s before the request, agent already XXX |
binding_failed, port removed |
ERROR after 20 s: Exceeded maximum number of retries…; Failed to bind port in neutron-server.log, PortBindingFailed in nova-compute.log |
| Right before the request | Bound (ovs, binding_host_id set), DOWN |
BUILD/spawning with a host and a Placement allocation, then ERROR after 324 s: Build of instance … aborted: Failed to allocate the network(s), not rescheduling; Timeout waiting for ['network-vif-plugged-<port-id>'], InstanceEventTimeout, VirtualInterfaceCreateException in nova-compute.log; host empty; no allocation left |
- A running instance kept answering ping through both (130 of 130 during the first).
sudo systemctl restart kolla-neutron_openvswitch_agent-container.serviceon the paused agent: 62 s, the container killed (Exited (137)) and started again, the agent alive 20 s later.openstack server deleteremoved bothERRORinstances. The verification test boot with--availability-zone nova:<compute-host>reachedACTIVEin 16 s with its portACTIVE.
Validity and limits¶
The mechanism (Nova waiting for network-vif-plugged, bounded by vif_plugging_timeout) is stable across recent
releases. It was first written from an operator's description of the provisioning path, release not recorded.
- Observed on one lab build: OpenStack 2026.1, Kolla-Ansible 22.2.0, ML2/OVS (VXLAN), Docker, Ubuntu Noble, 3 controllers and 2 computes. Cause observed: the Open vSwitch agent of one compute frozen.
- Also observed on an all-in-one lab (see Lab observations): the same fault chain, the quick checks, the binding
failure of an agent already dead,
systemctl restartof the paused agent, and the test boot of Verification. - Observed steps: the symptom and fault chain, the agent shown not alive, resuming the agent (
docker unpause), and deleting the failed instance. - Not run on a lab:
docker restartof the agent, every OVN step, the bridge-mapping, stale-RPC, Nova-notification and slow-host causes, disabling the compute service. - OVN commands assume the Neutron OVN driver; ML2/OVS commands assume the OVS agent. Not tested with Linux Bridge or SR-IOV ports, where binding failures have additional causes, nor on other releases, other deployment tools or at production scale.
Related¶
- No valid host with free capacity: the other common build failure, which happens before a host is selected.
- What a VM creation traverses, and tracing it by request id.
- Restarting the OVS agent does not cut instances: what a restart of the agent does to running traffic.
- ovs-vswitchd down on a compute
- Nova:
vif_plugging_timeoutandvif_plugging_is_fatal - Source: OpenStack, the Day After Tomorrow (S. Zaouam), chapter 5, technical note on the provisioning path.
- Source: lab reproduction by @soufian-zaouam (OpenStack 2026.1, Kolla-Ansible 22.2.0).
Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow and a lab reproduction by @soufian-zaouam on 2026.1. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.