Reading the Open vSwitch logs (ovs-vswitchd, ovsdb-server) and changing their levels with ovs-appctl¶
Every compute (and, with ML2/OVS, every network node) runs two Open vSwitch daemons, whatever the Neutron backend:
ovsdb-serverholds the switch configuration (bridges, ports, interfaces, controllers) in theOpen_vSwitchdatabase;ovs-vswitchdreads that database, creates the ports in the kernel datapath and forwards packets, following the OpenFlow rules set by the Neutron Open vSwitch agent (ML2/OVS) or byovn-controller(ML2/OVN).
Both log through the same library, vlog, with the same line format and the same ovs-appctl vlog/* commands. The
logs of ovn-controller and of the OVN databases use the same format; what their messages mean is covered on the OVN
page.
Where the logs are¶
Kolla-Ansible: the containers openvswitch_vswitchd and openvswitch_db start the daemons with
-vconsole:emer -vsyslog:err -vfile:info and a log file on the kolla_logs volume
(openvswitch-vswitchd.json.j2,
openvswitch-db-server.json.j2):
/var/log/kolla/openvswitch/ovs-vswitchd.log
/var/log/kolla/openvswitch/ovsdb-server.log
So the file gets info and above, syslog (the journal) only err and above, and docker logs almost nothing.
Any (package install): without a file name, --log-file writes to <LOGDIR>/<program>.log, where LOGDIR is set
when Open vSwitch is built (lib/vlog.man); distribution
packages commonly use /var/log/openvswitch/. The running process shows the real path:
# On the compute
ps -o args= -C ovs-vswitchd | tr ' ' '\n' | grep -E '^--log-file|^-v'
ps -o args= -C ovsdb-server | tr ' ' '\n' | grep -E '^--log-file|^-v'
Reading one line¶
The default file pattern, from
include/openvswitch/vlog.h, is
%D{%Y-%m-%dT%H:%M:%S.###Z}|%05N|%c%T|%p|%m, which gives this shape (format, not a captured line):
<YYYY-MM-DDTHH:MM:SS.mmmZ>|<serial>|<module>[(<thread>)]|<LEVEL>|<message>
| Field | Meaning (ovs-appctl(8)) |
|---|---|
| Timestamp | UTC (Z), whatever the host's time zone |
<serial> |
Message number, from 00001. Each thread counts on its own: a handler thread's first message is 00001 too (|00001|ofproto_dpif_xlate(handler3)|WARN|... on the lab). A main-thread line 00001|vlog|INFO|opened log file means the daemon restarted |
<module> |
The vlog module that logged the message (bridge, connmgr, rconn, timeval, …); ovs-appctl vlog/ lists them |
(<thread>) |
Present when the message comes from a thread other than the main one, for example (handler12) or (revalidator3) |
<LEVEL> |
EMER, ERR, WARN, INFO or DBG |
The levels, as ovs-appctl(8) defines them: emer (a major failure forced the process to abort), err (a high-level
operation or subsystem failed; attention is warranted), warn (a low-level operation failed, higher levels may
recover), info (useful in retrospect), dbg (too voluminous for normal use, never logged by default).
The first line of each run is the quickest way to spot a restart (a plain |00001| also matches the first message of
every handler thread):
# On the compute (Kolla-Ansible path; adapt for a package install)
grep -n '|00001|vlog|INFO|opened log file' /var/log/kolla/openvswitch/ovs-vswitchd.log | tail -5
grep -E '\|(EMER|ERR|WARN)\|' /var/log/kolla/openvswitch/ovs-vswitchd.log | tail -30
Messages worth knowing¶
All the messages below are format strings from Open vSwitch v3.7.1; the blocks of lines are captured on a lab (Kolla-Ansible 22.2.0, Open vSwitch 3.7.1, ML2/OVS with the hybrid iptables firewall), redacted.
Ports and interfaces (module bridge)¶
| Message (format) | Meaning |
|---|---|
bridge <bridge>: added interface <name> on port <n> |
A port from the database is now in the datapath. For an instance, <name> is its tap device (tap<port-id prefix>) on br-int, or qvo<port-id prefix> with the hybrid iptables firewall (OVSHybridIptablesFirewallDriver), where the tap sits on a Linux bridge qbr<port-id prefix> |
bridge <bridge>: deleted interface <name> on port <n> |
The port was removed |
could not open network device <name> (<error>) |
The database has an interface that ovs-vswitchd cannot open. No such device for a tap… (or qvo…) name usually means the instance or its tap is gone while the port is still in the database; on the lab, after a host reboot, for the port of an instance that was shut off |
could not add network device <name> to ofproto (<error>) |
The device exists but could not be added to the bridge |
bridge <bridge>: using datapath ID <dpid> |
The bridge was (re)configured; seen at start-up |
On the lab, an instance booted and deleted 23 seconds later:
2026-10-27T07:49:51.527Z|00232|bridge|INFO|bridge br-int: added interface qvo<port-id-prefix> on port 15
2026-10-27T07:50:01.575Z|00233|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 2 s starting 10 s ago (12 adds, 4 deletes)
2026-10-27T07:50:14.619Z|00234|bridge|INFO|bridge br-int: deleted interface qvo<port-id-prefix> on port 15
and, at the start of ovs-vswitchd after the host rebooted, for an instance that was shut off:
2026-10-27T07:24:33.670Z|00054|bridge|WARN|could not open network device qvo<port-id-prefix> (No such device)
To match a tap device to a Neutron port: Nova names the tap tap followed by the port id, cut to 14 characters, so
tap and the first 11 characters of the port id (NIC_NAME_LEN in
nova/network/neutron.py and
model.py). The database keeps the full id in external_ids:iface-id:
# On the compute. Kolla-Ansible: prefix with docker exec openvswitch_vswitchd
ovs-vsctl --columns=name,external_ids find Interface name=tap<port-id-prefix>
ovs-vsctl --columns=name,external_ids find Interface name=qvo<port-id-prefix> # hybrid iptables firewall
With the hybrid firewall, the tap search prints nothing; the qvo one prints the port:
name : qvo<port-id-prefix>
external_ids : {attached-mac="fa:16:3e:00:00:01", iface-id="<port-id>", iface-status=active, vm-uuid="<server-id>"}
Controllers and database connections (modules rconn, reconnect, connmgr, jsonrpc)¶
ovs-vswitchd is an OpenFlow switch: its controller is the Neutron Open vSwitch agent (ML2/OVS, which listens by
default on 127.0.0.1:6633, options of_listen_address and of_listen_port in
ovs_conf.py),
or ovn-controller (ML2/OVN). A controller connection is named <bridge><-><target>, for example
br-int<->tcp:127.0.0.1:6633 (connmgr.c).
With ML2/OVN, ovn-controller is not in the bridge's Controller table: it connects as a client to the bridge's
management socket, unix:<rundir>/br-int.mgmt
(ovn-controller.c). Failures on that link
are logged by ovn-controller, in ovn-controller.log, not as br-int<->… lines in ovs-vswitchd.log.
| Message (format) | Meaning | Source |
|---|---|---|
<name>: connecting... then <name>: connected |
(Re)connection to a controller or to the database. For OpenFlow, rconn.c logs connected at INFO only for reliable connections (such as a configured controller), and at DBG otherwise |
rconn.c, reconnect.c |
<name>: connection failed (<error>), <name>: connection timed out |
An OpenFlow controller is not answering. Connection refused on 127.0.0.1:6633 means the Neutron agent is not listening (stopped, restarting) |
rconn.c |
<name>: connection attempt failed (<error>), <name>: connection attempt timed out |
The same for a database (JSON-RPC) connection, such as a client of ovsdb-server |
reconnect.c |
<name>: connection closed by peer, <name>: connection dropped (<error>) |
The other side went away | reconnect.c |
<name>: no response to inactivity probe after <n> seconds, disconnecting |
The other side is alive enough to keep the socket open but did not answer for <n> seconds: an overloaded or blocked agent, ovsdb-server or ovs-vswitchd |
reconnect.c, rconn.c |
<name>: waiting <n> seconds before reconnect |
Back-off between attempts; the wait doubles up to a maximum. Logged by rconn.c for OpenFlow connections (whole seconds) and by reconnect.c for database connections |
rconn.c, reconnect.c |
<name>: receive error: <error>, <name>: send error: <error> |
A JSON-RPC (database) connection failed; <name> such as unix#<n> is a client of the unix socket |
jsonrpc.c |
On the lab, systemctl restart kolla-neutron_openvswitch_agent-container.service gave, for each of br-int,
br-ex and br-tun (br-int shown):
2026-10-27T07:51:28.578Z|00236|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connection closed by peer
2026-10-27T07:51:29.667Z|00239|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting...
2026-10-27T07:51:29.667Z|00240|rconn|WARN|br-int<->tcp:127.0.0.1:6633: connection failed (Connection refused)
2026-10-27T07:51:29.667Z|00241|rconn|INFO|br-int<->tcp:127.0.0.1:6633: waiting 2 seconds before reconnect
2026-10-27T07:51:31.667Z|00249|rconn|WARN|br-int<->tcp:127.0.0.1:6633: connection failed (Connection refused)
2026-10-27T07:51:31.667Z|00250|rconn|INFO|br-int<->tcp:127.0.0.1:6633: waiting 4 seconds before reconnect
2026-10-27T07:51:35.671Z|00257|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting...
2026-10-27T07:51:36.171Z|00260|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connected
2026-10-27T07:51:46.175Z|00263|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 38 flow_mods in the 9 s starting 10 s ago (18 adds, 20 deletes)
A controller connection that drops and comes back while the Neutron agent restarts is expected; the agent's restart does not cut the instances (restarting the OVS agent).
The daemon is struggling (modules timeval, ovs_rcu, memory, vlog)¶
| Message (format) | Meaning | Source |
|---|---|---|
Unreasonably long <n>ms poll interval (<u>ms user, <s>ms system), sometimes followed by faults: …, disk: …, context switches: … |
The main loop of the daemon did not come back for at least one second. user high: the daemon itself was busy (huge flow table, many ports). system high, or faults with major faults: the host is short of memory or CPU. Correlate with the kernel log and the host load |
lib/timeval.c |
blocked <n> ms waiting for <thread> to quiesce |
A thread is stuck; with repeated lines, the daemon is close to hung | lib/ovs-rcu.c |
<n> kB peak resident set size after <t> seconds, then peak resident set size grew <p>% in last <t> seconds, from <a> kB to <b> kB |
Memory growth report (the first at start-up, then each growth of 50 % or more) | lib/memory.c |
Dropped <n> log messages in last <t> seconds (most recently, <s> seconds ago) due to excessive rate |
Rate limiting hid messages: the problem is bigger than the lines around it show | lib/vlog.c |
<datapath>: failed to add <name> as port: <error> |
The kernel datapath refused a port | lib/dpif.c |
Seen on the lab at the start of ovs-vswitchd, just after the host booted, with a cold disk rather than a busy daemon:
2026-10-27T07:24:32.404Z|00009|timeval|WARN|Unreasonably long 1409ms poll interval (0ms user, 1ms system)
2026-10-27T07:24:32.404Z|00010|timeval|WARN|faults: 5 minor, 2 major
2026-10-27T07:24:32.404Z|00011|timeval|WARN|disk: 360 reads, 0 writes
2026-10-27T07:24:32.404Z|00012|timeval|WARN|context switches: 2 voluntary, 0 involuntary
When ovs-vswitchd stops, its log simply ends; the next start begins again at serial 00001. See
ovs-vswitchd down on a compute for what that does to the instances.
ovsdb-server¶
ovsdb-server logs its version at start-up (ovsdb-server (Open vSwitch) <version>; on the lab
2026-10-27T07:24:32.927Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 3.7.1), loading configuration from
'<file>', and failures such as failed to open database '<name>': <error> or failed to configure remotes: <error>
(ovsdb-server.c). Its client connections use
the jsonrpc and reconnect messages above. ovs-vswitchd, ovs-vsctl, the Neutron agent and ovn-controller are
all clients.
Changing the log levels with ovs-appctl¶
ovs-appctl talks to a running daemon through its control socket. By default it targets ovs-vswitchd; -t
ovsdb-server targets the database server. From
ovs-appctl(8):
# On the compute (read-only). Kolla-Ansible: docker exec openvswitch_vswitchd ovs-appctl ...
# and docker exec openvswitch_db ovs-appctl -t ovsdb-server ...
ovs-appctl vlog/list # modules, and the level of each for console, syslog and file
ovs-appctl vlog/list-pattern # the pattern of each destination
ovs-appctl -t ovsdb-server vlog/list
vlog/set takes up to one word of each kind: a module (from vlog/list), a destination (syslog, console,
file), a level (off, emer, err, warn, info, dbg). Messages at the level or above are kept. Logging to a
file only happens if the daemon was started with --log-file.
⚠️ State-changing. Debug logging can grow the log file very fast on a busy compute (some modules log every packet or flow) and costs CPU. On Kolla-Ansible,
/var/log/kollais shared by every service of the host, so a runaway log can fill the disk for all of them. Instances are not affected. The change lasts until you undo it or the daemon restarts. Raise one module at a time, on the destinationfile, and note the level shown byvlog/listbefore you change it. Many debug messages are rate-limited;ovs-appctl vlog/disable-rate-limit <module>lifts that and adds even more volume: use it only briefly, and undo it withvlog/enable-rate-limit <module>.
# On the compute. Kolla-Ansible: docker exec openvswitch_vswitchd ovs-appctl ...
ovs-appctl vlog/list | grep -E 'console|bridge|rconn' # note the current levels (columns: console, syslog, file)
ovs-appctl vlog/set bridge:file:dbg # one module, file only
ovs-appctl vlog/set rconn:file:dbg
Undo by setting back the level you noted, info with Kolla-Ansible's -vfile:info:
ovs-appctl vlog/set bridge:file:info
ovs-appctl vlog/set rconn:file:info
ovs-appctl vlog/set without any argument sets every module and destination to dbg: avoid it on a production
compute.
Rotation¶
ovs-appctl vlog/reopen makes the daemon close and reopen its log file, which is what a log rotation needs after
moving the file. Neither it nor vlog/close has an effect if the daemon was started without --log-file.
Kolla-Ansible's logrotate configuration uses copytruncate
(cron-logrotate-global.conf.j2),
so vlog/reopen is rarely needed there.
Do not use vlog/close on a production host: the daemon then stops writing its log file until vlog/reopen or a
restart, and the next incident leaves no trace.
A short reading order¶
- Network lost for every instance of one compute: is
ovs-vswitchdrunning, and did its log restart at00001? - One instance without network:
added interface tap…for its port at boot, orcould not open network device? - Flows missing or stale: with ML2/OVS,
connection failedorno response to inactivity probetowards the Neutron agent inovs-vswitchd.log, then the agent's own log. With ML2/OVN, readovn-controller.log: the OpenFlow link tobr-int.mgmtis logged there. - Slowness, drops under load:
Unreasonably long … poll intervalandblocked … waiting … to quiesce, then the kernel log and host load for the same UTC minute.
Related runbooks: ovs-vswitchd down on a compute, tunnel MTU mismatch, restarting the OVS agent, after a compute reboot.
Lab observations¶
On an all-in-one lab (OpenStack 2026.1, Kolla-Ansible 22.2.0, Ubuntu 24.04, Docker, ML2/OVS with VXLAN and the hybrid iptables firewall, Open vSwitch 3.7.1 in the Kolla images):
| Step | Observed |
|---|---|
Log paths and ps options |
As written: -vconsole:emer -vsyslog:err -vfile:info, the two files under /var/; docker logs shows only the start wrapper; nothing in the journal |
grep '|00001|' |
Also matched the first message of each handler thread (ofproto_dpif_xlate(handlerN)); only vlog|INFO|opened log file marked the restart |
grep of EMER|ERR|WARN |
As written; could not open network device qvo... (No such device), removing policing failed: No such device, and dropping VLAN 0 packet warnings at start-up |
| Instance boot and delete | added interface / deleted interface for qvo<prefix>, not tap<prefix> (hybrid firewall); find Interface name=tap... printed nothing, name=qvo... the port |
Neutron agent restart (systemctl restart) |
connection closed by peer, Connection refused, back-off 2 then 4 seconds, connected 7.6 s after the drop |
vlog/list, vlog/, -t ovsdb-server vlog/ |
As written |
vlog/, rconn:file:dbg, then back to info |
As written; 1799 bytes in 30 s on an idle lab |
vlog/reopen |
Returned without error; no line written |
Not run: the package-install paths, ML2/OVN (ovn-controller and br-int.mgmt), DPDK, and the messages that need a
fault the lab did not cause (inactivity probe, blocked ... to quiesce, Dropped ... log messages, failed to add).
Validity and limits¶
- Written from upstream source and documentation: Open vSwitch v3.7.1, Kolla-Ansible and Neutron
stable/2026.1. The blocks of lines are captured on the lab (Open vSwitch 3.7.1 in the Kolla 2026.1 Ubuntu images); the table entries are format strings from the source. Older Open vSwitch releases may word some lines differently (ovs-vswitchd --versiongives yours). The Open vSwitch version on a 2026.1 compute depends on the base distribution of the images or packages, so it may not be 3.7.1: the run saw only the Ubuntu images. - DPDK (
ovs-vswitchdwith a userspace datapath) adds its own modules and messages; not covered. - The meaning given for
Unreasonably long … poll intervalwith highsystemtime is an interpretation of what the numbers measure (getrusageof the thread), not an upstream statement.
Provenance: written by agent:ozarland-author, from upstream documentation and source code (Open vSwitch, Kolla-Ansible, Neutron). Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.