Skip to content

Reading the Open vSwitch logs (ovs-vswitchd, ovsdb-server) and changing their levels with ovs-appctl

Every compute (and, with ML2/OVS, every network node) runs two Open vSwitch daemons, whatever the Neutron backend:

  • ovsdb-server holds the switch configuration (bridges, ports, interfaces, controllers) in the Open_vSwitch database;
  • ovs-vswitchd reads that database, creates the ports in the kernel datapath and forwards packets, following the OpenFlow rules set by the Neutron Open vSwitch agent (ML2/OVS) or by ovn-controller (ML2/OVN).

Both log through the same library, vlog, with the same line format and the same ovs-appctl vlog/* commands. The logs of ovn-controller and of the OVN databases use the same format; what their messages mean is covered on the OVN page.

Where the logs are

Kolla-Ansible: the containers openvswitch_vswitchd and openvswitch_db start the daemons with -vconsole:emer -vsyslog:err -vfile:info and a log file on the kolla_logs volume (openvswitch-vswitchd.json.j2, openvswitch-db-server.json.j2):

/var/log/kolla/openvswitch/ovs-vswitchd.log
/var/log/kolla/openvswitch/ovsdb-server.log

So the file gets info and above, syslog (the journal) only err and above, and docker logs almost nothing.

Any (package install): without a file name, --log-file writes to <LOGDIR>/<program>.log, where LOGDIR is set when Open vSwitch is built (lib/vlog.man); distribution packages commonly use /var/log/openvswitch/. The running process shows the real path:

# On the compute
ps -o args= -C ovs-vswitchd | tr ' ' '\n' | grep -E '^--log-file|^-v'
ps -o args= -C ovsdb-server | tr ' ' '\n' | grep -E '^--log-file|^-v'

Reading one line

The default file pattern, from include/openvswitch/vlog.h, is %D{%Y-%m-%dT%H:%M:%S.###Z}|%05N|%c%T|%p|%m, which gives this shape (format, not a captured line):

<YYYY-MM-DDTHH:MM:SS.mmmZ>|<serial>|<module>[(<thread>)]|<LEVEL>|<message>
Field Meaning (ovs-appctl(8))
Timestamp UTC (Z), whatever the host's time zone
<serial> Message number, from 00001. Each thread counts on its own: a handler thread's first message is 00001 too (|00001|ofproto_dpif_xlate(handler3)|WARN|... on the lab). A main-thread line 00001|vlog|INFO|opened log file means the daemon restarted
<module> The vlog module that logged the message (bridge, connmgr, rconn, timeval, …); ovs-appctl vlog/list lists them
(<thread>) Present when the message comes from a thread other than the main one, for example (handler12) or (revalidator3)
<LEVEL> EMER, ERR, WARN, INFO or DBG

The levels, as ovs-appctl(8) defines them: emer (a major failure forced the process to abort), err (a high-level operation or subsystem failed; attention is warranted), warn (a low-level operation failed, higher levels may recover), info (useful in retrospect), dbg (too voluminous for normal use, never logged by default).

The first line of each run is the quickest way to spot a restart (a plain |00001| also matches the first message of every handler thread):

# On the compute (Kolla-Ansible path; adapt for a package install)
grep -n '|00001|vlog|INFO|opened log file' /var/log/kolla/openvswitch/ovs-vswitchd.log | tail -5
grep -E '\|(EMER|ERR|WARN)\|' /var/log/kolla/openvswitch/ovs-vswitchd.log | tail -30

Messages worth knowing

All the messages below are format strings from Open vSwitch v3.7.1; the blocks of lines are captured on a lab (Kolla-Ansible 22.2.0, Open vSwitch 3.7.1, ML2/OVS with the hybrid iptables firewall), redacted.

Ports and interfaces (module bridge)

Message (format) Meaning
bridge <bridge>: added interface <name> on port <n> A port from the database is now in the datapath. For an instance, <name> is its tap device (tap<port-id prefix>) on br-int, or qvo<port-id prefix> with the hybrid iptables firewall (OVSHybridIptablesFirewallDriver), where the tap sits on a Linux bridge qbr<port-id prefix>
bridge <bridge>: deleted interface <name> on port <n> The port was removed
could not open network device <name> (<error>) The database has an interface that ovs-vswitchd cannot open. No such device for a tap… (or qvo…) name usually means the instance or its tap is gone while the port is still in the database; on the lab, after a host reboot, for the port of an instance that was shut off
could not add network device <name> to ofproto (<error>) The device exists but could not be added to the bridge
bridge <bridge>: using datapath ID <dpid> The bridge was (re)configured; seen at start-up

On the lab, an instance booted and deleted 23 seconds later:

2026-10-27T07:49:51.527Z|00232|bridge|INFO|bridge br-int: added interface qvo<port-id-prefix> on port 15
2026-10-27T07:50:01.575Z|00233|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 2 s starting 10 s ago (12 adds, 4 deletes)
2026-10-27T07:50:14.619Z|00234|bridge|INFO|bridge br-int: deleted interface qvo<port-id-prefix> on port 15

and, at the start of ovs-vswitchd after the host rebooted, for an instance that was shut off:

2026-10-27T07:24:33.670Z|00054|bridge|WARN|could not open network device qvo<port-id-prefix> (No such device)

To match a tap device to a Neutron port: Nova names the tap tap followed by the port id, cut to 14 characters, so tap and the first 11 characters of the port id (NIC_NAME_LEN in nova/network/neutron.py and model.py). The database keeps the full id in external_ids:iface-id:

# On the compute. Kolla-Ansible: prefix with docker exec openvswitch_vswitchd
ovs-vsctl --columns=name,external_ids find Interface name=tap<port-id-prefix>
ovs-vsctl --columns=name,external_ids find Interface name=qvo<port-id-prefix>   # hybrid iptables firewall

With the hybrid firewall, the tap search prints nothing; the qvo one prints the port:

name                : qvo<port-id-prefix>
external_ids        : {attached-mac="fa:16:3e:00:00:01", iface-id="<port-id>", iface-status=active, vm-uuid="<server-id>"}

Controllers and database connections (modules rconn, reconnect, connmgr, jsonrpc)

ovs-vswitchd is an OpenFlow switch: its controller is the Neutron Open vSwitch agent (ML2/OVS, which listens by default on 127.0.0.1:6633, options of_listen_address and of_listen_port in ovs_conf.py), or ovn-controller (ML2/OVN). A controller connection is named <bridge><-><target>, for example br-int<->tcp:127.0.0.1:6633 (connmgr.c).

With ML2/OVN, ovn-controller is not in the bridge's Controller table: it connects as a client to the bridge's management socket, unix:<rundir>/br-int.mgmt (ovn-controller.c). Failures on that link are logged by ovn-controller, in ovn-controller.log, not as br-int<->… lines in ovs-vswitchd.log.

Message (format) Meaning Source
<name>: connecting... then <name>: connected (Re)connection to a controller or to the database. For OpenFlow, rconn.c logs connected at INFO only for reliable connections (such as a configured controller), and at DBG otherwise rconn.c, reconnect.c
<name>: connection failed (<error>), <name>: connection timed out An OpenFlow controller is not answering. Connection refused on 127.0.0.1:6633 means the Neutron agent is not listening (stopped, restarting) rconn.c
<name>: connection attempt failed (<error>), <name>: connection attempt timed out The same for a database (JSON-RPC) connection, such as a client of ovsdb-server reconnect.c
<name>: connection closed by peer, <name>: connection dropped (<error>) The other side went away reconnect.c
<name>: no response to inactivity probe after <n> seconds, disconnecting The other side is alive enough to keep the socket open but did not answer for <n> seconds: an overloaded or blocked agent, ovsdb-server or ovs-vswitchd reconnect.c, rconn.c
<name>: waiting <n> seconds before reconnect Back-off between attempts; the wait doubles up to a maximum. Logged by rconn.c for OpenFlow connections (whole seconds) and by reconnect.c for database connections rconn.c, reconnect.c
<name>: receive error: <error>, <name>: send error: <error> A JSON-RPC (database) connection failed; <name> such as unix#<n> is a client of the unix socket jsonrpc.c

On the lab, systemctl restart kolla-neutron_openvswitch_agent-container.service gave, for each of br-int, br-ex and br-tun (br-int shown):

2026-10-27T07:51:28.578Z|00236|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connection closed by peer
2026-10-27T07:51:29.667Z|00239|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting...
2026-10-27T07:51:29.667Z|00240|rconn|WARN|br-int<->tcp:127.0.0.1:6633: connection failed (Connection refused)
2026-10-27T07:51:29.667Z|00241|rconn|INFO|br-int<->tcp:127.0.0.1:6633: waiting 2 seconds before reconnect
2026-10-27T07:51:31.667Z|00249|rconn|WARN|br-int<->tcp:127.0.0.1:6633: connection failed (Connection refused)
2026-10-27T07:51:31.667Z|00250|rconn|INFO|br-int<->tcp:127.0.0.1:6633: waiting 4 seconds before reconnect
2026-10-27T07:51:35.671Z|00257|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting...
2026-10-27T07:51:36.171Z|00260|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connected
2026-10-27T07:51:46.175Z|00263|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 38 flow_mods in the 9 s starting 10 s ago (18 adds, 20 deletes)

A controller connection that drops and comes back while the Neutron agent restarts is expected; the agent's restart does not cut the instances (restarting the OVS agent).

The daemon is struggling (modules timeval, ovs_rcu, memory, vlog)

Message (format) Meaning Source
Unreasonably long <n>ms poll interval (<u>ms user, <s>ms system), sometimes followed by faults: …, disk: …, context switches: … The main loop of the daemon did not come back for at least one second. user high: the daemon itself was busy (huge flow table, many ports). system high, or faults with major faults: the host is short of memory or CPU. Correlate with the kernel log and the host load lib/timeval.c
blocked <n> ms waiting for <thread> to quiesce A thread is stuck; with repeated lines, the daemon is close to hung lib/ovs-rcu.c
<n> kB peak resident set size after <t> seconds, then peak resident set size grew <p>% in last <t> seconds, from <a> kB to <b> kB Memory growth report (the first at start-up, then each growth of 50 % or more) lib/memory.c
Dropped <n> log messages in last <t> seconds (most recently, <s> seconds ago) due to excessive rate Rate limiting hid messages: the problem is bigger than the lines around it show lib/vlog.c
<datapath>: failed to add <name> as port: <error> The kernel datapath refused a port lib/dpif.c

Seen on the lab at the start of ovs-vswitchd, just after the host booted, with a cold disk rather than a busy daemon:

2026-10-27T07:24:32.404Z|00009|timeval|WARN|Unreasonably long 1409ms poll interval (0ms user, 1ms system)
2026-10-27T07:24:32.404Z|00010|timeval|WARN|faults: 5 minor, 2 major
2026-10-27T07:24:32.404Z|00011|timeval|WARN|disk: 360 reads, 0 writes
2026-10-27T07:24:32.404Z|00012|timeval|WARN|context switches: 2 voluntary, 0 involuntary

When ovs-vswitchd stops, its log simply ends; the next start begins again at serial 00001. See ovs-vswitchd down on a compute for what that does to the instances.

ovsdb-server

ovsdb-server logs its version at start-up (ovsdb-server (Open vSwitch) <version>; on the lab 2026-10-27T07:24:32.927Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 3.7.1), loading configuration from '<file>', and failures such as failed to open database '<name>': <error> or failed to configure remotes: <error> (ovsdb-server.c). Its client connections use the jsonrpc and reconnect messages above. ovs-vswitchd, ovs-vsctl, the Neutron agent and ovn-controller are all clients.

Changing the log levels with ovs-appctl

ovs-appctl talks to a running daemon through its control socket. By default it targets ovs-vswitchd; -t ovsdb-server targets the database server. From ovs-appctl(8):

# On the compute (read-only). Kolla-Ansible: docker exec openvswitch_vswitchd ovs-appctl ...
#                                         and docker exec openvswitch_db ovs-appctl -t ovsdb-server ...
ovs-appctl vlog/list                 # modules, and the level of each for console, syslog and file
ovs-appctl vlog/list-pattern         # the pattern of each destination
ovs-appctl -t ovsdb-server vlog/list

vlog/set takes up to one word of each kind: a module (from vlog/list), a destination (syslog, console, file), a level (off, emer, err, warn, info, dbg). Messages at the level or above are kept. Logging to a file only happens if the daemon was started with --log-file.

⚠️ State-changing. Debug logging can grow the log file very fast on a busy compute (some modules log every packet or flow) and costs CPU. On Kolla-Ansible, /var/log/kolla is shared by every service of the host, so a runaway log can fill the disk for all of them. Instances are not affected. The change lasts until you undo it or the daemon restarts. Raise one module at a time, on the destination file, and note the level shown by vlog/list before you change it. Many debug messages are rate-limited; ovs-appctl vlog/disable-rate-limit <module> lifts that and adds even more volume: use it only briefly, and undo it with vlog/enable-rate-limit <module>.

# On the compute. Kolla-Ansible: docker exec openvswitch_vswitchd ovs-appctl ...
ovs-appctl vlog/list | grep -E 'console|bridge|rconn'          # note the current levels (columns: console, syslog, file)
ovs-appctl vlog/set bridge:file:dbg                            # one module, file only
ovs-appctl vlog/set rconn:file:dbg

Undo by setting back the level you noted, info with Kolla-Ansible's -vfile:info:

ovs-appctl vlog/set bridge:file:info
ovs-appctl vlog/set rconn:file:info

ovs-appctl vlog/set without any argument sets every module and destination to dbg: avoid it on a production compute.

Rotation

ovs-appctl vlog/reopen makes the daemon close and reopen its log file, which is what a log rotation needs after moving the file. Neither it nor vlog/close has an effect if the daemon was started without --log-file. Kolla-Ansible's logrotate configuration uses copytruncate (cron-logrotate-global.conf.j2), so vlog/reopen is rarely needed there.

Do not use vlog/close on a production host: the daemon then stops writing its log file until vlog/reopen or a restart, and the next incident leaves no trace.

A short reading order

  1. Network lost for every instance of one compute: is ovs-vswitchd running, and did its log restart at 00001?
  2. One instance without network: added interface tap… for its port at boot, or could not open network device?
  3. Flows missing or stale: with ML2/OVS, connection failed or no response to inactivity probe towards the Neutron agent in ovs-vswitchd.log, then the agent's own log. With ML2/OVN, read ovn-controller.log: the OpenFlow link to br-int.mgmt is logged there.
  4. Slowness, drops under load: Unreasonably long … poll interval and blocked … waiting … to quiesce, then the kernel log and host load for the same UTC minute.

Related runbooks: ovs-vswitchd down on a compute, tunnel MTU mismatch, restarting the OVS agent, after a compute reboot.

Lab observations

On an all-in-one lab (OpenStack 2026.1, Kolla-Ansible 22.2.0, Ubuntu 24.04, Docker, ML2/OVS with VXLAN and the hybrid iptables firewall, Open vSwitch 3.7.1 in the Kolla images):

Step Observed
Log paths and ps options As written: -vconsole:emer -vsyslog:err -vfile:info, the two files under /var/log/kolla/openvswitch/; docker logs shows only the start wrapper; nothing in the journal
grep '|00001|' Also matched the first message of each handler thread (ofproto_dpif_xlate(handlerN)); only vlog|INFO|opened log file marked the restart
grep of EMER|ERR|WARN As written; could not open network device qvo... (No such device), removing policing failed: No such device, and dropping VLAN 0 packet warnings at start-up
Instance boot and delete added interface / deleted interface for qvo<prefix>, not tap<prefix> (hybrid firewall); find Interface name=tap... printed nothing, name=qvo... the port
Neutron agent restart (systemctl restart) connection closed by peer, Connection refused, back-off 2 then 4 seconds, connected 7.6 s after the drop
vlog/list, vlog/list-pattern, -t ovsdb-server vlog/list As written
vlog/set bridge:file:dbg, rconn:file:dbg, then back to info As written; 1799 bytes in 30 s on an idle lab
vlog/reopen Returned without error; no line written

Not run: the package-install paths, ML2/OVN (ovn-controller and br-int.mgmt), DPDK, and the messages that need a fault the lab did not cause (inactivity probe, blocked ... to quiesce, Dropped ... log messages, failed to add).

Validity and limits

  • Written from upstream source and documentation: Open vSwitch v3.7.1, Kolla-Ansible and Neutron stable/2026.1. The blocks of lines are captured on the lab (Open vSwitch 3.7.1 in the Kolla 2026.1 Ubuntu images); the table entries are format strings from the source. Older Open vSwitch releases may word some lines differently (ovs-vswitchd --version gives yours). The Open vSwitch version on a 2026.1 compute depends on the base distribution of the images or packages, so it may not be 3.7.1: the run saw only the Ubuntu images.
  • DPDK (ovs-vswitchd with a userspace datapath) adds its own modules and messages; not covered.
  • The meaning given for Unreasonably long … poll interval with high system time is an interpretation of what the numbers measure (getrusage of the thread), not an upstream statement.

Provenance: written by agent:ozarland-author, from upstream documentation and source code (Open vSwitch, Kolla-Ansible, Neutron). Reviewed in its pull request (see the page history). Checked by a human operator: not yet.

Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.