Control plane down. What still works for running instances, and what does not¶
The facts that make "stop the APIs, protect the running workloads" a credible option rather than a euphemism for an outage. Verify them for your own platform before an incident; they depend on the networking backend.
What keeps working when the APIs, the bus and the databases are down¶
- Running instances keep running, keep their disks, keep exchanging traffic. The data plane (hypervisors, disks, network paths) does not depend on the control plane to carry existing state.
- Existing DHCP leases keep being served: with ML2/OVS, by the DHCP agent's
dnsmasqon the network node as long as that host runs; with OVN, DHCP is answered locally on each compute byovn-controller. - Router failover with L3 HA (keepalived) or OVN gateway chassis is a data-plane mechanism and does not need the API.
- Security groups, floating IPs, routes already programmed stay in effect.
What is lost¶
- Creating, modifying, migrating or deleting any resource.
- Propagation of new state to the data plane: a new port, a changed security-group rule, a newly associated floating IP will not take effect until the control plane returns.
- Metadata: an instance that boots or reboots during the outage may fail to reach the metadata service (cloud-init keys, hostname, user data), because the metadata agent or proxy needs Nova and Neutron APIs.
- DHCP for new instances or expired leases with ML2/OVS, if
neutron-dhcp-agentcannot reachneutron-serverfor new port information (existing leases continue). - Console access (VNC/SPICE proxies need the API).
- Monitoring that reads the APIs goes blind; the instances are fine, the dashboards are not.
Two cautions¶
- "Stopping the control plane" means the APIs, the bus and the databases, not the agents. Neutron agents and
ovn-controllerrun on the data-plane nodes. Restarting some of them, depending on version and configuration, can interrupt traffic briefly (OVS agent restarts may reprogram flows). Leave them alone during a controlled stop. - The exact behaviour depends on the backend and the deployment. Verify on a lab or during a recovery
exercise: stop the APIs and the bus, then check that an instance keeps its connectivity, renews its lease, and
that a router failover still works. Write the result down for your platform.
Kolla-Ansible 2026.1: each container is wrapped in a systemd unit
kolla-<container>-container.servicewithRestart=always, so a plaindocker stopis undone afterRestartSec(11 s with the Kolla defaults, 59 s on the lab, which setsdocker_client_timeout: 600). Stop the unit instead (systemctl stop kolla-<container>-container.service) and start it again the same way. This stops the control plane: do it on a lab or in a planned recovery exercise only, never to diagnose a live incident.
How to check on a live platform, read-only¶
# From a host that can reach the instance network (or from another instance)
ping -c 3 <instance-address>
# Lease state inside the instance (console or existing session)
ip -4 addr # valid_lft resets to the lease time after a renewal
cat /var/lib/dhcp/dhclient*.leases 2>/dev/null | tail -5 # dhclient; or networkctl / nmcli; CirrOS uses dhcpcd
# ML2/OVS: dnsmasq still serving on the network node
pgrep -af dnsmasq | grep -F "<network-id>" # the command line is not always "dnsmasq --no-hosts" (Kolla: "dnsmasq -k --no-hosts")
# OVN: controller still up on the compute, ports still bound
ovs-vsctl show | head; ovn-sbctl show 2>/dev/null # sbctl needs the southbound DB, which may be up independently
# (OVN not run on the lab)
# L3 HA: which node is master for the router (not run on the lab: the lab router was not HA)
ip netns exec qrouter-<router-id> ip addr | grep -F "<router-external-address>"
Lab observations¶
Kolla-Ansible 2026.1 all-in-one, ML2/OVS, legacy (non-HA) router. The APIs, conductor, scheduler, Keystone,
Glance, Placement, Heat, Horizon, RabbitMQ, MariaDB and memcached were stopped for about 8 minutes; Neutron agents,
nova-compute, libvirt and Open vSwitch were left running.
- A running instance answered 1 ping per second on its fixed and floating addresses with no loss for the whole stop.
- A forced lease renewal inside the instance was answered by dnsmasq (
DHCPREQUEST/DHCPACK) during the stop. - A metadata request from the running instance got no answer (timeout). An instance rebooted during the stop got its
address by DHCP but failed every metadata request; CirrOS waited roughly 50 s per try, and did not start
sshduntil metadata answered, about a minute after the control plane was started again (8 minutes after its boot). - When RabbitMQ and
neutron-servercame back, the DHCP agent loggedAgent has just been revived. Scheduling full syncand restarted dnsmasq; no lease was lost.
Validity and limits¶
General behaviour of Nova and Neutron with ML2/OVS and OVN as documented upstream and observed by an operator in a real control-plane incident. The lab confirmation (OpenStack 2026.1, Kolla-Ansible all-in-one) covers ML2/OVS with a non-HA (legacy) router only: the OVN behaviour, L3 HA and the router or gateway failover check were not tested. Other backends (Linux Bridge, SR-IOV, vendor SDN) differ. Octavia load balancers, Designate records and other services with their own data-plane components have their own behaviour, not covered here.
Related¶
- Reopening provisioning after a control-plane incident.
- VM creation fails platform-wide.
- Source: OpenStack, the Day After Tomorrow (S. Zaouam), chapter 1, technical note on control plane and data plane.
Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.