Potentially dangerous operations on a running Kolla-Ansible Ussuri platform: their impact and the precautions to take
before running them. Ratings are relative to a production cloud with running VMs.
Note
Key answers
25 risks (R1–R25), each with what actually happens, its blast radius, VM and API impact, likelihood, severity,
precautions and evidence (Matrix).
Rated Critical:R7 (openvswitch_vswitchd restart or recreate), R12 (changing the image tag and
running deploy instead of upgrade), R14 (bootstrap-servers re-run on a live host), R15
(kolla-ansible stop) and R16 (kolla-ansible destroy).
Computes and VMs:R1. Any config change touching nova-compute, followed by a deploy/reconfigure with
a nova tag or no tag, restarts nova_compute, nova_libvirt and nova_ssh on every affected compute, all at once
with the default serial 0; whether live migrations in flight fail and whether running VMs keep running are
NOT AVAILABLE.
Network data plane:R5 (an ml2_conf.ini override restarts every OVS agent), R6 (with
enable_neutron_agent_ha: yes, the L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 before
the agent starts) and R7; whether VM traffic is interrupted is NOT AVAILABLE.
Likelihood = how likely the operation is to cause the impact when the triggering condition is present.
Severity = worst realistic outcome.
Evidence cites the source lines, or the study section that cites them. Nothing was run. NOT AVAILABLE marks
outcomes that the sources do not establish.
Any config change touching nova-compute (override, global flag, password, RabbitMQ list) followed by deploy/reconfigure with a nova tag or no tag
What actually happens
nova_computeand nova_libvirt and nova_ssh restart on every affected compute, all at once (serial 0). The cause is the loop-notify behaviour
Blast radius
all computes
VM impact
libvirtd restart; NOT AVAILABLE whether live migrations in flight fail and whether running VMs keep running
API impact
nova-compute restarts on each host; NOT AVAILABLE how long it is down
Likelihood
High
Severity
High
Precautions
genconfig + diff, then deploy-containers --limit in batches, or ANSIBLE_SERIAL; drain live migrations first; validate libvirt restart behaviour in staging
Evidence
roles/nova-cell/tasks/check-containers.yml:2-20, roles/nova-cell/defaults/main.yml:3-54, nova.yml:132; ansible 2.9.27 lib/ansible/executor/playbook_executor.py:276-278 (serial 0 = one batch); Tags §2.4; Nova deep dive §5
R2 — Enabling or disabling a service with a logrotate or fluentd entry¶
Operation / trigger
Enabling or disabling a service that has a logrotate or fluentd input entry (enable_X for most OpenStack services, e.g. barbican) in globals.yml, then running any play that includes common
What actually happens
td-agent.conf and logrotate.conf render differently on every host → fluentd, cronand kolla_toolbox restart (loop-notify) on every host where a play that runs common executes: the whole fleet for an untagged run, only the tagged service's hosts for a tagged run (-t nova with the multinode inventory: the control and compute hosts, which the nova.yml plays target; the others keep the old files until a later run includes common there). Nova/Cinder/Glance configs gain sections (e.g. [barbican]) → those services restart too. Flags in neither list (e.g. enable_prometheus, enable_ovn, enable_openvswitch, enable_memcached, enable_placement, enable_redis) leave both files unchanged
Blast radius
fleet-wide (untagged run); the service's hosts (tagged run)
the services whose configs changed restart (stop up to 10 s, then start; R24); NOT AVAILABLE how long their APIs are unavailable, and whether the fluentd, cron and kolla_toolbox restarts affect APIs (Apply common changes (fluentd, cron, kolla_toolbox) §3)
Likelihood
High
Severity
Medium–High
Precautions
Treat enable_* changes as fleet-wide changes. Roll out with --limit batches
R3 — Changing RabbitMQ membership, password or port¶
Operation / trigger
Changing RabbitMQ group membership (adding/removing a controller), RabbitMQ password or port
What actually happens
rpc_transport_url changes in every messaging service on every host → all nova, neutron (and any compute-side agent) containers restart; update_cell rewrites the cell's MQ URL
Blast radius
all hosts
VM impact
as R1 plus neutron agent restart; NOT AVAILABLE its effect on flows
API impact
yes
Likelihood
Certain
Severity
High
Precautions
Plan as a maintenance; batch with --limit; verify RabbitMQ cluster health first
38 roles embed the memcached host list in their configs (37 template loops plus monasca's monasca_memcached_servers; fact sheet B §3.10 counts 36, corrected in Per-role fact sheets), so their services restart when their plays run; NOT AVAILABLE whether cached tokens are lost. Not only the control plane: neutron.conf is shared by every neutron agent, so neutron_openvswitch_agent restarts on every compute and network node, and the dhcp, l3 and metadata agents, which get the same file, restart on network nodes; cinder.conf reaches cinder-volume/backup and ceilometer.conf ceilometer-compute. Compute nova.conf has no memcached list
Blast radius
controllers + network nodes + all computes (+ storage)
VM impact
as R5: OVS agent restart; NOT AVAILABLE its effect on flows, and whether router/DHCP traffic is interrupted on network nodes; with enable_neutron_agent_ha, the R6 netns cleanup
API impact
memcached server list changes; NOT AVAILABLE effect on auth latency and Horizon sessions
Likelihood
Certain
Severity
High
Precautions
Plan as a dataplane change, like R3/R5: batch with --limit or ANSIBLE_SERIAL, network nodes one at a time; low-traffic window
Evidence
fact sheet B §3.9 item 2, §3.10; roles/neutron/templates/neutron.conf.j2:119, roles/neutron/tasks/config.yml:73-98
R5 — ml2_conf.ini override meant for neutron-server¶
Operation / trigger
/etc/kolla/config/neutron/ml2_conf.ini override (meant for neutron-server)
What actually happens
also merged into openvswitch_agent.ini → every OVS agent restarts (computes + network). On network nodes, loop-notify also restarts dhcp, l3 and metadata agents
Blast radius
network + all computes
VM impact
OVS agent restart; NOT AVAILABLE its effect on flows, and whether router/DHCP traffic is interrupted on network nodes
API impact
neutron-server restart
Likelihood
Certain
Severity
High
Precautions
Put server-only options in neutron/neutron-server.conf, or check the rendered openvswitch_agent.ini diff
R6 — neutron_l3_agent restart with enable_neutron_agent_ha: yes¶
Operation / trigger
Any restart of neutron_l3_agent with enable_neutron_agent_ha: yes
What actually happens
the wrapper runs neutron-netns-cleanup --force --agent-type l3before the agent starts (the wrapper's comment: "Remove all l3 related netns"), then neutron-l3-agent starts CODE. NOT AVAILABLE What the command deletes: it is neutron code, which is not in these sources (F12)
Blast radius
network nodes (and DVR computes)
VM impact
NOT AVAILABLE whether north-south and floating-IP traffic of the routers on that node stops, whether and when neutron-l3-agent rebuilds the namespaces, and whether HA routers fail over
API impact
none
Likelihood
Certain (if HA flag)
Severity
High
Precautions
Restart network nodes one at a time; fail routers over first
openvswitch_vswitchd restart or recreate (image change, start-ovs change, -t openvswitch with a changed image; also any openvswitch_db-only drift, e.g. a start-ovsdb-server change or a db image, volume or dimension change: check-containers loops over both OVS services, so loop-notify (Tags §2.4) also notifies the vswitchd handler, which runs after the db restart and wait)
What actually happens
ovs-vswitchd (the userspace daemon) restarts. The image only runs modprobe openvswitch (the kernel module is loaded, never unloaded or reloaded), and start-ovs launches ovs-vswitchd with no flow save/restore. Bridges and ports persist in conf.db on the openvswitch_db volume. NOT AVAILABLE Whether the OpenFlow tables held by the old daemon are lost, and whether neutron-openvswitch-agent (or ovn-controller with OVN) reprograms them
Blast radius
per host (all OVS hosts at once with serial 0)
VM impact
NOT AVAILABLE whether all VM network traffic on the host is interrupted until flows are restored, and for how long
API impact
none
Likelihood
Certain on image change
Severity
Critical
Precautions
Upgrade or recreate OVS host by host with evacuation or acceptance; never fleet-wide with serial 0
rolling restart in 4 waves by index % 4, one wave after another; each wave waits up to 10 × 6 s for WSREP Synced. A node that fails its wait marks only its own host failed, so the next wave still restarts its nodes. HAProxy's single active backend is node 0. NOT AVAILABLE Whether node 0's wave runs first
Blast radius
DB cluster
VM impact
none directly
API impact
DB connection resets
Likelihood
Certain if changed
Severity
High
Precautions
Check wsrep_cluster_size and Synced on all nodes before; take a backup (mariadb_backup); watch between waves
Evidence
fact sheet B §1.2, §1.5, §1.8 items 1 and 3; roles/mariadb/handlers/main.yml:66-78, roles/mariadb/tasks/restart_services.yml:28-39; ansible 2.9.27 lib/ansible/playbook/included_file.py:51-56,92-95 (one included file per loop item), lib/ansible/plugins/strategy/__init__.py:1045-1063 (included files run one after another); :548-561 (a failure marks only that host failed), :1019-1020 (each notified host is checked on its own); no MariaDB play sets any_errors_fatal (only nova.yml:75,265)
R12 — Image tag change run with deploy instead of upgrade¶
Operation / trigger
Changing openstack_tag / *_tag and running deploy instead of upgrade
What actually happens
every container is recreated (fleet-wide); upgrade-specific steps (RPC pin removal, nova-status check, online migrations except Placement's, which run inside bootstrap_placement, and, with the non-default ironic_enable_rolling_upgrade: no, Ironic's inside bootstrap_ironic) do not run. Neutron: instead of the rolling expand → restart → contract (default neutron_enable_rolling_upgrade: yes), bootstrap_neutron runs neutron-db-manage … upgrade head for neutron, neutron-dynamic-routing, neutron-fwaas and neutron-vpnaas with the new image before the old neutron_server containers restart; NOT AVAILABLE whether upgrade head applies the expand and contract branches together, which would run contract migrations under live old-version servers
Blast radius
fleet-wide
VM impact
as R1 + R7 + R8 together
API impact
yes
Likelihood
Certain
Severity
Critical
Precautions
Use pull then upgrade (Ussuri docs); per-service tags; ANSIBLE_SERIAL
can restart the Docker daemon (if daemon.json changes) → every container on the host restarts, on all hosts at once by default; also disables firewalld/ufw, sets SELinux permissive, removes host packages (open-iscsi, libvirt*, chrony, …)
Blast radius
per host / fleet
VM impact
Docker restart = nova_libvirt + OVS restart → see R1/R7
API impact
yes
Likelihood
Medium
Severity
Critical
Precautions
Run only with --limit on new hosts, or -e kolla_serial=1; diff daemon.json first
toggles global MariaDB log_bin_trust_function_creators 1→0 around db sync; with the file backend only glance-api[0] exists → any restart is a full Image API outage
fact gathering for all hosts is executed sequentially by the limited hosts (delegated loop) → long runtime; /etc/hosts and other cross-host files are not updated on non-limited hosts
Blast radius
–
VM impact
none
API impact
none
Likelihood
Certain
Severity
Low (time), Medium (drift)
Precautions
Enable fact caching (ansible-tuning.rst); follow up with a full run when cross-host config changed
Evidence
gather-facts.yml:29-50 (each limited host loops over its share of groups['all']: one delegated setup per host without facts); ansible 2.9.27 lib/ansible/executor/task_executor.py:317-367 (loop items run one after another)
Effective container stop timeout is 10 s, not docker_graceful_timeout
What actually happens
on every restart, a service still running 10 s after SIGTERM is SIGKILLed (heat-engine, cinder-volume copies, glance uploads); for nova_libvirt this matters only if libvirtd has not exited by then (Nova deep dive §5). NOT AVAILABLE Whether services started by a shell wrapper that does not exec them (e.g. neutron-l3-agent, haproxy, ovs-vswitchd, ovsdb-server) receive SIGTERM at all (Kolla images §2.6)
Blast radius
all restarts
VM impact
in-flight operations killed
API impact
–
Likelihood
Certain
Severity
Medium
Precautions
Drain long operations before restarts; patch kolla_docker or pass graceful_timeout if needed
Lumo: if you keep only 5 things from this page, keep these.
Rated Critical: openvswitch_vswitchd restart or recreate (R7), image tag change run with deploy instead of upgrade (R12), bootstrap-servers re-run (R14), stop (R15), destroy (R16).
A config change touching nova-compute, then deploy/reconfigure, restarts nova_compute, nova_libvirt and nova_ssh on every affected compute, all at once with the default serial 0.
Whether running VMs keep running, and whether live migrations in flight fail, during that restart is NOT AVAILABLE
Before a nova-compute change: genconfig + diff, then deploy-containers --limit in batches; drain live migrations first; validate libvirt restart behaviour in staging.
Changing the image tag and running deploy instead of upgrade recreates every container fleet-wide and skips the upgrade-specific steps; use pull then upgrade (Ussuri docs).
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one
you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page.
Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first
choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.