Reconfigure Neutron¶
Impact Restarts on drift Critical
Lumo: pause here
This page covers an operation rated Critical in the risk register. Read its risks before you run anything, and try it on a test platform first.
What kolla-ansible -t neutron reconfigure does: it runs the loadbalancer play (neutron-server frontend) and the
neutron
play on neutron-server, every agent group, every compute and manila-share. The page gives the restart set per change
and what is known of the impact on networking.
The scenario sections are written for the assumed deployment; the page uses the definition of drift and the notion of code guarantees and environment dependence.
Note
Key answers
-t neutron reconfigure(§2 to §5) targets every compute, and runs thebootstrap_neutronDB-sync container on every reconfigure CODE (§2). In 2023.1 anml2_conf.inioverride no longer reachesopenvswitch_agent.ini, so it restarts onlyneutron_serverCODE (§3); aneutron.confoverride still restarts every neutron container on every host. What an agent restart does to existing connectivity is NOT AVAILABLE (§4).
Warning
R6: with enable_neutron_agent_ha, the L3 agent wrapper runs
neutron-netns-cleanup before the agent starts, at every neutron_l3_agent restart; what that command deletes is
NOT AVAILABLE — see §4.
1. Commands¶
1.1 Neutron¶
kolla-ansible -i multinode -t neutron reconfigure # server + all agents (+ computes)
kolla-ansible -i multinode -t neutron reconfigure --limit network01 # one network node at a time
kolla-ansible -i multinode -t openvswitch reconfigure --limit compute042 # OVS on one host
-t neutrontargets every compute (the neutron play includescompute).- Network nodes should be done one at a time. With
enable_neutron_agent_ha, the L3 agent wrapper runsneutron-netns-cleanupbefore the agent starts, at everyneutron_l3_agentrestart; what that command deletes is NOT AVAILABLE (R6).
2. Execution and targets¶
kolla-ansible -t neutron reconfigure runs:
- the loadbalancer play (neutron-server frontend;
site.yml:238-242); - the neutron play on
neutron-server, every agent group, every compute andmanila-share(site.yml:590-608).
Per host the steps are: the ML2 plugin check (container facts for neutron_openvswitch_agent/ovn_controller and
the OVN DB volumes; it fails the host if the running agent type does not match neutron_plugin_agent), register
(run_once), config-host (ip6_tables module on L3/linuxbridge/OVS agent hosts; sysctl on L3 hosts), config,
check-containers, bootstrap, flush (roles/neutron/tasks/deploy.yml; neutron_plugin_agent_check.yml:2-35;
config-host.yml:2-42). common no longer runs under -t neutron: it is its own play, tagged common only
(site.yml:86-97). The bootstrap step always runs the Neutron DB and user/grant tasks (mysql_db/mysql_user via
kolla_toolbox, run_once, delegated to neutron-server[0], skipped only with use_preconfigured_databases,
roles/neutron/tasks/bootstrap.yml:2-36) CODE, so MariaDB must be reachable.
Change from Ussuri: the bootstrap_neutron DB-sync container has no when in 2023.1, so it runs on every
reconfigure (and deploy), run_once on neutron-server[0] (roles/neutron/tasks/bootstrap_service.yml:2-21;
reconfigure.yml imports deploy.yml). neutron_enable_rolling_upgrade (roles/neutron/defaults/main.yml:710) is
now read only by upgrade.yml:5, :8. The separate SFC bootstrap container is gone: the container runs
neutron-db-manage --subproject neutron upgrade head, then the same for each enabled subproject in
NEUTRON_BOOTSTRAP_SERVICES (networking-sfc with enable_neutron_sfc, neutron-dynamic-routing, neutron-vpnaas,
vmware-nsx) (roles/neutron/defaults/main.yml:656-664, :705;
kolla docker/neutron/neutron-server/extend_start.sh:5-13).
What a no-op upgrade head does to a running neutron-server is NOT AVAILABLE.
3. What restarts CODE¶
| Change | Restarted | Source |
|---|---|---|
/etc/ (meant for the server) |
neutron_server only (controllers), plus neutron_infoblox_ipam_agent where enabled. Change from Ussuri: openvswitch_agent.ini and linuxbridge_agent.ini no longer merge ml2_conf.ini, so the agents on computes and network nodes do not restart |
roles/; roles/; Tags §2.4 |
/etc/ |
neutron_openvswitch_agent on all computes and network nodes | roles/ |
/etc/ (also global.conf, neutron/) |
every neutron container on every host | roles/ |
| RabbitMQ/memcached membership change | every neutron container (transport URL) | group_vars/; roles/; Change control-plane membership §2 |
4. Impact on networking¶
| Component | On restart (code-guaranteed when drift) | Effect on existing connectivity (environment-dependent) |
|---|---|---|
| neutron-server | API unavailable during the stop/start, on all controllers at once | NOT AVAILABLE Whether existing traffic is unaffected, and whether port bindings and new instances fail during the window |
| neutron-openvswitch-agent | container stop/start | NOT AVAILABLE Whether flows in OVS persist while the agent is down, whether and how the agent resyncs and re-installs flows on start with the 2023.1 defaults, and whether a VM loses packets during the agent restart |
| neutron-l3-agent | stop/start; with enable_neutron_agent_ha: yes the wrapper runs neutron-netns-cleanup --force --agent-type l3 before the agent starts (the wrapper's comment: "Remove all l3 related netns"; neutron-l3-agent-wrapper.sh.j2:5-11) |
CODE Without the HA cleanup nothing in the wrapper deletes namespaces, and /run/netns is shared with the host (roles/). NOT AVAILABLE Whether the namespaces then persist, whether keepalived/haproxy child processes die with the container, whether with the cleanup all routers on that node lose N-S/FIP connectivity until rebuilt, and whether HA routers fail over. NOT AVAILABLE What the cleanup command deletes: it is neutron code, which is not in these sources (F12) |
| neutron-dhcp-agent | stop/start | NOT AVAILABLE Whether dnsmasq processes die with the container, whether VMs renewing leases in the window fail to renew, and whether new ports wait for the agent to come back |
| neutron-metadata-agent | stop/start | NOT AVAILABLE Whether metadata requests (cloud-init) fail during the window |
| Open vSwitch | not touched by -t neutron (separate play/tag) |
– |
OVN (if neutron_plugin_agent: ovn) |
-t neutron touches neutron-server and neutron-ovn-metadata-agent. In 2023.1 the ovn role is split: -t ovn runs both the ovn-controller and the ovn-db plays (site.yml:575-576, :587-588). The ovn-controller role re-applies br-int and 9 external_ids keys on every ovn-controller host, including ovn-cms-options, through the openvswitch_bridge/openvswitch_db modules instead of the Ussuri ovs-vsctl commands (roles/); whether those report changed on an unchanged host is NOT AVAILABLE (the modules are not in these sources) (fact sheet A §5.8) |
ovn-controller restart is not triggered by -t neutron |
5. Risks and verification¶
- Risks: R5, R6, R7.
- Before:
openstack network agent list(all alive);- identify the network nodes that host HA router masters;
- render with
genconfig -t neutronand diffopenvswitch_agent.inion a compute. If it changed, every compute will restart its agent; - expect the
bootstrap_neutronDB-sync container on neutron-server[0] even with no change (§2).
- Apply: network nodes one by one (
--limit network01, thennetwork02). Computes in batches. - After:
- agent list;
ip netnscount on network nodes matches the number of routers/networks;- ping a VM's floating IP continuously during each batch.
Related¶
- Risk register — the risks this page names: R5
(
ml2_conf.inioverride meant for neutron-server; in 2023.1 it no longer reaches the agents, §3), R6 (restart ofneutron_l3_agentwithenable_neutron_agent_ha: yes) and R7 (openvswitch_vswitchdrestart or recreate) - Neutron — service card and where to look
- Open vSwitch — service card and where to look; Open vSwitch has a separate play and
tag and is not touched by
-t neutron - Pre-execution checklist — the checks before every production run
- Write config first, apply in controlled batches — how to apply in batches (
genconfig, thendeploy-containers --limit;ANSIBLE_SERIAL); its examples use the nova tags - Kolla images §5.3 — neutron-l3-agent: the wrapper
that runs the namespace cleanup with
enable_neutron_agent_ha - What is not known — what the sources do not establish about the network data plane