Reconfigure Neutron¶
Impact Restarts on drift Critical
Lumo: pause here
This page covers an operation rated Critical in the risk register. Read its risks before you run anything, and try it on a test platform first.
What kolla-ansible -t neutron reconfigure does in 2024.1: it runs the loadbalancer play (neutron-server frontend) and
the neutron play on neutron-server, every agent group, every compute and manila-share. The page gives the restart
set per change and what is known of the impact on networking.
The scenario sections are written for the assumed deployment; the page uses the definition of drift and the notion of code guarantees and environment dependence.
Note
Key answers
-t neutron reconfigure(§2 to §5) targets every compute, and still runs thebootstrap_neutronDB-sync container on every reconfigure CODE (§2). Anml2_conf.inioverride still restarts onlyneutron_server(and the other neutron containers of its host) CODE (§3); aneutron.confoverride still restarts every neutron container on every host.- New in 2024.1 CODE (§3): FWaaS (
fwaas_driver.ini) and TaaS (neutron_taas.conf, which reaches the Open vSwitch agent on every compute);neutron.confnow always carriesdns_domain; the L3-host sysctl values go through the newsysctlrole (§2). The handlers are unchanged apart from the module name. What an agent restart does to existing connectivity is NOT AVAILABLE (§4).
Warning
R6: with enable_neutron_agent_ha, the L3 agent wrapper runs
neutron-netns-cleanup before the agent starts, at every neutron_l3_agent restart; what that command deletes is
NOT AVAILABLE — see §4.
1. Commands¶
1.1 Neutron¶
kolla-ansible -i multinode -t neutron reconfigure # server + all agents (+ computes)
kolla-ansible -i multinode -t neutron reconfigure --limit network01 # one network node at a time
kolla-ansible -i multinode -t openvswitch reconfigure --limit compute042 # OVS on one host
-t neutrontargets every compute (the neutron play includescompute,site.yml:671-693).- Network nodes should be done one at a time. With
enable_neutron_agent_ha, the L3 agent wrapper runsneutron-netns-cleanupbefore the agent starts, at everyneutron_l3_agentrestart; what that command deletes is NOT AVAILABLE (R6).
2. Execution and targets¶
kolla-ansible -t neutron reconfigure runs:
- the loadbalancer play (neutron-server frontend;
site.yml:105-116, neutron include:243-247). Changed in 2024.1: the external frontend now namesneutron_external_fqdnandneutron_server_public_port, which follows the single external frontend when it is enabled (roles/neutron/defaults/main.yml:19-25,group_vars/all.yml:502-505); - the neutron play on
neutron-server, every agent group, every compute andmanila-share, withserial: kolla_serial(default"0") and, new in 2024.1,max_fail_percentage(default100) (site.yml:671-693).
Per host the steps are (roles/neutron/tasks/deploy.yml:2-23; reconfigure.yml:2 imports that file):
- the ML2 plugin check: container facts for
neutron_openvswitch_agent/ovn_controllerand the OVN DB volumes; it fails the host if the running agent type does not matchneutron_plugin_agent(neutron_plugin_agent_check.yml:2-35); - register (run_once). New in 2024.1: it also grants the neutron user the
servicerole in theserviceproject (register.yml:2-8,roles/neutron/defaults/main.yml:923-926,roles/service-ks-register/tasks/main.yml:103-117); - config-host: the
ip6_tablesmodule on L3, linuxbridge and Open vSwitch agent hosts (config-host.yml:2-13); an IPv6 check on every host; thegc_threshsysctl values on L3 agent hosts whenset_sysctl(defaultyes,group_vars/all.yml:703). Changed in 2024.1: the sysctl values are set by the newsysctlrole, included with the list assettings(config-host.yml:20-34). The role runs its ownCheck IPv6 support, skips a valueKOLLA_SKIP, removes a key whose value isKOLLA_UNSET, and skips thenet.ipv6.*keys whennet.ipv6.conf.all.disable_ipv6is1, writing tokolla_sysctl_conf_path(/etc/sysctl.conf) (roles/sysctl/tasks/main.yml:1-20,group_vars/all.yml:706). The neutron role's own IPv6 check (config-host.yml:15-18) still runs, but no task of the role reads its result any more CODE; - config, check-containers, the fake-driver config when
enable_nova_fake, bootstrap, then a flush of the handlers (deploy.yml:11-23).
common does not run under -t neutron: it is its own play, tagged common only (site.yml:88-104). The bootstrap
step always runs the Neutron DB and user/grant tasks (mysql_db/mysql_user via kolla_toolbox, run_once, delegated to
neutron-server[0], skipped only with use_preconfigured_databases, roles/neutron/tasks/bootstrap.yml:2-36) CODE,
so MariaDB must be reachable.
Still holds in 2024.1: the bootstrap_neutron DB-sync container has no when, so it runs on every reconfigure
(and deploy), run_once on neutron-server[0] (roles/neutron/tasks/bootstrap.yml:38, bootstrap_service.yml:2-21).
neutron_enable_rolling_upgrade (roles/neutron/defaults/main.yml:732) is read only by upgrade.yml:5, :8. The
container runs neutron-db-manage --subproject neutron upgrade head, then the same for each enabled subproject in
NEUTRON_BOOTSTRAP_SERVICES (kolla docker/neutron/neutron-server/extend_start.sh:5-13). Changed in 2024.1: the
subproject list gains neutron-fwaas (enable_neutron_fwaas) and tap-as-a-service (enable_neutron_taas), next to
networking-sfc, neutron-dynamic-routing, neutron-vpnaas and vmware-nsx (roles/neutron/defaults/main.yml:674-686,
:727); both flags default to no (group_vars/all.yml:814, :822). What a no-op upgrade head does to a running
neutron-server is NOT AVAILABLE.
3. What restarts CODE¶
| Change | Restarted | Source |
|---|---|---|
/etc/ (meant for the server) |
neutron_server only (controllers), plus neutron_infoblox_ipam_agent where enabled; through loop-notify in the check step, also the other neutron containers of a host that runs neutron-server. Still holds in 2024.1: openvswitch_agent.ini and linuxbridge_agent.ini do not merge ml2_conf.ini, so the agents on computes and network nodes do not restart |
roles/; roles/; Tags §2.4 |
/etc/ |
neutron_openvswitch_agent on all computes and network nodes | roles/ |
/etc/ (also global.conf, neutron/) |
every neutron container on every host (all but neutron_tls_proxy render neutron.conf) |
roles/ |
/etc/, e.g. neutron/ |
that container only, on every host that runs it | roles/ |
Files in /etc/ |
neutron_server, and neutron_openvswitch_agent or neutron_linuxbridge_agent on every host that runs one, computes included | roles/ |
New in 2024.1: /etc/, with enable_neutron_fwaas |
neutron_server and neutron_l3_agent (no per-host override file) | roles/ |
New in 2024.1: /etc/ (or neutron/), with enable_neutron_taas |
neutron_server and neutron_openvswitch_agent, so every compute | roles/ |
New in 2024.1: turning enable_neutron_fwaas or enable_neutron_taas on or off |
every container that renders neutron.conf (its service_plugins gains firewall_v2 or taas); TaaS also adds taas to the Open vSwitch agent extensions, FWaaS adds fwaas_v2 to the L3 agent extensions; the next bootstrap_neutron migrates the subproject (§2) |
roles/; roles/, openvswitch_agent.ini.j2:9-10, l3_agent.ini.j2:17-18 |
Changed in 2024.1: /etc/ |
neutron_server and neutron_l3_agent, now only with enable_neutron_vpnaas; without it the file is no longer rendered or passed on the command line, and no task removes a copy written earlier |
roles/; roles/, neutron-l3-agent-wrapper.sh.j2:16-20 |
| RabbitMQ/memcached membership change | every neutron container that renders neutron.conf (transport URLs, memcached list) |
group_vars/; roles/; Change control-plane membership §2 |
First run of the 2024.1 code on configs rendered by 2023.1 CODE: neutron.conf changes for every container
that renders it, because dns_domain = {{ neutron_dns_domain }} (default openstacklocal) is now written
unconditionally (2023.1 wrote it only with Designate) and because RabbitMQ quorum queues are on by default
(F17) (roles/neutron/templates/neutron.conf.j2:64, :146-155,
roles/neutron/defaults/main.yml:958, group_vars/all.yml:268). Without VPNaaS, the neutron-server command and the L3
agent wrapper also change (§3 row neutron_vpnaas.conf). And the systemd unit of a container
written by 2023.1 is reported as drift at its first 2024.1 check (F16). Every neutron
container on every host of the play can therefore restart on that run.
Handlers, still hold in 2024.1: one recreate_or_restart_container handler per container, skipped when
kolla_action is config; only the module name changed (kolla_container) (roles/neutron/handlers/main.yml:1-341).
The L3 agent handler is a chain: container facts, a group_by on whether neutron_l3_agent runs, start of the stopped
ones, a wait of neutron_l3_agent_failover_delay when both kinds exist, then the running ones
(roles/neutron/handlers/main.yml:90-157). For the running ones, each host loops over the list of hosts with a running
L3 agent and restarts only on its own item, with a pause of neutron_l3_agent_failover_delay seconds (default 0,
roles/neutron/defaults/main.yml:651) before every item but the first (handlers/main.yml:136-157; ansible-core
lib/ansible/executor/task_executor.py:318-322). With the default 0 nothing spaces the restarts.
4. Impact on networking¶
| Component | On restart (code-guaranteed when drift) | Effect on existing connectivity (environment-dependent) |
|---|---|---|
| neutron-server | API unavailable during the stop/start, on all controllers at once (no serial unless ANSIBLE_SERIAL is set, site.yml:686) |
NOT AVAILABLE Whether existing traffic is unaffected, and whether port bindings and new instances fail during the window |
| neutron-openvswitch-agent | container stop/start | NOT AVAILABLE Whether flows in OVS persist while the agent is down, whether and how the agent resyncs and re-installs flows on start with the 2024.1 defaults, and whether a VM loses packets during the agent restart |
| neutron-l3-agent | stop/start; with enable_neutron_agent_ha: yes the wrapper runs neutron-netns-cleanup --force --agent-type l3 before the agent starts (the wrapper's comment: "Remove all l3 related netns"), with fwaas_driver.ini added when FWaaS is enabled (neutron-l3-agent-wrapper.sh.j2:5-14) |
CODE Without the HA cleanup nothing in the wrapper deletes namespaces, and /run/netns is shared with the host (roles/). NOT AVAILABLE Whether the namespaces then persist, whether keepalived/haproxy child processes die with the container, whether with the cleanup all routers on that node lose N-S/FIP connectivity until rebuilt, and whether HA routers fail over. NOT AVAILABLE What the cleanup command deletes: it is neutron code, which is not in these sources (F12) |
| neutron-dhcp-agent | stop/start | NOT AVAILABLE Whether dnsmasq processes die with the container, whether VMs renewing leases in the window fail to renew, and whether new ports wait for the agent to come back |
| neutron-metadata-agent | stop/start | NOT AVAILABLE Whether metadata requests (cloud-init) fail during the window |
| Open vSwitch | not touched by -t neutron (separate play/tag, site.yml:611-624) |
– |
OVN (if neutron_plugin_agent: ovn) |
-t neutron touches neutron-server and neutron-ovn-metadata-agent (plus neutron_ovn_agent with neutron_enable_ovn_agent, default no, group_vars/). -t ovn runs both the ovn-controller and the ovn-db plays (site.yml:641-653, :655-669). The ovn-controller role, unchanged in 2024.1, re-applies br-int and 9 external_ids keys on every ovn-controller host, including ovn-cms-options, through the openvswitch_bridge/openvswitch_db modules (roles/); whether those report changed on an unchanged host is NOT AVAILABLE (the modules are not in these sources) (OVN) |
ovn-controller restart is not triggered by -t neutron |
5. Risks and verification¶
- Risks: R5, R6, R7.
- Before:
openstack network agent list(all alive);- identify the network nodes that host HA router masters;
- render with
genconfig -t neutronand diffopenvswitch_agent.iniandneutron.confon a compute (andneutron_taas.confwith TaaS). If one changed, every compute will restart its agent; - on the first run of the 2024.1 code, expect
neutron.confto change everywhere (§3); - expect the
bootstrap_neutronDB-sync container on neutron-server[0] even with no change (§2).
- Apply: network nodes one by one (
--limit network01, thennetwork02). Computes in batches. - After:
- agent list;
ip netnscount on network nodes matches the number of routers/networks;- ping a VM's floating IP continuously during each batch.
Related¶
- Risk register — the risks this page names: R5
(
ml2_conf.inioverride meant for neutron-server; in 2024.1 it still does not reach the agents, §3), R6 (restart ofneutron_l3_agentwithenable_neutron_agent_ha: yes) and R7 (openvswitch_vswitchdrestart or recreate) - Neutron — service card and where to look
- Open vSwitch — service card and where to look; Open vSwitch has a separate play and
tag and is not touched by
-t neutron - Pre-execution checklist — the checks before every production run
- Write config first, apply in controlled batches — how to apply in batches (
genconfig, thendeploy-containers --limit;ANSIBLE_SERIAL); its examples use the nova tags - Kolla images §5.3 — neutron-l3-agent: the wrapper
that runs the namespace cleanup with
enable_neutron_agent_ha - What is not known — what the sources do not establish about the network data plane