Technical findings¶
The study consolidates three things. This page holds the first:
- what was established and how (evidence level): the headline findings below;
- implementation limitations and bugs found in
2023.1-eol: Known bugs and limitations; - what the sources do not establish: Open questions and What is not known.
Note
Key answers
- Fifteen headline findings (F1–F15), first established on
ussuri-eoland re-checked against2023.1-eol, each with its evidence and why it matters (Headline findings). - Still hold in 2023.1: loop-notify amplification (F1), same-tag image rebuilds (F4),
reconfigure=deploy(F5, now including Neutron's DB sync), F6 to F10, F12, F13, and per-image bootstrap behaviour (F15). - No longer hold in 2023.1: the
-t commonno-op (F2:commonis its own tagged play), the ignored stop timeout (F3: the 60 sdocker_graceful_timeoutreachesdocker stopthrough a systemd unit) and theml2_conf.inimerge intoopenvswitch_agent.ini(F11). F14 changed in part.
Evidence levels: CODE and NOT AVAILABLE, defined once, in Evidence labels. The study rests only on reading source code; no run against a host is used as evidence.
Headline findings¶
| # | Finding |
|---|---|
| F1 | Loop-notify amplification |
| F2 | -t common is a no-op (no longer holds in 2023.1) |
| F3 | docker_graceful_timeout and docker_client_timeout are ignored (no longer holds in 2023.1) |
| F4 | Same-tag image rebuilds are not picked up by deploy/reconfigure/upgrade |
| F5 | reconfigure = deploy |
| F6 | Global inputs are rendered into per-host files |
| F7 | --list-tasks cannot show what a tag runs |
| F8 | --limit does not confine actions |
| F9 | With --limit, all hosts' facts are gathered sequentially |
| F10 | kolla_serial makes run_once steps repeat per batch |
| F11 | The openvswitch_agent.ini merge includes the operator's ml2_conf.ini overrides (no longer holds in 2023.1) |
| F12 | The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes |
| F13 | The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes |
| F14 | kolla_set_configs --check has blind spots (changed in part in 2023.1) |
| F15 | What the bootstrap containers run differs per image |
F1 — Loop-notify amplification¶
Still holds in 2023.1 CODE.
| Finding | Loop-notify amplification. When a config/check task loops over a role's services and one item changes, ansible-core 2.14 notifies the handlers of all executed items. In Kolla this means a change for one container restarts every container of that role on that host. A nova-compute-only change restarts nova_libvirt and nova_ssh; a scheduler-only change restarts nova_api; a change on a network node restarts all neutron agents there |
| Evidence | CODE ansible-core 2.14.18 lib/, lib/; roles/, roles/, roles/; walk-through in Tags §2.4 |
| Why it matters | Restart blast radius is per role-on-host, not per container. libvirt restarts far more often than operators expect |
F2 — -t common is a no-op¶
No longer holds in 2023.1 CODE. common is a play of its own, with play-level tags: common
(site.yml:86-97), and no role has a meta/ directory, so nothing else reaches the role. The included tasks inherit
the play tag, so -t common renders the common configs and restarts what changed.
| Finding | Ussuri: -t common was a no-op, because tags on the dynamic include_tasks do not propagate to the included tasks. 2023.1: roles/ is a bare include_tasks: "{{ kolla_action }}.yml", reached only from the Apply role common play; -t common runs it on the cron, fluentd, kolla-logs and kolla-toolbox hosts, and a service tag (e.g. -t nova) skips it |
| Evidence | CODE site.yml:86-97; roles/; ansible-core 2.14.18 lib/, lib/, lib/ (Tags §2.1; R25) |
| Why it matters | common changes (fluentd/cron/toolbox) are applied by -t common or by an untagged run, and never by a service-tagged run: a fluentd/logrotate change made when enabling a service with -t <svc> deploy waits for the next run of the common play (R2) |
F3 — docker_graceful_timeout and docker_client_timeout are ignored¶
No longer holds in 2023.1 CODE. The two options have no argument default any more, so the values from
docker_common_options are used; containers (except one-shots) run under a systemd unit whose ExecStop carries the
graceful timeout (R24).
| Finding | Ussuri: the kolla_docker argument defaults (10 s, 120 s) overrode common_options, so every stop sent SIGKILL 10 s after SIGTERM. 2023.1: graceful_timeout and client_timeout are declared without a default, so the common_options values (docker_graceful_timeout: 60, docker_client_timeout: 120) are taken; the 10 s and 120 s values apply only when neither is given. A restart or stop goes through the unit kolla-<name>-container.service, whose ExecStop is docker stop <name> -t <graceful_timeout>; without a unit file the module calls Docker stop with the same timeout |
| Evidence | CODE library/; group_vars/; module_utils/; module_utils/ |
| Why it matters | A service still running 60 s after SIGTERM is killed with its in-flight work (heat-engine, cinder copies, glance uploads); raising docker_graceful_timeout now has an effect (it reaches the unit on the next restart, R24). For nova_libvirt this matters only if libvirtd has not exited by then (Nova deep dive §5). NOT AVAILABLE systemd's own stop timeout for these units (the unit sets none). NOT AVAILABLE Whether services started by a shell wrapper that does not exec them (e.g. neutron-l3-agent, haproxy) receive SIGTERM at all (Kolla images §2.6) |
F4 — Same-tag image rebuilds are not picked up by deploy/reconfigure/upgrade¶
Still holds in 2023.1 CODE.
| Finding | Same-tag image rebuilds are not picked up by deploy/reconfigure/upgrade. Only absent images are pulled; an explicit pull is required |
| Evidence | CODE module_utils/ (the container's image id is compared with the local image of that tag), :362-388 (check_image, then pull_image only when the image is absent) (R13) |
| Why it matters | Mutable tags create silent drift between hosts. A later unrelated pull turns into a mass recreate |
F5 — reconfigure = deploy¶
Holds in 2023.1, without the Neutron exception CODE.
| Finding | reconfigure = deploy for 57 of the 59 roles that have a tasks/ (56 via import_tasks: deploy.yml, plus skyline via include_tasks: deploy.yml; only swift and bifrost have their own logic). A reconfigure re-runs Keystone registration, DB creation/grants and the DB-sync bootstrap containers of the services. The Ussuri Neutron exception is gone: bootstrap_neutron is imported without any condition and runs on every deploy and reconfigure; neutron_enable_rolling_upgrade is read only by upgrade.yml |
| Evidence | CODE roles/; roles/, roles/, roles/, roles/ (Repository architecture §4.2). One-shot containers counted from the code: -t nova starts 6 on control01 when cell0 is mapped and the cell is unchanged (API DB sync, map_cell0, list_cells ×3 — the cell0 lookup new in 2023.1, create_cells and discover_computes —, cell DB sync: roles/, roles/, roles/, create_cells.yml:2-4, discover_computes.yml:4-5,, get_cell_settings.yml:2-21); on -t neutron reconfigure the neutron role starts one, bootstrap_neutron, on the first neutron-server host (roles/). NOT AVAILABLE Totals for a run without a tag, which depend on the enabled services |
| Why it matters | A "reconfigure" applies schema migrations if the image tag moved, for Neutron too (neutron-db-manage … upgrade head for neutron and each enabled subproject, kolla docker/; the separate Ussuri bootstrap_neutron_sfc container no longer exists). There is never a "config only" path except genconfig + deploy-containers (tools/) |
F6 — Global inputs are rendered into per-host files¶
Still holds in 2023.1 CODE.
| Finding | Global inputs are rendered into per-host files. enable_* flags, RabbitMQ/memcached membership, service passwords and the external FQDN end up in compute-side files |
| Evidence | CODE group_vars/ (rpc_transport_url, notify_transport_url: every RabbitMQ host, user and password); nova-cell/; neutron/ (memcached list, shared by the OVS agent on computes); common/ (td-agent.conf and logrotate.conf from the global enable_* flags) |
| Why it matters | Small central changes cause fleet-wide restarts |
F7 — --list-tasks cannot show what a tag runs¶
Still holds in 2023.1 CODE.
| Finding | --list-tasks cannot show what a tag runs, and --list-hosts shows 0 hosts for &enable_* plays |
| Evidence | CODE ansible-core 2.14.18 lib/ (the listing walks the statically compiled play and resolves host patterns before any task runs), lib/; site.yml:16-20 (the enable_* groups are created by group_by at run time); list-tasks-nova.txt |
| Why it matters | Operators need genconfig or staging to predict a run |
F8 — --limit does not confine actions¶
Still holds in 2023.1 CODE.
| Finding | --limit does not confine actions. delegate_to: groups[x][0] runs work on hosts outside the limit. With deploy --limit set to a compute host, control01 (the first neutron-server and nova-conductor host of the multinode inventory) runs the Neutron DB create/grant and the bootstrap_neutron container, plus cell discovery (nova_list_cells, openstack compute service list, nova-manage cell_v2 discover_hosts). The nova-cell RabbitMQ user/vhost creation is also delegated (to the first RabbitMQ host), but it runs only when the first nova-conductor is inside the limit, so a limit to compute hosts does not run it |
| Evidence | CODE site.yml:590-605 and nova.yml:117-136 (both plays include compute); roles/, roles/; roles/, discover_computes.yml:4-23, get_cell_settings.yml:2-21; roles/; inventory/ (Add a compute node §2) |
| Why it matters | "Only the new compute" still executes DB-sync containers on a controller |
F9 — With --limit, all hosts' facts are gathered sequentially¶
Still holds in 2023.1 CODE.
| Finding | With --limit, all hosts' facts are gathered sequentially by the limited hosts (delegated loop) |
| Evidence | CODE gather-facts.yml:30-54 (each limited host loops over its share of groups['all']; skipped for hosts that already have facts); loop items run one after another (ansible-core 2.14.18 lib/) (R23) |
| Why it matters | O(total hosts) serial latency on every limited run without fact caching |
F10 — kolla_serial makes run_once steps repeat per batch¶
Still holds in 2023.1 CODE.
| Finding | kolla_serial makes run_once steps repeat per batch (API DB sync ×3 with serial 1 and the three controllers of the multinode inventory) and orders a play's restarts by host-pattern order (computes before conductors in the nova-cell play) |
| Evidence | CODE ansible-core 2.14.18 lib/, lib/, lib/; roles/; nova.yml:29-51,; inventory/ (Tags §2.5) |
| Why it matters | Batching reduces blast radius but multiplies bootstrap work, and exposes ordering within a play |
F11 — The openvswitch_agent.ini merge includes the operator's ml2_conf.ini overrides¶
No longer holds in 2023.1 CODE. ml2_conf.ini is rendered only for neutron-server and
neutron-infoblox-ipam-agent; openvswitch_agent.ini has its own template and override files
(R5).
| Finding | Ussuri: the openvswitch_agent.ini merge included the operator's ml2_conf.ini overrides. 2023.1: openvswitch_agent.ini merges templates/ with neutron/ and neutron/ only |
| Evidence | CODE roles/ (services_need_ml2_conf_ini), :197-212 (openvswitch_agent.ini) |
| Why it matters | Server-side ML2 changes no longer restart every OVS agent; they restart neutron_server, and through loop-notify every neutron container on a host that also runs neutron-server (F1, R5). OVS agent options belong in neutron/ |
F12 — The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes¶
Still holds in 2023.1 CODE.
| Finding | The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes (default no); its comment says this removes all L3 namespaces (what the command does is neutron code: NOT AVAILABLE) |
| Evidence | CODE roles/; group_vars/ |
| Why it matters | NOT AVAILABLE Whether an L3 agent restart interrupts router traffic on that node (Q4). 2023.1 adds neutron_l3_agent_failover_delay to space the restarts across hosts when set above its default 0 (roles/) (R6) |
F13 — The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes¶
Still holds in 2023.1 CODE.
| Finding | The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes |
| Evidence | CODE site.yml:544 (import_playbook: nova.yml, whose Apply role nova-cell play is nova.yml:117-136), then site.yml:546-555 (Apply role openvswitch) and site.yml:590-608 (Apply role neutron) |
| Why it matters | NOT AVAILABLE Whether instances can be scheduled on such a compute before the later plays finish (Execution flow §2.4); relevant when adding computes one by one |
F14 — kolla_set_configs --check has blind spots¶
Holds in 2023.1, changed in part CODE: a missing optional destination is now drift (exit 1), and a restart now removes most dropped entries.
| Finding | kolla_set_configs --check has blind spots. It compares only the config_files entries of config.json (content, mode, owner). A change confined to config.json itself (command, permissions; all of memcached's settings) is never detected, nor is an entry dropped from config.json. A newly added optional file now reports drift (exit 1), like any missing destination; the Ussuri exit 2 is gone. preserve_properties entries (prometheus extras) and directory sources with subdirectories report drift on every run. A restart now removes (or resets to the original) the destination of a dropped entry when it was recorded at the container's first start; glob sources are never recorded and keep their stale files |
| Evidence | CODE kolla docker/; module_utils/; roles/ (Kolla images §3) |
| Why it matters | "genconfig + --check" (Predict and inspect a run) under-reports some changes and over-reports others. A config.json-only change is applied by a normal deploy/reconfigure, because the changed template task notifies the restart handler. Once genconfig has written it, neither deploy-containers nor a later run (its template task is then unchanged) applies it until the container restarts for another reason |
F15 — What the bootstrap containers run differs per image¶
Still holds in 2023.1; the Neutron gap is closed CODE.
| Finding | What the bootstrap containers run differs per image. Nova's bootstrap containers run with KOLLA_UPGRADE (schema sync, no online data migrations outside upgrade). Placement's KOLLA_BOOTSTRAP container also runs online_data_migrations, on every deploy/reconfigure. Heat's does not re-apply a changed heat_domain_admin_password to Keystone. neutron-server's migrates neutron and each enabled subproject; the Ussuri gap (neutron-dynamic-routing migrated by the bootstrap but never by the rolling upgrade) is closed, because both lists are the enabled neutron_subprojects |
| Evidence | CODE kolla docker/, docker/, docker/, docker/, docker/; roles/, roles/, roles/, roles/, roles/ |
| Why it matters | What a "reconfigure" or a tag-changing "deploy" does to databases depends on the image, not only on kolla-ansible (Kolla images §4) |
Related¶
- Known bugs and limitations — implementation limitations and bugs
found in
2023.1-eol - Open questions — Q1–Q14, among them Q4, which F12 refers to
- What is not known — everything marked NOT AVAILABLE, by topic
- Risk register — the risks R1–R25
- Kolla images — the image side of F14 and F15, and the answer to Q14
- Per-role fact sheets — errata with their evidence
To remember
Lumo: if you keep only 5 things from this page, keep these.
- Restart blast radius is per role-on-host, not per container: a nova-compute-only change restarts
nova_libvirtandnova_ssh. CODE - Same-tag image rebuilds are not picked up by
deploy/reconfigure/upgrade: only absent images are pulled; an explicitpullis required. CODE - A
reconfigureis adeploy: it applies schema migrations if the image tag moved, for Neutron too. CODE --limitdoes not confine actions: "only the new compute" still executes DB-sync containers on a controller. CODE- A service still running 60 s after SIGTERM is killed with its in-flight work; raising
docker_graceful_timeoutnow has an effect. CODE
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.