Fifteen headline findings (F1–F15), each with its evidence and why it matters, among them loop-notify
amplification (F1), the -t common no-op (F2), the ignored stop timeout (F3), same-tag image
rebuilds (F4) and reconfigure = deploy (F5) (Headline findings).
Evidence levels: CODE and NOT AVAILABLE, defined once, in Evidence
labels. The study rests only on reading source code; no run against a host is
used as evidence.
Loop-notify amplification. When a config/check task loops over a role's services and one item changes, Ansible 2.9 notifies the handlers of all executed items. In Kolla this means a change for one container restarts every container of that role on that host. A nova-compute-only change restarts nova_libvirt and nova_ssh; a scheduler-only change restarts nova_api; a change on a network node restarts all neutron agents there
common changes (fluentd/cron/toolbox) are applied by whatever run first reaches the host, often unexpectedly
F3 — docker_graceful_timeout and docker_client_timeout are ignored¶
Finding
docker_graceful_timeout and docker_client_timeout are ignored. The kolla_docker argument defaults (10 s, 120 s) override common_options, so every stop sends SIGKILL to a container still running 10 s after SIGTERM
Evidence
CODEkolla_docker.py:991,1011,1037-1047; group_vars/all.yml:154,166-167; ansible 2.9.27 lib/ansible/module_utils/basic.py:755,1929-1939 (argument defaults are filled in before the merge, so they are never None)
Why it matters
On every restart, a service still running 10 s after SIGTERM is SIGKILLed with its in-flight work (heat-engine, cinder copies, glance uploads); for nova_libvirt this matters only if libvirtd has not exited by then (Nova deep dive §5). NOT AVAILABLE Whether services started by a shell wrapper that does not exec them (e.g. neutron-l3-agent, haproxy) receive SIGTERM at all (Kolla images §2.6). The release note claiming 60 s (releasenotes/notes/docker-stop-timeout-c8089f6bf92d6c28.yaml) has no effect
F4 — Same-tag image rebuilds are not picked up by deploy/reconfigure/upgrade¶
Finding
Same-tag image rebuilds are not picked up by deploy/reconfigure/upgrade. Only absent images are pulled; an explicit pull is required
Evidence
CODEkolla_docker.py:805-808, :403-413
Why it matters
Mutable tags create silent drift between hosts. A later unrelated pull turns into a mass recreate
reconfigure = deploy for 70 of the 72 roles that have a tasks/reconfigure.yml (67 via import_tasks: deploy.yml, plus congress, mongodb and nova-hyperv via include_tasks: deploy.yml; only swift and bifrost have their own logic). A reconfigure re-runs Keystone registration, DB creation/grants and the DB-sync bootstrap containers of most services. Neutron is the exception: with the default neutron_enable_rolling_upgrade: "yes", bootstrap_neutron runs on deploy but is skipped on reconfigure, so a reconfigure re-runs Neutron's DB create/grant tasks but not its DB sync
Evidence
CODEroles/*/tasks/reconfigure.yml; roles/neutron/tasks/bootstrap_service.yml:21-22,24-46, roles/neutron/defaults/main.yml:411. One-shot containers counted from the code: -t nova starts 5 on control01 when cell0 is mapped and the cell is unchanged (API DB sync, map_cell0, cell DB sync, list_cells ×2: roles/nova/tasks/bootstrap.yml:52-54, map_cell0.yml:2-26, roles/nova-cell/tasks/bootstrap.yml:38-40, create_cells.yml:2-3, discover_computes.yml:5); on -t neutron reconfigure with default flags the neutron role starts none (roles/neutron/tasks/bootstrap_service.yml:21-22,41-42, group_vars/all.yml:677). NOT AVAILABLE Totals for a run without a tag, which depend on the enabled services
Why it matters
A "reconfigure" can apply schema migrations if the image tag moved, for most services; for Neutron only with non-default flags: enable_neutron_sfc (the bootstrap_neutron_sfc container runs neutron-db-manage --subproject networking-sfc … upgrade head on every reconfigure) or neutron_enable_rolling_upgrade: "no" (full bootstrap_neutron). There is never a "config only" path except genconfig + deploy-containers
F6 — Global inputs are rendered into per-host files¶
Finding
Global inputs are rendered into per-host files.enable_* flags, RabbitMQ/memcached membership, service passwords and the external FQDN end up in compute-side files
Evidence
CODEgroup_vars/all.yml:206,216; nova-cell/templates/nova.conf.j2; neutron/templates/neutron.conf.j2:119 (memcached list, shared by the OVS agent on computes); common/tasks/config.yml:109-240
--list-tasks cannot show what a tag runs, and --list-hosts shows 0 hosts for &enable_* plays
Evidence
CODE ansible 2.9.27 lib/ansible/cli/playbook.py:145-147,180-185 (the listing walks the statically compiled play and resolves host patterns before any task runs), lib/ansible/inventory/manager.py:425; site.yml:21-25 (the enable_* groups are created by group_by at run time); list-tasks-nova.txt
Why it matters
Operators need genconfig or staging to predict a run
--limit does not confine actions.delegate_to: groups[x][0] runs work on hosts outside the limit. With deploy --limit set to a compute host, control01 (the first neutron-server and nova-conductor host of the multinode inventory) runs the Neutron DB create/grant and the bootstrap_neutron container, plus cell discovery (nova_list_cells, openstack compute service list, nova-manage cell_v2 discover_hosts). The nova-cell RabbitMQ user/vhost creation is also delegated (to the first RabbitMQ host), but it runs only when the first nova-conductor is inside the limit, so a limit to compute hosts does not run it
Evidence
CODEsite.yml:761-775 and nova.yml:117-126 (both plays include compute); roles/neutron/tasks/bootstrap.yml:12-13,31-32, bootstrap_service.yml:19-22; roles/nova-cell/tasks/wait_discover_computes.yml:41-42,86-89, discover_computes.yml:4-21, get_cell_settings.yml:2-16; roles/nova-cell/tasks/rabbitmq.yml:10-12; inventory/multinode:3-5,138-139,290-291,312-313
Why it matters
"Only the new compute" still executes DB-sync containers on a controller
F9 — With --limit, all hosts' facts are gathered sequentially¶
Finding
With --limit, all hosts' facts are gathered sequentially by the limited hosts (delegated loop)
Evidence
CODEgather-facts.yml:29-50; loop items run one after another (ansible 2.9.27 lib/ansible/executor/task_executor.py:317,367)
Why it matters
O(total hosts) serial latency on every limited run without fact caching
F10 — kolla_serial makes run_once steps repeat per batch¶
Finding
kolla_serial makes run_once steps repeat per batch (API DB sync ×3 with serial 1 and the three controllers of the multinode inventory) and orders a play's restarts by host-pattern order (computes before conductors in the nova-cell play)
Batching reduces blast radius but multiplies bootstrap work, and exposes ordering within a play
F11 — The openvswitch_agent.ini merge includes the operator's ml2_conf.ini overrides¶
Finding
The openvswitch_agent.ini merge includes the operator's ml2_conf.ini overrides
Evidence
CODEroles/neutron/tasks/config.yml:172-190
Why it matters
Server-side ML2 changes restart every OVS agent
F12 — The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes¶
Finding
The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes; its comment says this removes all L3 namespaces (what the command does is neutron code: NOT AVAILABLE)
Evidence
CODEneutron-l3-agent-wrapper.sh.j2:5-12
Why it matters
NOT AVAILABLE Whether an L3 agent restart interrupts router traffic on that node (Q4)
F13 — The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes¶
Finding
The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes
Evidence
CODEsite.yml:710-780
Why it matters
NOT AVAILABLE Whether instances can be scheduled on such a compute before the later plays finish (Execution flow §2.4); relevant when adding computes one by one
kolla_set_configs --check has blind spots. It compares only the config_files entries of config.json (content, mode, owner). A change confined to config.json itself (command, permissions; all of memcached's settings) is never detected. A newly added optional file makes it exit 2, which fails check-containers, unless an entry listed before it already differs (then 1). preserve_properties entries (prometheus extras) and directory sources with subdirectories report drift on every run. A restart never removes the files of a dropped entry
"genconfig + --check" (Predict and inspect a run) under-reports some changes and over-reports others. A config.json-only change is applied by a normal deploy/reconfigure, because the changed template task notifies the restart handler. Once genconfig has written it, neither deploy-containers nor a later run (its template task is then unchanged) applies it until the container restarts for another reason
F15 — What the bootstrap containers run differs per image¶
Finding
What the bootstrap containers run differs per image. Nova's bootstrap containers run with KOLLA_UPGRADE (schema sync, no online data migrations outside upgrade). Placement's KOLLA_BOOTSTRAP container also runs online_data_migrations, on every deploy/reconfigure. Heat's does not re-apply a changed heat_domain_admin_password to Keystone. neutron-server's migrates neutron-dynamic-routing, which the rolling upgrade never migrates
Lumo: if you keep only 5 things from this page, keep these.
Restart blast radius is per role-on-host, not per container: a nova-compute-only change restarts nova_libvirt and nova_ssh. CODE
-t common is a no-op: common changes (fluentd/cron/toolbox) are applied by whatever run first reaches the host, often unexpectedly. CODE
docker_graceful_timeout is ignored: every stop sends SIGKILL to a container still running 10 s after SIGTERM. CODE
Same-tag image rebuilds are not picked up by deploy/reconfigure/upgrade: only absent images are pulled; an explicit pull is required. CODE
--limit does not confine actions: "only the new compute" still executes DB-sync containers on a controller. CODE
Check yourself
Lumo: 3 quick questions to check the main ideas of this page.
Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first
choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.