Why a small change creates a large workload¶
Why a small change can create a large workload in Kolla-Ansible 2024.1: seven small changes, each with why it is large and the source lines and sections that show it. The six rows of 2023.1 still hold, two with changes; the row on container-engine options is new on this page. The per-host work that every run pays, independently of restarts, closes the page.
| Small change | Why it is large | Source |
|---|---|---|
| One option for nova-compute | /etc/ targets every compute; loop-notify adds libvirt + ssh restarts (with the default Nova settings, which enable the nova_libvirt and nova_ssh containers). New in 2024.1: the same option in /etc/ is merged into that host's nova-compute/ only, so it reaches one compute |
nova.yml:125-147; roles/ (sources at :78, :80); roles/; Tags §2.4; Reconfigure Nova §3 |
enable_<new service>: yes |
changes fluentd/logrotate on every host (at the next run of the common play: no tag or -t common) and nova.conf sections on every compute (e.g. [barbican]) |
roles/, roles/; site.yml:88-103; roles/; Reconfigure Nova §3 |
| One extra controller | changes the transport URL in every messaging service everywhere | group_vars/; Change control-plane membership §2 |
| A server-side ML2 option | still does not reach the agents: ml2_conf.ini (and its overrides) is rendered for neutron-server and neutron-infoblox-ipam-agent only, and openvswitch_agent.ini and linuxbridge_agent.ini do not merge the ml2_conf.ini overrides. A file added under neutron/ in the custom config is still copied to every Open vSwitch and Linux bridge agent and restarts them |
roles/, :181-213, :15-21, :485-506; Reconfigure Neutron §3 |
| Rebuilt image of one service | once the new image is on the host (a same-tag rebuild needs an explicit pull), its siblings on the same host restart too: the image ID differs for one item of the role's check loop, and that loop notifies the handlers of every item |
module_utils/ (Podman: module_utils/); roles/; Tags §2.4; Update container images §2; F4 |
Any OpenStack API service -t <svc> |
runs the service's include in the loadbalancer play (haproxy configuration; new in 2024.1: also the service's block of external-frontend-map when haproxy_single_external_frontend is set, default false) and Keystone registration even with no change. Still holds in 2024.1: it does not run common, since no role has a meta directory and the common play is tagged common only; the DB sync runs on every deploy and, for most services, on every reconfigure. Neutron too: its bootstrap (neutron-db-manage ... upgrade head) has no kolla_action condition and runs on every deploy and reconfigure. Infrastructure tags such as -t haproxy run the loadbalancer play, whose 39 service includes are all evaluated and run for the enabled services, but no Keystone registration or DB sync |
Reconfigure Neutron §2, Reconfigure a single service §2; site.yml:88-103, :105-326; roles/, group_vars/; roles/, roles/, roles/; kolla docker/ |
One container-engine option (docker_graceful_timeout, docker_client_timeout, docker_restart_policy, docker_restart_policy_retry; new on this page) |
the value goes into docker_common_options, which every container task passes, and into the systemd unit text of every container. compare_container compares the whole unit file with the rendered one, so the next check of every long-running container reports it as changed and its restart handler restarts it, rewriting the unit; loop-notify then restarts the rest of its role on that host. One-shot containers have no unit; Swift deploy has no check task. A docker_restart_policy change between unless-stopped and always renders the same Restart=always and changes nothing |
group_vars/, :142-153; library/; module_utils/, :56-79, :154-167, :172-185; module_utils/, :58-65; module_utils/; F16 |
Note
The first 2024.1 deploy or reconfigure after an upgrade from 2023.1 is a large workload with no operator change
at all CODE: the 2024.1 unit template differs from the one 2023.1 wrote, so every container whose unit 2023.1 wrote
is reported as changed at its first 2024.1 check (F16;
module_utils/kolla_systemd_worker.py:20-39, :154-167); and om_enable_rabbitmq_quorum_queues now defaults to
true (group_vars/all.yml:268), which adds rabbit_quorum_queue = true to the messaging config of a cloud that
kept the 2023.1 default (e.g. roles/nova-cell/templates/nova.conf.j2:207-216;
F17).
And independently of restarts, every run pays the per-host work CODE. Still holds in 2024.1: every full deploy
or reconfigure makes 10 compare_container calls per compute (common 3, iscsid with Cinder LVM, openvswitch 2,
neutron 1, nova-cell 3), each one a kolla_set_configs --check exec in the container once it exists and no compared
parameter differs (roles/common/tasks/check-containers.yml:2-17, roles/nova-cell/tasks/check-containers.yml:2-22,
roles/openvswitch/tasks/check-containers.yml:2-18, roles/neutron/tasks/check-containers.yml:2-19,
roles/iscsi/tasks/check-containers.yml:2-18; module_utils/kolla_container_worker.py:20, :58-65;
module_utils/kolla_docker_worker.py:146-181, Podman module_utils/kolla_podman_worker.py:428-453). The common
three run only when the common play runs (no tag or -t common, site.yml:88-103). The remote module invocations
of a no-change -t nova run, 27 per compute in 2024.1 (31 on the first compute), are counted from the task files in
Compute scaling §2.
The number of task results per compute is NOT AVAILABLE.
Related¶
- Compute scaling §5 — the work multiplier and the restart multiplier
- Tags §2.4 — loop-notify
- Technical findings — F6: global inputs are rendered into per-host files; F16: the unit text is part of the container check
- Write config first, apply in controlled batches —
genconfig, thendeploy-containers --limitbatch by batch - Risk register — the risks, from R1
To remember
Lumo: if you keep only 5 things from this page, keep these.
- One option for nova-compute in
nova/nova-compute.confis not small: it targets every compute, and loop-notify adds the libvirt and ssh restarts. New in 2024.1:nova/<host>/nova-compute.confreaches that compute only. CODE - A rebuilt image of one service, once pulled, restarts its sibling containers on the same host too.
- One extra controller changes the transport URL in every messaging service everywhere.
- One container-engine option such as
docker_graceful_timeoutchanges the systemd unit of every container, and the next check of each container reports it as changed. CODE - Every run pays the per-host work: every full
deployorreconfiguremakes 10compare_containercalls per compute, independently of restarts. CODE
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.