The open questions of the study, Q1 to Q21. The summary table lists them, and each question has a section that gives
why it matters and its status. Q1 to Q14 are the questions of the 2023.1 study, re-checked against the 2024.1 sources;
Q15 to Q21 are new in 2024.1.
Note
Key answers
Open questions Q1–Q21: all are NOT AVAILABLE, except Q14, which is answered from the source
CODE, the control-flow part of Q6 and the handler part of Q17, also answered from the source
CODE, and Q10, answered in part: F2 no longer applies, and ansible-core
2.15 is not in these sources (Questions). Q1 asks whether running VMs survive a
nova_libvirt restart or recreate.
New in 2024.1: how the Podman path behaves on a host (Q15), what oslo.messaging does with existing
classic queues under the new quorum-queue default (Q16), the real order of the staged etcd restarts
(Q17), where the Keystone service role comes from (Q18), what happens to containers of removed
services (Q19), what else drifts on the first 2024.1 run (Q20) and which RabbitMQ version a cloud
runs before its upgrade (Q21).
These questions cannot be settled by reading the sources, except Q14, which is answered in its section by reading the
kolla image sources. For Q6, the 2024.1 plays and the ansible-core 2.16.20 code answer whether later batches proceed;
the state of the failed node and the availability risk stay NOT AVAILABLE. For Q17, the 2024.1 handlers show what
each host includes, not the order across hosts. Q10 was asked for the Ansible versions Ussuri allows (2.8 and 2.9);
2024.1 allows ansible-core 2.15 and 2.16, and only 2.16.20 is in these sources, so its 2024.1 form is answered only in
part. The others are NOT AVAILABLE.
Control flow answered from the sourceCODE. In 2024.1 the handlers only put the hosts into the mariadb_restart and mariadb_start groups, and separate plays with serial: "33%" restart them; no MariaDB play sets any_errors_fatal, and each sets max_fail_percentage from mariadb_max_fail_percentage, then kolla_max_fail_percentage, default 100 (roles/mariadb/handlers/main.yml:55-72, mariadb.yml:18-52). A node that fails its port or WSREP wait (roles/mariadb/tasks/restart_services.yml:18-51) marks only its own host failed. With the default of 100: when that is every host of the batch (always so with three nodes, one host per batch, ansible-core 2.16.20 lib/ansible/utils/helpers.py:25-34), ansible-core ends the playbook and the later batches are not restarted (lib/ansible/executor/playbook_executor.py:204-209, :219-220); when other hosts of the batch succeed, the next batch still restarts its nodes. New in 2024.1: with -e mariadb_max_fail_percentage=0 (or kolla_max_fail_percentage), the first failed node marks the rest of its batch failed and ends the run before the next batch (lib/ansible/plugins/strategy/linear.py:337-348, lib/ansible/executor/playbook_executor.py:196-198) (MariaDB, Controlled batches, fact sheet B §1.5). NOT AVAILABLE Whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is
Predict and inspect a run, Tags §2.6: kolla_container, which replaces kolla_docker in 2024.1, does not declare check-mode support (library/kolla_container.py:355-359), so ansible-core skips it under --check (lib/ansible/module_utils/basic.py:506-507) CODE; yet three kolla_container tasks set check_mode: false and start one-shot containers even under --check (roles/nova-cell/tasks/get_cell_settings.yml:21, roles/rabbitmq/tasks/version-check.yml:36, roles/nova/tasks/upgrade.yml:30) CODE
Ussuri allows 2.8. 2024.1 no longer runs Ansible 2.8 or 2.9: kolla-ansible accepts ansible-core 2.15 to 2.16 (tools/kolla-ansible:10-11), so the question becomes whether 2.15 and 2.16 behave the same
Status
Answered in partCODE. F2 no longer applies in 2024.1, whatever the Ansible version: common is its own play tagged common (site.yml:88-103), so -t common does not depend on tag inheritance through the dynamic include_tasks (roles/common/tasks/main.yml:2). F1 holds in ansible-core 2.16.20: one changed loop item notifies the handlers of every executed item (lib/ansible/plugins/strategy/__init__.py:648-673, lib/ansible/executor/task_executor.py:816-820); the tag inheritance of dynamic includes is in lib/ansible/playbook/block.py:294-361 (Tags §2.1, §2.4). The 2023.1 answer, read in ansible-core 2.14.18, does not carry over. NOT AVAILABLE Whether ansible-core 2.15 has the same code: only 2.16.20 is in these sources
Answered from the sourceCODE, see Kolla images §4. Still holds in 2024.1: Nova's bootstrap containers run with KOLLA_UPGRADE (roles/nova/tasks/bootstrap_service.yml:10-11, roles/nova-cell/tasks/bootstrap_service.yml:10-11): api_db sync + db sync --local_cell (nova-api, kolla docker/nova/nova-api/extend_start.sh:16-20), db sync --local_cell (nova-conductor, kolla docker/nova/nova-conductor/extend_start.sh:9-12), so Nova's online data migrations never run outside upgrade; the images' KOLLA_BOOTSTRAP branches are unreachable. Placement's KOLLA_BOOTSTRAP runs db sync + online_data_migrations on every deploy/reconfigure (roles/placement/tasks/bootstrap_service.yml:11, kolla docker/placement/placement-api/extend_start.sh:5-9). nova-api's KOLLA_UPGRADE_CHECK runs nova-status upgrade check, upgrade only (roles/nova/tasks/upgrade.yml:18, kolla docker/nova/nova-api/extend_start.sh:27-30). Glance adds db_load_metadefs (kolla docker/glance/glance-api/extend_start.sh:5-9). Heat creates heat_user_domain/heat_domain_admin only if the domain is absent, so a changed heat_domain_admin_password is never pushed to Keystone (kolla docker/heat/heat-api/extend_start.sh:5-17). neutron-server's KOLLA_BOOTSTRAP migrates neutron and every enabled subproject, and the rolling upgrade migrates the same list; changed in 2024.1, the list adds neutron-fwaas and tap-as-a-service to networking-sfc, neutron-dynamic-routing, neutron-vpnaas and vmware-nsx (roles/neutron/defaults/main.yml:674-686,727,733). New in 2024.1, octavia-api's KOLLA_BOOTSTRAP also runs octavia-db-manage upgrade_persistence (kolla docker/octavia/octavia-api/extend_start.sh:5-9). RabbitMQ writes the Erlang cookie only in its bootstrap container (kolla docker/rabbitmq/rabbitmq/extend_start.sh:7-18). Agent, OVS and OVN images have no one-shot branch
How do the Podman worker, the Podman Python client, Podman and systemd behave on a real host when kolla_container_engine: podman?
Why it matters
kolla_container_engine defaults to "docker" (group_vars/all.yml:92), and any other value makes kolla_container use the Podman worker (library/kolla_container.py:415-420), which talks to the Podman socket through the Podman Python client (module_utils/kolla_podman_worker.py:21,73) CODE. With Podman the worker writes and enables a systemd unit at creation for every container, one-shots included (module_utils/kolla_podman_worker.py:497-505), and stops every long-running container through its unit (:575-593); image pruning becomes a podman image prune command (Update container images §1.2). Every Podman statement of the study is code-only (Repository architecture §6, Limits of the study)
Status
NOT AVAILABLE The Podman Python client, Podman and systemd are not in these sources, and no Podman host was used
Q16 — Existing classic queues under the quorum-queue default¶
New in 2024.1.
Question
What does oslo.messaging do with existing classic queues when the services restart with rabbit_quorum_queue = true?
Why it matters
F17: om_enable_rabbitmq_quorum_queues defaults to true (group_vars/all.yml:266-268), so a cloud that kept the 2023.1 default gets a changed messaging config, and a restart, for every service that renders [oslo_messaging_rabbit]. prechecks fails when HA and quorum are equal (roles/prechecks/tasks/service_checks.yml:67-72) and, with quorum on, asserts that every queue except reply_ and _fanout ones is a quorum queue; its message calls the migration manual (roles/rabbitmq/tasks/precheck.yml:131-152,185-189) CODE (RabbitMQ, Pre-execution checklist §2)
Status
NOT AVAILABLE oslo.messaging is not in these sources; nor is how long a manual migration takes
Handlers answered from the sourceCODE. The handlers look up the leader (roles/etcd/handlers/main.yml:9-17); outside upgrade and config they restart the non-leaders in a loop over the four values of groups.etcd.index(host) % 4 (:38-55), then the leader, whose handler is defined last (:57-65). On upgrade the non-leader handler loops over groups.etcd without using item (:28-36), so each non-leader includes the restart tasks once per etcd member; its comment says "one by one". Each restart waits for the client port and for etcdctl endpoint health (roles/etcd/tasks/restart_services.yml:16-41). NOT AVAILABLE How ansible-core orders, across hosts, the files included by one looped handler, so whether the % 4 waves really restart one after another and whether the upgrade restarts are really one by one; whether etcd clients fail while the members restart
Q18 — Which step creates the Keystone service role¶
New in 2024.1.
Question
Which step creates the Keystone service role that 2024.1 roles grant to their users?
Why it matters
The nova role no longer creates the role itself (2023.1 passed nova_ks_roles: [service]) but still grants it to nova (roles/nova/tasks/register.yml:2-8, roles/nova/defaults/main.yml:253-256); service-ks-register creates only the roles of its users plus service_ks_register_roles, empty by default (roles/service-ks-register/tasks/main.yml:87-97, roles/service-ks-register/defaults/main.yml:29) CODE (Nova deep dive §2.1)
Status
NOT AVAILABLE No Kolla-Ansible task found passes it, and the Keystone sources, where its own bootstrap is defined, are not part of this study
Q19 — Containers of removed services left on hosts¶
New in 2024.1.
Question
What happens to a chrony container, or to a container of a service removed in 2024.1, left on a host by an earlier release?
Why it matters
2024.1 has no chrony role or play, and no file under ansible/ or tools/ names freezer, monasca, murano, sahara, senlin, solum, vitrage, kafka, storm, zookeeper or outward RabbitMQ; the only chrony reference is the precheck that looks for an active host time daemon (roles/prechecks/tasks/timesync_checks.yml:1-47) CODE. So no tagged run stops, removes or checks those containers (Chrony, Tags §3, row "removed services")
Status
NOT AVAILABLE What such a container does on a host, for example a chrony container next to the host NTP daemon that the precheck requires
Q20 — What else drifts on the first 2024.1 run of a 2023.1 cloud¶
New in 2024.1.
Question
Besides the unit change of F16, what makes containers differ on the first 2024.1 deploy, reconfigure or deploy-containers of a cloud deployed by 2023.1, and how does the new unit behave?
Why it matters
F16: the 2024.1 unit template differs from the 2023.1 one and the whole unit text is compared, so every container whose unit 2023.1 wrote is restarted at its first check (module_utils/kolla_systemd_worker.py:20-39,154-167) CODE. The nova_libvirt bind of /run is now written /run:/run:shared with Docker (roles/nova-cell/defaults/main.yml:374; 2023.1 /run/:/run/:shared), and the Docker worker compares the bind strings as sets, without normalising them (module_utils/kolla_docker_worker.py:124-144) CODE (Nova, iSCSI, Predict and inspect a run §2)
Status
NOT AVAILABLE Whether Docker reports the 2023.1 /run/ binds with their trailing slash, and so whether they alone make nova_libvirt and iscsid differ; whether systemd treats the 2023.1 StartLimitIntervalSec= and the 2024.1 StartLimitInterval= key the same (Kolla images §2.3)
Q21 — RabbitMQ version before the upgrade, and the 3.12 step¶
New in 2024.1.
Question
Which RabbitMQ version does a cloud run before its first 2024.1 run, and which image would a kolla-ansible rabbitmq-upgrade 3.12 run with 2024.1 code find?
Why it matters
upgrade now runs the version check first (roles/rabbitmq/tasks/upgrade.yml:2), which fails unless the running version is at most one minor version behind the target, or the step is 3.13 to 4.0 or 4.1, and names the rabbitmq-upgrade step to run first (roles/rabbitmq/tasks/version-check.yml:44-64) CODE. The 2024.1 rabbitmq image pins RabbitMQ 3.13 (kolla docker/rabbitmq/rabbitmq/apt_preferences_rabbitmq.ubuntu:2), the 2023.1 one pinned 3.11, and the kolla 2024.1 tree has no rabbitmq-3-12 image CODE (RabbitMQ)
Status
NOT AVAILABLE The running version is a property of each cloud, and the registry content is not in these sources