What is not known¶
What the sources do not establish, grouped by topic. Each line restates the open point and links to the sections that mark it NOT AVAILABLE; nothing here is an answer. The open questions Q1–Q21 are in Open questions. Every point was checked against the 2024.1 pages that mark it: points the 2024.1 code settles, or that concern 2023.1 code which no longer exists, were removed or narrowed, and the row says so. Points new in 2024.1 (the Podman path, quorum queues, the staged etcd restart, the systemd unit change, the removed services) are marked "New in 2024.1".
Running VMs and compute hosts¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
Whether running VMs survive a nova_libvirt restart or recreate (Q1): where libvirtd puts QEMU processes, whether QEMU survives the stop of the container, and whether libvirtd re-attaches to running domains when it starts. |
Nova deep dive §5, Reconfigure Nova §5, Nova, Risk register (R1), Open questions, Kolla images §5.1, Kolla images §6 (item 11), Kolla images §7 |
How the stop of nova_libvirt plays out: whether dumb-init forwards SIGTERM to libvirtd only, whether libvirtd itself is SIGKILLed if it is still running after docker_graceful_timeout (60 s, group_vars/; the stop goes through the container's systemd unit, which sets no TimeoutStopSec, so systemd's own stop timeout, host configuration, also applies), and whether the engine kills what is left in the container's cgroup under pid_mode: host. |
Nova deep dive §5, Kolla images §2.6, Repository architecture §6, Kolla images §7 |
Whether the image's /var/run is a symlink to /run, which would put libvirt's running-domain state on the host /run bind mount. |
Nova deep dive §5, Kolla images §7 |
| Whether surviving QEMU processes keep running the old QEMU binary, and keep the old container's mount namespace, after a recreate with a new image. | Nova deep dive §5, Kolla images §7 |
Whether live migrations in progress fail, whether attach/detach and snapshot operations in progress fail, and whether virsh and nova-compute calls fail until libvirtd is back. |
Nova deep dive §5, Nova deep dive §6, Reconfigure Nova §5, Tags §3.1 (row nova), Risk register (R1) |
What a nova_compute restart does: whether running VMs are affected, how long the service is down and whether it is reported down, whether in-flight operations fail or are left in a transitional state, and whether init_host reconciles state on start. |
Nova deep dive §5, Reconfigure Nova §5, Risk register (R1) |
Whether a nova_ssh restart interrupts resizes or cold migrations that are copying disks. |
Nova deep dive §6, Reconfigure Nova §5 |
Whether API calls and instance actions fail while the nova API, scheduler and conductors restart; whether the scheduler SIGHUP is harmless; whether discover_hosts affects existing instances. |
Reconfigure Nova §5, Nova deep dive §6, Tags §3.1 (row nova) |
Whether the recursive chown of /var/lib/nova at every nova_compute start also enters the shares mounted under /var/, and how start time grows with the number of files. |
Nova deep dive §5, Kolla images §6 (item 10), Kolla images §7 |
Upstream Nova behaviour when a dependency is missing: MariaDB, RabbitMQ liveness reporting, Keystone, Placement (whether scheduling stops while it restarts), host mappings, the network-vif-plugged event, libvirt; Nova's upstream default list of scheduler filters, which Kolla leaves unset. |
Nova deep dive §2.2, Nova deep dive §3, Placement, Tags §3.1 (row <svc> API service) |
| Whether instances can be scheduled on a fresh compute before the openvswitch and neutron plays have finished there. | Execution flow §2.4, Component dependencies §6, Add a compute node §2, Findings (F13) |
Whether live migration to a new compute fails while, with libvirt_tls, the other computes' /etc/hosts lacks it. |
Add a compute node §2 |
Whether restarts that touch none of the containers the study traces to running VMs (the common containers, the control-plane services; "none" in Tags §3.1) affect running VMs, APIs or traffic another way. |
Tags §3.1, Apply common changes (fluentd, cron, kolla_toolbox) §4, Reconfigure a single service §3, Pre-execution checklist §2, Limits of the study, Risk register (R2) |
The effects of a bootstrap-servers re-run on a live compute (Q13), among them whether every container on the host stops when Docker restarts (restarted by default) and how the kolla-<name>-container.service units bring them back; the runtime effects of the baremetal role's limitations. |
Compute scaling §3 (row "Docker daemon restart"), Open questions, Risk register (R14), Known bugs and limitations (row "baremetal") |
Network data plane¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
neutron_openvswitch_agent restart (Q2): whether flows in OVS persist while the agent is down, whether and how the agent resyncs and re-installs flows, and whether a VM loses packets. |
Reconfigure Neutron §4, Tags §3.1 (row neutron), Component dependencies §4, Neutron, Risk register (R3, R4, R5), Open questions, Kolla images §5 (row neutron-openvswitch-agent) |
openvswitch_vswitchd restart or recreate (Q3): whether the OpenFlow tables are lost, whether the OVS agent or ovn-controller reprograms them, and whether all VM traffic on the host is interrupted, and for how long; whether VM networking is interrupted on every compute at once when the openvswitch play restarts it with no serial. |
Tags §3.1 (row openvswitch), Open vSwitch, Update container images §4, Risk register (R7, R15), Open questions, Kolla images §5 (row openvswitch-vswitchd), Kolla images §6 (item 13), Kolla images §7 |
New in 2024.1: whether the first 2024.1 run reports the new external_ids:hostname key changed, and so restarts openvswitch_vswitchd (a change of it notifies that restart, roles/, roles/); what setting fail_mode does to an existing bridge; whether the openvswitch_db, openvswitch_bridge and openvswitch_port modules report changed when a value is already set (their code is not in the studied sources); whether the Open vSwitch and OVN images have the openvswitch user that their new chown steps need. |
Open vSwitch, OVN, Kolla images §7 |
neutron_l3_agent restart (Q4): whether the namespaces persist without the HA cleanup, whether keepalived/haproxy child processes die with the container, whether with the cleanup the routers on that node lose north-south and floating-IP connectivity until rebuilt, and whether HA routers fail over. What neutron-netns-cleanup itself does is neutron code. |
Reconfigure Neutron §4, Risk register (R6), Findings (F12), Open questions, Component dependencies §4, Kolla images §5.3 |
neutron_dhcp_agent restart: whether dnsmasq processes die with the container, whether lease renewals fail during the window, whether new ports wait for the agent, and whether qdhcp- namespaces persist. |
Reconfigure Neutron §4, Component dependencies §4 |
neutron_metadata_agent restart: whether metadata requests fail during the window. |
Reconfigure Neutron §4 |
neutron_server restart: whether existing traffic is unaffected, and whether port bindings and new instances fail during the window. Whether the RPC layer tolerates agents that start before the server. |
Reconfigure Neutron §4, Execution flow §2.5 |
| Whether the processes that agent containers spawn (dnsmasq, keepalived, haproxy for metadata) die with the container and are respawned by the agent; the DHCP and router outage per agent restart, and whether router and DHCP traffic is interrupted on network nodes. | Component dependencies §4, Risk register (R4, R5) |
OVN: whether an ovn-controller restart re-programs flows on the host; whether Kolla-Ansible's direct ovn-ctl call creates and upgrades the DB schema; what set-connection does to established client connections; the impact of losing a manual chassis value. |
Tags §3.1 (row ovn), OVN, Reconfigure a single service §3, Kolla images §5 (row ovn-nb-db-server), Kolla images §7 |
Whether VM network traffic stops under kolla-ansible stop. |
Risk register (R15) |
Storage and images¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
iscsid restart: whether existing sessions are kernel-held, and whether attach/detach during the window fails. |
Tags §3.1 (row iscsi), iSCSI, Reconfigure a single service §3 |
tgtd restart: whether tgtd keeps targets only in memory, and so whether a restart cuts disk I/O for VMs that use LVM volumes from that host; whether tgtd needs configfs. |
Tags §3.1 (row iscsi), iSCSI, Kolla images §5 (row tgtd) |
Whether removing the iscsi_info volume produces a new IQN at the next start, and whether that breaks backend ACLs. |
iSCSI, Kolla images §6 (item 19) |
| Glance: whether snapshots in progress fail (R20); whether existing file-store images stay readable, and boots from them still work, after Swift is enabled (R21, Q12). | Glance, Swift, Risk register, Open questions |
Whether a [barbican] section rendered before Barbican exists has any effect before encrypted volumes are used. |
Execution flow §2.2 |
Cinder, Swift, Manila, Octavia and Barbican: service-side behaviour the roles do not establish (for example whether cinder-backup works with default settings, what changing swift_hash_path_suffix/prefix after data is written does, what Octavia amphorae do after a change of the health-manager host list). |
Cinder, Swift, Manila, Octavia, Barbican |
Control plane: HAProxy, MariaDB, RabbitMQ, Keystone, memcached, Heat¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
HAProxy restart (Q5): whether keepalived drops or moves the VIP, and for how long; whether the node goes to FAULT; whether established connections are cut; whether nova and neutron get DB errors and instance metadata requests fail meanwhile. How HAProxy treats backends that are not up yet on a first deploy. The exit status of keepalived's check_alive.sh when no check script is present. (The Ussuri haproxy upgrade bugs stay gone in 2024.1: roles/ imports deploy.yml.) |
HAProxy and keepalived, Risk register (R8), Tags §3.1 (row haproxy), Component dependencies §6, Reconfigure a single service §3, Open questions, Known bugs and limitations (row "haproxy"), Kolla images §6 (item 15), Kolla images §7 |
MariaDB (Q6): in which order group_by places the hosts in mariadb_restart, hence in which batch node 0 (the active HAProxy backend) restarts; after a failed WSREP wait in a batch that does not stop the run (with the default max_fail_percentage of 100, a batch whose hosts all fail ends the playbook; otherwise the next batch still restarts its nodes CODE, R9), whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and how large the quorum and availability risk is. Whether the DB containers come back as a non-primary cluster after a full stop, and whether a manual start left with --wsrep-new-cluster bootstraps a new cluster. What mariabackup does when the server it talks to is not the local one. |
MariaDB, Risk register (R9), Tags §3.1 (row mariadb), fact sheet B §1.5, Database operations, Known bugs and limitations (row "mariadb"), Open questions, Kolla images §6 (item 16), Kolla images §7 |
RabbitMQ: whether nova-compute heartbeats are lost and services are reported down while the brokers are drained and recreated batch by batch during upgrade (R10), and whether API calls fail meanwhile; whether, during a one-by-one restart, nova-compute and the agents lose RPC, RPC-backed API calls time out and the agents reconnect by themselves; which node starts first after an uncontrolled full stop; the consequences of a changed Erlang cookie, and whether rabbitmq-server uses RABBITMQ_CLUSTER_COOKIE or only the cookie file; new in 2024.1, what happens to the cookie when rabbitmq_datadir_volume is changed on an existing cloud. |
RabbitMQ, Risk register, Tags §3.1 (row rabbitmq), Change control-plane membership §2, Known bugs and limitations (second row rabbitmq), Kolla images §6 (item 9), Kolla images §7 |
New in 2024.1, quorum queues (Q16): what oslo.messaging does with existing classic queues when the services restart with rabbit_quorum_queue = true, now rendered by default (group_vars/). |
Findings (F17), RabbitMQ, Nova deep dive §7, Pre-execution checklist §2, Predict and inspect a run §2, Open questions |
New in 2024.1, RabbitMQ version (Q21): which RabbitMQ version a given cloud runs before its first 2024.1 run, which the new version check compares with the target image, and which image a kolla-ansible rabbitmq-upgrade 3.12 run with 2024.1 code would find in the registry. |
RabbitMQ, Open questions |
Keystone: whether authentication, and so the other services' API requests, fails while keystone restarts or is down (R19); Fernet key distribution failure modes (Q11); whether Keystone still reads a removed policy file after the restart; the upstream semantics of keystone-manage bootstrap, and so whether the bootstrap wrapper can ever report unchanged; whether the keystone_fernet container restarts when key 0 is missing; whether an admin endpoint, which 2024.1 code no longer removes, is still in a platform's catalogue; whether the SSH clients accept the new keystone_ssh host keys after a recreate. |
Keystone, Risk register, Tags §3.1 (row <svc> API service), Component dependencies §1, Nova deep dive §3, Reconfigure a single service §1.1, fact sheet B §3.10, Open questions, Kolla images §3.5, Kolla images §5 (row keystone-fernet), Kolla images §7 |
New in 2024.1, the Keystone service role (Q18): which step creates it. The roles only grant it to their users; no Kolla-Ansible task found creates it (roles/, roles/). |
Nova deep dive §2.1, Open questions |
| memcached: whether cached tokens are lost when the membership changes or the container restarts, and the effect on auth latency and Horizon sessions. | Memcached, Horizon, Risk register (R4), Component dependencies §1, Tags §3.1 (row memcached / memcache), fact sheet A §3.8, Kolla images §5 (row memcached), Kolla images §6 (item 14) |
Heat: whether stack-domain operations fail authentication after a changed heat_domain_admin_password; what heat-engine restarts do to stacks in progress. |
Heat, Known bugs and limitations (row "heat"), Kolla images §6 (item 7), Kolla images §7 |
New in 2024.1, etcd (Q17): the order in which the staged handlers really restart the members across hosts, on upgrade and outside it, and whether etcd clients fail while the members restart. |
Execution flow §7, Controlled batches §1.2, Change control-plane membership §2, Tags §3.1 (row etcd), Open questions |
The impact on the Neutron API of the upgrade contract step, and whether a neutron contract migration is safe against running servers (in particular with no subproject enabled, when no server is stopped first); the runtime effects of the swift limitations; the effect on the masakari instancemonitor of its hard-coded qemu+tcp URI (roles/) with libvirt_tls. |
Neutron, Known bugs and limitations (rows "neutron", "swift", "masakari"), Kolla images §6 (item 8), Kolla images §7 |
Upgrades and database migrations¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
Whether neutron-db-manage … upgrade head, run by a tag change applied with deploy, applies contract migrations under live old-version servers, and what that does to them. |
Update container images §4, Risk register (R12) |
Whether glance-manage db_load_metadefs, without --merge, only adds namespaces that are not yet in the DB. |
Kolla images §4 (row glance-api) |
New in 2024.1, the first 2024.1 run on a cloud deployed by 2023.1 (Q20): the changed systemd unit flags, at its check, every container whose unit 2023.1 wrote and that no 2024.1 run has recreated or restarted yet (Swift deploy has no check) CODE (F16); not established are whether Docker reports the 2023.1 /run/ binds of nova_libvirt and iscsid with their trailing slash, and so whether those alone also differ, and whether systemd treats the old StartLimitIntervalSec= and the new StartLimitInterval= key the same. |
Nova, iSCSI, Findings (F16), Kolla images §2.3, Predict and inspect a run §2, Open questions |
Signals, the container runtime and image start behaviour¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
dumb-init signal forwarding: whether --single-child forwards SIGTERM to the direct child only, and what happens to grandchildren; the same for mysqld_safe under the MariaDB image's own entrypoint. |
Kolla images §2.6, Kolla images §7, Nova deep dive §5 |
Whether services started by a shell wrapper that does not exec them (neutron-l3-agent, haproxy, ovs-vswitchd, ovsdb-server) receive SIGTERM at all before the container stops. |
Component dependencies §4, HAProxy and keepalived, Open vSwitch, Risk register (R24), Kolla images §2.6, Kolla images §5.3, Kolla images §6 (item 12) |
Whether a container whose kolla_set_configs fails at start then restarts in a loop under its systemd unit or stops at the start limit; what the engine (Docker or Podman) does with a container whose healthcheck reports it unhealthy, beyond reporting it. |
Kolla images §2.3, Kolla images §2.7, Kolla images §3.3, Kolla images §7 |
Whether systemd starts a container stopped by kolla-ansible stop again at the next boot of the host: the stop path does not disable the unit, which was enabled for multi-user.target. |
Stop and destroy |
New in 2024.1, the Podman path (Q15): how the Podman worker, the Podman Python client, Podman itself and systemd behave on a real host (kolla_container_engine: podman); what systemd does with the unit the Podman worker writes for a one-shot container left in place; what podman image prune --force --filter 'label=kolla_version' removes. |
Repository architecture §6, Limits of the study, Kolla images §4, Update container images §1.2, Update container images §4, Open questions |
| Whether the generated CA trust bundle keeps the last custom CA once it is removed. | Kolla images §2.4, Kolla images §6 (item 4), Kolla images §7 |
Whether the SSH clients accept the new nova_ssh and keystone_ssh host keys after a recreate without failing. |
Kolla images §6 (item 18), Keystone |
| Whether xtrace writes one-shot secrets (the heat domain admin password, the MariaDB root password in the upgrade one-shot; the MariaDB bootstrap turns xtrace off around its root-password commands) to the container's stderr. | Kolla images §2.2 |
Horizon: how long a start that regenerates the static assets takes; in which order Horizon loads local_settings.py and the files of local_settings.d. |
Horizon, Kolla images §5 (row horizon), Kolla images §5.2, Kolla images §7 |
| fluentd: whether the tail read positions survive a config-only restart, and whether they are lost on a recreate; whether fluentd can read the journal. | Apply common changes (fluentd, cron, kolla_toolbox) §3, Common |
| The cost of the recursive chown on large trees (glance store, NFS instance directory, MariaDB datadir, RabbitMQ mnesia). | Kolla images §6 (item 10), Kolla images §7 |
| What the image scripts do against real services and data. | Limits of the study, Kolla images §1 |
Tooling, collections and removed services¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
ansible-core 2.15: Kolla-Ansible 2024.1 accepts 2.15 and 2.16 (tools/), and only 2.16.20 was read, so whether 2.15 behaves the same (Q10). |
Limits of the study, Tags §2, Open questions |
What the galaxy collections do (community.docker, containers.podman, among them the docker_prune that prune-images calls): what a prune removes, so whether the images of the previous release are kept after an upgrade; which kolla_version label value the images carry. |
Limits of the study, Repository architecture §6, Update container images §1.2 |
New in 2024.1, containers of removed services (Q19): what happens to a chrony container, or to a container of a removed service (freezer, monasca, murano, sahara, senlin, solum, vitrage, kafka, storm, zookeeper, outward RabbitMQ), left on a host by an earlier release: no 2024.1 file under ansible/ or tools/ names them, except the precheck that looks for a host time daemon. Which time source the hosts use. |
Chrony, Tags §3 (row "removed services"), Component dependencies §2, Open questions |
How getopt parses --include-dev on the command line of kolla-ansible destroy (declared as an option that takes a value, read as a flag). |
Stop and destroy |
Predicting a run¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
Whether, on an unchanged cloud and apart from the preserve_properties exception, every task that notifies a restart handler reports ok, so that nothing restarts. |
Tags §3, Tags §3.1, Reconfigure Nova §3, Reconfigure a single service §2, Find it fast (row "Will containers restart or be recreated?") |
Whether a check-mode run (--check --diff) fails or gives partial results (Q9), and which of the check_mode: false tasks it reaches. |
Predict and inspect a run §1, Open questions, Tags §2.6 |
| The number of one-shot containers of a run without a tag. | Findings (F5) |
Timings, counts and scale¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
| Task-result totals per compute, per controller and per fleet, and the remote invocations of a full run. | Compute scaling §2, Compute scaling §2.1, Compute scaling §4.2, Compute scaling §6, Compute scaling §8, Execution flow §2.4, Reconfigure Nova §4, Add a compute node §2, Why a small change creates a large workload |
| Any duration of an initial deploy or of a reconfigure (image pulls, first container starts, per-task overhead). | Execution flow §6, Command quick lookup |
The latency of one remote invocation (t_inv), the render time of a template (t_render), play and run durations, and real timing at scale (Q7). |
Compute scaling §4.1, Compute scaling §6, Open questions |
The CPU cost per host of the group_by loop, and how many attempts the until/retries loops need. |
Compute scaling §1 |
| Whether, and at which forks value, the deploy host's CPU becomes the bottleneck; whether forks overlap SSH and remote-execution latency in production; the share of run time taken by the compute-wide plays. | Compute scaling §6, Compute scaling §7 |
| Image sizes and registry throughput (Q8). | Compute scaling §4.2, Open questions |
How close together fleet-wide restarts land in time, and the duration of the delegated fact loop under --limit. |
Compute scaling §6 |
In the fact sheets¶
The fact sheets mark their own NOT AVAILABLE points inline. Fact sheets C and D also have their own NOT AVAILABLE list, and fact sheet E a register of open questions.
| Fact sheet | Own list | Sections that contain NOT AVAILABLE marks |
|---|---|---|
| A — infra, load balancing, OVN | – | §1.3, §1.4, §1.8, §1.9, §2.4, §3.8, §4.8, §5.8, §6.8, §7.3, §8.3, §9 |
| B — MariaDB, RabbitMQ, Keystone | – | §0, §1.5, §1.6, §1.8, §2.2, §2.5, §2.8, §3.3, §3.6, §3.7, §3.9, §3.10 |
| C — Glance, Placement, Cinder, Horizon, Heat | §8 | §0, §1.4, §1.6, §1.7, §1.8, §2.8, §3.6, §3.8, §4.4, §4.8, §5.6, §5.8, §6, §8 |
| D — Barbican, Octavia, Swift, Manila, compute-side roles | §8 | §1.6, §2.3, §2.5, §2.6, §3.4, §3.5, §3.8, §5.0, §5.1, §5.2, §5.3, §5.5, §5.6, §5.7, §5.8, §5.9, §5.12, §5.15, §6.2, §7, §8 |
| E — host bootstrap, operational playbooks | §14 | the sections that the fact sheet itself marks; the pages above cite §3 (/etc/hosts) and §6 (destroy) |
Related¶
- Open questions — the open questions Q1 to Q21, one section each, with why each one matters
- Limits of this study — why these points are open: source code only, no timings, no container or VM behaviour used as evidence, the Podman path read in the code only
- Conventions and evidence labels — the definition of NOT AVAILABLE
- Kolla images §7 — what is not available from the kolla image sources
- Known bugs and limitations — the bugs whose effect the Status column marks NOT AVAILABLE
- Risk register — the risks whose VM impact and API impact rows carry the NOT AVAILABLE marks listed here
- Findings — F16 and F17, the 2024.1 findings behind Q20 and Q16
- Troubleshooting — the symptom index
To remember
Lumo: if you keep only 5 things from this page, keep these.
- This page lists what the sources do not establish; nothing here is an answer.
- Whether running VMs survive a
nova_libvirtrestart or recreate is NOT AVAILABLE - What oslo.messaging does with existing classic queues once the services restart with
rabbit_quorum_queue = true, the 2024.1 default, is NOT AVAILABLE - How the Podman path new in 2024.1 behaves on a real host is NOT AVAILABLE: its code was read, no Podman host was used
- Any duration of an initial deploy or of a reconfigure (image pulls, first container starts, per-task overhead) is NOT AVAILABLE
Check yourself
Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: this is the last page of the journey. Well done for coming this far!
Lumo: nicely done! One more page on your journey.