What is not known¶
What the sources do not establish, grouped by topic. Each line restates the open point and links to the sections that mark it NOT AVAILABLE; nothing here is an answer. The open questions Q1–Q14 are in Open questions.
Running VMs and compute hosts¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
Whether running VMs survive a nova_libvirt restart or recreate (Q1): where libvirtd puts QEMU processes, whether QEMU survives the stop of the container, and whether libvirtd re-attaches to running domains when it starts. |
Nova deep dive §5, Reconfigure Nova §5, Risk register (R1), Open questions, Kolla images §6 (item 11), Kolla images §7 |
How the stop of nova_libvirt plays out: whether dumb-init forwards SIGTERM to libvirtd only, whether libvirtd itself is SIGKILLed if it is still running after 10 s, and whether the runtime kills what is left in the container's cgroup under pid_mode: host. |
Nova deep dive §5, Kolla images §2.6, Kolla images §7 |
Whether the image's /var/run is a symlink to /run, which would put libvirt's running-domain state on the host /run bind mount. |
Nova deep dive §5, Kolla images §7 |
| Whether surviving QEMU processes keep running the old QEMU binary after a recreate with a new image. | Nova deep dive §5, Kolla images §7 |
Whether live migrations in progress fail, whether attach/detach and snapshot operations in progress fail, and whether virsh and nova-compute calls fail until libvirtd is back. |
Nova deep dive §5, Nova deep dive §6, Reconfigure Nova §5, Tags §3.1 (row nova), Risk register (R1) |
What a nova_compute restart does: whether running VMs are affected, how long the service is down and whether it is reported down, whether in-flight operations fail or are left in a transitional state, and whether init_host reconciles state on start. |
Nova deep dive §5, Reconfigure Nova §5, Risk register (R1) |
Whether a nova_ssh restart interrupts resizes or cold migrations that are copying disks. |
Nova deep dive §6, Reconfigure Nova §5 |
Whether API calls and instance actions fail while the nova API, scheduler and conductors restart; whether the scheduler SIGHUP is harmless; whether discover_hosts affects existing instances. |
Reconfigure Nova §5, Nova deep dive §6, Tags §3.1 (row nova) |
Whether the recursive chown of /var/lib/nova at every nova_compute start also enters the shares mounted under /var/, and how start time grows with the number of files. |
Nova deep dive §5, Kolla images §6 (item 10), Kolla images §7 |
Upstream Nova behaviour when a dependency is missing: MariaDB, RabbitMQ liveness reporting, Keystone, Placement (whether scheduling stops while it restarts), host mappings, the network-vif-plugged event, libvirt. |
Nova deep dive §3, Tags §3.1 (row <svc> API service) |
| Whether instances can be scheduled on a fresh compute before the openvswitch and neutron plays have finished there. | Execution flow §2.4, Component dependencies §6, Findings (F13) |
Whether live migration to a new compute fails while, with libvirt_tls, the other computes' /etc/hosts lacks it. |
Add a compute node §2, fact sheet E §3 |
Whether restarts that touch none of the containers the study traces to running VMs (the common containers, the control-plane services; "none" in Tags §3.1) affect running VMs, APIs or traffic another way. |
Tags §3.1, Apply common changes (fluentd, cron, kolla_toolbox) §3, Apply common changes (fluentd, cron, kolla_toolbox) §4, Reconfigure a single service §3, Limits of the study, Risk register (R2) |
The effects of a bootstrap-servers re-run on a live compute (Q13), and the runtime effects of the baremetal role's limitations. |
Open questions, Known bugs and limitations (row "baremetal") |
Network data plane¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
neutron_openvswitch_agent restart (Q2): whether flows in OVS persist while the agent is down, whether and how the agent resyncs and re-installs flows, and whether a VM loses packets. |
Reconfigure Neutron §4, Tags §3 and §3.1 (rows neutron), Component dependencies §4, Risk register (R3, R4, R5), Open questions, Kolla images §5 (row neutron-openvswitch-agent) |
openvswitch_vswitchd restart or recreate (Q3): whether the OpenFlow tables are lost, whether the OVS agent or ovn-controller reprograms them, and whether all VM traffic on the host is interrupted, and for how long. |
Tags §3 (row openvswitch), Tags §3.1 (row openvswitch), Update container images §4, Risk register (R7, R15), Open questions, Kolla images §5 (row openvswitch-vswitchd), Kolla images §6 (item 13), Kolla images §7 |
neutron_l3_agent restart (Q4): whether the namespaces persist without the HA cleanup, whether keepalived/haproxy child processes die with the container, whether with the cleanup the routers on that node lose north-south and floating-IP connectivity until rebuilt, and whether HA routers fail over. What neutron-netns-cleanup itself does is neutron code. |
Reconfigure Neutron §4, Risk register (R6), Findings (F12), Open questions, Component dependencies §4, Kolla images §5.3 |
neutron_dhcp_agent restart: whether dnsmasq processes die with the container, whether lease renewals fail during the window, whether new ports wait for the agent, and whether qdhcp- namespaces persist. |
Reconfigure Neutron §4, Component dependencies §4, Kolla images §5 (row neutron-dhcp-agent) |
neutron_metadata_agent restart: whether metadata requests fail during the window. |
Reconfigure Neutron §4 |
neutron_server restart: whether existing traffic is unaffected, and whether port bindings and new instances fail during the window. Whether the RPC layer tolerates agents that start before the server. |
Reconfigure Neutron §4, Execution flow §2.5 |
| Whether the processes that agent containers spawn (dnsmasq, keepalived, haproxy for metadata) die with the container and are respawned by the agent; the DHCP and router outage per agent restart, and whether router and DHCP traffic is interrupted on network nodes. | Component dependencies §4, Risk register (R4, R5) |
OVN: whether an ovn-controller restart re-programs flows on the host, and whether Kolla-Ansible's direct ovn-ctl call creates and upgrades the DB schema. |
Tags §3.1 (row ovn), Reconfigure a single service §3, Kolla images §5 (row ovn-nb-db-server), Kolla images §7 |
Whether rules in the old iptables backend are migrated when neutron_legacy_iptables changes. |
Kolla images §5 (row neutron-base) |
Whether VM network traffic stops under kolla-ansible stop. |
Risk register (R15) |
Storage and images¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
iscsid restart: whether existing sessions are kernel-held, and whether attach/detach during the window fails. |
Tags §3.1 (row iscsi), Reconfigure a single service §3 |
tgtd restart: whether tgtd loads targets.conf itself or keeps targets only in memory, and so whether a restart cuts disk I/O for VMs that use LVM volumes from that host. |
Tags §3.1 (row iscsi), Kolla images §5 (row tgtd) |
Whether removing the iscsi_info volume produces a new IQN at the next start, and whether that breaks backend ACLs. |
Kolla images §6 (item 19) |
| Glance: whether snapshots in progress fail (R20); whether existing file-store images stay readable, and boots from them still work, after Swift is enabled (R21, Q12). | Risk register, Open questions |
Whether a [barbican] section rendered before Barbican exists has any effect before encrypted volumes are used. |
Execution flow §2.2 |
Control plane: HAProxy, MariaDB, RabbitMQ, Keystone, memcached, Heat¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
| HAProxy restart (Q5): whether keepalived drops or moves the VIP, and for how long; whether the node goes to FAULT; whether established connections are cut; whether nova and neutron get DB errors and instance metadata requests fail meanwhile. How HAProxy treats backends that are not up yet on a first deploy. The effect of the haproxy upgrade bugs. | Risk register (R8), Tags §3 and §3.1 (rows haproxy), Component dependencies §6, Open questions, Known bugs and limitations (row "haproxy"), Kolla images §6 (item 15) |
MariaDB (Q6): whether node 0's restart wave runs first; after a failed WSREP wait (the next wave still restarts its nodes CODE, R9), whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and how large the quorum and availability risk is. Whether the DB containers come back as a non-primary cluster after a full stop, and whether a manual start left with --wsrep-new-cluster bootstraps a new cluster. |
Risk register (R9), Tags §3.1 (row mariadb), fact sheet B §1.5, Known bugs and limitations (row "mariadb"), Open questions, Kolla images §6 (item 16), Kolla images §7 |
| RabbitMQ: whether nova-compute heartbeats are lost and services are reported down during the upgrade outage (R10); whether, during a one-by-one restart, nova-compute and the agents lose RPC, RPC-backed API calls time out and the agents reconnect by themselves; the consequences of a changed Erlang cookie. | Risk register, Tags §3.1 (row rabbitmq), Known bugs and limitations (second row rabbitmq), Kolla images §6 (item 9), Kolla images §7 |
Keystone: whether authentication, and so the other services' API requests, fails while keystone restarts or is down (R19); Fernet key distribution failure modes (Q11); whether Keystone still reads a removed policy file after the restart; the upstream semantics of keystone-manage bootstrap; whether the keystone_fernet container restarts when key 0 is missing. |
Risk register, Tags §3.1 (row <svc> API service), Component dependencies §1, Nova deep dive §3, Reconfigure a single service §1.1, fact sheet B §3.10, Open questions, Kolla images §3.5, Kolla images §5 (row keystone-fernet), Kolla images §7 |
| memcached: whether cached tokens are lost when the membership changes or the container restarts, and the effect on auth latency and Horizon sessions. | Risk register (R4), Component dependencies §1, Tags §3 and §3.1 (rows memcached / memcache), fact sheet A §3.8, Kolla images §5 (row memcached), Kolla images §6 (item 14) |
Heat: whether stack-domain operations fail authentication after a changed heat_domain_admin_password. |
Known bugs and limitations (row "heat"), Kolla images §6 (item 7), Kolla images §7 |
The impact on the Neutron API of the upgrade gaps; the runtime effects of the swift limitations; the effect on the masakari instancemonitor. |
Known bugs and limitations (rows "neutron", "swift", "masakari") |
Upgrades and database migrations¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
Whether neutron-db-manage … upgrade head, run by a tag change applied with deploy, applies contract migrations under live old-version servers, and what that does to them. |
Update container images §4, Risk register (R12) |
Whether glance-manage db_load_metadefs, without --merge, only adds namespaces that are not yet in the DB. |
Kolla images §4 (row glance-api) |
Signals, the container runtime and image start behaviour¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
dumb-init signal forwarding: whether --single-child forwards SIGTERM to the direct child only, and what happens to grandchildren; the same for mysqld_safe under the MariaDB image's own entrypoint. |
Kolla images §2.6, Kolla images §7, Nova deep dive §5 |
Whether services started by a shell wrapper that does not exec them (neutron-l3-agent, haproxy, ovs-vswitchd, ovsdb-server) receive SIGTERM at all before the container stops. |
Component dependencies §4, Risk register (R24), Findings (F3), Kolla images §2.6, Kolla images §5.3, Kolla images §6 (item 12) |
Whether a container whose kolla_set_configs fails at start then restarts in a loop. |
Kolla images §2.3, Kolla images §3.3, Kolla images §7 |
| Whether the generated CA trust bundle keeps the last custom CA once it is removed. | Kolla images §2.4, Kolla images §3.5, Kolla images §6 (item 4), Kolla images §7 |
Whether the SSH clients accept the new nova_ssh and keystone_ssh host keys after a recreate without failing. |
Kolla images §6 (item 18) |
| Whether xtrace writes one-shot secrets (the heat domain admin password, the MariaDB root password) to the container's stderr. | Kolla images §2.2 |
| Horizon: how long a start that regenerates the static assets takes. | Kolla images §5 (row horizon), Kolla images §5.2, Kolla images §7 |
| fluentd: whether the tail read positions survive a config-only restart, and whether they are lost on a recreate. | Apply common changes (fluentd, cron, kolla_toolbox) §3, Kolla images §5 (row fluentd) |
| The cost of the recursive chown on large trees (glance store, NFS instance directory, MariaDB datadir, RabbitMQ mnesia). | Kolla images §6 (item 10), Kolla images §7 |
| What the image scripts do against real services and data. | Limits of the study, Kolla images §1 |
Predicting a run¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
Whether, on an unchanged cloud and apart from the preserve_properties exception, every task that notifies a restart handler reports ok, so that nothing restarts. |
Tags §3, Tags §3.1, Reconfigure Nova §3, Reconfigure a single service §2, Find it fast (row "Will containers restart or be recreated?") |
Whether a check-mode run (--check --diff) fails or gives partial results (Q9). |
Predict and inspect a run, Open questions, Tags §2.6 |
| The number of one-shot containers of a run without a tag. | Findings (F5) |
Timings, counts and scale¶
| Not established by the sources | Marked NOT AVAILABLE in |
|---|---|
| Task-result totals per compute, per controller and per fleet, and the remote invocations of a full run. | Compute scaling §2, Compute scaling §2.1, Compute scaling §4.2, Compute scaling §6, Compute scaling §8, Execution flow §2.4, Reconfigure Nova §4, Add a compute node §2, Why a small change creates a large workload |
| Container counts of an initial deploy. | Execution flow §6 |
The latency of one remote invocation (t_inv), the render time of a template (t_render), play and run durations, and real timing at scale (Q7). |
Compute scaling §4.1, Compute scaling §6, Open questions |
The CPU cost per host of the group_by loop, and how many attempts the until/retries loops need. |
Compute scaling §1 |
| Whether, and at which forks value, the deploy host's CPU becomes the bottleneck; whether forks overlap SSH and remote-execution latency in production; the share of run time taken by the compute-wide plays. | Compute scaling §6, Compute scaling §7 |
| Image sizes and registry throughput (Q8). | Compute scaling §4.2, Open questions |
How close together fleet-wide restarts land in time, and the duration of the delegated fact loop under --limit. |
Compute scaling §6 |
In the fact sheets¶
The fact sheets mark their own NOT AVAILABLE points inline. Fact sheets C and D also have their own NOT AVAILABLE list, and fact sheet E a register of open questions.
| Fact sheet | Own list | Sections that contain NOT AVAILABLE marks |
|---|---|---|
| A — infra, load balancing, OVN | – | §1.8, §1.9, §2.4, §3.8, §4.7, §4.8, §5.8, §6.3, §6.8, §7.3, §8.3, §9 |
| B — MariaDB, RabbitMQ, Keystone | – | §0, §1.3, §1.5, §1.8, §2.2, §2.4, §2.5, §2.8, §3.6, §3.7, §3.9 |
| C — Glance, Placement, Cinder, Horizon, Heat | §8 | §0, §1.4, §1.6, §1.7, §1.8, §2.8, §3.8, §4.8, §5.4, §5.6, §5.8, §6, §8 |
| D — Barbican, Octavia, Swift, Manila, compute-side roles | §8 | §1.6, §2.3, §2.5, §2.6, §3.4, §3.5, §3.8, §4.3, §5.0, §5.1, §5.2, §5.3, §5.5, §5.6, §5.7, §5.8, §5.9, §5.10, §5.11, §5.12, §5.13, §6.2, §7, §8 |
| E — host bootstrap, operational playbooks | §14 | §0, §1, §1.2, §1.3, §2.2, §2.3, §2.4, §3, §4, §6, §7, §9, §13, §14, §15 |
Related¶
- Open questions — the open questions Q1 to Q14, one section each, with why each one matters
- Limits of this study — why these points are open: source code only, no timings, no container or VM behaviour used as evidence
- Conventions and evidence labels — the definition of NOT AVAILABLE
- Kolla images §7 — what is not available from the kolla image sources
- Known bugs and limitations — the bugs whose effect the Status column marks NOT AVAILABLE
- Risk register — the risks whose VM impact and API impact rows carry the NOT AVAILABLE marks listed here
- Troubleshooting — the symptom index
To remember
Lumo: if you keep only 5 things from this page, keep these.
- This page lists what the sources do not establish; nothing here is an answer.
- Whether running VMs survive a
nova_libvirtrestart or recreate is NOT AVAILABLE - Whether an
openvswitch_vswitchdrestart interrupts all VM traffic on the host, and for how long, is NOT AVAILABLE - Whether a check-mode run (
--check --diff) fails or gives partial results is NOT AVAILABLE - Play and run durations, and real timing at scale, are NOT AVAILABLE
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: this is the last page of the journey. Well done for coming this far!
Lumo: nicely done! One more page on your journey.