Skip to content

Troubleshooting

A white stone aqueduct bridge spans turquoise water between green island cliffs.

Where to look when a kolla-ansible run did something unexpected, or did nothing. Each row of the symptom index names what an operator sees, restates what the study says the code does, and links to the section that establishes it. The page also says where to look for one component, what is already known as a bug or as an open point, and which checks the study names.

Important

The study rests only on reading source code: kolla-ansible 2024.1-eom, the kolla image repository 2024.1-eom, ansible-collection-kolla 2024.1-eom and the ansible-core 2.16.20 source. Nothing was run against a host, there are no timings, and every statement about data-plane or VM impact is NOT AVAILABLE (Limits of this study). All conclusions come from the 2024.1 implementation, not from 2023.1 or from newer releases. Where a row says "no longer", "fixed" or "changed in 2024.1", it keeps a symptom that the 2023.1 code produced and says what the 2024.1 code does instead; "new in 2024.1" marks a symptom that the 2023.1 code did not produce. Docker is the default engine; the Podman paths new in 2024.1 were read in the code only.

During an incident, the portal can tell what Kolla will do given the inputs: tasks and restarts. What that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE in these sources (Code guarantees and environment dependence).

This page is an index. It adds no diagnosis and no procedure of its own.

Symptom index

Three mechanisms explain almost every surprise on the operation pages: config drift means a full container stop/start, never a reload; loop-notify, where one changed container in a role's loop restarts every container of that role on that host; and global inputs, which are rendered into per-host files on every host (Three mechanisms that explain almost every surprise).

In the tables, "What the code does" restates what the sections linked under "Read" say, and keeps their evidence labels. Rows that restate a fact sheet carry the words of the fact sheets, FACT and NOT AVAILABLE, which name the same two levels as CODE and NOT AVAILABLE (Evidence labels); for 2024.1 each such row was checked against the 2024.1 code, and where the code is cited in the row it carries CODE. A sentence without a label restates a sentence that carries none.

Containers restarted that should not have

What you see What the code does Read
A container was stopped and started for a config change; nothing was reloaded A config change is applied only by stopping and starting the container; kolla_container, the module that replaced kolla_docker, has no reload action CODE. The stop timeout is the 60 s of docker_graceful_timeout: containers run under a systemd unit whose stop is <engine> stop <name> -t <docker_graceful_timeout> (group_vars/all.yml:139,151; module_utils/kolla_systemd_worker.py:32), for Docker and Podman CODE. Repository architecture §4.3; Ten things to know (item 2); R24
Containers restarted although their own config did not change; every container of the role restarted on the host When a config or check task loops over a role's services and one item changes, ansible-core 2.16.20 notifies the handlers of all executed items: a change for one container restarts every container of that role on that host CODE. Tags §2.4; F1
nova_libvirt and nova_ssh restarted on every compute after a change that concerned only nova-compute With the default Nova settings, a nova-compute-only override restarts nova_compute, nova_libvirt and nova_ssh on every compute host CODE. Kolla's own handler ordering restarts libvirt before nova-compute CODE. Tags §2.4; Reconfigure Nova §4; R1; Nova deep dive §2.4 (the other triggers)
nova_api restarted after a scheduler-only override A scheduler-only override rewrites nova-scheduler/nova.conf only and restarts nova_scheduler and, by loop-notify, nova_api CODE. Reconfigure Nova §3; Tags §2.4
HAProxy restarted during a run with a service tag such as -t keystone Handlers ignore tags CODE: the haproxy-config task, imported under tags: always through loadbalancer-config, notifies Restart haproxy container, a listen topic of the loadbalancer role's handlers, which is why -t keystone can restart HAProxy. HAProxy restarts on every loadbalancer host if the rendered services.d/<svc>*.cfg changed (or, new in 2024.1 and only with the single external frontend, external-frontend-map): the backup hosts first, all at once, then the host that holds the VIP CODE. Tags §2.2; Reconfigure a single service §3; Reconfigure a single service §1.2 (--skip-tags haproxy); R8
HAProxy was restarted, not reloaded, on all haproxy hosts at the same time HAProxy is restarted instead of reloaded CODE. Every haproxy-relevant change goes to Restart haproxy container: a change in any one service's services.d/<svc>.cfg, a cert, haproxy.cfg or haproxy_run.sh. The restart is staged, not simultaneous: the handlers find the host that holds kolla_internal_vip_address, stop and start haproxy on every backup host at once and wait for haproxy_monitor_port, then do the same on the VIP holder (roles/loadbalancer/handlers/main.yml:9-34, :54-118, :179-253) CODE. Known bugs and limitations (row haproxy); fact sheet A §1.8 (A); R8; HAProxy + Keepalived
fluentd, cron and kolla_toolbox restarted during an unrelated run such as -t nova Not from a service-tagged run in 2024.1: common is its own play, tagged common only, and no service play runs it (site.yml:88-103) CODE. Enabling a service that appears in fluentd_input_openstack_services or cron_logrotate_services (most OpenStack services, e.g. barbican, roles/common/defaults/main.yml:157-158, roles/common/tasks/config.yml:215) changes td-agent.conf and/or logrotate.conf on every host of the common group. The common containers then restart at the next -t common or untagged run, whatever that run was meant to change. New in 2024.1, enable_redis (which the Octavia jobboard needs) and enable_letsencrypt are among those flags. Apply common changes §3; R2; Repository architecture §5.1
kolla_toolbox restarted although only the fluentd or the cron config changed Any drift restarts all three common containers on that host (loop-notify in check-containers.yml) CODE. Apply common changes §3 ("What restarts")
neutron_openvswitch_agent restarted on all computes and network nodes after an ml2_conf.ini override meant for the server No longer, as in 2023.1 CODE: ml2_conf.ini is rendered only for neutron-server and neutron-infoblox-ipam-agent, and openvswitch_agent.ini does not merge it, so the override restarts neutron_server on the controllers (plus neutron_infoblox_ipam_agent where enabled) and no agent on the computes or network nodes (roles/neutron/tasks/config.yml:159-179, :198-213). An override of neutron/openvswitch_agent.ini restarts neutron_openvswitch_agent on all computes and network nodes; on network nodes the dhcp, l3 and metadata agents restart too (loop-notify in check-containers.yml) CODE. Reconfigure Neutron §3; R5; F11
Nova and neutron containers restarted on all computes after a controller was added, removed or replaced Changing the rabbitmq membership changes the transport URL everywhere, and changing memcached changes the memcache server lists. In the default inventory both are children of control, so a controller change does both. A memcached-only change does not touch nova-compute (the compute nova.conf has no memcache list) but still restarts the neutron agents on computes (neutron.conf). Change control-plane membership §1.1; Change control-plane membership §2; R3; R4
Fleet-wide restarts after an enable_* flag changed, for example enable_barbican: yes enable_barbican: yes, with Barbican not even deployed, rewrites nova.conf everywhere (adds [barbican], roles/nova/templates/nova.conf.j2:217-218, roles/nova-cell/templates/nova.conf.j2:259-260): all nova containers, libvirt included, restart on the next run of the nova plays CODE. It also changes td-agent.conf and logrotate.conf, so fluentd, cron and kolla_toolbox restart on every host of the common group when the common play next runs (-t common or an untagged run) CODE. Reconfigure Nova §3; R2; F6; Why a small change creates a large workload
A container restarts on every deploy/reconfigure, for example prometheus_server; kolla_set_configs --check exits 1 on every run kolla_set_configs --check exits 1 on every run for a config_files entry with preserve_properties and no perm, and for a directory source that contains subdirectories (kolla set_configs.py is unchanged from 2023.1). So prometheus_server restarts on every deploy/reconfigure whenever {{ node_custom_config }}/prometheus/extras/ holds files CODE. Tags §3; Kolla images §3.1; Kolla images §6 (item 2); F14
A container restarted on a run where nothing was changed A restart handler runs only when a task that notifies it reports changed CODE. check-containers reports changed only when the container is missing, is not running (Podman), differs from the task in a compared parameter, has a missing or different systemd unit file, or kolla_set_configs --check reports drift: exit 1, or with Docker also 137 or a client error CODE. NOT AVAILABLE Whether, on an unchanged cloud and apart from the exception of the row above and the first 2024.1 run (next row), every task that notifies a restart handler reports ok, so that nothing restarts. Tags §3; Repository architecture §4.3; What is not known, "Predicting a run"
New in 2024.1: the first deploy or reconfigure after the upgrade restarted containers whose config had not changed The systemd unit template changed in 2024.1 (the start-limit line is StartLimitInterval= where 2023.1 wrote StartLimitIntervalSec=; the template now names the engine, but with Docker those lines read as in 2023.1), and check_unit_change compares the whole unit text, so a container whose unit 2023.1 wrote, and that no 2024.1 run has recreated or restarted yet, reports drift at its first 2024.1 check and is restarted, its unit rewritten first; loop-notify then restarts the rest of its role on that host (module_utils/kolla_systemd_worker.py:20-39, :154-167) CODE. With the RabbitMQ default flipped to quorum queues, rabbit_quorum_queue = true is also rendered into the [oslo_messaging_rabbit] section of 26 templates in 24 roles, computes included, on a cloud that kept the 2023.1 default (group_vars/all.yml:268) CODE. F16; F17; Predict and inspect a run §2; Pre-execution checklist §2
A container that was stopped, or that keeps exiting, was restarted by the run With Docker, kolla_container treats as drift an exit code 137 of the check (the exec was killed because the container exited) and a client error (container stopped or restarting) (module_utils/kolla_docker_worker.py:155-163, :173-177): a stopped container is restarted by the next run CODE. New in 2024.1, with Podman a container that is not running is drift before any exec (module_utils/kolla_podman_worker.py:432-433), but an exit code of 137 fails the task (next table) CODE. Kolla images §3.3
After an image pull, the other containers of the role restarted too After a pull that downloaded something, the next deploy, reconfigure or upgrade recreates the affected containers, and loop-notify restarts their siblings CODE. With only the nova-compute image rebuilt, -t nova deploy after that pull recreates nova_compute and restarts nova_libvirt and nova_ssh on every compute (default Nova settings) CODE. Update container images §1.1; Update container images §2
A config change written earlier by genconfig, or by hand, was applied by a later, unrelated run A config file changed by an earlier genconfig, or by hand, still restarts the container on the next deploy/reconfigure/deploy-containers CODE. Between genconfig and the last deploy-containers batch, any other Kolla run that reaches those hosts applies the pending drift in files listed in config_files CODE. Repository architecture §4.3; Controlled batches §2 (caveats)
A container picked up new config after a Docker restart or a reboot, without a Kolla run Containers started again by their systemd unit after a container engine restart or a reboot also pick up the new config, because COPY_ALWAYS copies the files and rewrites /run_command at every container start CODE. Controlled batches §2 (caveats); Kolla images §3.5; fact sheet E §2.4
With COPY_ONCE, the container was recreated, not restarted COPY_ONCE: /configured lives in the container layer, so only a recreate re-applies config. kolla_container always recreates under COPY_ONCE (Docker module_utils/kolla_docker_worker.py:353-361; Podman module_utils/kolla_podman_worker.py:507-524) CODE. Kolla images §3.5
After a HAProxy restart, frontends changed that the run had not rendered The restart re-reads all services.d/* files: any drift left in /etc/kolla/haproxy/ by earlier runs (e.g. a previous genconfig, or stale files, which haproxy-config never removes, roles/haproxy-config/tasks/main.yml:2-23) goes live too CODE. fact sheet A §1.9; HAProxy + Keepalived
genconfig restarted services on Hyper-V hosts Does not apply to 2024.1: genconfig writes config but does not restart any container (every restart handler carries when: kolla_action != "config"), and there is no nova-hyperv role in ansible/roles/ CODE. Pre-execution checklist (step 1); Execution flow §4 (row genconfig)
openvswitch_vswitchd restarted although only openvswitch_db had drifted check-containers loops over both OVS services (roles/openvswitch/tasks/check-containers.yml:2-18), so with any openvswitch_db-only drift (e.g. a start-ovsdb-server change or a db image, volume or dimension change) loop-notify also notifies the vswitchd handler, which runs after the db restart and wait (roles/openvswitch/handlers/main.yml:2-29) CODE. R7; Tags §2.4
mariadb_clustercheck restarted on all nodes at once The clustercheck restart is not rolling: the handler runs in the role play, not in the batched restart plays, and restarts mariadb_clustercheck on all notified nodes at once (roles/mariadb/handlers/main.yml:75-91) CODE. With clustercheck enabled (default: follows enable_haproxy), HAProxy checks the backends on the clustercheck port (roles/mariadb/defaults/main.yml:64, :132). NOT AVAILABLE: whether HAProxy marks every backend down when that restart outlasts the check settings, which would mean a short total DB outage through the VIP. fact sheet B §1.8 (item 2); MariaDB/Galera
keepalived restarted after a host was inserted, removed or re-ordered in the haproxy group, or after a CA change keepalived priority is the index in groups['loadbalancer'] + 1 (roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15) CODE. Inserting, removing or re-ordering a loadbalancer host changes keepalived.conf on every host after it in the list, and each changed host goes through Restart keepalived container: that also stops and starts haproxy (and proxysql) on each notified host, backups first, and stops the master's keepalived last, so the VIP moves CODE. How long the VIP is unavailable is NOT AVAILABLE. A CA bundle change (e.g. a root CA rotation with kolla_copy_ca_into_containers: yes) notifies the restart handlers of haproxy, proxysql and keepalived on the loadbalancer hosts (roles/loadbalancer/tasks/copy-certs.yml:2-6, roles/service-cert-copy/tasks/main.yml:2-12), with the same staged sequence CODE. fact sheet A §1.8 (C); fact sheet A §7.3; R22; HAProxy + Keepalived
keystone restarted after a memcached, rabbitmq or keystone node was added or removed keystone.conf embeds every memcached host and the RabbitMQ transport URLs (roles/keystone/templates/keystone.conf.j2:4, :51, :54), and the fernet crontab and fernet-push.sh embed the keystone host list and index (roles/keystone/tasks/config.yml:195-200, roles/keystone/templates/fernet-push.sh.j2:7-22) CODE. Adding or removing a memcached, rabbitmq or keystone node therefore restarts keystone, and keystone-fernet as well for keystone node changes. fact sheet B §3.9 (item 2); R19
All DB nodes restarted after the Galera membership changed Galera membership changes restart all DB nodes: galera.cnf lists every shard host (roles/mariadb/templates/galera.cnf.j2:29) CODE. The restart runs in the separate Restart mariadb services play, in serial: "33%" batches (mariadb.yml:18-34) CODE. Change control-plane membership §2; fact sheet B §1.8 (item 11); R9
The RabbitMQ brokers restarted one by one after the group was scaled or a password changed Scaling the rabbitmq group changes rabbitmq.conf (the classic_config node list, roles/rabbitmq/templates/rabbitmq.conf.j2:18-21) on every broker, and a password change changes definitions.json; the config tasks notify Restart rabbitmq container, which only groups the host (roles/rabbitmq/handlers/main.yml:2-4) CODE. The restart then runs in the Restart rabbitmq services play, serial: "33%" (one broker at a time with three), each broker put into maintenance mode (drained) first (rabbitmq.yml:30-49, roles/rabbitmq/tasks/restart_services.yml:9-39) CODE. fact sheet B §2.8 (items 4 and 5); RabbitMQ
New in 2024.1: etcd members restarted in waves, the leader last The etcd play has no serial any more (site.yml:470-481), and its handlers are include_tasks with listen: outside upgrade and config, the non-leaders restart in four waves by index % 4, then the leader (roles/etcd/handlers/main.yml:40-65) CODE. On upgrade, the handler at :28-36 loops over groups.etcd without using item, so each non-leader includes the restart tasks once per member; its comment says "one by one", but the order across hosts is NOT AVAILABLE. ANSIBLE_SERIAL does not batch the etcd play. Change control-plane membership §2; Execution flow §5
Every heat-engine restarted in the same step FACT: with serial 0, every heat-engine restarts in the same handler flush on any heat.conf change; a memcached or rabbit membership change, a password rotation or an openstack_service_workers change is enough. fact sheet C §5.8
glance_api was recreated after the Cinder iSCSI/LVM backend was enabled With enable_cinder, enabling enable_cinder_backend_iscsi (or cinder_backend_ceph) flips glance-api to privileged, and the iSCSI backend also adds the iscsi_info volume and /dev (roles/glance/defaults/main.yml:10, :202-203), so glance-api is recreated CODE. fact sheet C §3.8 (item 7)
Every container of several services restarted after a CA or backend certificate rotation A CA or backend cert rotation restarts every container of every service using it: service-cert-copy loops over the role's services and notifies Restart <service> container for each (roles/service-cert-copy/tasks/main.yml:2-56) CODE; whole fleet, per-play parallel. fact sheet A §7.3
All OVN DB members restarted at once; ovn_controller restarted on all computes and network nodes in parallel OVN is two roles, ovn-db and ovn-controller. Any change to ovn-nb-db/ovn-sb-db restarts every member at once under serial 0 (the ovn-db play's serial is kolla_serial, default 0, site.yml:662; roles/ovn-db/handlers/main.yml:2-45) CODE. ovn-controller restarts happen on image or container-parameter change or when a deploy/reconfigure rewrites its config.json, and hit all its hosts in parallel. fact sheet A §5.8 (A and C); OVN
On ovs-dpdk hosts, ovs-vswitchd restarted after the ovsdpdk-db restart Any ovsdpdk-db restart (an image change or config.json drift) also restarts ovs-vswitchd, on all DPDK hosts in parallel: the db restart handler notifies Restart ovsdpdk-vswitchd container (roles/ovs-dpdk/handlers/main.yml:2-22) CODE. Changed in 2024.1: the same handler notifies Wait for dpdk tunnel ip, spelled like the handler (:21, :88), so the 2023.1 missing-handler error is gone and the wait runs (next tables) CODE. NOT AVAILABLE: whether the restarts are a data-plane outage for all VMs on that host. fact sheet A §6.8; Open vSwitch
The Docker daemon restarted during bootstrap-servers The work comes from ansible-collection-kolla (kolla-host.yml:4-14 applies openstack.kolla.baremetal), which installs the engine named by kolla_container_engine (ansible-collection-kolla roles/baremetal/defaults/main.yml:37, roles/baremetal/tasks/install.yml:38). With Docker, Restart docker is a handler, notified when the /etc/docker/daemon.json write reports changed (the rendered content differs, or the mode is not 0644), when the registry CA file changes, and through Reload docker service file when the docker.service.d/kolla.conf drop-in is written or removed (ansible-collection-kolla roles/docker/tasks/config.yml:9-114, roles/docker/handlers/main.yml:2-15) CODE. It is flushed at Flush handlers (roles/docker/tasks/config.yml:116-117), so it runs at the same moment on every host in the batch, and the batch is all hosts by default (kolla-host.yml:6) CODE. fact sheet E §0 (item 1); fact sheet E §2.4; R14; Q13
A service restarted on every host at once instead of one host at a time Without batching, handlers restart a service on every host of the play at once. Only MariaDB and RabbitMQ roll by themselves: their restarts run in separate plays with serial: "33%" (one node at a time with three nodes), and each RabbitMQ node is drained first CODE. New in 2024.1, etcd restarts its non-leaders in waves inside its handlers (row above) CODE. Ten things to know (item 4); Execution flow §5; Execution flow §7; Controlled batches §1.2
Containers restarted in an order that is not the order of the changes Within a role, restart order is handler definition order, not notification order CODE: nova-cell restarts nova_ssh, then nova_libvirt, then nova_compute (roles/nova-cell/handlers/main.yml:74, :92, :151); openvswitch restarts openvswitch_db, waits, then restarts openvswitch_vswitchd (roles/openvswitch/handlers/main.yml:2, :20, :29). Execution flow §7; Tags §5

Nothing happened, or the change was not applied

Rebuilt images that a run did not pick up are under Images.

What you see What the code does Read
-t common reconfigure changed nothing: a fluentd, cron or kolla_toolbox change was not applied (no-op) Does not hold in 2024.1 CODE, as in 2023.1: common is its own play with play-level tags: common (site.yml:88-103), and the tasks of the dynamic include in roles/common/tasks/main.yml:2 inherit that tag through their parent block (ansible-core 2.16.20 lib/ansible/playbook/block.py:352-361, lib/ansible/plugins/strategy/linear.py:271-273). -t common reconfigure reconfigures fluentd, cron and kolla_toolbox on every host of the common group. If nothing changed, check that the change is really in the rendered files (genconfig -t common, which in 2024.1 can pull the fluentd image, see Images). Apply common changes §2; Apply common changes §1.1 (the -t common command); Tags §2.1; F2; R25
The common change was applied later, by a run that was not meant to apply it common changes (fluentd/cron/toolbox) are applied only by a run that includes the common play: -t common or an untagged run, on every host of the common group. A service-tagged run never applies them CODE (site.yml:88-103). F2 ("Why it matters"); R25; Apply common changes §3
kolla_set_configs --check returns 0 and nothing restarts after a change to a container's command line, for example a memcached setting; genconfig then deploy-containers did not apply it --check compares only the config_files entries of config.json: a change confined to config.json itself (command, permissions; all of memcached's settings) is never detected CODE. A normal deploy/reconfigure applies it, because the changed template task notifies the restart handler; once genconfig has written it, neither deploy-containers nor a later run applies it until the container restarts for another reason. F14; Kolla images §3.2; Kolla images §6 (item 1); Controlled batches §2 (caveats)
An OVN command-line change was not applied FACT: the OVN manifests have no config_files, so kolla_set_configs --check always exits 0: a command-line change written by genconfig is applied neither by deploy-containers nor by a later run, until the container restarts for another reason. fact sheet A §5.8 (C)
A file that was removed from the config is still in the container after the restart A restart removes most dropped files CODE: every start deletes each recorded destination that did not exist at the container's first start, and restores the others to their original. Exceptions CODE: glob sources are never recorded (e.g. prometheus extras/*), so their destinations stay; a destination on a volume that already held a file when the container was created is restored rather than deleted; nothing happens after the first start under COPY_ONCE. --check does not flag dropped entries, so the removal waits for a restart for another reason CODE. Deleting a custom file such as /etc/kolla/config/keystone/keystone-paste.ini also leaves its rendered copy in /etc/kolla/keystone/, which the optional keystone-paste.ini entry copies in again at every start (roles/keystone/templates/keystone.json.j2:18-24) CODE. Kolla images §3.5; Kolla images §6 (item 4); Repository architecture §7 (item 4)
A custom CA that was deleted is still installed in the containers A CA deleted from kolla_certificates_dir/ca/, or left there after kolla_copy_ca_into_containers is turned off, stays in /etc/kolla/<svc>/ca-certificates/, because the copy task never deletes files (roles/service-cert-copy/tasks/main.yml:2-12) CODE, so it is re-added at every start until it is deleted there by hand. Kolla images §3.5
The HAProxy frontend of a disabled service is still loaded Stale services.d files are never removed CODE: haproxy-config only templates (roles/haproxy-config/tasks/main.yml:2-23); the only removal is mariadb.cfg when ProxySQL is enabled (roles/loadbalancer/tasks/config.yml:60-72). If a service is disabled (enable_X: no), its include_role is skipped; the old /etc/kolla/haproxy/services.d/X.cfg stays and is loaded by haproxy_run.sh on the next restart. New in 2024.1: turning haproxy_single_external_frontend off leaves services.d/external-frontend.cfg in place, and a disabled service's block stays in external-frontend-map (roles/loadbalancer/tasks/config.yml:162-175, roles/haproxy-config/tasks/main.yml:25-53) CODE. Known bugs and limitations (row haproxy); fact sheet A §1.8 (D); fact sheet A §2.4
A Keystone domain config file that was removed is still copied into the container Removing a file from /etc/kolla/config/keystone/domains is never propagated: no task deletes stale files in /etc/kolla/keystone/domains/ (roles/keystone/tasks/config.yml:106-128 only copies). Under the default COPY_ALWAYS the stale file is copied back into the container at every start, even after a recreate, because the domains directory is a listed source (roles/keystone/templates/keystone.json.j2:26-27) CODE. fact sheet B §3.9 (item 8)
A swift config or ring change was not applied by deploy No handlers are notified in the swift role: deploy does not apply config or ring changes CODE; runtime effects NOT AVAILABLE. Known bugs and limitations (row swift); fact sheet D §3.4; fact sheet D §7 (item 4); Swift
A changed heat_domain_admin_password was not applied in Keystone bootstrap_heat creates heat_user_domain/heat_domain_admin only when the domain is absent: a changed heat_domain_admin_password reaches heat.conf but never Keystone (kolla docker/heat/heat-api/extend_start.sh:8-14, roles/heat/tasks/bootstrap_service.yml:22) CODE; impact NOT AVAILABLE. Known bugs and limitations (row heat); Kolla images §6 (item 7)
A changed rabbitmq_cluster_cookie was not applied to the brokers The image writes .erlang.cookie only in the bootstrap container (new volume). A cookie change recreates the brokers (env compare) but they keep the old cookie, while kolla_toolbox gets the new one CODE; consequences NOT AVAILABLE. Known bugs and limitations (second row rabbitmq); Kolla images §6 (item 9)
An ovs-dpdk NIC mapping change was not applied The NIC mapping file is written once CODE: ovs-dpdkctl.conf is generated only if it is absent (roles/ovs-dpdk/files/ovs-dpdkctl.sh:350-353), so mapping changes require deleting it manually. Known bugs and limitations (row ovs-dpdk); fact sheet A §6.8
After -t keystone, the HAProxy backend list of another service was not updated Under -t keystone, other services' .cfg files are not re-rendered: each per-service include_role … tasks_from: loadbalancer of the loadbalancer play carries its own service tag (site.yml:122-323) CODE. If another service's backend list changed (say, a new controller), it is not updated. fact sheet A §1.9
/etc/hosts on the existing hosts does not list the new compute after a run with --limit /etc/hosts on existing hosts is not updated with the new compute, because bootstrap-servers was limited: the block is written by ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:25-41 on the hosts the run reaches only CODE. With --limit, /etc/hosts and other cross-host files are not updated on non-limited hosts. Add a compute node §1.1; Add a compute node §2; R23; fact sheet E §3
An expired certificate was not renewed by a new run of the certificates playbook; a new backend host is not in the backend certificate Test CA only; certs never renewed; new backend hosts not added CODE. Every command step of the certificates role is creates:-guarded (14 of 14 in roles/certificates/tasks/), so re-running never renews expired certificates; the backend certificate's host list (roles/certificates/templates/openssl-kolla-backend.cnf.j2:16) is used only when backend.csr does not exist yet (roles/certificates/tasks/generate-backend.yml:23-48). Known bugs and limitations (row certificates); fact sheet A §8.3
New certificates were generated, but the containers still use the old ones Distribution needs a separate run: certs only reach containers through kolla-ansible -t haproxy reconfigure (pem files, roles/loadbalancer/tasks/config.yml:211-247) or a per-service reconfigure (backend and CA, service-cert-copy) CODE. Each of those means a container restart. fact sheet A §8.3
Neutron's DB sync did not run on reconfigure No longer, as in 2023.1 CODE: reconfigure.yml imports deploy.yml, and bootstrap_neutron has no when, so a reconfigure re-runs Neutron's DB create/grant tasks and its DB sync; neutron_enable_rolling_upgrade is read only by upgrade.yml (roles/neutron/tasks/reconfigure.yml:2, roles/neutron/tasks/bootstrap_service.yml:2-21, roles/neutron/tasks/upgrade.yml:4-8). F5; Reconfigure Neutron §2
The tag service-rabbitmq ran nothing The tag is on the RabbitMQ user/vhost block inside the service-rabbitmq role (roles/service-rabbitmq/tasks/main.yml:44-45), which is reached through dynamic includes: on its own it executes nothing and is not usable as a selector CODE. Tags §3 (row service-rabbitmq)
-t memcache did not update the HAProxy file of memcached -t memcache reconfigures the container but not its HAProxy file; only -t memcached also selects the memcached include of the loadbalancer play (site.yml:233-237). That include renders a frontend only with enable_haproxy_memcached (default no, group_vars/all.yml:737), so with the defaults the two tags differ in nothing visible CODE. fact sheet A §3.8; Tags §3 (row memcached / memcache); Memcached
The nova-reload, nova-api-reload and nova-cell-reload tags did nothing The reload plays run recreate_or_restart of nova services only when kolla_action == upgrade (nova.yml:180-253): a no-op outside upgrade CODE. Tags §3 (row nova-reload)
deploy-containers did not update the HAProxy frontends deploy-containers writes no config, and the per-service loadbalancer includes run only for deploy, reconfigure, upgrade and config (site.yml:324-326): only the loadbalancer role's check-containers.yml runs, and only under -t haproxy (or keepalived, loadbalancer) or no tags CODE. Execution flow §4 (row deploy-containers); fact sheet A §1.9
A frontend change written by genconfig never reached HAProxy, even after -t <svc> reconfigure Recorded in the 2024.1 study CODE: genconfig re-renders services.d for the tagged services (the loadbalancer block runs for config, site.yml:324-326), while every loadbalancer handler is skipped under config (roles/loadbalancer/handlers/main.yml:16-17). A later -t <svc> run finds the file unchanged and notifies nothing, and deploy-containers runs no loadbalancer work. The change reaches HAProxy only through -t haproxy reconfigure or an untagged run, whose check compares services.d in the haproxy container (roles/loadbalancer/tasks/check-containers.yml:2-18), with the staged restart (roles/loadbalancer/handlers/main.yml:9-34). Predict and inspect a run (row "Render configs without restarting"); Controlled batches §2 (caveats); Known bugs and limitations (row haproxy (genconfig))
After the compute batches, config drift is still pending on the controllers genconfig -t nova-cell also renders the conductor and console config on controllers. Include them in a batch too, or the drift stays pending there. Controlled batches §1.1 (caveat)
After a tagged run, some hosts still have the old fluentd or logrotate files After enabling or disabling a service that has a logrotate or fluentd input entry (enable_X), td-agent.conf and logrotate.conf change on the hosts of the common group only when the common play runs: -t common or an untagged run, on every host of that group CODE. A service-tagged run leaves every host with the old files (site.yml:88-103), and fluentd, cron and kolla_toolbox restart only when the common play next runs. R2; Apply common changes §3
mariadb_backup did nothing: every task was skipped The backup container runs only on mariadb_backup_host, the first host of the MariaDB shard group (roles/mariadb/defaults/main.yml:123, roles/mariadb/tasks/backup.yml:33-35), and the playbook applies the role only with enable_mariabackup: yes (default no; mariadb_backup.yml:8-11, group_vars/all.yml:804); otherwise every task is skipped silently CODE. fact sheet E §9; Database operations
deploy -t barbican alone: the barbican_notifications topic is not enabled Running kolla-ansible deploy -t barbican alone does not run the keystone play, so the barbican_notifications topic, which the keystone role enables (roles/keystone/defaults/main.yml:191-192), is not enabled in keystone.conf CODE. NOT AVAILABLE: whether barbican-keystone-listener then receives project-deletion events. fact sheet D §1.6; Barbican
No amphora flavor, image, network or security group exists after an Octavia deploy As in 2023.1 CODE: with octavia_auto_configure (default: true when amphora is in octavia_provider_drivers, group_vars/all.yml:1323), deploy runs prepare.yml, which creates the amphora flavor, the nova keypair, the security groups and rules, and the load-balancer management network and subnet (plus a router for IPv6) (roles/octavia/tasks/deploy.yml:4-5, roles/octavia/tasks/prepare.yml:2-161). The amphora image is not created by the role. With octavia_auto_configure off, none of these is created. What the os_* modules do with resources that already exist with other settings is NOT AVAILABLE: the modules are not in these sources. Known bugs and limitations (row octavia); fact sheet D §2.5; Octavia
No share type, service image or network exists after a Manila deploy Kolla does not create share types, the service image or networks CODE (no such task under roles/manila/tasks/). fact sheet D §4.6; Manila
A service tag did not touch MariaDB, RabbitMQ, memcached or Open vSwitch A service tag never touches MariaDB, RabbitMQ or memcached containers (other plays), and it never touches OVS (unless -t openvswitch): Open vSwitch is a separate play with its own tag (site.yml:611-624), not touched by -t neutron CODE. Reconfigure a single service §2; Reconfigure Neutron §4

The run failed or a task failed

What you see What the code does Read
The check-containers task failed on a host; kolla_set_configs --check exit code 2 Exit 2 means an environment problem, not drift: a config.json that cannot be opened, or a uid or gid with no passwd or group entry; kolla_container then fails the check task on that host instead of restarting (kolla docker/base/set_configs.py:188, :193, :285, :579-581; module_utils/kolla_docker_worker.py:178-181; library/kolla_container.py:427-429) CODE. An optional file rendered for the first time (e.g. nova-compute release) is drift, exit 1, and the restart copies the new file (set_configs.py:166-169) CODE. Kolla images §3.3; Kolla images §6 (item 3); Predict and inspect a run (row "See what will restart"); Controlled batches §1.1
New in 2024.1, with Podman: the check-containers task failed with "Failed to compare container configuration" and an exit code that is not 2, for example 137 The Podman worker treats only rc 0 (no drift) and rc 1 (drift) as answers; any other exit code of the check exec, 137 included, raises and fails the task on that host, where the Docker worker reports 137 as drift (module_utils/kolla_podman_worker.py:446-453; Docker module_utils/kolla_docker_worker.py:169-181) CODE. What makes a Podman exec return 137 is NOT AVAILABLE. Kolla images §3.3; Known bugs and limitations (row kolla_container (Podman))
Every later run fails on the same check task Not for a newly added prometheus extras file: a missing destination is drift (exit 1), and an extras entry with preserve_properties exits 1 on every run, so prometheus_server restarts on every deploy/reconfigure instead of failing CODE. A check task that fails on every run has exit 2 (previous rows), or, with Podman, any exit code other than 0 or 1; a later run does not clear it by itself. Tags §3; Kolla images §3.3
docker exec <c> kolla_set_configs --check run by hand exits 2 In an image with a service USER (e.g. USER nova), a plain docker exec <c> kolla_set_configs --check cannot open the files in /etc/kolla/<service>/, which are owned by root with mode 0660, and exits 2. kolla_container execs as root; by hand, use docker exec -u root (or podman exec -u root). Kolla images §3.4; Predict and inspect a run (row "See what will restart")
After the restart, the container fails at every start Exit 1 is also returned for a required source that is missing and for an invalid config.json; the restarted container then fails in kolla_set_configs at every start CODE. What its systemd unit then does is NOT AVAILABLE. Kolla images §3.3; Predict and inspect a run (row "See what will restart")
deploy or reconfigure fails in the mariadb role after kolla-ansible stop, or while a node is not Synced Reconfigure aborts if the cluster is fully stopped (multi-node) or any alive node is not Synced (roles/mariadb/tasks/lookup_cluster.yml:35-42, :72-78) CODE. kolla-ansible stop stops every node at once, so the next deploy/reconfigure fails until mariadb_recovery is run. fact sheet B §1.8 (item 4); R15; Database operations
The MariaDB restart handler failed while it waited for WSREP sync The wait is a task of the Restart mariadb services play, not a handler: about 60 s (10 × 6 s, roles/mariadb/tasks/restart_services.yml:31-45) CODE; a restart that needs IST or SST longer than that fails. The restart plays set no any_errors_fatal; new in 2024.1 they set max_fail_percentage, default 100 (mariadb.yml:18-52) CODE, which ansible-core 2.16.20 compares with the failures of the current batch only, so it never stops the play early (lib/ansible/plugins/strategy/linear.py:336-348, lib/ansible/executor/task_queue_manager.py:330-338) CODE. A node that fails its wait therefore marks only its own host failed; when that is every host of the batch (always so with three nodes, one host per batch), ansible-core ends the playbook and the later batches are not restarted; when other hosts of the batch succeed, the next batch still restarts its nodes (lib/ansible/executor/playbook_executor.py:204-209, lib/ansible/utils/helpers.py:25-32) CODE. NOT AVAILABLE: whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is. fact sheet B §1.8 (item 3); fact sheet B §1.5 (failure mode); R9; Known bugs and limitations (row mariadb); Q6; Controlled batches §1.3
kolla-ansible pull with outward RabbitMQ enabled: outward_rabbitmq_services No longer applies in 2024.1 CODE: outward RabbitMQ is removed. rabbitmq.yml has three plays, all on the rabbitmq group (rabbitmq.yml:3-70), and pull looks up rabbitmq_services through project_name: "rabbitmq" (roles/rabbitmq/vars/main.yml:2, roles/rabbitmq/tasks/pull.yml:2-3, roles/service-images-pull/tasks/main.yml:14). Known bugs and limitations (first row rabbitmq); RabbitMQ
pull for ovs-dpdk: ovs_services Fixed, as in 2023.1 CODE: pull imports service-images-pull, which reads ovsdpdk_services through kolla_role_name: "ovsdpdk" (roles/ovs-dpdk/tasks/pull.yml:2-3, roles/ovs-dpdk/vars/main.yml:4, roles/service-images-pull/tasks/main.yml:14). Known bugs and limitations (row ovs-dpdk); fact sheet A §6; Open vSwitch
prechecks passed, but deploy failed in the keystone role at set_fact cron_jobs The precheck validates the wrong variable: it asserts fernet_token_expiry <= 604800 (roles/keystone/tasks/precheck.yml:43-53), while the generator is fed fernet_key_rotation_interval, which is fernet_token_expiry + fernet_token_allow_expired_window (roles/keystone/tasks/config.yml:195-200, group_vars/all.yml:977-984) and must not exceed one week (roles/keystone/files/fernet_rotate_cron_generator.py:58-62). The generator exits 0 with {"failed": true} (:31-36, :117), so the command task succeeds and set_fact cron_jobs then errors on a missing key (roles/keystone/tasks/config.yml:207-210) CODE. Any fernet_token_expiry above 5 days with the default window (2 days) passes precheck but breaks deploy. fact sheet B §3.9 (item 5); Known bugs and limitations (row keystone)
New in 2024.1: prechecks failed on the RabbitMQ variables or on the queue types om_enable_rabbitmq_quorum_queues now defaults to true (group_vars/all.yml:266-268). The precheck fails when HA and quorum are equal, so also when both are off (roles/prechecks/tasks/service_checks.yml:67-72), and, when a rabbitmq container exists and quorum is on, it asserts that every queue except reply_ and _fanout ones is a quorum queue, so it fails while classic queues exist CODE. F17; RabbitMQ; Pre-execution checklist §2
New in 2024.1: prechecks failed with "Redis must be enabled when using octavia jobboard" With the amphora provider, enable_octavia_jobboard is true by default (group_vars/all.yml:832), and the octavia precheck asserts enable_redis, which defaults to no (group_vars/all.yml:840; roles/octavia/tasks/precheck.yml:67-73) CODE: with the default settings, Octavia fails prechecks until Redis is enabled or the jobboard turned off. The requirement is checked only by prechecks; what Octavia does when Redis is unreachable is NOT AVAILABLE. Known bugs and limitations (row octavia); Octavia; fact sheet D §7 (item 5)
New in 2024.1: deploy with cyborg enabled, around the Keystone registration of cyborg The cyborg endpoints are built from cyborg_internal_fqdn and cyborg_external_fqdn (roles/cyborg/defaults/main.yml:147-148), which no file of the 2024.1 tree defines, and they feed the registration that deploy runs first (roles/cyborg/defaults/main.yml:169-175, roles/cyborg/tasks/register.yml:2-7, roles/cyborg/tasks/deploy.yml:2) CODE. Whether deploy then fails at cyborg registration unless both variables are defined is NOT AVAILABLE. Known bugs and limitations (row cyborg); fact sheet D §5.9
The swift reconfigure fails The swift reconfigure fails if any listed container is not running (failed_when: not container_state.Running, roles/swift/tasks/reconfigure.yml:32-42), so reconfigure cannot repair a crashed container CODE. fact sheet D §3.4; fact sheet D §7 (item 4)
upgrade: the task "Stopping all slave keepalived containers" with enable_keepalived: no, or with an api_interface whose name contains - Does not exist in 2024.1 CODE: the loadbalancer role's upgrade.yml stops and removes prometheus_haproxy_exporter, then imports deploy.yml, and has no "Stopping all slave keepalived containers" task (roles/loadbalancer/tasks/upgrade.yml:4-21). Known bugs and limitations (row haproxy); fact sheet A §1.8 (F); HAProxy + Keepalived
bootstrap-servers failed on every host at the /etc/hosts task The block loops over all groups['baremetal'] hosts and calls kolla_address for each (ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:25-41), so every baremetal host's facts are required CODE. A baremetal host outside bifrost without gathered facts (for example unreachable during fact gathering, with no fact cache) makes kolla_address raise on every host (kolla-ansible kolla_ansible/kolla_address.py:52-95), which fails the task on every host and ends the play for them. fact sheet E §0 (item 4); fact sheet E §3
A run with --limit failed because a host outside the limit is unreachable With --limit, the limited hosts gather the facts of the other hosts in a delegated loop (gather-facts.yml:34-62). An unreachable other host without cached facts makes its loop item unreachable; ansible-core counts a loop result as unreachable if any item is, and then marks the limited host unreachable and removes it from the play and from every later play (ansible-core 2.16.20 lib/ansible/executor/task_result.py:72-73, :96-106, lib/ansible/plugins/strategy/__init__.py:632-636) CODE. So --limit newnode can fail because an unrelated node is down. fact sheet E §4; fact sheet E §14 (H4)
Templating failed in kolla_address: interface not present on host kolla_address raises FilterError if the interface or address family is missing on the host (kolla_ansible/kolla_address.py:66-95) CODE. Every octavia host must have octavia_network_interface with an address, or templating fails. The setup filter documented as an example in group_vars/all.yml:29 ("ansible_[!qt]*", not the default, :31) also hides real interfaces whose names start with q or t; kolla_address then fails for api_interface, tunnel_interface and so on if they are named that way. fact sheet D §2.6; fact sheet E §4; fact sheet E §14 (H16)
Jinja errors on groups['ovn-…'] There is no group validation precheck: the prechecks of ovn-db and ovn-controller check only free ports (roles/ovn-db/tasks/precheck.yml; roles/ovn-controller/tasks/precheck.yml is empty) CODE, so an old inventory without the ovn-* groups only surfaces as Jinja errors on groups['ovn-…']. fact sheet A §5.8 (H); OVN
-t nova failed at Keystone registration on a fresh cloud Nothing enforces a dependency when you use tags CODE: -t nova on a fresh cloud fails at Keystone registration if Keystone was never deployed. On an existing cloud it assumes MariaDB, RabbitMQ and Keystone are healthy. Component dependencies §6
A bootstrap or registration task failed on the first host of a group Bootstrap and registration tasks target groups[...][0] CODE: the first host of a group is a hidden dependency; it must be reachable and have kolla_toolbox running. Component dependencies §6
mariadb_recovery tried to start mariadb_clustercheck although it is disabled The recovery restart tasks loop over mariadb_services with no item.value.enabled or group check (roles/mariadb/tasks/recover_cluster.yml:172-186, :203-217) CODE. With enable_mariadb_clustercheck: no, recovery still tries to start a mariadb_clustercheck container; if the image pull returns an error, kolla_container fails the task on that host. In deploy and reconfigure, only enabled items are notified (roles/mariadb/tasks/config.yml:42-53, roles/mariadb/tasks/check-containers.yml:13-18). fact sheet B §1.8 (item 6); Known bugs and limitations (row mariadb)
docker.service is masked after a failed or interrupted bootstrap-servers run on Debian/Ubuntu With the Docker engine on Debian/Ubuntu, the run masks docker.service when no unit file exists in /etc/systemd/system/ yet, and unmasks it later, either when the package install changed something or at the final Start and enable docker (ansible-collection-kolla roles/docker/tasks/install.yml:15-30, :44-53, roles/docker/tasks/config.yml:119-125) CODE. A failure or interruption between those points leaves the symlink in place. NOT AVAILABLE: whether Docker starts after the next reboot. fact sheet E §2.3; fact sheet E §0 (item 3); Known bugs and limitations (row baremetal)
Prechecks: the port or the address that a precheck tests The swift precheck probes rsync port 873, but rsyncd uses 10873 (roles/swift/tasks/precheck.yml:89, group_vars/all.yml:628) CODE. The octavia health-manager precheck tests the port on api_interface_address while the health manager binds octavia_network_interface_address (roles/octavia/tasks/precheck.yml:29-38, roles/octavia/templates/octavia.conf.j2:80); they differ only when octavia_network_interface is not api_interface CODE. The iscsi precheck tests that iscsi_port (3260) is free on the iscsid hosts, while 3260 is the target (tgtd) port; new in 2024.1, it tests it on tgtd_interface_address (roles/iscsi/tasks/precheck.yml:17-27) CODE. Known bugs and limitations (row swift); fact sheet D §2.6; fact sheet D §7 (items 2, 5 and 11)
The RabbitMQ handler blocks after an uncontrolled full stop After an uncontrolled full stop (kolla-ansible stop, power loss) followed by deploy, the brokers come back through the batched Restart rabbitmq services play (serial: "33%", one broker per batch with three), each start followed by rabbitmqctl wait --timeout 60 (rabbitmq.yml:30-49, roles/rabbitmq/tasks/restart_services.yml:34-39) CODE. Which node starts first follows the group_by order: NOT AVAILABLE. NOT AVAILABLE (RabbitMQ semantics): whether, if that node was not the last to stop, it waits for its peers' Mnesia tables before giving up, so that the first batch stalls or fails before the other brokers are started. fact sheet B §2.8 (item 3); RabbitMQ
stop and the swift account-auditor The swift stop.yml uses the wrong container name CODE: the account-auditor stop checks 'swift_object_auditor' not in skip_stop_containers (roles/swift/tasks/stop.yml:23-31). Known bugs and limitations (row swift); fact sheet D §3.4

The run is slow

What you see What the code does Read
A run with --limit is slow on a large inventory: it spends a long time gathering facts With --limit, the other hosts' facts are gathered sequentially by the limited hosts (delegated loop, the hosts being divided between the limited hosts, gather-facts.yml:34-62) CODE. With --limit compute001 on the 30-compute inventory (38 hosts, localhost included), compute001 runs 38 setup calls in sequence: its own and one for each of the 37 other hosts CODE; its duration is NOT AVAILABLE. F9; Compute scaling §6 (item 5); R23; fact sheet E §4
A run with no change is still slow on a large fleet Every run that includes the compute group executes the full per-host chain on every compute, even with no change: Kolla has no "only if changed" short-circuit at play level. Per compute, for the default 2024.1 service set with Cinder LVM, nova_compute_virt_type: kvm and a non-Ubuntu host, a no-change -t nova reconfigure makes 27 remote module invocations (31 on compute001), one more than in 2023.1 for the new hostnqn file of nova_compute CODE. Compute scaling §5; Compute scaling §2; Why a small change creates a large workload
The run is slower with ANSIBLE_SERIAL or kolla_serial kolla_serial bounds simultaneous restarts per play, but adds time: every batch re-runs run_once steps and the compute registration wait, and each batch waits for the slowest host. Compute scaling §7; Tags §2.5
One slow host slows the whole run A task runs on all hosts of the play (up to forks at a time), and the next task starts only when every host has finished the current one CODE: the slowest host sets the pace of every task. Compute scaling §1
The restart window of an upgrade includes image downloads Upgrade without a prior pull: hosts download lazily inside the handler phase (recreate_or_restart_container pulls before stopping), which lengthens and staggers the restart window. Update container images §4; Nova deep dive §7; Pre-execution checklist (step 4)
The run waits for compute registration (up to 20 × 10 s) The compute registration wait runs for the first compute of each batch, delegated to the conductor, for up to 20 × 10 s per batch, multiplied by the number of batches when serial is used: it waits until all computes of the batch appear in compute service list. Compute scaling §3
A 300 s wait on ovs-dpdk hosts The Wait for dpdk tunnel ip handler is a wait_for on a host with no port and a 300 s timeout (roles/ovs-dpdk/handlers/main.yml:88-92) CODE. Changed in 2024.1: the ovsdpdk-db restart handler notifies it by its exact name (:21), so it runs after every ovsdpdk-db restart, and ansible-core 2.16.20 wait_for with neither port nor path sleeps for the whole timeout (lib/ansible/modules/wait_for.py:541-542): each such restart adds 300 s on that host CODE. Known bugs and limitations (row ovs-dpdk); fact sheet A §6.8
bootstrap-servers is slow on a large inventory /etc/hosts lists every baremetal host, rendered on every host: O(N²) templating (ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:25-41) CODE. Known bugs and limitations (row baremetal); fact sheet E §0 (item 4); Compute scaling §3
A container is slow to start kolla_set_configs makes one chown per file of the glance image store, of /var/lib/nova on computes, of the MariaDB datadir, of the RabbitMQ mnesia directory and of the cinder volume (twice for cinder-volume), on every COPY_ALWAYS start CODE. NOT AVAILABLE How start time grows with the number of files, especially on NFS-backed trees. Kolla images §6 (item 10)
Horizon is slow to start after a recreate or a settings change FACT: collectstatic and compress run on the first start after a recreate (new image, plugin flag change, COPY_ONCE) or after a settings change, and not on a plain restart with unchanged settings and plugin flags; new in 2024.1, compress is tried up to five times. NOT AVAILABLE: how long they take. fact sheet C §4.8; Kolla images §5.2
Every task costs several SSH round trips Kolla ships no ansible.cfg, so pipelining stays off and each invocation still makes several round trips (tmp dir, transfer, exec, cleanup); ansible-core multiplexes the SSH connection by default (ControlPersist=60s, ansible-core 2.16.20 lib/ansible/plugins/connection/ssh.py:78-80) CODE. Compute scaling §3 (row "SSH connection setup"); Compute scaling §8
The deploy host is busy during the config tasks template and merge_configs render on the deploy host for every host and item, then copy compares checksums remotely CODE: deploy-host CPU scales with hosts × templates. Whether, and at which forks value, the deploy host's CPU becomes the bottleneck is NOT AVAILABLE. Compute scaling §1; Compute scaling §6 (item 3)
How long a run should take No timings are given. Work is counted as operations read from the source; task-result totals and wall times are NOT AVAILABLE. Compute scaling; What is not known, "Timings, counts and scale"; Q7; Q8

--limit seems ignored

--limit does not confine actions (F8). The rows about --limit are in four tables of this index:

Scope surprises: hosts or services touched outside what was asked

What you see What the code does Read
Tasks ran on a controller (control01) although --limit named only a compute --limit does not confine actions: delegate_to: groups[x][0] runs work on hosts outside the limit. With deploy --limit set to a compute host, control01 (the first neutron-server and nova-conductor host of the multinode inventory) runs the Neutron DB create/grant and the bootstrap_neutron container, plus cell discovery CODE. F8; Add a compute node §2; Reconfigure a single service §2 (row "Actions on hosts outside --limit")
DB-sync (bootstrap) containers and Keystone registration ran on a reconfigure reconfigure = deploy: in 2024.1, 54 roles have a tasks/reconfigure.yml, 52 of them run deploy.yml and only swift and bifrost have their own logic, so a reconfigure re-runs Keystone registration, DB creation/grants and the DB-sync bootstrap containers of most services, Neutron's included CODE. F5; Execution flow §6; Repository architecture §4.2; Tags §4 (item 5)
A service tag also ran common (fluentd, cron, kolla_toolbox) on every host of the service play Does not happen in 2024.1 CODE: no role declares common as a meta dependency (no roles/*/meta directory), and common is its own play tagged common only (site.yml:88-103). A service tag leaves fluentd, cron and kolla_toolbox untouched; their pending changes wait for -t common or an untagged run. Reconfigure a single service §2; Tags §4 (item 1); Repository architecture §5.1
A service tag rendered the HAProxy frontend of the service on all haproxy hosts -t <svc> runs that service's loadbalancer include, and so its haproxy-config, on all loadbalancer hosts: the per-service include_role … tasks_from: loadbalancer tasks of the loadbalancer play carry the service tag (site.yml:122-323) CODE. Tags §2.3; fact sheet A §1.9
A tag ran tasks on every compute nova, nova-cell, neutron, openvswitch, ovn (through ovn-controller), iscsi (when iscsid is enabled) and common target compute hosts, as do baremetal (bootstrap-servers, all hosts) and nova-reload/nova-cell-reload (work only on upgrade); nova-api does not touch computes. chrony does not exist: 2024.1 has no chrony play. A tag never restricts which hosts a play targets; use --limit for that. Tags §3; Tags §4 (item 2)
-t keystone re-distributed the Fernet keys -t keystone runs distribute_fernet.yml: on a reconfigure the Fernet keys are re-distributed from the first running keystone_fernet node to the others. Reconfigure a single service §2; Execution flow §6; R19
-t glance changed a global MariaDB variable The glance role toggles the global MariaDB variable log_bin_trust_function_creators 1→0 around its db sync on every deploy/reconfigure (roles/glance/tasks/bootstrap_service.yml:2-17, :39-54) CODE. Known bugs and limitations (row glance); R20; fact sheet C §1
nova_scheduler received SIGHUP on a run with no change Any deploy/reconfigure with the nova, nova-api or nova-refresh-scheduler-cell-cache tag (or no tag) whose --limit, if any, includes the nova-scheduler hosts, even with no change, sends SIGHUP to nova_scheduler (cell-cache refresh, roles/nova/tasks/refresh_scheduler_cell_cache.yml:10-12, nova.yml:152-172); the PID is now read with kolla_container_engine inspect CODE. It is the only signal Kolla-Ansible sends to a running containerised service, and it loads no new config files. Nova deep dive §2.4; Repository architecture §4.3
-t haproxy also reconfigured fluentd, cron and kolla-toolbox on the haproxy hosts Does not happen in 2024.1 CODE: the loadbalancer play (site.yml:105-326) does not include common, which is its own play tagged common only (site.yml:88-103). fact sheet A §1.8 (E); Apply common changes §3
genconfig with a service tag also changed files on the haproxy hosts genconfig writes /etc/kolla/<svc>/ on the targets and also re-renders the HAProxy services.d files for the tagged services: the per-service loadbalancer includes also run for kolla_action config (site.yml:324-326) CODE. HAProxy itself is not restarted (see Nothing happened). Predict and inspect a run (row "Render configs without restarting")
genconfig ran NIC rebinding on ovs-dpdk hosts NIC rebinding (ovs-dpdkctl.sh bind_nics) and unit installation (ovs-dpdkctl.sh install) are tasks of the ovs-dpdk config.yml (roles/ovs-dpdk/tasks/config.yml:35-59), so they run on every deploy, reconfigure, upgrade and genconfig CODE. They are idempotent only as long as the driver already matches. fact sheet A §6.8
A bootstrap-servers re-run changed the host: firewall, SELinux, packages Re-running bootstrap-servers changes the host (ansible-collection-kolla, openstack.kolla.baremetal) CODE: it disables ufw/firewalld (disable_firewall, roles/baremetal/tasks/install.yml:2-32), sets SELinux to selinux_state (permissive by default) when change_selinux is set (roles/baremetal/tasks/post-install.yml:21-28), removes the libvirt AppArmor profile on Ubuntu with apparmor_remove_libvirt_profile (roles/apparmor_libvirt/tasks/main.yml:2-5) and removes host packages (roles/packages/defaults/main.yml:25-34): lxd, lxc and open-iscsi on Debian/Ubuntu, and libvirt* (plus iscsi-initiator-utils on RedHat) only with the libvirt container. chrony is not removed. None of these tasks restarts containers; a Docker restart is a separate handler (row "The Docker daemon restarted during bootstrap-servers"). fact sheet E §0 (item 6); R14
destroy also ran on the deploy host destroy acts on hosts: all (destroy.yml:3), including the deploy host (localhost is in [deployment], inventory/multinode:32-33) CODE. tools/cleanup-host deletes every network namespace and every link whose line matches tap, brq or vxlan (tools/cleanup-host:5-22), and rm -rfs /etc/kolla/* except six names (passwords.yml, globals.yml, globals.d, kolla-build.conf, config, certificates; tools/cleanup-host:89-102) and, new in 2024.1, the inventory paths passed by the CLI (tools/cleanup-host:82-87) CODE. fact sheet E §0 (item 10); fact sheet E §6; R16; Stop and destroy
-t ovn overwrote or removed a manual ovn-cms-options value ovn-cms-options is set to enable-chassis-as-gw (plus availability-zones when set) on the ovn-controller-network hosts, and removed entirely on the other ovn-controller hosts, on every run (roles/ovn-controller/tasks/setup-ovs.yml:12-39) CODE. Any manual ovn-cms-options value is overwritten or removed on every deploy or reconfigure; the impact of that loss is NOT AVAILABLE. fact sheet A §5.8 (E); Tags §3.1 (row ovn)
A manual net.ipv4.tcp_retries2 line disappeared from /etc/sysctl.conf on the haproxy hosts With set_sysctl and the default haproxy_host_ipv4_tcp_retries2: "KOLLA_UNSET", the sysctl task removes any manual net.ipv4.tcp_retries2 line from /etc/sysctl.conf on every run; in 2024.1 it runs through the new sysctl role, with the same KOLLA_UNSET logic (roles/loadbalancer/tasks/config-host.yml:8-18, roles/sysctl/tasks/main.yml:7-20, roles/loadbalancer/defaults/main.yml:179, group_vars/all.yml:706) CODE. fact sheet A §1.8 (G)
A keystone reconfigure re-applied the admin credentials and the endpoints Reconfigure re-asserts identity state: it re-applies the admin credentials and endpoints, and the DB user password, from passwords.yml and globals. NOT AVAILABLE: whether out-of-band changes are silently reverted. fact sheet B §3.9 (item 10)
Enabling Swift, or glance_backend_ceph, changed the Glance backend Enabling Swift (or glance_backend_ceph, or, new in 2024.1, glance_backend_s3) on a running cloud silently switches Glance's backend (group_vars/all.yml:1001,1006; roles/glance/defaults/main.yml:247) CODE; NOT AVAILABLE whether existing file-store images stay readable. R21; Known bugs and limitations (row glance); fact sheet C §1.8 (item 1); Q12
After the inventory was reordered: keepalived priorities, MariaDB restart waves, the glance host or the bootstrap target changed Inventory reordering (loadbalancer, mariadb, glance, nova-conductor, keystone groups): keepalived priorities change; the MariaDB active backend (the first shard host, the others are backup, roles/mariadb/defaults/main.yml:64) moves; glance moves (file backend); bootstrap/discovery target changes CODE. Which MariaDB restart batch a host falls in follows the group_by order: NOT AVAILABLE. R22
A task reports changed on every run although nothing changed openvswitch sets system-id, hostname (new in 2024.1) and hw-offload through the openvswitch_db module and restarts openvswitch_vswitchd when it reports changed (roles/openvswitch/tasks/post-config.yml:3-23), and ovn-controller sets the chassis options through the openvswitch_bridge/openvswitch_db modules in kolla_toolbox (roles/ovn-controller/tasks/setup-ovs.yml:2-40) CODE. Whether those modules report changed on an unchanged host is NOT AVAILABLE (the modules are not in these sources). Tasks that do report changed on every run include the --check of containers with preserve_properties entries (prometheus extras) CODE. Known bugs and limitations (row openvswitch); Tags §3.1 (row openvswitch); fact sheet A §5.8 (D); Open vSwitch
An empty placement-api.cfg was written on the haproxy hosts although Placement is disabled The placement loadbalancer include lacks when: enable_placement (site.yml:248-251) CODE: under -t haproxy, the placement frontend file is always rendered, and is empty if enable_placement is false. Known bugs and limitations (second row site.yml); fact sheet A §1.9; Placement
A password rotation needed a reconfigure of the whole compute fleet FACT: placement and cinder credentials are embedded in nova-compute config on every compute, so password rotation needs a reconfigure of the whole compute fleet. fact sheet C §7; fact sheet C §2.8
On a new compute, nova_compute started before Open vSwitch and the Neutron agents The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes: site.yml:609 imports nova.yml before the openvswitch play (:611) and the neutron play (:671) CODE. NOT AVAILABLE Whether instances can be scheduled on such a compute before the later plays finish. F13; Execution flow §2.4; Component dependencies §6

Images

What you see What the code does Read
deploy or reconfigure did not pick up a rebuilt image pushed under the same tag deploy/reconfigure never pull an image that is already present locally, even if the registry has a newer build under the same tag, with Docker and with Podman CODE. Only absent images are pulled; an explicit pull is required CODE. Update container images §1.1; Update container images §2; F4; R13
An unrelated pull was followed by a mass recreate Mutable tags create silent drift between hosts. A later unrelated pull turns into a mass recreate. F4 ("Why it matters"); R13
A new openstack_tag applied with deploy recreated every container on every host New openstack_tag, deploy: every container on every host is recreated CODE. Changing the tag and running deploy (instead of upgrade) recreates the whole fleet in one go, without the upgrade-specific ordering: no nova-status upgrade check, no RPC-pin reload plays, and no online data migrations, except Placement's (and Ironic's only with ironic_enable_rolling_upgrade: no). Update container images §2; Update container images §4; R12
An OVS image change recreated openvswitch_vswitchd on every OVS host at the same time An OVS image change recreates openvswitch_vswitchd on every OVS host simultaneously (serial 0). NOT AVAILABLE Whether VM networking is interrupted on every compute at the same time. Update container images §4; R7
pull did not change the running containers pull never changes running containers CODE. After a pull that downloaded something, the next deploy, reconfigure or upgrade recreates the affected containers CODE. Update container images §1.1; R11
-t nova pull downloaded more than the nova images No longer, as in 2023.1 CODE: -t nova pull downloads on every host in the nova plays, for nova images only. The common images (fluentd, kolla-toolbox, cron) are not pulled, because common is a separate play tagged common and no role declares it as a dependency; add -t common to pull them. Update container images §2
genconfig pulled an image Changed in 2024.1 CODE: when the common play runs (no tag, or -t common), its config step runs kolla_container ensure_image on the fluentd image, then reads its labels (roles/common/tasks/config.yml:42-91); ensure_image pulls the image when that tag is not on the host (module_utils/kolla_docker_worker.py:501-503, module_utils/kolla_podman_worker.py:675-677). So genconfig can need the registry, and fails on a host where that pull fails. A service-tagged genconfig does not run the common play. Predict and inspect a run (row "Render configs without restarting"); Execution flow §4 (row genconfig); Apply common changes §4
The SSH host keys of nova_ssh and keystone_ssh changed after a recreate SSH host keys of nova_ssh and keystone_ssh change on every recreate, because /etc/ssh is on no volume. The clients set StrictHostKeyChecking no and keep no known-hosts file CODE; whether the SSH client then accepts the new keys without failing is NOT AVAILABLE (OpenSSH semantics). Kolla images §6 (item 18)
prune-images did not remove the images of the previous release With Docker, prune-images runs docker_prune on images with the kolla_version label filter; new in 2024.1, with Podman it runs podman image prune --force --filter 'label=kolla_version' (roles/prune-images/tasks/prune_images.yml:2-9, :13-16) CODE. Whether either prune removes only dangling images, so that old release tags still referenced by a tag are not reclaimed, is the module's or the engine's behaviour: NOT AVAILABLE (neither is in these sources). fact sheet E §0 (item 10); fact sheet E §7; R17

Settings that are ignored

What you see What the code does Read
docker_graceful_timeout (60 s) is ignored: containers get SIGKILL 10 s after the stop Still fixed in 2024.1 CODE: docker_graceful_timeout and docker_client_timeout take effect. Handlers pass docker_common_options, which carries both values, and kolla_container declares both options without an argument default, so the module defaults (10 s, 120 s) apply only when neither is given (group_vars/all.yml:139-153; library/kolla_container.py:313, :335, :361-373). Each container's systemd unit stops it with <engine> stop <name> -t <docker_graceful_timeout> (module_utils/kolla_systemd_worker.py:32), so a container still running 60 s after SIGTERM is killed. A changed timeout also changes the unit file, which check-containers detects. F3; Repository architecture §6; R24; Known bugs and limitations (row kolla_container)
-e kolla_serial=N is ignored by reconfigure and upgrade -e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade CODE: the wrapper appends -e kolla_serial=${ANSIBLE_SERIAL} after the user's -e options (tools/kolla-ansible:296, :522, :531). Known bugs and limitations (first row CLI); Repository architecture §2; fact sheet E §11
ANSIBLE_SERIAL is ignored by deploy kolla_serial is not passed for deploy (tools/kolla-ansible:490-493) CODE. Batching is ANSIBLE_SERIAL for reconfigure/upgrade and -e kolla_serial=N for deploy, which ignores ANSIBLE_SERIAL. Known bugs and limitations (first row CLI); Ten things to know (item 4); Execution flow §5
mariadb, rabbitmq and cyborg are not batched by ANSIBLE_SERIAL Not batched by ANSIBLE_SERIAL CODE: mariadb and rabbitmq, whose plays set no serial and whose restarts run in their own plays with a fixed serial: "33%" (mariadb.yml:3-52, rabbitmq.yml:3-49); cyborg, whose play uses serial instead of kolla_serial and so ignores ANSIBLE_SERIAL (bug, site.yml:876); and, new in 2024.1, etcd, whose play has no serial (site.yml:470-481). Outward RabbitMQ no longer exists. Controlled batches §1.2; Known bugs and limitations (first row site.yml); Execution flow §5
A per-host or per-group inventory value is ignored for a variable set in globals.yml globals.yml and passwords.yml are passed with -e @file (tools/kolla-ansible:602-604), that is extra-vars, the highest precedence in Ansible: a value in globals.yml cannot be overridden per host or per group in the inventory CODE. Repository architecture §2.1
A restart_policy change is not applied As in 2023.1 CODE: the FIXME "restart_policy is not checked in the container" remains (library/kolla_container.py:19), but the restart policy lives in the container's systemd unit (docker_restart_policy becomes Restart= and StartLimitBurst), and compare_container detects a unit file that differs, so a policy change that renders differently restarts the container at its next check; unless-stopped and always both render Restart=always, so a change between them changes nothing (module_utils/kolla_systemd_worker.py:28, :33, :56-58, :154-167). Known bugs and limitations (row kolla_container); Repository architecture §6
A docker_custom_option change is not applied Does not apply to 2024.1: the docker_custom_option variable was removed (ansible-collection-kolla releasenotes/notes/drop-docker-custom-option-6c8251c750473558.yaml:4), and no task in kolla-ansible or the collection reads it CODE. The docker.service.d/kolla.conf drop-in carries only the proxy settings and the Zun options; writing or removing it notifies Reload docker service file, then Restart docker (ansible-collection-kolla roles/docker/tasks/config.yml:55-92, roles/docker/templates/docker_systemd_service.j2, roles/docker/handlers/main.yml:2-15) CODE. fact sheet E §0 (item 2); Known bugs and limitations (row baremetal)
-v flags are lost with --ask-vault-pass --ask-vault-pass overwrites VERBOSITY with a copy of EXTRA_OPTS (tools/kolla-ansible:391-392) CODE, which duplicates all options seen so far and drops earlier -v flags. Known bugs and limitations (second row CLI); Repository architecture §2; fact sheet E §11
--include-dev and the next word of the command line --include-dev: is declared with an argument in LONG_OPTS but used as a flag (tools/kolla-ansible:275, :355-358); the command line is executed unquoted ($CMD, :146-148) CODE. Known bugs and limitations (third row CLI); fact sheet E §11
enable_swift_recon with its default "no" enable_swift_recon is tested without the bool filter ({% if enable_swift_recon %}, roles/swift/templates/account.conf.j2:15, :17, roles/swift/templates/container.conf.j2:15, :17, roles/swift/templates/object.conf.j2:19, :22), and its default is the string "no", not a boolean (group_vars/all.yml:844) CODE. NOT AVAILABLE: whether recon is then always on. fact sheet D §3.8; fact sheet D §7 (item 3)
A service variable overridden only on the backend hosts, and the HAProxy frontend The haproxy-config variables come from the service role's defaults, evaluated on the loadbalancer host; inventory or group_vars overrides of service variables therefore apply when set for the loadbalancer hosts or for all. NOT AVAILABLE: whether per-host overrides set only on backend hosts are seen by haproxy-config, outside the places where the code explicitly uses hostvars[groups[…][0]] (nova-cell, roles/nova-cell/tasks/loadbalancer.yml:37-48). fact sheet A §2.4
New in 2024.1: an haproxy_processes or haproxy_process_cpu_map override has no effect The two variables were renamed haproxy_threads and haproxy_thread_cpu_map (roles/loadbalancer/defaults/main.yml:66-67), which haproxy_main.cfg.j2 now reads (roles/loadbalancer/templates/haproxy/haproxy_main.cfg.j2:9-10); no file of the tree reads the old names any more CODE. HAProxy + Keepalived; fact sheet A §1.4
New in 2024.1: a Horizon override in /etc/kolla/config/horizon/local_settings or custom_local_settings has no effect The role now renders _9998-kolla-settings.py and _9999-custom-settings.py, each looked up with with_first_found under those new names (roles/horizon/tasks/config.yml:75-109); an override kept at the 2023.1 names is no longer read by any task, and the lookup falls through to the role template with no warning CODE. Horizon

Listings, check mode and prechecks

What you see What the code does Read
--list-tasks shows only include_tasks placeholders ansible-playbook site.yml --list-tasks -t nova lists only include_tasks placeholders for each role: the listing walks only the statically compiled play, while dynamic includes are loaded at run time CODE. Tags §2.6; F7; Predict and inspect a run
--list-hosts shows 0 hosts; Could not match supplied host pattern, ignoring: enable_nova_True Plays whose host pattern depends on the group_by groups (&enable_nova_True) resolve to 0 hosts at list time: those groups are created at run time CODE. Tags §2.6; F7; Predict and inspect a run
--list-tags does not show a tag --list-tags lists only the tags it can see statically; tags that occur only inside dynamically included files, such as service-rabbitmq, do not appear. Tags §3.1 (paragraph "Validation")
A --check --diff run skipped the kolla_container tasks kolla_container, kolla_container_facts and kolla_toolbox do not declare check-mode support (library/kolla_container.py:355-359), so Ansible skips them under --check (ansible-core 2.16.20 lib/ansible/module_utils/basic.py:506-507) CODE. A few tasks set check_mode: false and run for real even under --check, among them three kolla_container tasks that start one-shot containers: nova_list_cells, rabbitmq_version_check and nova_upgrade_checks (roles/nova-cell/tasks/get_cell_settings.yml:2-21, roles/rabbitmq/tasks/version-check.yml:18-36, roles/nova/tasks/upgrade.yml:9-30) CODE; rabbitmq_version_check is not reached, because its block depends on a kolla_container_facts result that check mode skips (roles/rabbitmq/tasks/version-check.yml:3-9, :75) CODE. NOT AVAILABLE Whether such a run fails or gives partial results. Predict and inspect a run (row "Check-mode run"); Tags §2.6; Q9
genconfig plus kolla_set_configs --check did not predict what the run did "genconfig + --check" under-reports some changes and over-reports others. In 2024.1 it also misses the unit-file drift of the first run after 2023.1, which check-containers compares outside the container (F16) CODE. F14 ("Why it matters"); Kolla images §3.2; Predict and inspect a run
docker ps shows no health status Docker healthchecks are wired CODE, as in 2023.1. With the default enable_container_healthchecks: yes, docker ps shows a health status for the services that define a test; a container without one shows none. No Kolla-Ansible task gates on it, so service health must still be checked from outside. Kolla images §6 (item 17); Kolla images §2.7
prechecks with -t <svc> skipped the general prechecks -t <svc> skips the prechecks role (untagged play, site.yml:77-86) but runs <svc>/precheck.yml CODE. The RabbitMQ variable check of F17 is in that general role (roles/prechecks/tasks/service_checks.yml:67-81), so -t rabbitmq prechecks does not run it CODE. Predict and inspect a run (row "Pre-deployment checks")
The tag name did not predict what the run did A tag name does not describe the work a run performs. What runs depends on which plays carry or inherit the tag, the loadbalancer play's per-service includes, handlers, which are never filtered by tags, the MariaDB and RabbitMQ restart plays that the handlers fill through group_by, the etcd handlers that include task files, and ansible-core's loop-notify semantics. Tags ("The single most important rule"); Tags §3
The same command was a no-op one day and a fleet-wide restart another day The same command can be a no-op or a fleet-wide restart, depending on whether a rendered file changed. That in turn depends on global variables (enable_*, group membership, passwords), not only on the tagged service CODE. Tags §4 (item 3); Component dependencies §5

Batching with serial

What serial does not reach is under Settings that are ignored; what it costs is under The run is slow.

What you see What the code does Read
With serial, the DB sync, Keystone registration or discovery ran several times With kolla_serial > 0, a run_once task runs once per batch, not once per play CODE. This applies to bootstrap containers, Keystone registration and DB creation. With kolla_serial=1 and the three controllers of the multinode inventory, nova_api_bootstrap runs 3 times (roles/nova/tasks/bootstrap_service.yml:16-21), once per controller batch CODE. Tags §2.5; F10; Controlled batches §1.2
With serial, the computes restarted before the conductors kolla_serial orders a play's restarts by host-pattern order (computes before conductors in the nova-cell play) CODE. F10
With serial, the neutron agents of the first hosts restarted only after every compute had finished the earlier plays A batch runs the complete play before the next batch starts, but each play is still traversed batch by batch for the whole fleet before the next play starts. The neutron agents on compute001 are restarted only after all computes have finished the nova-cell and openvswitch plays. Execution flow §5
New in 2024.1: a batch with failed hosts did not stop the run Every play now sets max_fail_percentage from <service>_max_fail_percentage, then kolla_max_fail_percentage, then 100 (e.g. site.yml:96-99); neither variable is set by default (etc/kolla/globals.yml:27-31 is commented out). ansible-core 2.16.20 compares the failed hosts of the current batch with the batch size and breaks only above the percentage, so 100 never stops a play early; the "whole batch failed" rule still ends the playbook (lib/ansible/plugins/strategy/linear.py:336-348, lib/ansible/executor/task_queue_manager.py:330-338, lib/ansible/executor/playbook_executor.py:204-209) CODE. A lower value, for example -e kolla_max_fail_percentage=0, makes the first failed host of a batch end the run before the next batch. Controlled batches §1.3; Execution flow §5

Upgrades and database migrations

What you see What the code does Read
A schema migration ran during a reconfigure A "reconfigure" can apply schema migrations if the image tag moved, for most services, Neutron included: bootstrap_neutron runs on every deploy and reconfigure (roles/neutron/tasks/bootstrap_service.yml:2-21, no when) CODE. There is never a "config only" path except genconfig + deploy-containers. F5 ("Why it matters")
Placement's online data migrations ran on a deploy or reconfigure Placement's online data migrations run on every deploy/reconfigure, inside bootstrap_placement, and twice on upgrade CODE. Kolla images §6 (item 6); F15
Nova's online data migrations did not run on deploy or reconfigure Nova's online data migrations never run outside upgrade. Deploy and reconfigure run the Nova bootstrap containers with KOLLA_UPGRADE (api_db sync + db sync, db sync --local_cell) CODE. New in 2024.1, the "Run Nova API online data migrations" play of upgrade sets any_errors_fatal: true (nova.yml:258-270) CODE. Kolla images §6 (item 5); F15
neutron-dynamic-routing was not migrated by upgrade Still fixed in 2024.1 CODE: upgrade with the default neutron_enable_rolling_upgrade: "yes" expands/contracts neutron plus every enabled subproject of neutron_subprojects (now neutron-fwaas, networking-sfc, neutron-dynamic-routing, neutron-vpnaas, vmware-nsx and tap-as-a-service), the same list the bootstrap container migrates (roles/neutron/defaults/main.yml:674-686, :727, :733; kolla docker/neutron/neutron-server/extend_start.sh:5-34). Known bugs and limitations (row neutron); Kolla images §6 (item 8); F15
An upgrade stopped neutron_server on every neutron-server host has_offline_migrations loops over the enabled subprojects only, never neutron itself, on the first neutron-server host with ignore_errors: any non-zero rc (a failed exec included) stops neutron_server on every neutron-server host in the play until the handler flush after the contract container; with no subproject enabled no server is stopped before upgrade --contract (roles/neutron/tasks/rolling_upgrade.yml:38-89) CODE. Impact on the Neutron API, and whether a neutron contract migration is safe against running servers: NOT AVAILABLE. Known bugs and limitations (row neutron); Kolla images §6 (item 8)
The RabbitMQ upgrade stopped all brokers No longer, as in 2023.1 CODE: upgrade checks the version, removes the ha-all policy when HA is off, renders the config, enables feature flags and checks the containers, then restarts the brokers in the serial: "33%" restart play, draining each one first; roles/rabbitmq/tasks/upgrade.yml:2-12 has no cluster-wide stop. NOT AVAILABLE Whether nova-compute heartbeats are lost and services are reported down meanwhile. R10; fact sheet B §2.8 (item 2); RabbitMQ
New in 2024.1: the RabbitMQ upgrade (or deploy, reconfigure) failed with "Looks like you're trying to run a skip-release upgrade!" upgrade.yml now imports version-check.yml first, as deploy.yml does (roles/rabbitmq/tasks/upgrade.yml:2, roles/rabbitmq/tasks/deploy.yml:2). When a rabbitmq container exists, it compares the running version with the one in a one-shot rabbitmq_version_check container of the target image and fails unless both have the same major version and at most one minor version between them, or the step is 3.13 to 4.0 or 4.1; the message names the kolla-ansible rabbitmq-upgrade step to run first (roles/rabbitmq/tasks/version-check.yml:12-36, :44-64) CODE. RabbitMQ; Pre-execution checklist §2
On upgrade, keepalived was stopped on the hosts that do not hold the VIP No longer, as in 2023.1 CODE: the loadbalancer role's upgrade.yml removes the prometheus_haproxy_exporter container and its config, then imports deploy.yml (roles/loadbalancer/tasks/upgrade.yml:4-21), with no upgrade-specific keepalived stop. A keepalived change on any action goes through the staged handlers: backups stopped and started first, the VIP holder's keepalived last (roles/loadbalancer/handlers/main.yml:36-52, :154-177, :210-222). fact sheet A §1.8 (F); HAProxy + Keepalived
The octavia upgrade removed the octavia admin role No longer, as in 2023.1 CODE: roles/octavia/tasks/upgrade.yml:2-46 is config, check-containers, the creation of the persistence database and its user (new in 2024.1), bootstrap_service and a flush of handlers, with no "Removing octavia user from admin project" task. Known bugs and limitations (row octavia); fact sheet D §2.3; Octavia
mysql_upgrade ran on all MariaDB nodes at once, with an unchanged image mysql_upgrade (the upgrade_mariadb container, KOLLA_UPGRADE) runs in the Apply mariadb post-configuration play, which has no serial, so on all nodes concurrently, on every upgrade, with no condition on an image change (mariadb.yml:70-92, roles/mariadb/tasks/post-upgrade.yml:2-24) CODE. NOT AVAILABLE: whether concurrent mysql_upgrade on Galera nodes conflicts. fact sheet B §1.8 (item 8)
Glance rolling upgrade with serial: the contract ran before the last batches were upgraded Answered from code: Glance rolling upgrade with ANSIBLE_SERIAL=1 runs the contract at the end of batch 1, before later batches are upgraded: the contract container is run_once, delegated to glance_api_hosts[0], after the flush of handlers (roles/glance/tasks/rolling_upgrade.yml:80-102) CODE. fact sheet C §8 (item 2); fact sheet C §1.8 (item 4)
Keystone upgrade with serial: log_bin_trust_function_creators With serial, log_bin_trust_function_creators is reset to 0 at the end of every batch (run_once), while Enable runs only in the batch whose first host is groups['keystone'][0] (run_once with when: inventory_hostname == groups['keystone'][0]) (roles/keystone/tasks/upgrade.yml:6-21, :36-50) CODE. The contract step, notified on groups['keystone'][-1] (:28-31), therefore runs with it disabled when that host is in a later batch. The impact is NOT AVAILABLE. fact sheet B §3.9 (item 9)
The swift "rolling" upgrade stopped all storage containers at once Despite the name, the swift rolling upgrade (swift_enable_rolling_upgrade: "yes" by default) stops all storage containers on all storage hosts at once (roles/swift/tasks/rolling_upgrade.yml:28-39, with the TODO "should be run in serial nodes") CODE. fact sheet D §3.4; fact sheet D §7 (item 4)
The cinder upgrade restarted every cinder-volume and cinder-backup at once The upgrade restarts every cinder container of the play at once (serial 0 unless ANSIBLE_SERIAL is set), and then a second time: after the flush, reload.yml restarts every enabled cinder container "to remove RPC version pin", after a cinder_rpc_version_startup_delay pause (30 s) that a handler runs only on upgrade (roles/cinder/tasks/upgrade.yml:20-23, roles/cinder/tasks/reload.yml:2-10, roles/cinder/handlers/main.yml:78-85, roles/cinder/defaults/main.yml:232) CODE. fact sheet C §3.8 (item 4); Cinder
A reconfigure with a bumped image tag ran cinder's db sync without its online data migrations A reconfigure with a bumped image tag runs cinder's db sync without its online data migration step: the bootstrap container sets KOLLA_BOOTSTRAP, not KOLLA_OSM, which only upgrade.yml sets (roles/cinder/tasks/bootstrap_service.yml:11, kolla docker/cinder/cinder-api/extend_start.sh:6-14, roles/cinder/tasks/upgrade.yml:25-43) CODE, while placement's bootstrap container runs both db sync and db online_data_migrations. Whether this leaves cinder in a partially upgraded state is NOT AVAILABLE. fact sheet C §6

Services, VMs and traffic during and after a restart

Every statement about data-plane or VM impact is NOT AVAILABLE (Limits of this study). The rows say what the code shows and point to what is open.

What you see What the code does Read
Running VMs after a nova_libvirt restart or recreate NOT AVAILABLE Whether running VMs survive a nova_libvirt restart or recreate; what the container and its start path do is CODE. Nova deep dive §5; Q1; What is not known, "Running VMs and compute hosts"
VM network traffic after a restart of the OVS agent, of openvswitch_vswitchd or of the L3 agent NOT AVAILABLE Whether VM network traffic is interrupted when the OVS agent, openvswitch_vswitchd or the L3 agent restarts; that the restarts happen is CODE. Running L3 agents restart in host-list order, spaced by neutron_l3_agent_failover_delay, default 0, so not spaced by default (roles/neutron/defaults/main.yml:651, roles/neutron/handlers/main.yml:129, :156) CODE. Reconfigure Neutron §4; R5; R6; R7; Q2; Q3; Q4; What is not known, "Network data plane"
Open vSwitch flows after an openvswitch_vswitchd restart An openvswitch_vswitchd restart replaces only the userspace daemon. The kernel module stays loaded and conf.db keeps bridges and ports CODE. NOT AVAILABLE Whether the OpenFlow tables are lost and must be reprogrammed by the OVS agent or ovn-controller. New in 2024.1, the openvswitch_db image runs ovsdb-tool convert at start when the schema needs a conversion (kolla docker/openvswitch/openvswitch-db-server/extend_start.sh:8-10) CODE. Kolla images §6 (item 13); R7; Q3
Router namespaces after a neutron_l3_agent restart The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes CODE; its comment says this removes all L3 namespaces. What the command does is neutron code: NOT AVAILABLE. F12; R6; Reconfigure Neutron §4; Kolla images §5.3; Q4
Work in progress was killed at a restart: SIGKILL after 10 s The delay is 60 s, not 10 s CODE: on every restart, a service still running docker_graceful_timeout (60 s) after SIGTERM is killed with its in-flight work (heat-engine, cinder copies, glance uploads). The unit sets no TimeoutStopSec, so systemd's own stop timeout also applies; its value is host configuration NOT AVAILABLE. NOT AVAILABLE Whether services started by a shell wrapper that does not exec them (e.g. neutron-l3-agent, haproxy) receive SIGTERM at all. F3 ("Why it matters"); R24; Repository architecture §6; Kolla images §2.6
The API of a service was unavailable during a run with its tag The service API is unavailable while its containers restart (up to 60 s stop, then start) on all its hosts at once. Add the HAProxy restart window if the frontend changed. Reconfigure a single service §3 ("Interruption")
Authentication while keystone restarts All keystone containers restart on all hosts at once; NOT AVAILABLE whether authentication fails while keystone restarts. R19; Reconfigure a single service §1.1
APIs, DB traffic and the VIP while HAProxy restarts haproxy is stopped and started (never reloaded): on every backup host at once, then on the host that holds the VIP CODE. On a haproxy-only change keepalived is not stopped, so the VIP stays on a node whose haproxy is down until it is back; keepalived runs a check_alive track script (interval 2, fall 2) by default CODE. Changed in 2024.1: with no script under /checks, check_alive.sh no longer tests the HAProxy socket itself CODE. NOT AVAILABLE Whether that script moves the VIP during the master restart, and for how long the APIs are unreachable. Blast radius: all APIs and all DB traffic (database_address is kolla_internal_fqdn; enable_proxysql defaults to no). R8; Q5; fact sheet A §1.8 (B); Kolla images §6 (item 15)
Token cache and Horizon sessions after a memcached restart memcached has no data volume and no persistence option CODE. NOT AVAILABLE (memcached semantics) Whether every restart therefore empties the cache, with token-cache misses and lost Horizon sessions where they are kept in memcached. Kolla images §6 (item 14); R4; fact sheet A §3.8
RPC while RabbitMQ restarts RabbitMQ restarts run in serial: "33%" batches (one broker at a time with three), each broker drained first CODE; NOT AVAILABLE whether nova-compute and the agents lose RPC while their broker restarts, and whether RPC-backed API calls time out meanwhile. New in 2024.1, services use quorum queues by default (F17); what oslo.messaging does with them during a restart is NOT AVAILABLE. Tags §3.1 (row rabbitmq); fact sheet B §2.8 (item 1)
DB connections during a MariaDB rolling restart MariaDB config change or image change: the handlers only group the hosts, and the restart runs in the separate Restart mariadb services play in serial: "33%" batches, one batch after another (roles/mariadb/handlers/main.yml:56-73, mariadb.yml:18-34) CODE. HAProxy's single active backend is the first shard host, the others being backup (roles/mariadb/defaults/main.yml:64); NOT AVAILABLE in which batch that host restarts (group_by order). API impact: DB connection resets. R9; fact sheet B §1.8 (item 1)
MariaDB after all DB containers stopped (power loss, Docker restart on every DB host) MariaDB has no automatic Galera recovery. NOT AVAILABLE Whether, after all DB containers stopped, they come back as plain mysqld_safe with the cluster non-primary until mariadb_recovery. A container left with BOOTSTRAP_ARGS=--wsrep-new-cluster by an interrupted run passes that option to mysqld_safe again on any manual docker start CODE; whether that bootstraps a new cluster (split-brain risk) is NOT AVAILABLE. Kolla images §6 (item 16); Kolla images §5.4; fact sheet E §2.4; R18
Image API outage when glance_api restarts With the file backend on the default glance volume, only the first host of glance-api runs it (glance_api_hosts, group_vars/all.yml:1011), so any restart is a full Image API outage CODE; NOT AVAILABLE whether snapshots in progress fail. R20; fact sheet C §1.8 (item 3)
Volumes and iSCSI sessions after an iscsid or tgtd restart iscsid: NOT AVAILABLE whether existing sessions are kernel-held and whether attach/detach during the window fails. tgtd: nothing in its start path runs tgt-admin to load /etc/tgt/targets.conf; NOT AVAILABLE whether a tgtd restart cuts disk I/O for VMs using LVM volumes from that host until the targets are re-created. Tags §3.1 (row iscsi); What is not known, "Storage and images"
The iSCSI initiator name (IQN) after a restart, a recreate or the removal of the iscsi_info volume The iSCSI initiator name is stable per iscsi_info volume across restart and recreate CODE. NOT AVAILABLE Whether removing the volume produces a new IQN at the next start, and whether that breaks backend ACLs keyed on the old one. Kolla images §6 (item 19)
Heat stacks in progress when heat_engine restarted The engine sends SIGTERM, then SIGKILL if heat-engine is still running after docker_graceful_timeout (60 s) CODE. NOT AVAILABLE (upstream heat): whether stacks that a killed engine owned stay *_IN_PROGRESS until some engine starts again. fact sheet C §5.8
Logs after a restart of the common containers fluentd: log shipping gap; the Elasticsearch and OpenSearch outputs buffer to the fluentd_data volume, which a restart or a recreate keeps, while the tail read positions are on no volume CODE. cron: log rotation is delayed. VM and API impact: NOT AVAILABLE. Apply common changes §3 ("Impact")
The clock after a chrony restart or a VIP failover Does not apply to 2024.1: there is no chrony role, play, container or VIP time source; host time sync is outside Kolla-Ansible and only checked by the prechecks CODE. Which time source the hosts use is NOT AVAILABLE. Chrony; fact sheet A §4.8 (Ussuri role)
After kolla-ansible stop kolla-ansible stop stops all Kolla containers (incl. nova_libvirt, OVS, MariaDB) on targeted hosts; NOT AVAILABLE whether VM network traffic stops. MariaDB then needs mariadb_recovery. R15; Stop and destroy
mariadb_recovery stopped healthy MariaDB nodes mariadb_recovery stops every MariaDB node, even healthy ones (roles/mariadb/tasks/recover_cluster.yml:18-25), and bootstraps from the highest seqno: full DB outage CODE. R18; Database operations; fact sheet B §1.8 (item 5)
Anything else about VMs, traffic, storage or the control plane after a restart What the sources do not establish is listed by topic, with the sections that mark each point NOT AVAILABLE. What is not known

Other bugs and oddities, by service

What you see What the code does Read
Keystone Fernet keys: push, rotation times, expiry limits Fernet cron generator hour calculation (hour = time % HOUR_SPAN, roles/keystone/files/fernet_rotate_cron_generator.py:69); precheck vs generator expiry limits (roles/keystone/tasks/precheck.yml:43-53, roles/keystone/tasks/config.yml:195-200); push script stops at the first unreachable peer (set -o errexit, roles/keystone/templates/fernet-push.sh.j2:3, :17-22) CODE. All three unchanged in 2024.1. Known bugs and limitations (row keystone); fact sheet B §3.9 (items 3 to 6); Q11
Cinder: host, etcd coordination, backup driver No host/backend_host/cluster in cinder.conf; Ceph backup driver default (group_vars/all.yml:1050) while the Ceph files are copied only with cinder_backend_ceph (roles/cinder/tasks/config.yml:15-19) CODE. The Ussuri etcd item still does not hold as written: with cinder_coordination_backend: etcd the backend_url is the internal VIP, where the etcd role has a HAProxy frontend (roles/cinder/templates/cinder.conf.j2:271-281, roles/etcd/defaults/main.yml:11-17) CODE. Known bugs and limitations (row cinder); fact sheet C §3.8; Cinder
Glance with the file backend File backend = single host CODE: with the file backend on the default glance volume (not a shared path), glance runs only on the first host of groups['glance-api'], so reordering the inventory moves glance-api to another host, and the HAProxy members follow it (group_vars/all.yml:1011). Known bugs and limitations (row glance); fact sheet C §1.8 (item 2); R22
Glance image cache FACT: the image cache is never pruned: no task or image runs a glance cache pruner or cleaner. fact sheet C §1.8 (item 5); Glance
Masakari instancemonitor and libvirt SASL As in 2023.1 CODE: the instancemonitor still connects with qemu+tcp (roles/masakari/templates/masakari-monitors.conf.j2:18), but with libvirt_enable_sasl (default true, roles/nova-cell/defaults/main.yml:569, roles/masakari/defaults/main.yml:185) it gets a libvirt auth.conf with the SASL credentials (roles/masakari/templates/masakari-instancemonitor.json.j2:9-15, roles/masakari/templates/auth.conf.j2:1-3). Whether the instancemonitor then authenticates is NOT AVAILABLE. Known bugs and limitations (row masakari); fact sheet D §7 (item 7)
Barbican secrets and barbican_crypto_key The crypto KEK lives in passwords.yml (etc/kolla/passwords.yml:56) CODE. NOT AVAILABLE: whether losing or changing barbican_crypto_key makes stored secrets undecryptable. fact sheet D §1.6
MariaDB backup and the VIP Backup runs only on the first host of the MariaDB shard group (reading that node's datadir, roles/mariadb/defaults/main.yml:123), but its client config points at database_address (roles/mariadb/templates/backup.my.cnf.j2:5) CODE. NOT AVAILABLE: whether, if HAProxy has failed over away from that node, the backup tool connects to a different server than the one whose files it copies. fact sheet B §1.8 (item 9)
HAProxy on the backup nodes with set_sysctl: no ip_nonlocal_bind=1 is set only when set_sysctl is true (roles/loadbalancer/tasks/config-host.yml:8-18), and backup nodes bind VIP addresses CODE. NOT AVAILABLE: whether haproxy starts on non-VIP nodes with set_sysctl: no and no manual sysctl. fact sheet A §1.8 (H)
Two Kolla clouds on the same L2 segment: keepalived keepalived_virtual_router_id defaults to 51 (group_vars/all.yml:224) with the default keepalived_traffic_mode: multicast (roles/loadbalancer/defaults/main.yml:62); two Kolla clouds on the same L2 segment collide unless it is changed (unicast is an option) CODE. fact sheet A §1.8 (I)
OVN DB after the first host was replaced, or its volume lost A new cluster is seeded from the first ovn-nb-db/ovn-sb-db host, while a member re-created on an existing cluster joins the current leader (roles/ovn-db/tasks/bootstrap-initial.yml:5-13, :59-67) CODE. NOT AVAILABLE: what ovn-ctl does in those cases. There is no code for member removal and none for backups. fact sheet A §5.8 (B); OVN
OVN with DPDK The Ussuri inference that OVN + DPDK cannot work rested on the OVN bootstrap's docker exec into openvswitch_vswitchd, which 2024.1 does not do: ovn-controller sets the chassis options through the openvswitch_bridge/openvswitch_db modules in kolla_toolbox (roles/ovn-controller/tasks/setup-ovs.yml:2-40) CODE. Whether OVN works with ovs-dpdk is NOT AVAILABLE. fact sheet A §5.8 (G); OVN
New in 2024.1: cyborg endpoints The cyborg endpoints use cyborg_internal_fqdn/cyborg_external_fqdn, which nothing defines, and cyborg_services still has no haproxy key (roles/cyborg/defaults/main.yml:2-27, :147-148) CODE. Effect on registration: NOT AVAILABLE. Known bugs and limitations (row cyborg); fact sheet D §5.9

For any other role, the fact sheets list the notable behaviour in these sections (the component pages say, item by item, which of them still hold in 2024.1):

  • Fact sheet A: §1.8 (haproxy and keepalived; the loadbalancer role), §2.4 (haproxy-config), §3.8 (memcached), §4.8 (chrony: the Ussuri role, which 2024.1 does not have), §5.8 (ovn; ovn-db and ovn-controller), §6.8 (ovs-dpdk), §7.3 (service-cert-copy), §8.3 (certificates), §9 (restart triggers and blast radius)
  • Fact sheet B: §1.8 (MariaDB), §2.8 (RabbitMQ), §3.9 (Keystone)
  • Fact sheet C: §1.8 (Glance), §2.8 (Placement), §3.8 (Cinder), §4.8 (Horizon), §5.8 (Heat), §6 (what reconfigure does per role)
  • Fact sheet D: §1.6 (Barbican), §3.8 (Swift), §4.6 (Manila), §5.0 (the roles that place containers on compute hosts), §7 (consolidated bugs and oddities)
  • Fact sheet E: §0 (most important findings), §15 (operational recommendations)

Investigate a component

Each of the 21 component pages says where the study covers the component and ends with its service card. Under "Where to look", the sections of the study are listed under these sub-headings when the study has something to list: Operate (the scenario and the command lines), Impact and risk (what the tag executes, and the risks), Dependencies, Evidence (the fact sheet, the image side), Before and after a run (the checks, prerequisites and precautions the study gives) and Known limits and open points (known bugs, open questions, findings, and what the sources do not establish) (How to read a component page).

In the table, "Risks" gives the risk numbers that the page lists in its "Risks" bullets, and "Per-role evidence" the fact sheet that the page names, or the page that it names in its place. A dash means that the page has no "Risks" bullet.

Component Role and tag Risks Per-role evidence
Keystone role keystone, tag keystone R19, R8, R24 fact sheet B §3
Glance role glance, tag glance R20, R21, R8, R24 fact sheet C §1
Placement role placement, tag placement R8, R24 fact sheet C §2
Nova roles nova and nova-cell; tags nova, nova-api(-*), nova-cell(-*) nova: R1, R2, R3, R24; nova-cell: R1 Nova deep dive
Neutron role neutron, tag neutron R5, R6, R7; membership changes: R3, R4 Reconfigure Neutron
Cinder role cinder, tag cinder R8, R24 fact sheet C §3
Horizon role horizon, tag horizon R8, R24 fact sheet C §4
Heat role heat, tag heat R8, R24 fact sheet C §5
Barbican role barbican, tag barbican – fact sheet D §1
Octavia role octavia, tag octavia – fact sheet D §2
Swift role swift, tag swift – fact sheet D §3
Manila role manila, tag manila – fact sheet D §4
MariaDB/Galera role mariadb, tag mariadb R9; mariadb_recovery: R18 fact sheet B §1
RabbitMQ role rabbitmq, tag rabbitmq (outward RabbitMQ removed in 2024.1) R3, R10 fact sheet B §2
Memcached role memcached, tags memcache, memcached R4 fact sheet A §3
HAProxy + Keepalived role loadbalancer (haproxy, keepalived, proxysql and, new in 2024.1, haproxy-ssh with Let's Encrypt); tags haproxy, keepalived, loadbalancer R8 fact sheet A §1, §2
Common: fluentd, cron, kolla_toolbox role common, tag common (its own play; no service tag runs it) R2; R25 does not apply to 2024.1 Repository architecture §5.1
Open vSwitch role openvswitch, tag openvswitch R7 fact sheet A §6 for ovs-dpdk
OVN roles ovn-controller and ovn-db; tags ovn (both), ovn-controller, ovn-db – fact sheet A §5
Chrony no role, play or tag in 2024.1 (the page says so, with the code) R2 and R25 do not involve chrony fact sheet A §4 (the Ussuri role)
iscsid / tgtd role iscsi, tag iscsi (only on &enable_iscsid_True hosts) – fact sheet D §5.2

Two more pages cut across the components: Component dependencies, which maps what depends on what and keeps four kinds of dependency apart (installation, configuration, runtime, operational), and Nova deep dive. The roles without a component page are in fact sheet D §5 (the roles that place containers on compute hosts, cyborg included) and fact sheet E (bootstrap-servers, applied from ansible-collection-kolla, and the operational playbooks).

Cases the study does not solve

The study does not diagnose a runtime failure of OpenStack, such as a failed VM deployment: no container or VM behaviour is used as evidence (Limits of this study). For these classic cases it offers the dependencies and the restarts read from the code, and marks the rest as open.

Case What the study offers What it does not establish
An instance fails to boot or to be scheduled (a VM deployment fails) Nova deep dive §3 lists what must be available before Nova can operate, ordered by necessity: MariaDB, RabbitMQ, Keystone and memcached, Placement, cell mapping and host mapping, Neutron server and an L2 agent on the compute, Glance and Cinder, libvirt. Nova deep dive §2.2 gives what each Nova component requires at runtime. Component dependencies §1 and Component dependencies §3 give the shared substrate and the control-plane services. Upstream Nova behaviour is not in these sources and is marked NOT AVAILABLE. Diagnosing the runtime failure itself is outside what the sources establish.
A new compute gets no instances Nova deep dive §3 (item 5): the scheduler does not discover hosts by itself (discover_hosts_in_cells_interval = -1). Add a compute node §2 says what deploy --limit runs for a new compute. Reconfigure Nova §6 lists nova-manage cell_v2 list_hosts among the checks after a Nova run. NOT AVAILABLE Whether a compute that was not discovered, and so is not in a host mapping, gets no instances.
An instance has no network Component dependencies §4 gives the networking containers, their placement rule, what they need and their data-plane role. Reconfigure Neutron §4 gives, per component, what a restart does in the code. For neutron-server and the OVS, L3, DHCP and metadata agents, the effect of a restart on existing connectivity is marked NOT AVAILABLE (What is not known, "Network data plane").
A live migration, a resize or a console fails Nova deep dive §6 gives the mechanism and the dependency of live migration, cold migration and resize, the VNC console and scheduling. NOT AVAILABLE Whether a nova_libvirt or nova_ssh restart interrupts operations in progress. NOT AVAILABLE Whether, with libvirt_tls: true, live migration to a new compute fails while the other computes' /etc/hosts lacks it (Add a compute node §2).
An API rejects requests: authentication Component dependencies §1 says how every service reaches MariaDB, RabbitMQ, memcached, Keystone and HAProxy. fact sheet B §3.10 counts the roles whose templates point at Keystone and memcached. NOT AVAILABLE: whether those API services fail to authenticate requests while Keystone or memcached is unavailable.
A volume cannot be attached Component dependencies §2 (row "iscsid / tgtd"): nova-compute (volume attach) and cinder-volume (LVM) depend on it. Tags §3.1 (row iscsi) and fact sheet C §3.8 give what the code shows for iscsid, tgtd and cinder-volume. NOT AVAILABLE Whether existing iSCSI sessions are kernel-held and whether attach/detach fails during an iscsid restart. NOT AVAILABLE: whether existing attachments survive a cinder-volume restart.
The Galera cluster is down Database operations gives the mariadb_recovery command line, with the comment "ONLY when the whole Galera cluster is down". R18 and fact sheet B §1.3 say what recovery does, and Kolla images §5.4 what the MariaDB image does not do. NOT AVAILABLE Whether the DB containers come back as a non-primary cluster after a full stop (What is not known, "Control plane").

Is it known?

  • Known bugs and limitations — the implementation limitations and bugs found in 2024.1-eom, by area (CLI, site.yml, kolla_container, the kolla images and individual roles), with their status against 2023.1. Two 2023.1 bugs no longer hold in 2024.1 (the outward RabbitMQ pull lookup and the ovs-dpdk handler name), and the new rows cover the Podman check exit codes, the cyborg endpoint variables, the Octavia jobboard precheck and the genconfig/HAProxy interaction (rows above)
  • Findings — seventeen headline findings (F1–F17), each with its evidence and why it matters. In 2024.1, F2 (-t common is a no-op), F3 (the timeouts are ignored) and F11 (the openvswitch_agent.ini merge) still do not hold; F16 (the first 2024.1 check restarts a container whose unit 2023.1 wrote) and F17 (RabbitMQ quorum queues on by default) are new CODE (rows above)
  • Open questions — Q1–Q21, with their status in 2024.1. In the rows of this page, Q14 is answered from the source CODE and the control-flow part of Q6 is answered from the source with ansible-core 2.16.20 CODE (MariaDB/Galera). Q10 was asked for Ansible 2.8 and 2.9; in 2024.1 the behaviours it was about are read from ansible-core 2.16.20 (Tags §2.1, Tags §2.4), and the ansible-core 2.15 side, which the CLI also accepts, is NOT AVAILABLE
  • What is not known — what the sources do not establish, grouped by topic, with the sections that mark each point NOT AVAILABLE
  • Fact sheet corrections — eleven corrections to the Ussuri fact sheets, each restated for 2024.1. The one with the widest effect in Ussuri, that the effective stop timeout was 10 s and not the 60 s of docker_graceful_timeout, does not apply to 2024.1, where the 60 s is used (R24) CODE. Their evidence is under Verification and errata
  • The lists that the fact sheets keep themselves: fact sheet C §8 and fact sheet D §8 (NOT AVAILABLE lists), fact sheet D §7 (consolidated bugs and oddities) and fact sheet E §14 (register of open questions: NOT AVAILABLE, or answered from code)
  • Kolla images §7 — what is not available from the kolla image sources
  • Risk register — the risks (R1 onwards), each with what actually happens, its blast radius, VM and API impact, likelihood, severity, precautions and evidence, and its status in 2024.1 (R25 does not apply)

Checks the corpus names

The checks that the study names before and after a run, as links. This page adds none. Docker healthchecks are wired (enable_container_healthchecks: yes by default), so docker ps shows a health status for the services that define a test, but no Kolla-Ansible task waits on it: service health must still be checked from outside CODE (Kolla images §6, item 17).

Page Section What it lists
Pre-execution checklist step 5, "Health gates" Before the run, and again after each batch: Galera (wsrep_cluster_size, wsrep_local_state_comment=Synced); RabbitMQ (cluster status, no partitions); openstack compute service list and openstack network agent list all up; HAProxy backends all UP.
Pre-execution checklist step 6, "Workload gates" No live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts.
Pre-execution checklist step 1, "Know the diff before you run" kolla-ansible genconfig with the same tags and --limit, a diff of /etc/kolla/<service>/ against the previous version, and kolla_set_configs --check on each container; in 2024.1 also a copy of services.d before genconfig.
Pre-execution checklist §2 "First 2024.1 run on a cloud deployed by 2023.1" What to expect and check on the first 2024.1 run: unit-file drift, quorum queues and the stricter RabbitMQ prechecks.
Predict and inspect a run the table The commands that show what a run would target, render and restart, among them "See what will restart" (the exit codes of kolla_set_configs --check) and "Pre-deployment checks" (prechecks, read-only).
Reconfigure a single service §3, "Sequence, impact, risks and verification of -t <svc> reconfigure" Verification before (genconfig, the check in each <svc> container, a diff of /etc/kolla/haproxy/services.d/<svc>*.cfg) and after (openstack endpoint list --service <name>, the API through the VIP, the container start times, docker ps --filter name=haproxy).
Reconfigure Nova §6, "Operational risks and verification" Before (genconfig and a diff of nova.conf, the check in nova_compute on a few computes, no active live migrations) and after (openstack compute service list, docker ps --filter name=nova_ -a, virsh list --all inside nova_libvirt on a sample of computes, nova-manage cell_v2 list_hosts).
Reconfigure Neutron §5, "Risks and verification" Before (openstack network agent list, the network nodes that host HA router masters, a diff of openvswitch_agent.ini on a compute after genconfig -t neutron) and after (the agent list, the ip netns count on network nodes, a continuous ping of a VM's floating IP during each batch).
Apply common changes (fluentd, cron, kolla_toolbox) §4, "Sequence, risks and verification" Verification before (genconfig, then the check in each of fluentd, cron and kolla_toolbox) and after (docker ps start times, docker logs fluentd, logs still reaching the configured output).
Update container images §5, "Verification" Before (docker images on a sample of hosts after pull, image IDs against the registry) and after (docker ps --format shows the new tag everywhere, docker inspect of nova_compute equals the pulled image ID).
Risk register the row "Precautions" of each risk The precautions to take before running the operation.
Tags §3.1 columns "Safe on its own?" and "Prerequisites" Per tag: whether it is safe on its own, and its prerequisites.
Component pages "Before and after a run" The checks, prerequisites and precautions the study gives for the component, when it has some.

Add a compute node and Change control-plane membership say that they give no verification steps.