Skip to content

Troubleshooting

A white stone aqueduct bridge spans turquoise water between green island cliffs.

Where to look when a kolla-ansible run did something unexpected, or did nothing. Each row of the symptom index names what an operator sees, restates what the study says the code does, and links to the section that establishes it. The page also says where to look for one component, what is already known as a bug or as an open point, and which checks the study names.

Important

The study rests only on reading source code: kolla-ansible 2023.1-eol, the kolla image repository 2023.1-eol, ansible-collection-kolla 2023.1-eol and the ansible-core 2.14.18 source. Nothing was run against a host, there are no timings, and every statement about data-plane or VM impact is NOT AVAILABLE (Limits of this study). All conclusions come from the 2023.1 implementation, not from Ussuri or from newer releases. Where a row says "no longer", "fixed" or "changed in 2023.1", it keeps a symptom that the Ussuri code produced and says what the 2023.1 code does instead.

During an incident, the portal can tell what Kolla will do given the inputs: tasks and restarts. What that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE in these sources (Code guarantees and environment dependence).

This page is an index. It adds no diagnosis and no procedure of its own.

Symptom index

Three mechanisms explain almost every surprise on the operation pages: config drift means a full container stop/start, never a reload; loop-notify, where one changed container in a role's loop restarts every container of that role on that host; and global inputs, which are rendered into per-host files on every host (Three mechanisms that explain almost every surprise).

In the tables, "What the code does" restates what the sections linked under "Read" say, and keeps their evidence labels. Rows that restate a fact sheet carry the words of the fact sheets, FACT and NOT AVAILABLE, which name the same two levels as CODE and NOT AVAILABLE (Evidence labels); for 2023.1 each such row was checked against the 2023.1 code, and where the code is cited in the row it carries CODE. A sentence without a label restates a sentence that carries none.

Containers restarted that should not have

What you see What the code does Read
A container was stopped and started for a config change; nothing was reloaded A config change is applied only by stopping and starting the container; kolla_docker has no reload action CODE. The stop timeout is the 60 s of docker_graceful_timeout: containers run under a systemd unit whose stop is docker stop -t <docker_graceful_timeout>; the Ussuri bug that forced 10 s is fixed in 2023.1 CODE. Repository architecture §4.3; Ten things to know (item 2); R24
Containers restarted although their own config did not change; every container of the role restarted on the host When a config or check task loops over a role's services and one item changes, ansible-core 2.14.18 notifies the handlers of all executed items: a change for one container restarts every container of that role on that host CODE. Tags §2.4; F1
nova_libvirt and nova_ssh restarted on every compute after a change that concerned only nova-compute With the default Nova settings, a nova-compute-only override restarts nova_compute, nova_libvirt and nova_ssh on every compute host CODE. Kolla's own handler ordering restarts libvirt before nova-compute CODE. Tags §2.4; Reconfigure Nova §4; R1; Nova deep dive §2.4 (the other triggers)
nova_api restarted after a scheduler-only override A scheduler-only override rewrites nova-scheduler/nova.conf only and restarts nova_scheduler and, by loop-notify, nova_api CODE. Reconfigure Nova §3; Tags §2.4
HAProxy restarted during a run with a service tag such as -t keystone Handlers ignore tags CODE: the haproxy-config task, imported under tags: always through loadbalancer-config, notifies Restart haproxy container, a listen topic of the loadbalancer role's handlers, which is why -t keystone can restart HAProxy. HAProxy restarts on every loadbalancer host if the rendered services.d/<svc>*.cfg changed: the backup hosts first, all at once, then the host that holds the VIP CODE. Tags §2.2; Reconfigure a single service §3; Reconfigure a single service §1.2 (--skip-tags haproxy); R8
HAProxy was restarted, not reloaded, on all haproxy hosts at the same time HAProxy is restarted instead of reloaded CODE. Every haproxy-relevant change goes to Restart haproxy container: a change in any one service's services.d/<svc>.cfg, a cert, haproxy.cfg or haproxy_run.sh. In 2023.1 the restart is staged, not simultaneous: the handlers find the host that holds kolla_internal_vip_address, stop and start haproxy on every backup host at once and wait for haproxy_monitor_port, then do the same on the VIP holder (roles/loadbalancer/handlers/main.yml:9-34, :54-118, :179-253) CODE. Known bugs and limitations (row haproxy); fact sheet A §1.8 (A); R8; HAProxy + Keepalived
fluentd, cron and kolla_toolbox restarted during an unrelated run such as -t nova Not from a service-tagged run in 2023.1: common is its own play, tagged common only, and no service play runs it (site.yml:86-97) CODE. Enabling a service that appears in fluentd_input_openstack_services or cron_logrotate_services (most OpenStack services, e.g. barbican) changes td-agent.conf and/or logrotate.conf on every host of the common group. The common containers then restart at the next -t common or untagged run, whatever that run was meant to change. Apply common changes §3; R2; Repository architecture §5.1
kolla_toolbox restarted although only the fluentd or the cron config changed Any drift restarts all three common containers on that host (loop-notify in check-containers.yml) CODE. Apply common changes §3 ("What restarts")
neutron_openvswitch_agent restarted on all computes and network nodes after an ml2_conf.ini override meant for the server No longer in 2023.1 CODE: ml2_conf.ini is rendered only for neutron-server and neutron-infoblox-ipam-agent, and openvswitch_agent.ini no longer merges it, so the override restarts neutron_server on the controllers (plus neutron_infoblox_ipam_agent where enabled) and no agent on the computes or network nodes (roles/neutron/tasks/config.yml:158-178, :197-212). An override of neutron/openvswitch_agent.ini restarts neutron_openvswitch_agent on all computes and network nodes; on network nodes the dhcp, l3 and metadata agents restart too (loop-notify in check-containers.yml) CODE. Reconfigure Neutron §3; R5; F11
Nova and neutron containers restarted on all computes after a controller was added, removed or replaced Changing the rabbitmq membership changes the transport URL everywhere, and changing memcached changes the memcache server lists. In the default inventory both are children of control, so a controller change does both. A memcached-only change does not touch nova-compute (the compute nova.conf has no memcache list) but still restarts the neutron agents on computes (neutron.conf). Change control-plane membership §1.1; Change control-plane membership §2; R3; R4
Fleet-wide restarts after an enable_* flag changed, for example enable_barbican: yes enable_barbican: yes, with Barbican not even deployed, rewrites nova.conf everywhere (adds [barbican], roles/nova/templates/nova.conf.j2:204-205, roles/nova-cell/templates/nova.conf.j2:246-247): all nova containers, libvirt included, restart on the next run of the nova plays CODE. It also changes td-agent.conf and logrotate.conf, so fluentd, cron and kolla_toolbox restart on every host of the common group when the common play next runs (-t common or an untagged run) CODE. Reconfigure Nova §3; R2; F6; Why a small change creates a large workload
A container restarts on every deploy/reconfigure, for example prometheus_server; kolla_set_configs --check exits 1 on every run kolla_set_configs --check exits 1 on every run for a config_files entry with preserve_properties and no perm, and for a directory source that contains subdirectories. So prometheus_server restarts on every deploy/reconfigure whenever {{ node_custom_config }}/prometheus/extras/ holds files CODE. Tags §3; Kolla images §3.1; Kolla images §6 (item 2); F14
A container restarted on a run where nothing was changed A restart handler runs only when a task that notifies it reports changed CODE. check-containers reports changed only when the container is missing, is not running, differs from the task in a compared parameter, has a missing or different systemd unit file, or kolla_set_configs --check exits 1 or 137 or meets a Docker client error CODE. NOT AVAILABLE Whether, on an unchanged cloud and apart from the exception of the row above, every task that notifies a restart handler reports ok, so that nothing restarts. Tags §3; Repository architecture §4.3; What is not known, "Predicting a run"
A container that was stopped, or that keeps exiting, was restarted by the run kolla_docker also treats as drift an exit code 137 of the check (the exec was killed because the container exited) and a docker exec client error (container stopped or restarting): a stopped container is restarted by the next run. Kolla images §3.3
After an image pull, the other containers of the role restarted too After a pull that downloaded something, the next deploy, reconfigure or upgrade recreates the affected containers, and loop-notify restarts their siblings CODE. With only the nova-compute image rebuilt, -t nova deploy after that pull recreates nova_compute and restarts nova_libvirt and nova_ssh on every compute (default Nova settings) CODE. Update container images §1.1; Update container images §2
A config change written earlier by genconfig, or by hand, was applied by a later, unrelated run A config file changed by an earlier genconfig, or by hand, still restarts the container on the next deploy/reconfigure/deploy-containers CODE. Between genconfig and the last deploy-containers batch, any other Kolla run that reaches those hosts applies the pending drift in files listed in config_files CODE. Repository architecture §4.3; Controlled batches §2 (caveats)
A container picked up new config after a Docker restart or a reboot, without a Kolla run Containers started again by their systemd unit after a Docker restart or a reboot also pick up the new config, because COPY_ALWAYS copies the files and rewrites /run_command at every container start CODE. Controlled batches §2 (caveats); Kolla images §3.5; fact sheet E §2.4
With COPY_ONCE, the container was recreated, not restarted COPY_ONCE: /configured lives in the container layer, so only a recreate re-applies config. kolla_docker always recreates under COPY_ONCE (module_utils/kolla_docker_worker.py:374-382) CODE. Kolla images §3.5
After a HAProxy restart, frontends changed that the run had not rendered The restart re-reads all services.d/* files: any drift left in /etc/kolla/haproxy/ by earlier runs (e.g. a previous genconfig, or stale files, which haproxy-config never removes, roles/haproxy-config/tasks/main.yml:2-23) goes live too CODE. fact sheet A §1.9; HAProxy + Keepalived
genconfig restarted services on Hyper-V hosts Does not apply to 2023.1: genconfig writes config but does not restart any Docker container, and the Ussuri exception, the unguarded handlers of the deprecated nova-hyperv role, is gone: 2023.1 has no nova-hyperv role and no Hyper-V support in ansible/roles/ CODE. Pre-execution checklist (step 1); Execution flow §4 (row genconfig)
openvswitch_vswitchd restarted although only openvswitch_db had drifted check-containers loops over both OVS services (roles/openvswitch/tasks/check-containers.yml:2-17), so with any openvswitch_db-only drift (e.g. a start-ovsdb-server change or a db image, volume or dimension change) loop-notify also notifies the vswitchd handler, which runs after the db restart and wait (roles/openvswitch/handlers/main.yml:2-29) CODE. R7; Tags §2.4
mariadb_clustercheck restarted on all nodes at once The clustercheck restart is not rolling: the handler runs in the role play, not in the batched restart plays, and restarts mariadb_clustercheck on all notified nodes at once (roles/mariadb/handlers/main.yml:75-91) CODE. With clustercheck enabled (default: follows enable_haproxy), HAProxy checks the backends on the clustercheck port (roles/mariadb/defaults/main.yml:64, :132). NOT AVAILABLE: whether HAProxy marks every backend down when that restart outlasts the check settings, which would mean a short total DB outage through the VIP. fact sheet B §1.8 (item 2); MariaDB/Galera
keepalived restarted after a host was inserted, removed or re-ordered in the haproxy group, or after a CA change keepalived priority is the index in groups['loadbalancer'] + 1 (roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15) CODE. Inserting, removing or re-ordering a loadbalancer host changes keepalived.conf on every host after it in the list, and each changed host goes through Restart keepalived container: in 2023.1 that also stops and starts haproxy (and proxysql) on each notified host, backups first, and stops the master's keepalived last, so the VIP moves CODE. How long the VIP is unavailable is NOT AVAILABLE. A CA bundle change (e.g. a root CA rotation with kolla_copy_ca_into_containers: yes) notifies the restart handlers of haproxy, proxysql and keepalived on the loadbalancer hosts (roles/loadbalancer/tasks/copy-certs.yml:2-6, roles/service-cert-copy/tasks/main.yml:2-12), with the same staged sequence CODE. fact sheet A §1.8 (C); fact sheet A §7.3; R22; HAProxy + Keepalived
keystone restarted after a memcached, rabbitmq or keystone node was added or removed keystone.conf embeds every memcached host and the RabbitMQ transport URLs (roles/keystone/templates/keystone.conf.j2:4, :51, :54), and the fernet crontab and fernet-push.sh embed the keystone host list and index (roles/keystone/tasks/config.yml:195-200, roles/keystone/templates/fernet-push.sh.j2:7-22) CODE. Adding or removing a memcached, rabbitmq or keystone node therefore restarts keystone, and keystone-fernet as well for keystone node changes. fact sheet B §3.9 (item 2); R19
All DB nodes restarted after the Galera membership changed Galera membership changes restart all DB nodes: galera.cnf lists every shard host (roles/mariadb/templates/galera.cnf.j2:29) CODE. In 2023.1 the restart runs in the separate Restart mariadb services play, in serial: "33%" batches (mariadb.yml:14-26) CODE. Change control-plane membership §2; fact sheet B §1.8 (item 11); R9
The RabbitMQ brokers restarted one by one after the group was scaled or a password changed Scaling the rabbitmq group changes rabbitmq.conf (the classic_config node list, roles/rabbitmq/templates/rabbitmq.conf.j2:18-20) on every broker, and a password change changes definitions.json; the config tasks notify Restart rabbitmq container, which only groups the host (roles/rabbitmq/handlers/main.yml:2-4) CODE. The restart then runs in the Restart rabbitmq services play, serial: "33%" (one broker at a time with three), each broker put into maintenance mode (drained) first (rabbitmq.yml:26-41, roles/rabbitmq/tasks/restart_services.yml:9-39) CODE. fact sheet B §2.8 (items 4 and 5); RabbitMQ
Every heat-engine restarted in the same step FACT: with serial 0, every heat-engine restarts in the same handler flush on any heat.conf change; a memcached or rabbit membership change, a password rotation or an openstack_service_workers change is enough. fact sheet C §5.8
glance_api was recreated after the Cinder iSCSI/LVM backend was enabled With enable_cinder, enabling enable_cinder_backend_iscsi (or cinder_backend_ceph) flips glance-api to privileged, and the iSCSI backend also adds the iscsi_info volume and /dev (roles/glance/defaults/main.yml:10, :192-193), so glance-api is recreated CODE. fact sheet C §3.8 (item 7)
Every container of several services restarted after a CA or backend certificate rotation A CA or backend cert rotation restarts every container of every service using it: service-cert-copy loops over the role's services and notifies Restart <service> container for each (roles/service-cert-copy/tasks/main.yml:2-56) CODE; whole fleet, per-play parallel. fact sheet A §7.3
All OVN DB members restarted at once; ovn_controller restarted on all computes and network nodes in parallel In 2023.1 OVN is two roles, ovn-db and ovn-controller. Any change to ovn-nb-db/ovn-sb-db restarts every member at once under serial 0 (the ovn-db play's serial is kolla_serial, default 0, site.yml:585; roles/ovn-db/handlers/main.yml:2-45) CODE. ovn-controller restarts happen on image or container-parameter change or when a deploy/reconfigure rewrites its config.json, and hit all its hosts in parallel. fact sheet A §5.8 (A and C); OVN
On ovs-dpdk hosts, ovs-vswitchd restarted after the ovsdpdk-db restart Any ovsdpdk-db restart (an image change or config.json drift) also restarts ovs-vswitchd, on all DPDK hosts in parallel: the db restart handler notifies Restart ovsdpdk-vswitchd container (roles/ovs-dpdk/handlers/main.yml:16-23) CODE. The same handler also notifies wait for dpdk tunnel ip, while the handler is named Wait for dpdk tunnel ip (:88); ansible-core 2.14.18 matches handler names case-sensitively and raises an error for a notified handler it cannot find (lib/ansible/plugins/strategy/__init__.py:495-520, :651-656; ERROR_ON_MISSING_HANDLER defaults to true, lib/ansible/config/base.yml:1278-1280; Kolla ships no ansible.cfg) CODE. NOT AVAILABLE: how that error ends the run on such a host, and whether the restarts are a data-plane outage for all VMs on that host. fact sheet A §6.8; Open vSwitch
The Docker daemon restarted during bootstrap-servers In 2023.1 the work comes from ansible-collection-kolla (kolla-host.yml:4-10 applies openstack.kolla.baremetal). Restart docker is a handler, notified when the /etc/docker/daemon.json write reports changed (the rendered content differs, or the mode is not 0644), when the registry CA file changes, and through Reload docker service file when the docker.service.d/kolla.conf drop-in is written or removed (ansible-collection-kolla roles/docker/tasks/config.yml:9-107, roles/docker/handlers/main.yml:2-15) CODE. It is flushed at Flush handlers (config.yml:110-111), so it runs at the same moment on every host in the batch, and the batch is all hosts by default (kolla-host.yml:6) CODE. fact sheet E §0 (item 1); fact sheet E §2.4; R14; Q13
A service restarted on every host at once instead of one host at a time Without batching, handlers restart a service on every host of the play at once. Only MariaDB and RabbitMQ roll by themselves: their restarts run in separate plays with serial: "33%" (one node at a time with three nodes), and each RabbitMQ node is drained first CODE. Ten things to know (item 4); Execution flow §5; Execution flow §7; Controlled batches §1.2
Containers restarted in an order that is not the order of the changes Within a role, restart order is handler definition order, not notification order CODE: nova-cell restarts nova_ssh, then nova_libvirt, then nova_compute (roles/nova-cell/handlers/main.yml:74, :92, :151); openvswitch restarts openvswitch_db, waits, then restarts openvswitch_vswitchd (roles/openvswitch/handlers/main.yml:2, :20, :29). Execution flow §7; Tags §5

Nothing happened, or the change was not applied

Rebuilt images that a run did not pick up are under Images.

What you see What the code does Read
-t common reconfigure changed nothing: a fluentd, cron or kolla_toolbox change was not applied (no-op) Does not hold in 2023.1 CODE: common is its own play with play-level tags: common (site.yml:86-97), and the tasks of the dynamic include in roles/common/tasks/main.yml:2 inherit that tag through their parent block (ansible-core 2.14.18 lib/ansible/plugins/strategy/__init__.py:819-820, lib/ansible/playbook/block.py:355-363). -t common reconfigure reconfigures fluentd, cron and kolla_toolbox on every host of the common group. If nothing changed, check that the change is really in the rendered files (genconfig -t common). Apply common changes §2; Apply common changes §1.1 (the -t common command); Tags §2.1; F2; R25
The common change was applied later, by a run that was not meant to apply it common changes (fluentd/cron/toolbox) are applied only by a run that includes the common play: -t common or an untagged run, on every host of the common group. A service-tagged run never applies them CODE (site.yml:86-97). F2 ("Why it matters"); R25; Apply common changes §3
kolla_set_configs --check returns 0 and nothing restarts after a change to a container's command line, for example a memcached setting; genconfig then deploy-containers did not apply it --check compares only the config_files entries of config.json: a change confined to config.json itself (command, permissions; all of memcached's settings) is never detected CODE. A normal deploy/reconfigure applies it, because the changed template task notifies the restart handler; once genconfig has written it, neither deploy-containers nor a later run applies it until the container restarts for another reason. F14; Kolla images §3.2; Kolla images §6 (item 1); Controlled batches §2 (caveats)
An OVN command-line change was not applied FACT: the OVN manifests have no config_files, so kolla_set_configs --check always exits 0: a command-line change written by genconfig is applied neither by deploy-containers nor by a later run, until the container restarts for another reason. fact sheet A §5.8 (C)
A file that was removed from the config is still in the container after the restart In 2023.1 a restart removes most dropped files CODE: every start deletes each recorded destination that did not exist at the container's first start, and restores the others to their original. Exceptions CODE: glob sources are never recorded (e.g. prometheus extras/*), so their destinations stay; a destination on a volume that already held a file when the container was created is restored rather than deleted; nothing happens after the first start under COPY_ONCE. --check does not flag dropped entries, so the removal waits for a restart for another reason CODE. Deleting a custom file such as /etc/kolla/config/keystone/keystone-paste.ini also leaves its rendered copy in /etc/kolla/keystone/, which the optional keystone-paste.ini entry copies in again at every start (roles/keystone/templates/keystone.json.j2:18-24) CODE. Kolla images §3.5; Kolla images §6 (item 4); Repository architecture §7 (item 4)
A custom CA that was deleted is still installed in the containers A CA deleted from kolla_certificates_dir/ca/, or left there after kolla_copy_ca_into_containers is turned off, stays in /etc/kolla/<svc>/ca-certificates/, because the copy task never deletes files (roles/service-cert-copy/tasks/main.yml:2-12) CODE, so it is re-added at every start until it is deleted there by hand. Kolla images §3.5
The HAProxy frontend of a disabled service is still loaded Stale services.d files are never removed CODE: haproxy-config only templates (roles/haproxy-config/tasks/main.yml:2-23); the only removal is mariadb.cfg when ProxySQL is enabled (roles/loadbalancer/tasks/config.yml:60). If a service is disabled (enable_X: no), its include_role is skipped; the old /etc/kolla/haproxy/services.d/X.cfg stays and is loaded by haproxy_run.sh on the next restart. Known bugs and limitations (row haproxy); fact sheet A §1.8 (D); fact sheet A §2.4
A Keystone domain config file that was removed is still copied into the container Removing a file from /etc/kolla/config/keystone/domains is never propagated: no task deletes stale files in /etc/kolla/keystone/domains/ (roles/keystone/tasks/config.yml:106-128 only copies). Under the default COPY_ALWAYS the stale file is copied back into the container at every start, even after a recreate, because the domains directory is a listed source (roles/keystone/templates/keystone.json.j2:26-27) CODE. fact sheet B §3.9 (item 8)
A swift config or ring change was not applied by deploy No handlers are notified in the swift role: deploy does not apply config or ring changes CODE; runtime effects NOT AVAILABLE. Known bugs and limitations (row swift); fact sheet D §3.4; fact sheet D §7 (item 4); Swift
A changed heat_domain_admin_password was not applied in Keystone bootstrap_heat creates heat_user_domain/heat_domain_admin only when the domain is absent: a changed heat_domain_admin_password reaches heat.conf but never Keystone (kolla docker/heat/heat-api/extend_start.sh:8-14, roles/heat/tasks/bootstrap_service.yml:22) CODE; impact NOT AVAILABLE. Known bugs and limitations (row heat); Kolla images §6 (item 7)
A changed rabbitmq_cluster_cookie was not applied to the brokers The image writes .erlang.cookie only in the bootstrap container (new volume). A cookie change recreates the brokers (env compare) but they keep the old cookie, while kolla_toolbox gets the new one CODE; consequences NOT AVAILABLE. Known bugs and limitations (second row rabbitmq); Kolla images §6 (item 9)
An ovs-dpdk NIC mapping change was not applied The NIC mapping file is written once CODE: ovs-dpdkctl.conf is generated only if it is absent (roles/ovs-dpdk/files/ovs-dpdkctl.sh:350-353), so mapping changes require deleting it manually. Known bugs and limitations (row ovs-dpdk); fact sheet A §6.8
After -t keystone, the HAProxy backend list of another service was not updated Under -t keystone, other services' .cfg files are not re-rendered: each per-service include_role … tasks_from: loadbalancer of the loadbalancer play carries its own service tag (site.yml:110-341) CODE. If another service's backend list changed (say, a new controller), it is not updated. fact sheet A §1.9
/etc/hosts on the existing hosts does not list the new compute after a run with --limit /etc/hosts on existing hosts is not updated with the new compute, because bootstrap-servers was limited: the block is written by ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:25-41 on the hosts the run reaches only CODE. With --limit, /etc/hosts and other cross-host files are not updated on non-limited hosts. Add a compute node §1.1; Add a compute node §2; R23; fact sheet E §3
An expired certificate was not renewed by a new run of the certificates playbook; a new backend host is not in the backend certificate Test CA only; certs never renewed; new backend hosts not added CODE. Every command step of the certificates role is creates:-guarded (14 of 14 in roles/certificates/tasks/), so re-running never renews expired certificates; the backend certificate's host list (roles/certificates/templates/openssl-kolla-backend.cnf.j2:16) is used only when backend.csr does not exist yet (roles/certificates/tasks/generate-backend.yml:23-48). Known bugs and limitations (row certificates); fact sheet A §8.3
New certificates were generated, but the containers still use the old ones Distribution needs a separate run: certs only reach containers through kolla-ansible -t haproxy reconfigure (pem files, roles/loadbalancer/tasks/config.yml:193-225) or a per-service reconfigure (backend and CA, service-cert-copy) CODE. Each of those means a container restart. fact sheet A §8.3
Neutron's DB sync did not run on reconfigure No longer in 2023.1 CODE: reconfigure.yml imports deploy.yml, and bootstrap_neutron has no when, so a reconfigure re-runs Neutron's DB create/grant tasks and its DB sync; neutron_enable_rolling_upgrade is read only by upgrade.yml (roles/neutron/tasks/reconfigure.yml:2, bootstrap_service.yml:2-21, upgrade.yml:5, :8). F5; Reconfigure Neutron §2
The tag service-rabbitmq ran nothing The tag is on the RabbitMQ user/vhost block inside the service-rabbitmq role (roles/service-rabbitmq/tasks/main.yml:46-47), which is reached through dynamic includes: on its own it executes nothing and is not usable as a selector CODE. Tags §3 (row service-rabbitmq)
-t memcache did not update the HAProxy file of memcached -t memcache reconfigures the container but not its HAProxy file; only -t memcached also selects the memcached include of the loadbalancer play (site.yml:223-227). That include renders a frontend only with enable_haproxy_memcached (default no, group_vars/all.yml:637), so with the defaults the two tags differ in nothing visible CODE. fact sheet A §3.8; Tags §3 (row memcached / memcache); Memcached
The nova-reload, nova-api-reload and nova-cell-reload tags did nothing The reload plays run recreate_or_restart of nova services only when kolla_action == upgrade: a no-op outside upgrade. Tags §3 (row nova-reload)
deploy-containers did not update the HAProxy frontends deploy-containers writes no config, and the per-service loadbalancer includes run only for deploy, reconfigure, upgrade and config (site.yml:110-341): only the loadbalancer role's check-containers.yml runs, and only under -t haproxy (or keepalived, loadbalancer) or no tags CODE. Execution flow §4 (row deploy-containers); fact sheet A §1.9
After the compute batches, config drift is still pending on the controllers genconfig -t nova-cell also renders the conductor and console config on controllers. Include them in a batch too, or the drift stays pending there. Controlled batches §1.1 (caveat)
After a tagged run, some hosts still have the old fluentd or logrotate files After enabling or disabling a service that has a logrotate or fluentd input entry (enable_X), td-agent.conf and logrotate.conf change on the hosts of the common group only when the common play runs: -t common or an untagged run, on every host of that group CODE. A service-tagged run leaves every host with the old files (site.yml:86-97), and fluentd, cron and kolla_toolbox restart only when the common play next runs. R2; Apply common changes §3
mariadb_backup did nothing: every task was skipped The backup container runs only on mariadb_backup_host, the first host of the MariaDB shard group (roles/mariadb/defaults/main.yml:123, roles/mariadb/tasks/backup.yml:33-35), and the playbook applies the role only with enable_mariabackup: yes (default no; mariadb_backup.yml, group_vars/all.yml:709); otherwise every task is skipped silently CODE. fact sheet E §9; Database operations
deploy -t barbican alone: the barbican_notifications topic is not enabled Running kolla-ansible deploy -t barbican alone does not run the keystone play, so the barbican_notifications topic, which the keystone role enables (roles/keystone/defaults/main.yml:200-201), is not enabled in keystone.conf CODE. NOT AVAILABLE: whether barbican-keystone-listener then receives project-deletion events. fact sheet D §1.6; Barbican
No amphora flavor, image, network or security group exists after an Octavia deploy Changed in 2023.1 CODE: with octavia_auto_configure (default: true when amphora is in octavia_provider_drivers, group_vars/all.yml:1228), deploy runs prepare.yml, which creates the amphora flavor, the nova keypair, the security groups and rules, and the load-balancer management network and subnet (plus a router for IPv6) (roles/octavia/tasks/deploy.yml:4-5, roles/octavia/tasks/prepare.yml:2-161). The amphora image is not created by the role. With octavia_auto_configure off, none of these is created. What the os_* modules do with resources that already exist with other settings is NOT AVAILABLE: the modules are not in these sources. Known bugs and limitations (row octavia); fact sheet D §2.5; Octavia
No share type, service image or network exists after a Manila deploy Kolla does not create share types, the service image or networks CODE (no such task under roles/manila/tasks/). fact sheet D §4.6; Manila
A service tag did not touch MariaDB, RabbitMQ, memcached or Open vSwitch A service tag never touches MariaDB, RabbitMQ or memcached containers (other plays), and it never touches OVS (unless -t openvswitch): Open vSwitch is a separate play with its own tag (site.yml:546-555), not touched by -t neutron CODE. Reconfigure a single service §2; Reconfigure Neutron §4

The run failed or a task failed

What you see What the code does Read
The check-containers task failed on a host; kolla_set_configs --check exit code 2 Exit 2 means an environment problem, not drift: a config.json that cannot be opened, or a uid or gid with no passwd or group entry; kolla_docker then fails the check task on that host instead of restarting (kolla docker/base/set_configs.py:188, :193, :285, :579-581; module_utils/kolla_docker_worker.py:187-190) CODE. Changed in 2023.1: an optional file rendered for the first time (e.g. nova-compute release) no longer gives exit 2; it is drift, exit 1, and the restart copies the new file (set_configs.py:166-169) CODE. Kolla images §3.3; Kolla images §6 (item 3); Predict and inspect a run (row "See what will restart"); Controlled batches §1.1
Every later run fails on the same check task Not for a newly added prometheus extras file in 2023.1: a missing destination is drift (exit 1), and an extras entry with preserve_properties exits 1 on every run, so prometheus_server restarts on every deploy/reconfigure instead of failing CODE. A check task that fails on every run has exit 2 (previous row), which a later run does not clear by itself. Tags §3; Kolla images §3.3
docker exec <c> kolla_set_configs --check run by hand exits 2 In an image with a service USER (e.g. USER nova), a plain docker exec <c> kolla_set_configs --check cannot open the files in /etc/kolla/<service>/, which are owned by root with mode 0660, and exits 2. kolla_docker execs as root; by hand, use docker exec -u root. Kolla images §3.4; Predict and inspect a run (row "See what will restart")
After the restart, the container fails at every start Exit 1 is also returned for a required source that is missing and for an invalid config.json; the restarted container then fails in kolla_set_configs at every start CODE. What its systemd unit then does is NOT AVAILABLE. Kolla images §3.3; Predict and inspect a run (row "See what will restart")
deploy or reconfigure fails in the mariadb role after kolla-ansible stop, or while a node is not Synced Reconfigure aborts if the cluster is fully stopped (multi-node) or any alive node is not Synced (roles/mariadb/tasks/lookup_cluster.yml:35-42, :72-78) CODE. kolla-ansible stop stops every node at once, so the next deploy/reconfigure fails until mariadb_recovery is run. fact sheet B §1.8 (item 4); R15; Database operations
The MariaDB restart handler failed while it waited for WSREP sync In 2023.1 the wait is a task of the Restart mariadb services play, not a handler: about 60 s (10 × 6 s, roles/mariadb/tasks/restart_services.yml:31-45) CODE; a restart that needs IST or SST longer than that fails. The restart plays set neither any_errors_fatal nor max_fail_percentage (mariadb.yml:14-40) CODE: a node that fails its wait marks only its own host failed; when that is every host of the batch (always so with three nodes, one host per batch), ansible-core ends the playbook and the later batches are not restarted; when other hosts of the batch succeed, the next batch still restarts its nodes (ansible-core 2.14.18 lib/ansible/executor/playbook_executor.py:204-208, lib/ansible/utils/helpers.py:25-32) CODE. NOT AVAILABLE: whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is. fact sheet B §1.8 (item 3); fact sheet B §1.5 (failure mode); R9; Known bugs and limitations (row mariadb); Q6
kolla-ansible pull with outward RabbitMQ enabled: outward_rabbitmq_services Bug CODE: the outward plays set project_name: outward_rabbitmq and no kolla_role_name (rabbitmq.yml:71), so service-images-pull looks up outward_rabbitmq_services (roles/rabbitmq/tasks/pull.yml:2-3, roles/service-images-pull/tasks/main.yml:14), and that variable is not defined anywhere. The vars lookup raises an undefined-variable error for a name it cannot find when no default is given (ansible-core 2.14.18 lib/ansible/plugins/lookup/vars.py:94-104), so "Pull images" fails on the outward-rabbitmq hosts. fact sheet B §2.8 (item 6); Known bugs and limitations (first row rabbitmq)
pull for ovs-dpdk: ovs_services Fixed in 2023.1 CODE: pull imports service-images-pull, which reads ovsdpdk_services through kolla_role_name: "ovsdpdk" (roles/ovs-dpdk/tasks/pull.yml:2-3, roles/ovs-dpdk/vars/main.yml:4, roles/service-images-pull/tasks/main.yml:14). Known bugs and limitations (row ovs-dpdk); fact sheet A §6; Open vSwitch
prechecks passed, but deploy failed in the keystone role at set_fact cron_jobs The precheck validates the wrong variable: it asserts fernet_token_expiry <= 604800 (roles/keystone/tasks/precheck.yml:43-53), while the generator is fed fernet_key_rotation_interval, which is fernet_token_expiry + fernet_token_allow_expired_window (roles/keystone/tasks/config.yml:195-200, group_vars/all.yml:882-889) and must not exceed one week (roles/keystone/files/fernet_rotate_cron_generator.py:58-62). The generator exits 0 with {"failed": true} (:31-36, :117), so the command task succeeds and set_fact cron_jobs then errors on a missing key (config.yml:207-210) CODE. Any fernet_token_expiry above 5 days with the default window (2 days) passes precheck but breaks deploy. fact sheet B §3.9 (item 5); Known bugs and limitations (row keystone)
The swift reconfigure fails The swift reconfigure fails if any listed container is not running (failed_when: not container_state.Running, roles/swift/tasks/reconfigure.yml:32-39), so reconfigure cannot repair a crashed container CODE. fact sheet D §3.4; fact sheet D §7 (item 4)
upgrade: the task "Stopping all slave keepalived containers" with enable_keepalived: no, or with an api_interface whose name contains - Does not exist in 2023.1 CODE: the loadbalancer role's upgrade.yml imports deploy.yml and has no "Stopping all slave keepalived containers" task (roles/loadbalancer/tasks/upgrade.yml:2). Known bugs and limitations (row haproxy); fact sheet A §1.8 (F); HAProxy + Keepalived
bootstrap-servers failed on every host at the /etc/hosts task The block loops over all groups['baremetal'] hosts and calls kolla_address for each (ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:25-41), so every baremetal host's facts are required CODE. A baremetal host outside bifrost without gathered facts (for example unreachable during fact gathering, with no fact cache) makes kolla_address raise on every host (kolla-ansible kolla_ansible/kolla_address.py:52-95), which fails the task on every host and ends the play for them. fact sheet E §0 (item 4); fact sheet E §3
A run with --limit failed because a host outside the limit is unreachable With --limit, the limited hosts gather the facts of the other hosts in a delegated loop (gather-facts.yml:30-54). An unreachable other host without cached facts makes its loop item unreachable; ansible-core counts a loop result as unreachable if any item is, and then marks the limited host unreachable and removes it from the play and from every later play (ansible-core 2.14.18 lib/ansible/executor/task_result.py:72-73, lib/ansible/plugins/strategy/__init__.py:591-595) CODE. So --limit newnode can fail because an unrelated node is down. fact sheet E §4; fact sheet E §14 (H4)
Templating failed in kolla_address: interface not present on host kolla_address raises FilterError if the interface or address family is missing on the host (kolla_ansible/kolla_address.py:66-95) CODE. Every octavia host must have octavia_network_interface with an address, or templating fails. The setup filter documented as an example in group_vars/all.yml:29 ("ansible_[!qt]*", not the default, :31) also hides real interfaces whose names start with q or t; kolla_address then fails for api_interface, tunnel_interface and so on if they are named that way. fact sheet D §2.6; fact sheet E §4; fact sheet E §14 (H16)
Jinja errors on groups['ovn-…'] There is no group validation precheck: the prechecks of ovn-db and ovn-controller check only free ports (roles/ovn-db/tasks/precheck.yml; roles/ovn-controller/tasks/precheck.yml is empty) CODE, so an old inventory without the ovn-* groups only surfaces as Jinja errors on groups['ovn-…']. fact sheet A §5.8 (H); OVN
-t nova failed at Keystone registration on a fresh cloud Nothing enforces a dependency when you use tags CODE: -t nova on a fresh cloud fails at Keystone registration if Keystone was never deployed. On an existing cloud it assumes MariaDB, RabbitMQ and Keystone are healthy. Component dependencies §6
A bootstrap or registration task failed on the first host of a group Bootstrap and registration tasks target groups[...][0] CODE: the first host of a group is a hidden dependency; it must be reachable and have kolla_toolbox running. Component dependencies §6
mariadb_recovery tried to start mariadb_clustercheck although it is disabled The recovery restart tasks loop over mariadb_services with no item.value.enabled or group check (roles/mariadb/tasks/recover_cluster.yml:171-185, :202-216) CODE. With enable_mariadb_clustercheck: no, recovery still tries to start a mariadb_clustercheck container; if the image pull returns an error, kolla_docker fails the task on that host. In deploy and reconfigure, only enabled items are notified (roles/mariadb/tasks/config.yml:42-53, check-containers.yml:13-18). fact sheet B §1.8 (item 6); Known bugs and limitations (row mariadb)
docker.service is masked after a failed or interrupted bootstrap-servers run on Debian/Ubuntu On Debian/Ubuntu the run masks docker.service when no unit file exists in /etc/systemd/system/ yet, and unmasks it later, either when the package install changed something or at the final Start and enable docker (ansible-collection-kolla roles/docker/tasks/install.yml:15-30, :44-53, roles/docker/tasks/config.yml:113-119) CODE. A failure or interruption between those points leaves the symlink in place. NOT AVAILABLE: whether Docker starts after the next reboot. fact sheet E §2.3; fact sheet E §0 (item 3); Known bugs and limitations (row baremetal)
Prechecks: the port or the address that a precheck tests The swift precheck probes rsync port 873, but rsyncd uses 10873 (roles/swift/tasks/precheck.yml:89, group_vars/all.yml:544) CODE. The octavia health-manager precheck tests the port on api_interface_address while the health manager binds octavia_network_interface_address (roles/octavia/tasks/precheck.yml:29-38, roles/octavia/templates/octavia.conf.j2:79); they differ only when octavia_network_interface is not api_interface CODE. The iscsi precheck tests that iscsi_port (3260) is free on the iscsid hosts, while 3260 is the target (tgtd) port (roles/iscsi/tasks/precheck.yml:17-27) CODE. Known bugs and limitations (row swift); fact sheet D §2.6; fact sheet D §7 (items 2, 5 and 11)
The RabbitMQ handler blocks after an uncontrolled full stop After an uncontrolled full stop (kolla-ansible stop, power loss) followed by deploy, the brokers come back through the batched Restart rabbitmq services play (serial: "33%", one broker per batch with three), each start followed by rabbitmqctl wait --timeout 60 (rabbitmq.yml:26-41, roles/rabbitmq/tasks/restart_services.yml:34-39) CODE. Which node starts first follows the group_by order: NOT AVAILABLE. NOT AVAILABLE (RabbitMQ semantics): whether, if that node was not the last to stop, it waits for its peers' Mnesia tables before giving up, so that the first batch stalls or fails before the other brokers are started. fact sheet B §2.8 (item 3); RabbitMQ
stop and the swift account-auditor The swift stop.yml uses the wrong container name CODE: the account-auditor stop checks 'swift_object_auditor' not in skip_stop_containers (roles/swift/tasks/stop.yml:23-31). Known bugs and limitations (row swift); fact sheet D §3.4

The run is slow

What you see What the code does Read
A run with --limit is slow on a large inventory: it spends a long time gathering facts With --limit, the other hosts' facts are gathered sequentially by the limited hosts (delegated loop, the hosts being divided between the limited hosts, gather-facts.yml:30-54) CODE. With --limit compute001 on the 30-compute inventory (38 hosts, localhost included), compute001 runs 38 setup calls in sequence: its own and one for each of the 37 other hosts CODE; its duration is NOT AVAILABLE. F9; Compute scaling §6 (item 5); R23; fact sheet E §4
A run with no change is still slow on a large fleet Every run that includes the compute group executes the full per-host chain on every compute, even with no change: Kolla has no "only if changed" short-circuit at play level. Per compute, for the default 2023.1 service set with Cinder LVM, nova_compute_virt_type: kvm and a non-Ubuntu host, a no-change -t nova reconfigure makes 26 remote module invocations (30 on compute001) CODE. Compute scaling §5; Compute scaling §2; Why a small change creates a large workload
The run is slower with ANSIBLE_SERIAL or kolla_serial kolla_serial bounds simultaneous restarts per play, but adds time: every batch re-runs run_once steps and the compute registration wait, and each batch waits for the slowest host. Compute scaling §7; Tags §2.5
One slow host slows the whole run A task runs on all hosts of the play (up to forks at a time), and the next task starts only when every host has finished the current one CODE: the slowest host sets the pace of every task. Compute scaling §1
The restart window of an upgrade includes image downloads Upgrade without a prior pull: hosts download lazily inside the handler phase (recreate_or_restart_container pulls before stopping), which lengthens and staggers the restart window. Update container images §4; Nova deep dive §7; Pre-execution checklist (step 4)
The run waits for compute registration (up to 20 × 10 s) The compute registration wait runs for the first compute of each batch, delegated to the conductor, for up to 20 × 10 s per batch, multiplied by the number of batches when serial is used: it waits until all computes of the batch appear in compute service list. Compute scaling §3
A 300 s wait on ovs-dpdk hosts The Wait for dpdk tunnel ip handler is a wait_for on a host with no port and a 300 s timeout (roles/ovs-dpdk/handlers/main.yml:88-92) CODE. In 2023.1 the ovsdpdk-db restart handler notifies it as wait for dpdk tunnel ip (:16-23), a name that ansible-core 2.14.18 does not match to it (case-sensitive); the notify raises a missing-handler error instead (see Containers restarted that should not have) CODE. Whether the wait handler runs at all, and what an operator then sees, is NOT AVAILABLE. Known bugs and limitations (row ovs-dpdk); fact sheet A §6.8
bootstrap-servers is slow on a large inventory /etc/hosts lists every baremetal host, rendered on every host: O(N²) templating (ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:25-41) CODE. Known bugs and limitations (row baremetal); fact sheet E §0 (item 4); Compute scaling §3
A container is slow to start kolla_set_configs makes one chown per file of the glance image store, of /var/lib/nova on computes, of the MariaDB datadir, of the RabbitMQ mnesia directory and of the cinder volume (twice for cinder-volume), on every COPY_ALWAYS start CODE. NOT AVAILABLE How start time grows with the number of files, especially on NFS-backed trees. Kolla images §6 (item 10)
Horizon is slow to start after a recreate or a settings change FACT: collectstatic and compress run on the first start after a recreate (new image, plugin flag change, COPY_ONCE) or after a local_settings/custom_local_settings change, and not on a plain restart with unchanged settings and plugin flags. NOT AVAILABLE: how long they take. fact sheet C §4.8; Kolla images §5.2
Every task costs several SSH round trips Kolla ships no ansible.cfg, so pipelining stays off and each invocation still makes several round trips (tmp dir, transfer, exec, cleanup); ansible-core multiplexes the SSH connection by default (ControlPersist=60s, ansible-core 2.14.18 lib/ansible/plugins/connection/ssh.py:75-77) CODE. Compute scaling §3 (row "SSH connection setup"); Compute scaling §8
The deploy host is busy during the config tasks template and merge_configs render on the deploy host for every host and item, then copy compares checksums remotely CODE: deploy-host CPU scales with hosts × templates. Whether, and at which forks value, the deploy host's CPU becomes the bottleneck is NOT AVAILABLE. Compute scaling §1; Compute scaling §6 (item 3)
How long a run should take No timings are given. Work is counted as operations read from the source; task-result totals and wall times are NOT AVAILABLE. Compute scaling; What is not known, "Timings, counts and scale"; Q7; Q8

--limit seems ignored

--limit does not confine actions (F8). The rows about --limit are in four tables of this index:

Scope surprises: hosts or services touched outside what was asked

What you see What the code does Read
Tasks ran on a controller (control01) although --limit named only a compute --limit does not confine actions: delegate_to: groups[x][0] runs work on hosts outside the limit. With deploy --limit set to a compute host, control01 (the first neutron-server and nova-conductor host of the multinode inventory) runs the Neutron DB create/grant and the bootstrap_neutron container, plus cell discovery CODE. F8; Add a compute node §2; Reconfigure a single service §2 (row "Actions on hosts outside --limit")
DB-sync (bootstrap) containers and Keystone registration ran on a reconfigure reconfigure = deploy: in 2023.1, 59 roles have a tasks/reconfigure.yml, 57 of them run deploy.yml and only swift and bifrost have their own logic, so a reconfigure re-runs Keystone registration, DB creation/grants and the DB-sync bootstrap containers of most services, Neutron's included CODE. F5; Execution flow §6; Repository architecture §4.2; Tags §4 (item 5)
A service tag also ran common (fluentd, cron, kolla_toolbox) on every host of the service play Does not happen in 2023.1 CODE: no role declares common as a meta dependency any more (no roles/*/meta directory), and common is its own play tagged common only (site.yml:86-97). A service tag leaves fluentd, cron and kolla_toolbox untouched; their pending changes wait for -t common or an untagged run. Reconfigure a single service §2; Tags §4 (item 1); Repository architecture §5.1
A service tag rendered the HAProxy frontend of the service on all haproxy hosts -t <svc> runs that service's loadbalancer include, and so its haproxy-config, on all loadbalancer hosts: the per-service include_role … tasks_from: loadbalancer tasks of the loadbalancer play carry the service tag (site.yml:110-341) CODE. Tags §2.3; fact sheet A §1.9
A tag ran tasks on every compute nova, nova-cell, neutron, openvswitch, ovn (through ovn-controller), iscsi (when iscsid is enabled) and common target compute hosts, as do baremetal (bootstrap-servers, all hosts) and nova-reload/nova-cell-reload (work only on upgrade); nova-api does not touch computes. chrony no longer exists: 2023.1 has no chrony play. A tag never restricts which hosts a play targets; use --limit for that. Tags §3; Tags §4 (item 2)
-t keystone re-distributed the Fernet keys -t keystone runs distribute_fernet.yml: on a reconfigure the Fernet keys are re-distributed from the first running keystone_fernet node to the others. Reconfigure a single service §2; Execution flow §6; R19
-t glance changed a global MariaDB variable The glance role toggles the global MariaDB variable log_bin_trust_function_creators 1→0 around its db sync on every deploy/reconfigure CODE. Known bugs and limitations (row glance); R20; fact sheet C §1
nova_scheduler received SIGHUP on a run with no change Any deploy/reconfigure with the nova, nova-api or nova-refresh-scheduler-cell-cache tag (or no tag) whose --limit, if any, includes the nova-scheduler hosts, even with no change, sends SIGHUP to nova_scheduler (cell-cache refresh, roles/nova/tasks/refresh_scheduler_cell_cache.yml:10-12, nova.yml:140-156) CODE. It is the only signal Kolla-Ansible sends to a running containerised service, and it loads no new config files. Nova deep dive §2.4; Repository architecture §4.3
-t haproxy also reconfigured fluentd, cron and kolla-toolbox on the haproxy hosts Does not happen in 2023.1 CODE: the loadbalancer play (site.yml:99-110) does not include common, which is its own play tagged common only (site.yml:86-97). fact sheet A §1.8 (E); Apply common changes §3
genconfig with a service tag also changed files on the haproxy hosts genconfig writes /etc/kolla/<svc>/ on the targets and also re-renders the HAProxy services.d files for the tagged services: the per-service loadbalancer includes also run for kolla_action config (site.yml:110-341) CODE. Predict and inspect a run (row "Render configs without restarting")
genconfig ran NIC rebinding on ovs-dpdk hosts NIC rebinding (ovs-dpdkctl.sh bind_nics) and unit installation (ovs-dpdkctl.sh install) are tasks of the ovs-dpdk config.yml (roles/ovs-dpdk/tasks/config.yml:35-52), so they run on every deploy, reconfigure, upgrade and genconfig CODE. They are idempotent only as long as the driver already matches. fact sheet A §6.8
A bootstrap-servers re-run changed the host: firewall, SELinux, packages Re-running bootstrap-servers changes the host (ansible-collection-kolla, openstack.kolla.baremetal) CODE: it disables ufw/firewalld (disable_firewall, roles/baremetal/tasks/install.yml:2-32), sets SELinux to selinux_state (permissive by default) when change_selinux is set ( roles/baremetal/tasks/post-install.yml:21-28), removes the libvirt AppArmor profile on Ubuntu with apparmor_remove_libvirt_profile (roles/apparmor_libvirt/tasks/main.yml:2-5) and removes host packages (roles/packages/defaults/main.yml:24-33): lxd, lxc and open-iscsi on Debian/Ubuntu, and libvirt* (plus iscsi-initiator-utils on RedHat) only with the libvirt container. chrony is no longer removed. None of these tasks restarts containers; a Docker restart is a separate handler (row "The Docker daemon restarted during bootstrap-servers"). fact sheet E §0 (item 6); R14
destroy also ran on the deploy host destroy acts on hosts: all (destroy.yml:3), including the deploy host (localhost is in [deployment], inventory/multinode:32-33) CODE. tools/cleanup-host deletes every network namespace and every link whose line matches tap, brq or vxlan (tools/cleanup-host:5-22), and rm -rfs /etc/kolla/* except six names (passwords.yml, globals.yml, globals.d, kolla-build.conf, config, certificates; tools/cleanup-host:77-89) CODE. fact sheet E §0 (item 10); fact sheet E §6; R16; Stop and destroy
-t ovn overwrote or removed a manual ovn-cms-options value ovn-cms-options is set to enable-chassis-as-gw (plus availability-zones when set) on the ovn-controller-network hosts, and removed entirely on the other ovn-controller hosts, on every run (roles/ovn-controller/tasks/setup-ovs.yml:12-39) CODE. Any manual ovn-cms-options value is overwritten or removed on every deploy or reconfigure; the impact of that loss is NOT AVAILABLE. fact sheet A §5.8 (E); Tags §3.1 (row ovn)
A manual net.ipv4.tcp_retries2 line disappeared from /etc/sysctl.conf on the haproxy hosts With set_sysctl and the default haproxy_host_ipv4_tcp_retries2: "KOLLA_UNSET", the sysctl task removes any manual net.ipv4.tcp_retries2 line from /etc/sysctl.conf on every run (roles/loadbalancer/tasks/config-host.yml:8-26, roles/loadbalancer/defaults/main.yml:143, group_vars/all.yml:605) CODE. fact sheet A §1.8 (G)
A keystone reconfigure re-applied the admin credentials and the endpoints Reconfigure re-asserts identity state: it re-applies the admin credentials and endpoints, and the DB user password, from passwords.yml and globals. NOT AVAILABLE: whether out-of-band changes are silently reverted. fact sheet B §3.9 (item 10)
Enabling Swift, or glance_backend_ceph, changed the Glance backend Enabling Swift (or glance_backend_ceph) on a running cloud silently switches Glance's backend; NOT AVAILABLE whether existing file-store images stay readable. R21; Known bugs and limitations (row glance); fact sheet C §1.8 (item 1); Q12
After the inventory was reordered: keepalived priorities, MariaDB restart waves, the glance host or the bootstrap target changed Inventory reordering (loadbalancer, mariadb, glance, nova-conductor, keystone groups): keepalived priorities change; the MariaDB active backend (the first shard host, the others are backup, roles/mariadb/defaults/main.yml:64) moves; glance moves (file backend); bootstrap/discovery target changes CODE. Which MariaDB restart batch a host falls in follows the group_by order: NOT AVAILABLE. R22
A task reports changed on every run although nothing changed The Ussuri Set system-id command (openvswitch) and the OVN chassis docker exec ovs-vsctl commands, which always reported changed, are gone in 2023.1 CODE: openvswitch uses the openvswitch_db module and restarts openvswitch_vswitchd when it reports changed (roles/openvswitch/tasks/post-config.yml:3-22), and ovn-controller sets the chassis options through the openvswitch_bridge/openvswitch_db modules in kolla_toolbox (roles/ovn-controller/tasks/setup-ovs.yml:2-40). Whether those modules report changed on an unchanged host is NOT AVAILABLE (the modules are not in these sources). Tasks that do report changed on every run in 2023.1 include the --check of containers with preserve_properties entries (prometheus extras) CODE. Known bugs and limitations (row openvswitch); Tags §3.1 (row openvswitch); fact sheet A §5.8 (D); Open vSwitch
An empty placement-api.cfg was written on the haproxy hosts although Placement is disabled The placement loadbalancer include lacks when: enable_placement (site.yml:243-246) CODE: under -t haproxy, the placement frontend file is always rendered, and is empty if enable_placement is false. Known bugs and limitations (second row site.yml); fact sheet A §1.9; Placement
A password rotation needed a reconfigure of the whole compute fleet FACT: placement and cinder credentials are embedded in nova-compute config on every compute, so password rotation needs a reconfigure of the whole compute fleet. fact sheet C §7; fact sheet C §2.8
On a new compute, nova_compute started before Open vSwitch and the Neutron agents The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes: site.yml:544 imports nova.yml before the openvswitch play (:546) and the neutron play (:590) CODE. NOT AVAILABLE Whether instances can be scheduled on such a compute before the later plays finish. F13; Execution flow §2.4; Component dependencies §6

Images

What you see What the code does Read
deploy or reconfigure did not pick up a rebuilt image pushed under the same tag deploy/reconfigure never pull an image that is already present locally, even if the registry has a newer build under the same tag CODE. Only absent images are pulled; an explicit pull is required CODE. Update container images §1.1; Update container images §2; F4; R13
An unrelated pull was followed by a mass recreate Mutable tags create silent drift between hosts. A later unrelated pull turns into a mass recreate. F4 ("Why it matters"); R13
A new openstack_tag applied with deploy recreated every container on every host New openstack_tag, deploy: every container on every host is recreated CODE. Changing the tag and running deploy (instead of upgrade) recreates the whole fleet in one go, without the upgrade-specific ordering: no nova-status upgrade check, no RPC-pin reload plays, and no online data migrations, except Placement's (and Ironic's only with ironic_enable_rolling_upgrade: no). Update container images §2; Update container images §4; R12
An OVS image change recreated openvswitch_vswitchd on every OVS host at the same time An OVS image change recreates openvswitch_vswitchd on every OVS host simultaneously (serial 0). NOT AVAILABLE Whether VM networking is interrupted on every compute at the same time. Update container images §4; R7
pull did not change the running containers pull never changes running containers CODE. After a pull that downloaded something, the next deploy, reconfigure or upgrade recreates the affected containers CODE. Update container images §1.1; R11
-t nova pull downloaded more than the nova images No longer in 2023.1 CODE: -t nova pull downloads on every host in the nova plays, for nova images only. The common images (fluentd, kolla-toolbox, cron) are not pulled, because common is a separate play tagged common and no role declares it as a dependency any more; add -t common to pull them. Update container images §2
genconfig pulled an image No longer in 2023.1 CODE: the Ussuri fluentd label check, which pulled the fluentd image if it was missing, is gone from the common role. Execution flow §4 (row genconfig)
The SSH host keys of nova_ssh and keystone_ssh changed after a recreate SSH host keys of nova_ssh and keystone_ssh change on every recreate, because /etc/ssh is on no volume. The clients set StrictHostKeyChecking no and keep no known-hosts file CODE; whether the SSH client then accepts the new keys without failing is NOT AVAILABLE (OpenSSH semantics). Kolla images §6 (item 18)
prune-images did not remove the images of the previous release prune-images runs docker_prune on images with the kolla_version label filter (roles/prune-images/tasks/prune_images.yml:2-8) CODE; whether that prune removes only dangling images, so that old release tags still referenced by a tag are not reclaimed, is the docker_prune module's behaviour: NOT AVAILABLE (the module is not in these sources). fact sheet E §0 (item 10); fact sheet E §7; R17

Settings that are ignored

What you see What the code does Read
docker_graceful_timeout (60 s) is ignored: containers get SIGKILL 10 s after the stop Fixed in 2023.1 CODE: docker_graceful_timeout and docker_client_timeout now take effect. Handlers pass docker_common_options, which carries both values, and the module default (10 s) applies only when no graceful_timeout is passed (group_vars/all.yml:140-153; library/kolla_docker.py:303, :325, :351-382). Each container's systemd unit stops it with docker stop <name> -t <docker_graceful_timeout> (module_utils/kolla_systemd_worker.py:33, :69), so a container still running 60 s after SIGTERM is killed. A changed timeout also changes the unit file, which check-containers detects. F3; Repository architecture §6; R24; Known bugs and limitations (row kolla_docker)
-e kolla_serial=N is ignored by reconfigure and upgrade -e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade CODE: the wrapper appends -e kolla_serial=${ANSIBLE_SERIAL} after the user's -e options (tools/kolla-ansible:290, :508, :517). Known bugs and limitations (first row CLI); Repository architecture §2; fact sheet E §11
ANSIBLE_SERIAL is ignored by deploy kolla_serial is not passed for deploy (tools/kolla-ansible:476-478) CODE. Batching is ANSIBLE_SERIAL for reconfigure/upgrade and -e kolla_serial=N for deploy, which ignores ANSIBLE_SERIAL. Known bugs and limitations (first row CLI); Ten things to know (item 4); Execution flow §5
mariadb, rabbitmq and cyborg are not batched by ANSIBLE_SERIAL Not batched by ANSIBLE_SERIAL CODE: mariadb, rabbitmq and outward rabbitmq, whose plays set no serial and whose restarts run in their own plays with a fixed serial: "33%" (mariadb.yml:3-40, rabbitmq.yml:3-41, :82-98); and cyborg, whose play uses serial instead of kolla_serial and so ignores ANSIBLE_SERIAL (bug, site.yml:784). Controlled batches §1.2; Known bugs and limitations (first row site.yml); Execution flow §5
A per-host or per-group inventory value is ignored for a variable set in globals.yml globals.yml and passwords.yml are passed with -e @file (tools/kolla-ansible:585-587), that is extra-vars, the highest precedence in Ansible: a value in globals.yml cannot be overridden per host or per group in the inventory CODE. Repository architecture §2.1
A restart_policy change is not applied Changed in 2023.1 CODE: the FIXME "restart_policy is not checked in the container" remains (module_utils/kolla_docker_worker.py:15), but the restart policy now lives in the container's systemd unit (docker_restart_policy becomes Restart= and StartLimitBurst), and compare_container detects a unit file that differs, so a changed policy restarts the container on the next run (module_utils/kolla_systemd_worker.py:26-34, :50-74, :149-162). Known bugs and limitations (row kolla_docker); Repository architecture §6
A docker_custom_option change is not applied Does not apply to 2023.1: the docker_custom_option variable was removed (ansible-collection-kolla releasenotes/notes/drop-docker-custom-option-6c8251c750473558.yaml:4), and no task in kolla-ansible or the collection reads it CODE. The docker.service.d/kolla.conf drop-in now carries only the proxy settings and the Zun options; writing or removing it notifies Reload docker service file, then Restart docker (ansible-collection-kolla roles/docker/tasks/config.yml:49-86, roles/docker/templates/docker_systemd_service.j2, roles/docker/handlers/main.yml:2-15) CODE. fact sheet E §0 (item 2); Known bugs and limitations (row baremetal)
-v flags are lost with --ask-vault-pass --ask-vault-pass overwrites VERBOSITY with a copy of EXTRA_OPTS (tools/kolla-ansible:385-386) CODE, which duplicates all options seen so far and drops earlier -v flags. Known bugs and limitations (second row CLI); Repository architecture §2; fact sheet E §11
--include-dev and the next word of the command line --include-dev: is declared with an argument in LONG_OPTS but used as a flag (tools/kolla-ansible:270, :349-352); the command line is executed unquoted ($CMD, :140-142) CODE. Known bugs and limitations (third row CLI); fact sheet E §11
enable_swift_recon with its default "no" enable_swift_recon is tested without the bool filter ({% if enable_swift_recon %}, roles/swift/templates/account.conf.j2:15, :17, container.conf.j2:15, :17, object.conf.j2:19, :22), and its default is the string "no", not a boolean (group_vars/all.yml:750) CODE. NOT AVAILABLE: whether recon is then always on. fact sheet D §3.8; fact sheet D §7 (item 3)
A service variable overridden only on the backend hosts, and the HAProxy frontend The haproxy-config variables come from the service role's defaults, evaluated on the loadbalancer host; inventory or group_vars overrides of service variables therefore apply when set for the loadbalancer hosts or for all. NOT AVAILABLE: whether per-host overrides set only on backend hosts are seen by haproxy-config, outside the places where the code explicitly uses hostvars[groups[…][0]] (nova-cell, roles/nova-cell/tasks/loadbalancer.yml:37-48). fact sheet A §2.4

Listings, check mode and prechecks

What you see What the code does Read
--list-tasks shows only include_tasks placeholders ansible-playbook site.yml --list-tasks -t nova lists only include_tasks placeholders for each role: the listing walks only the statically compiled play, while dynamic includes are loaded at run time CODE. Tags §2.6; F7; Predict and inspect a run
--list-hosts shows 0 hosts; Could not match supplied host pattern, ignoring: enable_nova_True Plays whose host pattern depends on the group_by groups (&enable_nova_True) resolve to 0 hosts at list time: those groups are created at run time CODE. Tags §2.6; F7; Predict and inspect a run
--list-tags does not show a tag --list-tags lists only the tags it can see statically; tags that occur only inside dynamically included files, such as service-rabbitmq, do not appear. Tags §3.1 (paragraph "Validation")
A --check --diff run skipped the kolla_docker tasks kolla_docker does not declare check-mode support (library/kolla_docker.py:345-349), so Ansible skips it under --check (ansible-core 2.14.18 lib/ansible/module_utils/basic.py:520-521) CODE. NOT AVAILABLE Whether such a run fails or gives partial results. Predict and inspect a run (row "Check-mode run"); Tags §2.6; Q9
genconfig plus kolla_set_configs --check did not predict what the run did "genconfig + --check" under-reports some changes and over-reports others. F14 ("Why it matters"); Kolla images §3.2; Predict and inspect a run
docker ps shows no health status Changed in 2023.1 CODE: Docker healthchecks are wired. With the default enable_container_healthchecks: yes, docker ps shows a health status for the services that define a test; a container without one shows none. No Kolla-Ansible task gates on it, so service health must still be checked from outside. Kolla images §6 (item 17); Kolla images §2.7
prechecks with -t <svc> skipped the general prechecks -t <svc> skips the prechecks role (untagged play, site.yml:79-84) but runs <svc>/precheck.yml CODE. Predict and inspect a run (row "Pre-deployment checks")
The tag name did not predict what the run did A tag name does not describe the work a run performs. What runs depends on which plays carry or inherit the tag, the loadbalancer play's per-service includes, handlers, which are never filtered by tags, the MariaDB and RabbitMQ restart plays that the handlers fill through group_by, and ansible-core's loop-notify semantics. Tags ("The single most important rule"); Tags §3
The same command was a no-op one day and a fleet-wide restart another day The same command can be a no-op or a fleet-wide restart, depending on whether a rendered file changed. That in turn depends on global variables (enable_*, group membership, passwords), not only on the tagged service CODE. Tags §4 (item 3); Component dependencies §5

Batching with serial

What serial does not reach is under Settings that are ignored; what it costs is under The run is slow.

What you see What the code does Read
With serial, the DB sync, Keystone registration or discovery ran several times With kolla_serial > 0, a run_once task runs once per batch, not once per play CODE. This applies to bootstrap containers, Keystone registration and DB creation. With kolla_serial=1 and the three controllers of the multinode inventory, nova_api_bootstrap runs 3 times (roles/nova/tasks/bootstrap_service.yml:16-21), once per controller batch CODE. Tags §2.5; F10; Controlled batches §1.2
With serial, the computes restarted before the conductors kolla_serial orders a play's restarts by host-pattern order (computes before conductors in the nova-cell play) CODE. F10
With serial, the neutron agents of the first hosts restarted only after every compute had finished the earlier plays A batch runs the complete play before the next batch starts, but each play is still traversed batch by batch for the whole fleet before the next play starts. The neutron agents on compute001 are restarted only after all computes have finished the nova-cell and openvswitch plays. Execution flow §5

Upgrades and database migrations

What you see What the code does Read
A schema migration ran during a reconfigure A "reconfigure" can apply schema migrations if the image tag moved, for most services, Neutron included: in 2023.1 bootstrap_neutron runs on every deploy and reconfigure (roles/neutron/tasks/bootstrap_service.yml:2-21, no when) CODE. There is never a "config only" path except genconfig + deploy-containers. F5 ("Why it matters")
Placement's online data migrations ran on a deploy or reconfigure Placement's online data migrations run on every deploy/reconfigure, inside bootstrap_placement, and twice on upgrade CODE. Kolla images §6 (item 6); F15
Nova's online data migrations did not run on deploy or reconfigure Nova's online data migrations never run outside upgrade. Deploy and reconfigure run the Nova bootstrap containers with KOLLA_UPGRADE (api_db sync + db sync, db sync --local_cell) CODE. Kolla images §6 (item 5); F15
neutron-dynamic-routing was not migrated by upgrade Fixed in 2023.1 CODE: upgrade with the default neutron_enable_rolling_upgrade: "yes" expands/contracts neutron plus every enabled subproject of neutron_subprojects (networking-sfc, neutron-dynamic-routing, neutron-vpnaas, vmware-nsx), the same list the bootstrap container migrates (roles/neutron/defaults/main.yml:656-664, :705, :711; kolla docker/neutron/neutron-server/extend_start.sh:5-34). Known bugs and limitations (row neutron); Kolla images §6 (item 8); F15
An upgrade stopped neutron_server on every neutron-server host has_offline_migrations loops over the enabled subprojects only, never neutron itself, on the first neutron-server host with ignore_errors: any non-zero rc (a failed docker exec included) stops neutron_server on every neutron-server host in the play until the handler flush after the contract container; with no subproject enabled no server is stopped before upgrade --contract (roles/neutron/tasks/rolling_upgrade.yml:38-89) CODE. Impact on the Neutron API, and whether a neutron contract migration is safe against running servers: NOT AVAILABLE. Known bugs and limitations (row neutron); Kolla images §6 (item 8)
The RabbitMQ upgrade stopped all brokers No longer in 2023.1 CODE: upgrade renders the config, enables feature flags and then restarts the brokers in the serial: "33%" restart play, draining each one first; roles/rabbitmq/tasks/upgrade.yml:2-10 has no cluster-wide stop. NOT AVAILABLE Whether nova-compute heartbeats are lost and services are reported down meanwhile. R10; fact sheet B §2.8 (item 2); RabbitMQ
On upgrade, keepalived was stopped on the hosts that do not hold the VIP No longer in 2023.1 CODE: the loadbalancer role's upgrade.yml imports deploy.yml (roles/loadbalancer/tasks/upgrade.yml:2), with no upgrade-specific keepalived stop. A keepalived change on any action goes through the staged handlers: backups stopped and started first, the VIP holder's keepalived last (roles/loadbalancer/handlers/main.yml:36-52, :154-177, :210-222). fact sheet A §1.8 (F); HAProxy + Keepalived
The octavia upgrade removed the octavia admin role No longer in 2023.1 CODE: roles/octavia/tasks/upgrade.yml:2-9 is config, check-containers, bootstrap_service and a flush of handlers, with no "Removing octavia user from admin project" task. Known bugs and limitations (row octavia); fact sheet D §2.3; Octavia
mysql_upgrade ran on all MariaDB nodes at once, with an unchanged image mysql_upgrade (the upgrade_mariadb container, KOLLA_UPGRADE) runs in the Apply mariadb post-configuration play, which has no serial, so on all nodes concurrently, on every upgrade, with no condition on an image change (mariadb.yml:54-72, roles/mariadb/tasks/post-upgrade.yml:2-24) CODE. NOT AVAILABLE: whether concurrent mysql_upgrade on Galera nodes conflicts. fact sheet B §1.8 (item 8)
Glance rolling upgrade with serial: the contract ran before the last batches were upgraded Answered from code: Glance rolling upgrade with ANSIBLE_SERIAL=1 runs the contract at the end of batch 1, before later batches are upgraded: the contract container is run_once, delegated to glance_api_hosts[0], after the flush of handlers (roles/glance/tasks/rolling_upgrade.yml:80-102) CODE. fact sheet C §8 (item 2); fact sheet C §1.8 (item 4)
Keystone upgrade with serial: log_bin_trust_function_creators With serial, log_bin_trust_function_creators is reset to 0 at the end of every batch (run_once), while Enable runs only in the batch whose first host is groups['keystone'][0] (run_once with when: inventory_hostname == groups['keystone'][0]) (roles/keystone/tasks/upgrade.yml:6-21, :36-50) CODE. The contract step, notified on groups['keystone'][-1] (:28-31), therefore runs with it disabled when that host is in a later batch. The impact is NOT AVAILABLE. fact sheet B §3.9 (item 9)
The swift "rolling" upgrade stopped all storage containers at once Despite the name, the swift rolling upgrade (swift_enable_rolling_upgrade: "yes" by default) stops all storage containers on all storage hosts at once (roles/swift/tasks/rolling_upgrade.yml:28-38, with the TODO "should be run in serial nodes") CODE. fact sheet D §3.4; fact sheet D §7 (item 4)
The cinder upgrade restarted every cinder-volume and cinder-backup at once The upgrade restarts every cinder container of the play at once (serial 0 unless ANSIBLE_SERIAL is set), and then a second time: after the flush, reload.yml restarts every enabled cinder container "to remove RPC version pin", after a cinder_rpc_version_startup_delay pause (30 s) (roles/cinder/tasks/upgrade.yml:21-44, roles/cinder/tasks/reload.yml:2-9, roles/cinder/defaults/main.yml:229) CODE. fact sheet C §3.8 (item 4); Cinder
A reconfigure with a bumped image tag ran cinder's db sync without its online data migrations A reconfigure with a bumped image tag runs cinder's db sync without its online data migration step: the bootstrap container sets KOLLA_BOOTSTRAP, not KOLLA_OSM, which only upgrade.yml sets (roles/cinder/tasks/bootstrap_service.yml:11, kolla docker/cinder/cinder-api/extend_start.sh:6-14, roles/cinder/tasks/upgrade.yml:26-44) CODE, while placement's bootstrap container runs both db sync and db online_data_migrations. Whether this leaves cinder in a partially upgraded state is NOT AVAILABLE. fact sheet C §6

Services, VMs and traffic during and after a restart

Every statement about data-plane or VM impact is NOT AVAILABLE (Limits of this study). The rows say what the code shows and point to what is open.

What you see What the code does Read
Running VMs after a nova_libvirt restart or recreate NOT AVAILABLE Whether running VMs survive a nova_libvirt restart or recreate; what the container and its start path do is CODE. Nova deep dive §5; Q1; What is not known, "Running VMs and compute hosts"
VM network traffic after a restart of the OVS agent, of openvswitch_vswitchd or of the L3 agent NOT AVAILABLE Whether VM network traffic is interrupted when the OVS agent, openvswitch_vswitchd or the L3 agent restarts; that the restarts happen is CODE. Reconfigure Neutron §4; R5; R6; R7; Q2; Q3; Q4; What is not known, "Network data plane"
Open vSwitch flows after an openvswitch_vswitchd restart An openvswitch_vswitchd restart replaces only the userspace daemon. The kernel module stays loaded and conf.db keeps bridges and ports CODE. NOT AVAILABLE Whether the OpenFlow tables are lost and must be reprogrammed by the OVS agent or ovn-controller. Kolla images §6 (item 13); R7; Q3
Router namespaces after a neutron_l3_agent restart The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes CODE; its comment says this removes all L3 namespaces. What the command does is neutron code: NOT AVAILABLE. F12; R6; Reconfigure Neutron §4; Kolla images §5.3; Q4
Work in progress was killed at a restart: SIGKILL after 10 s In 2023.1 the delay is 60 s, not 10 s CODE: on every restart, a service still running docker_graceful_timeout (60 s) after SIGTERM is killed with its in-flight work (heat-engine, cinder copies, glance uploads). The unit sets no TimeoutStopSec, so systemd's own stop timeout also applies; its value is host configuration NOT AVAILABLE. NOT AVAILABLE Whether services started by a shell wrapper that does not exec them (e.g. neutron-l3-agent, haproxy) receive SIGTERM at all. F3 ("Why it matters"); R24; Repository architecture §6; Kolla images §2.6
The API of a service was unavailable during a run with its tag The service API is unavailable while its containers restart (up to 60 s stop, then start) on all its hosts at once. Add the HAProxy restart window if the frontend changed. Reconfigure a single service §3 ("Interruption")
Authentication while keystone restarts All keystone containers restart on all hosts at once; NOT AVAILABLE whether authentication fails while keystone restarts. R19; Reconfigure a single service §1.1
APIs, DB traffic and the VIP while HAProxy restarts haproxy is stopped and started (never reloaded): on every backup host at once, then on the host that holds the VIP CODE. On a haproxy-only change keepalived is not stopped, so the VIP stays on a node whose haproxy is down until it is back; keepalived runs a check_alive track script (interval 2, fall 2) by default CODE. NOT AVAILABLE Whether that script moves the VIP during the master restart, and for how long the APIs are unreachable. Blast radius: all APIs and all DB traffic (database_address is kolla_internal_fqdn; enable_proxysql defaults to no). R8; Q5; fact sheet A §1.8 (B); Kolla images §6 (item 15)
Token cache and Horizon sessions after a memcached restart memcached has no data volume and no persistence option CODE. NOT AVAILABLE (memcached semantics) Whether every restart therefore empties the cache, with token-cache misses and lost Horizon sessions where they are kept in memcached. Kolla images §6 (item 14); R4; fact sheet A §3.8
RPC while RabbitMQ restarts RabbitMQ restarts run in serial: "33%" batches (one broker at a time with three), each broker drained first CODE; NOT AVAILABLE whether nova-compute and the agents lose RPC while their broker restarts, and whether RPC-backed API calls time out meanwhile. Tags §3.1 (row rabbitmq); fact sheet B §2.8 (item 1)
DB connections during a MariaDB rolling restart MariaDB config change or image change: the handlers only group the hosts, and the restart runs in the separate Restart mariadb services play in serial: "33%" batches, one batch after another (roles/mariadb/handlers/main.yml:56-73, mariadb.yml:14-26) CODE. HAProxy's single active backend is the first shard host, the others being backup (roles/mariadb/defaults/main.yml:64); NOT AVAILABLE in which batch that host restarts (group_by order). API impact: DB connection resets. R9; fact sheet B §1.8 (item 1)
MariaDB after all DB containers stopped (power loss, Docker restart on every DB host) MariaDB has no automatic Galera recovery. NOT AVAILABLE Whether, after all DB containers stopped, they come back as plain mysqld_safe with the cluster non-primary until mariadb_recovery. A container left with BOOTSTRAP_ARGS=--wsrep-new-cluster by an interrupted run passes that option to mysqld_safe again on any manual docker start CODE; whether that bootstraps a new cluster (split-brain risk) is NOT AVAILABLE. Kolla images §6 (item 16); Kolla images §5.4; fact sheet E §2.4; R18
Image API outage when glance_api restarts With the file backend on the default glance volume, only the first host of glance-api runs it (glance_api_hosts, group_vars/all.yml:915), so any restart is a full Image API outage CODE; NOT AVAILABLE whether snapshots in progress fail. R20; fact sheet C §1.8 (item 3)
Volumes and iSCSI sessions after an iscsid or tgtd restart iscsid: NOT AVAILABLE whether existing sessions are kernel-held and whether attach/detach during the window fails. tgtd: nothing in its start path runs tgt-admin to load /etc/tgt/targets.conf; NOT AVAILABLE whether a tgtd restart cuts disk I/O for VMs using LVM volumes from that host until the targets are re-created. Tags §3.1 (row iscsi); What is not known, "Storage and images"
The iSCSI initiator name (IQN) after a restart, a recreate or the removal of the iscsi_info volume The iSCSI initiator name is stable per iscsi_info volume across restart and recreate CODE. NOT AVAILABLE Whether removing the volume produces a new IQN at the next start, and whether that breaks backend ACLs keyed on the old one. Kolla images §6 (item 19)
Heat stacks in progress when heat_engine restarted Docker sends SIGTERM, then SIGKILL if heat-engine is still running after docker_graceful_timeout (60 s) CODE. NOT AVAILABLE (upstream heat): whether stacks that a killed engine owned stay *_IN_PROGRESS until some engine starts again. fact sheet C §5.8
Logs after a restart of the common containers fluentd: log shipping gap; the Elasticsearch and OpenSearch outputs buffer to the fluentd_data volume, which a restart or a recreate keeps, while the tail read positions are on no volume CODE. cron: log rotation is delayed. VM and API impact: NOT AVAILABLE. Apply common changes §3 ("Impact")
The clock after a chrony restart or a VIP failover Does not apply to 2023.1: there is no chrony role, play, container or VIP time source; host time sync is outside Kolla-Ansible and only checked by the prechecks CODE. Which time source the hosts use is NOT AVAILABLE. Chrony; fact sheet A §4.8 (Ussuri role)
After kolla-ansible stop kolla-ansible stop stops all Kolla containers (incl. nova_libvirt, OVS, MariaDB) on targeted hosts; NOT AVAILABLE whether VM network traffic stops. MariaDB then needs mariadb_recovery. R15; Stop and destroy
mariadb_recovery stopped healthy MariaDB nodes mariadb_recovery stops every MariaDB node, even healthy ones (roles/mariadb/tasks/recover_cluster.yml:18-24), and bootstraps from the highest seqno: full DB outage CODE. R18; Database operations; fact sheet B §1.8 (item 5)
Anything else about VMs, traffic, storage or the control plane after a restart What the sources do not establish is listed by topic, with the sections that mark each point NOT AVAILABLE. What is not known

Other bugs and oddities, by service

What you see What the code does Read
Keystone Fernet keys: push, rotation times, expiry limits Fernet cron generator hour calculation (hour = time % HOUR_SPAN, roles/keystone/files/fernet_rotate_cron_generator.py:69); precheck vs generator expiry limits (roles/keystone/tasks/precheck.yml:43-53, config.yml:195-200); push script stops at the first unreachable peer (set -o errexit, roles/keystone/templates/fernet-push.sh.j2:3, :17-22) CODE. Known bugs and limitations (row keystone); fact sheet B §3.9 (items 3 to 6); Q11
Cinder: host, etcd coordination, backup driver No host/backend_host/cluster in cinder.conf; Ceph backup driver default (group_vars/all.yml:954) while ceph.conf is copied only with cinder_backend_ceph (roles/cinder/tasks/config.yml:15-19) CODE. The Ussuri etcd item no longer holds as written: with cinder_coordination_backend: etcd the backend_url is the internal VIP, where the etcd role has a HAProxy frontend (roles/cinder/templates/cinder.conf.j2:245-253, roles/etcd/defaults/main.yml:26-32) CODE. Known bugs and limitations (row cinder); fact sheet C §3.8; Cinder
Glance with the file backend File backend = single host CODE: with the file backend on the default glance volume (not a shared path), glance runs only on the first host of groups['glance-api'], so reordering the inventory moves glance-api to another host, and the HAProxy members follow it (group_vars/all.yml:915). Known bugs and limitations (row glance); fact sheet C §1.8 (item 2); R22
Glance image cache The image cache is never pruned: no task or image of 2023.1 runs a glance cache pruner or cleaner CODE. fact sheet C §1.8 (item 5); Glance
Masakari instancemonitor and libvirt SASL Changed in 2023.1 CODE: the instancemonitor still connects with qemu+tcp (roles/masakari/templates/masakari-monitors.conf.j2:18), but with libvirt_enable_sasl (default true, roles/nova-cell/defaults/main.yml:562) it gets a libvirt auth.conf with the SASL credentials (roles/masakari/templates/masakari-instancemonitor.json.j2:9-15, auth.conf.j2:2-3). Whether the instancemonitor then authenticates is NOT AVAILABLE. Known bugs and limitations (row masakari); fact sheet D §7 (item 7)
Barbican secrets and barbican_crypto_key The crypto KEK lives in passwords.yml (etc/kolla/passwords.yml:56) CODE. NOT AVAILABLE: whether losing or changing barbican_crypto_key makes stored secrets undecryptable. fact sheet D §1.6
MariaDB backup and the VIP Backup runs only on the first host of the MariaDB shard group (reading that node's datadir, roles/mariadb/defaults/main.yml:123), but its client config points at database_address (roles/mariadb/templates/backup.my.cnf.j2:5) CODE. NOT AVAILABLE: whether, if HAProxy has failed over away from that node, the backup tool connects to a different server than the one whose files it copies. fact sheet B §1.8 (item 9)
HAProxy on the backup nodes with set_sysctl: no ip_nonlocal_bind=1 is set only when set_sysctl is true (roles/loadbalancer/tasks/config-host.yml:8-26), and backup nodes bind VIP addresses CODE. NOT AVAILABLE: whether haproxy starts on non-VIP nodes with set_sysctl: no and no manual sysctl. fact sheet A §1.8 (H)
Two Kolla clouds on the same L2 segment: keepalived keepalived_virtual_router_id defaults to 51 (group_vars/all.yml:189) with the default keepalived_traffic_mode: multicast (roles/loadbalancer/defaults/main.yml:50); two Kolla clouds on the same L2 segment collide unless it is changed (new in 2023.1: unicast is an option) CODE. fact sheet A §1.8 (I)
OVN DB after the first host was replaced, or its volume lost A new cluster is seeded from the first ovn-nb-db/ovn-sb-db host, while a member re-created on an existing cluster joins the current leader (roles/ovn-db/tasks/bootstrap-initial.yml:5-13, :59-67) CODE. NOT AVAILABLE: what ovn-ctl does in those cases. There is no code for member removal and none for backups. fact sheet A §5.8 (B); OVN
OVN with DPDK The Ussuri inference that OVN + DPDK cannot work rested on the OVN bootstrap's docker exec into openvswitch_vswitchd, which 2023.1 no longer does: ovn-controller sets the chassis options through the openvswitch_bridge/openvswitch_db modules in kolla_toolbox (roles/ovn-controller/tasks/setup-ovs.yml:2-40) CODE. Whether OVN works with ovs-dpdk is NOT AVAILABLE. fact sheet A §5.8 (G); OVN

For any other role, the fact sheets list the notable behaviour in these sections (the component pages say, item by item, which of them still hold in 2023.1):

  • Fact sheet A: §1.8 (haproxy and keepalived; the loadbalancer role in 2023.1), §2.4 (haproxy-config), §3.8 (memcached), §4.8 (chrony: the Ussuri role, which 2023.1 no longer has), §5.8 (ovn; ovn-db and ovn-controller in 2023.1), §6.8 (ovs-dpdk), §7.3 (service-cert-copy), §8.3 (certificates), §9 (restart triggers and blast radius)
  • Fact sheet B: §1.8 (MariaDB), §2.8 (RabbitMQ), §3.9 (Keystone)
  • Fact sheet C: §1.8 (Glance), §2.8 (Placement), §3.8 (Cinder), §4.8 (Horizon), §5.8 (Heat), §6 (what reconfigure does per role)
  • Fact sheet D: §1.6 (Barbican), §3.8 (Swift), §4.6 (Manila), §5.0 (the roles that place containers on compute hosts), §7 (consolidated bugs and oddities)
  • Fact sheet E: §0 (most important findings), §15 (operational recommendations)

Investigate a component

Each of the 21 component pages says where the study covers the component and ends with its service card. Under "Where to look", the sections of the study are listed under these sub-headings when the study has something to list: Operate (the scenario and the command lines), Impact and risk (what the tag executes, and the risks), Dependencies, Evidence (the fact sheet, the image side), Before and after a run (the checks, prerequisites and precautions the study gives) and Known limits and open points (known bugs, open questions, findings, and what the sources do not establish) (How to read a component page).

In the table, "Risks" gives the risk numbers that the page lists in its "Risks" bullets, and "Per-role evidence" the fact sheet that the page names, or the page that it names in its place. A dash means that the page has no "Risks" bullet.

Component Role and tag Risks Per-role evidence
Keystone role keystone, tag keystone R19, R8, R24 fact sheet B §3
Glance role glance, tag glance R20, R21, R8, R24 fact sheet C §1
Placement role placement, tag placement R8, R24 fact sheet C §2
Nova roles nova and nova-cell; tags nova, nova-api(-*), nova-cell(-*) nova: R1, R2, R3, R24; nova-cell: R1 Nova deep dive
Neutron role neutron, tag neutron R5, R6, R7; membership changes: R3, R4 Reconfigure Neutron
Cinder role cinder, tag cinder R8, R24 fact sheet C §3
Horizon role horizon, tag horizon R8, R24 fact sheet C §4
Heat role heat, tag heat R8, R24 fact sheet C §5
Barbican role barbican, tag barbican – fact sheet D §1
Octavia role octavia, tag octavia – fact sheet D §2
Swift role swift, tag swift – fact sheet D §3
Manila role manila, tag manila – fact sheet D §4
MariaDB/Galera role mariadb, tag mariadb R9; mariadb_recovery: R18 fact sheet B §1
RabbitMQ role rabbitmq, tag rabbitmq R3, R10 fact sheet B §2
Memcached role memcached, tags memcache, memcached R4 fact sheet A §3
HAProxy + Keepalived role loadbalancer (haproxy, keepalived, proxysql); tags haproxy, keepalived, loadbalancer R8 fact sheet A §1, §2
Common: fluentd, cron, kolla_toolbox role common, tag common (its own play; no service tag runs it) R2; R25 does not apply to 2023.1 Repository architecture §5.1
Open vSwitch role openvswitch, tag openvswitch R7 fact sheet A §6 for ovs-dpdk
OVN roles ovn-controller and ovn-db; tags ovn (both), ovn-controller, ovn-db – fact sheet A §5
Chrony no role, play or tag in 2023.1 (the page says so, with the code) R2 and R25 no longer involve chrony fact sheet A §4 (the Ussuri role)
iscsid / tgtd role iscsi, tag iscsi (only on &enable_iscsid_True hosts) – fact sheet D §5.2

Two more pages cut across the components: Component dependencies, which maps what depends on what and keeps four kinds of dependency apart (installation, configuration, runtime, operational), and Nova deep dive. The roles without a component page are in fact sheet D §5 (the roles that place containers on compute hosts) and fact sheet E (bootstrap-servers, now applied from ansible-collection-kolla, and the operational playbooks).

Cases the study does not solve

The study does not diagnose a runtime failure of OpenStack, such as a failed VM deployment: no container or VM behaviour is used as evidence (Limits of this study). For these classic cases it offers the dependencies and the restarts read from the code, and marks the rest as open.

Case What the study offers What it does not establish
An instance fails to boot or to be scheduled (a VM deployment fails) Nova deep dive §3 lists what must be available before Nova can operate, ordered by necessity: MariaDB, RabbitMQ, Keystone and memcached, Placement, cell mapping and host mapping, Neutron server and an L2 agent on the compute, Glance and Cinder, libvirt. Nova deep dive §2.2 gives what each Nova component requires at runtime. Component dependencies §1 and Component dependencies §3 give the shared substrate and the control-plane services. Upstream Nova behaviour is not in these sources and is marked NOT AVAILABLE. Diagnosing the runtime failure itself is outside what the sources establish.
A new compute gets no instances Nova deep dive §3 (item 5): the scheduler does not discover hosts by itself (discover_hosts_in_cells_interval = -1). Add a compute node §2 says what deploy --limit runs for a new compute. Reconfigure Nova §6 lists nova-manage cell_v2 list_hosts among the checks after a Nova run. NOT AVAILABLE Whether a compute that was not discovered, and so is not in a host mapping, gets no instances.
An instance has no network Component dependencies §4 gives the networking containers, their placement rule, what they need and their data-plane role. Reconfigure Neutron §4 gives, per component, what a restart does in the code. For neutron-server and the OVS, L3, DHCP and metadata agents, the effect of a restart on existing connectivity is marked NOT AVAILABLE (What is not known, "Network data plane").
A live migration, a resize or a console fails Nova deep dive §6 gives the mechanism and the dependency of live migration, cold migration and resize, the VNC console and scheduling. NOT AVAILABLE Whether a nova_libvirt or nova_ssh restart interrupts operations in progress. NOT AVAILABLE Whether, with libvirt_tls: true, live migration to a new compute fails while the other computes' /etc/hosts lacks it (Add a compute node §2).
An API rejects requests: authentication Component dependencies §1 says how every service reaches MariaDB, RabbitMQ, memcached, Keystone and HAProxy. fact sheet B §3.10 counts the roles whose templates point at Keystone and memcached. NOT AVAILABLE: whether those API services fail to authenticate requests while Keystone or memcached is unavailable.
A volume cannot be attached Component dependencies §2 (row "iscsid / tgtd"): nova-compute (volume attach) and cinder-volume (LVM) depend on it. Tags §3.1 (row iscsi) and fact sheet C §3.8 give what the code shows for iscsid, tgtd and cinder-volume. NOT AVAILABLE Whether existing iSCSI sessions are kernel-held and whether attach/detach fails during an iscsid restart. NOT AVAILABLE: whether existing attachments survive a cinder-volume restart.
The Galera cluster is down Database operations gives the mariadb_recovery command line, with the comment "ONLY when the whole Galera cluster is down". R18 and fact sheet B §1.3 say what recovery does, and Kolla images §5.4 what the MariaDB image does not do. NOT AVAILABLE Whether the DB containers come back as a non-primary cluster after a full stop (What is not known, "Control plane").

Is it known?

  • Known bugs and limitations — the implementation limitations and bugs found in ussuri-eol, by area (CLI, site.yml, kolla_docker, the kolla images and individual roles), with their status in 2023.1. Several no longer hold in 2023.1, among them the 10 s stop timeout, the ovs-dpdk pull bug, the haproxy upgrade task and the octavia admin-role removal (rows above)
  • Findings — fifteen headline findings (F1–F15), each with its evidence and why it matters. In 2023.1, F2 (-t common is a no-op), F3 (the timeouts are ignored) and F11 (the openvswitch_agent.ini merge) no longer hold, and the Neutron parts of F5 and F15 have changed CODE (rows above)
  • Open questions — Q1–Q14: all are NOT AVAILABLE, except Q14, which is answered from the source CODE, and the control-flow part of Q6, which is also answered from the source CODE. Q10 was asked for Ansible 2.8 and 2.9; in 2023.1 the behaviours it was about are read from ansible-core 2.14.18 (Tags §2.1, Tags §2.4), and the ansible-core 2.13 side is NOT AVAILABLE
  • What is not known — what the sources do not establish, grouped by topic, with the sections that mark each point NOT AVAILABLE
  • Fact sheet corrections — eleven corrections to the Ussuri fact sheets. The one with the widest effect in Ussuri, that the effective stop timeout was 10 s and not the 60 s of docker_graceful_timeout, no longer applies to 2023.1, where the 60 s is used (R24) CODE. Their evidence is under Verification and errata
  • The lists that the fact sheets keep themselves: fact sheet C §8 and fact sheet D §8 (NOT AVAILABLE lists), fact sheet D §7 (consolidated bugs and oddities) and fact sheet E §14 (register of open questions: NOT AVAILABLE, or answered from code)
  • Kolla images §7 — what is not available from the kolla image sources
  • Risk register — 25 risks (R1–R25), each with what actually happens, its blast radius, VM and API impact, likelihood, severity, precautions and evidence, and its status in 2023.1 (R24 no longer holds as written, R25 does not apply)

Checks the corpus names

The checks that the study names before and after a run, as links. This page adds none. Docker healthchecks are wired in 2023.1 (enable_container_healthchecks: yes by default), so docker ps shows a health status for the services that define a test, but no Kolla-Ansible task waits on it: service health must still be checked from outside CODE (Kolla images §6, item 17).

Page Section What it lists
Pre-execution checklist step 5, "Health gates" Before the run, and again after each batch: Galera (wsrep_cluster_size, wsrep_local_state_comment=Synced); RabbitMQ (cluster status, no partitions); openstack compute service list and openstack network agent list all up; HAProxy backends all UP.
Pre-execution checklist step 6, "Workload gates" No live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts.
Pre-execution checklist step 1, "Know the diff before you run" kolla-ansible genconfig with the same tags and --limit, a diff of /etc/kolla/<service>/ against the previous version, and kolla_set_configs --check on each container.
Predict and inspect a run the table The commands that show what a run would target, render and restart, among them "See what will restart" (the exit codes of kolla_set_configs --check) and "Pre-deployment checks" (prechecks, read-only).
Reconfigure a single service §3, "Sequence, impact, risks and verification of -t <svc> reconfigure" Verification before (genconfig, the check in each <svc> container, a diff of /etc/kolla/haproxy/services.d/<svc>*.cfg) and after (openstack endpoint list --service <name>, the API through the VIP, the container start times, docker ps --filter name=haproxy).
Reconfigure Nova §6, "Operational risks and verification" Before (genconfig and a diff of nova.conf, the check in nova_compute on a few computes, no active live migrations) and after (openstack compute service list, docker ps --filter name=nova_ -a, virsh list --all inside nova_libvirt on a sample of computes, nova-manage cell_v2 list_hosts).
Reconfigure Neutron §5, "Risks and verification" Before (openstack network agent list, the network nodes that host HA router masters, a diff of openvswitch_agent.ini on a compute after genconfig -t neutron) and after (the agent list, the ip netns count on network nodes, a continuous ping of a VM's floating IP during each batch).
Apply common changes (fluentd, cron, kolla_toolbox) §4, "Sequence, risks and verification" Verification before (genconfig, then the check in each of fluentd, cron and kolla_toolbox) and after (docker ps start times, docker logs fluentd, logs still reaching the configured output).
Update container images §5, "Verification" Before (docker images on a sample of hosts after pull, image IDs against the registry) and after (docker ps --format shows the new tag everywhere, docker inspect of nova_compute equals the pulled image ID).
Risk register the row "Precautions" of each risk The precautions to take before running the operation.
Tags §3.1 columns "Safe on its own?" and "Prerequisites" Per tag: whether it is safe on its own, and its prerequisites.
Component pages "Before and after a run" The checks, prerequisites and precautions the study gives for the component, when it has some.

Add a compute node and Change control-plane membership say that they give no verification steps.