Troubleshooting¶

Where to look when a kolla-ansible run did something unexpected, or did nothing. Each row of the symptom
index names what an operator sees, restates what the study says the code does, and links to the section
that establishes it. The page also says where to look for one component, what is already known as a bug or as an open
point, and which checks the study names.
Important
The study rests only on reading source code: kolla-ansible ussuri-eol, the kolla image repository ussuri-eol and
the Ansible 2.9.27 source. Nothing was run against a host, there are no timings, and every statement about data-plane
or VM impact is NOT AVAILABLE (Limits of this study). All
conclusions come from the Ussuri implementation, not from newer releases.
During an incident, the portal can tell what Kolla will do given the inputs: tasks and restarts. What that does to VMs and APIs depends on the environment and is mostly NOT AVAILABLE in these sources (Code guarantees and environment dependence).
This page is an index. It adds no diagnosis and no procedure of its own.
Symptom index¶
Three mechanisms explain almost every surprise on the operation pages: config drift means a full container stop/start, never a reload; loop-notify, where one changed container in a role's loop restarts every container of that role on that host; and global inputs, which are rendered into per-host files on every host (Three mechanisms that explain almost every surprise).
In the tables, "What the code does" restates what the sections linked under "Read" say, and keeps their evidence labels. Rows that restate a fact sheet carry the words of the fact sheets, FACT and NOT AVAILABLE, which name the same two levels as CODE and NOT AVAILABLE (Evidence labels). A sentence without a label restates a sentence that carries none.
Containers restarted that should not have¶
| What you see | What the code does | Read |
|---|---|---|
| A container was stopped and started for a config change; nothing was reloaded | A config change is applied only by stopping and starting the container; kolla_docker has no reload action. The effective stop timeout is 10 s, not the 60 s of docker_graceful_timeout. |
Repository architecture §4.3; Ten things to know (item 2) |
| Containers restarted although their own config did not change; every container of the role restarted on the host | When a config or check task loops over a role's services and one item changes, Ansible 2.9 notifies the handlers of all executed items: a change for one container restarts every container of that role on that host CODE. | Tags §2.4; F1 |
nova_libvirt and nova_ssh restarted on every compute after a change that concerned only nova-compute |
With the default Nova settings, a nova-compute-only override restarts nova_compute, nova_libvirt and nova_ssh on every compute host CODE. Kolla's own handler ordering restarts libvirt before nova-compute CODE. |
Tags §2.4; Reconfigure Nova §4; R1; Nova deep dive §2.4 (the other triggers) |
nova_api restarted after a scheduler-only override |
A scheduler-only override rewrites nova-scheduler/ only and restarts nova_scheduler and, by loop-notify, nova_api CODE. |
Reconfigure Nova §3; Tags §2.4 |
HAProxy restarted during a run with a service tag such as -t keystone |
Handlers ignore tags CODE: the haproxy-config task under tags: always notifies Restart haproxy container, which is why -t keystone can restart HAProxy. HAProxy restarts on all haproxy hosts at once if the rendered services.d/ changed. |
Tags §2.2; Reconfigure a single service §3; Reconfigure a single service §1.2 (--skip-tags haproxy); R8 |
| HAProxy was restarted, not reloaded, on all haproxy hosts at the same time | HAProxy is restarted instead of reloaded CODE. Every haproxy-relevant change goes to Restart haproxy container: a change in any one service's services.d/, a cert, haproxy.cfg or haproxy_run.sh. With the default serial 0, all haproxy hosts restart in the same handler step. |
Known bugs and limitations (row haproxy); fact sheet A §1.8 (A); R8 |
fluentd, cron and kolla_toolbox restarted during an unrelated run such as -t nova |
Enabling a service that appears in fluentd_input_openstack_services or cron_logrotate_services (most OpenStack services, e.g. barbican) changes td-agent.conf and/or logrotate.conf on every host. The common containers then restart on whatever hosts the next run reaches, which can be a seemingly unrelated -t nova run. |
Apply common changes §3 ("Hidden coupling"); R2; Repository architecture §5.1 |
kolla_toolbox restarted although only the fluentd or the cron config changed |
Any drift restarts all three common containers on that host (loop-notify in check-containers.yml) CODE. |
Apply common changes §3 ("What restarts") |
neutron_openvswitch_agent restarted on all computes and network nodes after an ml2_conf.ini override meant for the server |
openvswitch_agent.ini merges ml2_conf.ini, so the override restarts neutron_server on the controllers and neutron_openvswitch_agent on all computes and network nodes; on network nodes the dhcp, l3 and metadata agents restart too (loop-notify) CODE. |
Reconfigure Neutron §3; R5; F11 |
| Nova and neutron containers restarted on all computes after a controller was added, removed or replaced | Changing the rabbitmq membership changes the transport URL everywhere, and changing memcached changes the memcache server lists. In the default inventory both are children of control, so a controller change does both. A memcached-only change does not touch nova-compute (the compute nova.conf has no memcache list) but still restarts the neutron agents on computes (neutron.conf). |
Change control-plane membership §1.1; Change control-plane membership §2; R3; R4 |
Fleet-wide restarts after an enable_* flag changed, for example enable_barbican: yes |
enable_barbican: yes, with Barbican not even deployed, rewrites nova.conf everywhere (adds [barbican]), td-agent.conf and logrotate.conf: all nova containers, libvirt included, and fluentd, cron and kolla_toolbox restart CODE. |
Reconfigure Nova §3; R2; F6; Why a small change creates a large workload |
A container restarts on every deploy/reconfigure, for example prometheus_server; kolla_set_configs --check exits 1 on every run |
kolla_set_configs --check exits 1 on every run for a config_files entry with preserve_properties and no perm, and in practice for a directory source that contains subdirectories. So prometheus_server restarts on every deploy/reconfigure whenever {{ node_custom_config }}/ holds files already copied into the container CODE. |
Tags §3; Kolla images §3.1; Kolla images §6 (item 2); F14 |
| A container restarted on a run where nothing was changed | A restart handler runs only when a task that notifies it reports changed CODE. check-containers reports changed only when the container is missing, is not running, differs from the task in a compared parameter, or kolla_set_configs --check exits 1 or 137 or meets a Docker client error CODE. NOT AVAILABLE Whether, on an unchanged cloud and apart from the exception of the row above, every task that notifies a restart handler reports ok, so that nothing restarts. |
Tags §3; What is not known, "Predicting a run" |
| A container that was stopped, or that keeps exiting, was restarted by the run | kolla_docker also treats as drift an exit code 137 of the check (the exec was killed because the container exited) and a docker exec client error (container stopped or restarting): a stopped container is restarted by the next run. |
Kolla images §3.3 |
| After an image pull, the other containers of the role restarted too | After a pull that downloaded something, the next deploy, reconfigure or upgrade recreates the affected containers, and loop-notify restarts their siblings CODE. With only the nova-compute image rebuilt, -t nova deploy after that pull recreates nova_compute and restarts nova_libvirt and nova_ssh on every compute CODE. |
Update container images §1.1; Update container images §2 |
A config change written earlier by genconfig, or by hand, was applied by a later, unrelated run |
A config file changed by an earlier genconfig, or by hand, still restarts the container on the next deploy/reconfigure/deploy-containers CODE. Between genconfig and the last deploy-containers batch, any other Kolla run that reaches those hosts applies the pending drift in files listed in config_files CODE. |
Repository architecture §4.3; Controlled batches §2 (caveats) |
| A container picked up new config after a Docker restart or a reboot, without a Kolla run | Hosts restarted by Docker or reboot also pick up the new config, because COPY_ALWAYS copies the files and rewrites /run_command at container start CODE. |
Controlled batches §2 (caveats); fact sheet E §2.4 |
With COPY_ONCE, the container was recreated, not restarted |
COPY_ONCE: /configured lives in the container layer, so only a recreate re-applies config. kolla_docker always recreates under COPY_ONCE. |
Kolla images §3.5 |
| After a HAProxy restart, frontends changed that the run had not rendered | The restart re-reads all services.d/* files: any drift left in /etc/ by earlier runs (e.g. a previous genconfig, or stale files) goes live too. |
fact sheet A §1.9 |
genconfig restarted services on Hyper-V hosts |
genconfig writes config but does not restart any Docker container. Exception: the handlers of the deprecated nova-hyperv role are unguarded and are notified directly by its config.yml, so with enable_hyperv genconfig can restart the Windows nova-compute, neutron-hyperv-agent and FreeRDP-WebConnect services. |
Pre-execution checklist (step 1) |
openvswitch_vswitchd restarted although only openvswitch_db had drifted |
check-containers loops over both OVS services, so with any openvswitch_db-only drift (e.g. a start-ovsdb-server change or a db image, volume or dimension change) loop-notify also notifies the vswitchd handler, which runs after the db restart and wait. |
R7; Tags §2.4 |
mariadb_clustercheck restarted on all nodes at once |
The clustercheck restart is not rolling: the handler restarts mariadb_clustercheck on all notified nodes at once. NOT AVAILABLE: whether HAProxy marks every backend down when that restart outlasts the check settings, which would mean a short total DB outage through the VIP. |
fact sheet B §1.8 (item 2) |
keepalived restarted after a host was inserted, removed or re-ordered in the haproxy group, or after a CA change |
keepalived priority is the index in groups['haproxy'] + 1 (FACT). Inserting, removing or re-ordering a haproxy host changes keepalived.conf on every host after it in the list, and each changed host restarts keepalived (in parallel). Whether that moves or flaps the VIP is NOT AVAILABLE. A CA bundle change (e.g. a root CA rotation with kolla_copy_ca_into_containers: yes) triggers Restart keepalived container on all haproxy hosts in parallel, together with the haproxy restart. |
fact sheet A §1.8 (C); fact sheet A §7.3; R22 |
keystone restarted after a memcached, rabbitmq or keystone node was added or removed |
keystone.conf embeds every memcached host and every rabbitmq host, and the fernet crontab and fernet-push.sh embed the keystone host list and index. Adding or removing a memcached, rabbitmq or keystone node therefore restarts keystone, and keystone-fernet as well for keystone node changes. |
fact sheet B §3.9 (item 2); R19 |
| All DB nodes restarted after the Galera membership changed | Galera membership changes restart all DB nodes: galera.cnf lists all nodes CODE. |
Change control-plane membership §2; fact sheet B §1.8 (item 11) |
| The RabbitMQ brokers restarted one by one after the group was scaled or a password changed | Scaling the rabbitmq group changes rabbitmq.conf (classic_config node list) on every broker, which causes a rolling restart. definitions.json changes (a password change) cause a rolling broker restart in the rabbitmq play. |
fact sheet B §2.8 (items 4 and 5) |
| Every heat-engine restarted in the same step | FACT: with serial 0, every heat-engine restarts in the same handler flush on any heat.conf change; a memcached or rabbit membership change, a password rotation or an openstack_service_workers change is enough. |
fact sheet C §5.8 |
glance_api was recreated after the Cinder iSCSI/LVM backend was enabled |
FACT: enabling enable_cinder_backend_iscsi/LVM also flips glance-api to privileged with /dev, so glance-api is recreated. |
fact sheet C §3.8 (item 7) |
| Every container of several services restarted after a CA or backend certificate rotation | A CA or backend cert rotation restarts every container of every service using it (whole fleet, per-play parallel). | fact sheet A §7.3 |
All OVN DB members restarted at once; ovn_controller restarted on all computes and network nodes in parallel |
Any change to ovn-nb-db/ovn-sb-db restarts every member at once under serial 0. ovn-controller restarts happen on image or container-parameter change or when a deploy/reconfigure rewrites its config.json, and hit all computes and network nodes in parallel. |
fact sheet A §5.8 (A and C) |
On ovs-dpdk hosts, ovs-vswitchd restarted after the ovsdpdk-db restart |
Any ovsdpdk-db restart (an image change or config.json drift) also restarts ovs-vswitchd, on all DPDK computes in parallel, followed by a mandatory 5-minute sleep. NOT AVAILABLE: whether this is a data-plane outage for all VMs on that host. |
fact sheet A §6.8 |
The Docker daemon restarted during bootstrap-servers |
FACT: the restart task runs if and only if the /etc/ write reports changed (the rendered content differs, or the file mode is not 0644). It is a plain task, not a handler: it runs at the same moment on every host in the batch, and the batch is all hosts by default. |
fact sheet E §0 (item 1); fact sheet E §2.4; R14; Q13 |
| A service restarted on every host at once instead of one host at a time | Without batching, handlers restart a service on every host of the play at once. Only MariaDB (4 waves) and RabbitMQ (one by one) roll by themselves. | Ten things to know (item 4); Execution flow §7; Controlled batches §1.2 |
| Containers restarted in an order that is not the order of the changes | Within a role, restart order is handler definition order, not notification order CODE: nova-cell restarts nova_ssh, then nova_libvirt, then nova_compute; openvswitch restarts openvswitch_db, waits, then restarts openvswitch_vswitchd. |
Execution flow §7; Tags §5 |
Nothing happened, or the change was not applied¶
Rebuilt images that a run did not pick up are under Images.
| What you see | What the code does | Read |
|---|---|---|
-t common reconfigure changed nothing: a fluentd, cron or kolla_toolbox change was not applied (no-op) |
kolla-ansible -t common reconfigure executes, per play, only the include_tasks statement and the set_fact. The included tasks are filtered out because they do not carry the common tag CODE: no host is reconfigured. |
Apply common changes §2; Apply common changes §1.1 (the -t chrony command); Tags §2.1; F2; R25 |
The common change was applied later, by a run that was not meant to apply it |
common changes (fluentd/cron/toolbox) are applied by whatever run first reaches the host, often unexpectedly. |
F2 ("Why it matters"); R25; Apply common changes §3 |
kolla_set_configs --check returns 0 and nothing restarts after a change to a container's command line, for example a memcached setting; genconfig then deploy-containers did not apply it |
--check compares only the config_files entries of config.json: a change confined to config.json itself (command, permissions; all of memcached's settings) is never detected CODE. A normal deploy/reconfigure applies it, because the changed template task notifies the restart handler; once genconfig has written it, neither deploy-containers nor a later run applies it until the container restarts for another reason. |
F14; Kolla images §3.2; Kolla images §6 (item 1); Controlled batches §2 (caveats) |
| An OVN command-line change was not applied | FACT: the OVN manifests have no config_files, so kolla_set_configs --check always exits 0: a command-line change written by genconfig is applied neither by deploy-containers nor by a later run, until the container restarts for another reason. |
fact sheet A §5.8 (C) |
| A file that was removed from the config is still in the container after the restart | A restart does not apply a removal CODE. The destinations of entries dropped from config.json, and of optional sources that are gone, are never deleted: they stay in the container until it is recreated, and a destination on a volume survives recreates as well. --check does not flag them CODE. Deleting a custom file such as /etc/ also leaves its rendered copy in /etc/, which is copied in again even after a recreate. |
Kolla images §3.5; Kolla images §6 (item 4); Repository architecture §7 (item 4) |
| A custom CA that was deleted is still installed in the containers | A CA deleted from kolla_certificates_dir/, or left there after kolla_copy_ca_into_containers is turned off, stays in /etc/, because the copy task never deletes files CODE, so it is re-added at every start until it is deleted there by hand. |
Kolla images §3.5 |
| The HAProxy frontend of a disabled service is still loaded | Stale services.d files are never removed CODE. If a service is disabled (enable_X: no), its include_role is skipped; the old /etc/ stays and is loaded by haproxy_run.sh on the next restart. |
Known bugs and limitations (row haproxy); fact sheet A §1.8 (D); fact sheet A §2.4 |
| A Keystone domain config file that was removed is still copied into the container | Removing a file from /etc/ is never propagated: no task deletes stale files in /etc/. FACT: under the default COPY_ALWAYS the stale file is copied back into the container at every start, even after a recreate. |
fact sheet B §3.9 (item 8) |
A swift config or ring change was not applied by deploy |
No handlers are notified in the swift role: deploy does not apply config or ring changes CODE; runtime effects NOT AVAILABLE. |
Known bugs and limitations (row swift); fact sheet D §3.4; fact sheet D §7 (item 4) |
A changed heat_domain_admin_password was not applied in Keystone |
bootstrap_heat creates heat_user_domain/heat_domain_admin only when the domain is absent: a changed heat_domain_admin_password reaches heat.conf but never Keystone CODE; impact NOT AVAILABLE. |
Known bugs and limitations (row heat); Kolla images §6 (item 7) |
A changed rabbitmq_cluster_cookie was not applied to the brokers |
The image writes .erlang.cookie only in the bootstrap container (new volume). A cookie change recreates the brokers (env compare) but they keep the old cookie, while kolla_toolbox gets the new one CODE; consequences NOT AVAILABLE. |
Known bugs and limitations (second row rabbitmq); Kolla images §6 (item 9) |
| An ovs-dpdk NIC mapping change was not applied | The NIC mapping file is written once CODE: ovs-dpdkctl.conf is never regenerated, so mapping changes require deleting it manually. |
Known bugs and limitations (row ovs-dpdk); fact sheet A §6.8 |
After -t keystone, the HAProxy backend list of another service was not updated |
Under -t keystone, other services' .cfg files are not re-rendered: if another service's backend list changed (say, a new controller), it is not updated. |
fact sheet A §1.9 |
/etc/hosts on the existing hosts does not list the new compute after a run with --limit |
/etc/hosts on existing hosts is not updated with the new compute, because bootstrap-servers was limited. With --limit, /etc/hosts and other cross-host files are not updated on non-limited hosts. |
Add a compute node §1.1; Add a compute node §2; R23; fact sheet E §3 |
| An expired certificate was not renewed by a new run of the certificates playbook; a new backend host is not in the backend certificate | Test CA only; certs never renewed; new backend hosts not added CODE. Every generating step is creates:-guarded, so re-running never renews expired certificates. |
Known bugs and limitations (row certificates); fact sheet A §8.3 |
| New certificates were generated, but the containers still use the old ones | Distribution needs a separate run: certs only reach containers through kolla-ansible -t haproxy reconfigure (pem files) or a per-service reconfigure (backend and CA). Each of those means a container restart. |
fact sheet A §8.3 |
Neutron's DB sync did not run on reconfigure |
With the default neutron_enable_rolling_upgrade: "yes", bootstrap_neutron runs on deploy but is skipped on reconfigure: a reconfigure re-runs Neutron's DB create/grant tasks but not its DB sync CODE. |
F5; Reconfigure Neutron §2 |
The tag service-rabbitmq ran nothing |
The tag is on the RabbitMQ user/vhost blocks inside roles, which are inside dynamic includes: on its own it executes nothing and is not usable as a selector. | Tags §3 (row service-rabbitmq) |
-t memcache did not update the HAProxy file of memcached |
-t memcache reconfigures the container but not its haproxy file; only -t memcached does both. |
fact sheet A §3.8; Tags §3 (row memcached / memcache) |
The nova-reload, nova-api-reload and nova-cell-reload tags did nothing |
The reload plays run recreate_or_restart of nova services only when kolla_action == upgrade: a no-op outside upgrade. |
Tags §3 (row nova-reload) |
deploy-containers did not update the HAProxy frontends |
deploy-containers writes no config, and the haproxy includes are skipped: only haproxy check-containers.yml runs, and only under -t haproxy or no tags. |
Execution flow §4 (row deploy-containers); fact sheet A §1.9 |
| After the compute batches, config drift is still pending on the controllers | genconfig -t nova-cell also renders the conductor and console config on controllers. Include them in a batch too, or the drift stays pending there. |
Controlled batches §1.1 (caveat) |
| After a tagged run, some hosts still have the old fluentd or logrotate files | After enabling or disabling a service that has a logrotate or fluentd input entry (enable_X), fluentd, cron and kolla_toolbox restart on every host where a play that runs common executes: the whole fleet for an untagged run, only the tagged service's hosts for a tagged run. The other hosts keep the old files until a later run includes common there. |
R2 |
mariadb_backup did nothing: every task was skipped |
FACT: backup runs only on groups['mariadb'][0] and requires enable_mariabackup: yes (default no); otherwise every task is skipped silently. |
fact sheet E §9; Database operations |
deploy -t barbican alone: the barbican_notifications topic is not enabled |
FACT: running kolla-ansible deploy -t barbican alone does not run the keystone play, so the barbican_notifications topic is not enabled in keystone.conf. NOT AVAILABLE: whether barbican-keystone-listener then receives project-deletion events. |
fact sheet D §1.6 |
| No amphora flavor, image, network or security group exists after an Octavia deploy | No amphora resources are created CODE: in Ussuri the role does not create the amphora flavor, image, keypair, lb-mgmt network, security group or certificates. | Known bugs and limitations (row octavia); fact sheet D §2.5 |
| No share type, service image or network exists after a Manila deploy | FACT: Kolla does not create share types, the service image or networks. | fact sheet D §4.6 |
| A service tag did not touch MariaDB, RabbitMQ, memcached or Open vSwitch | A service tag never touches MariaDB, RabbitMQ or memcached containers (other plays), and it never touches OVS (unless -t openvswitch). Open vSwitch is not touched by -t neutron (separate play/tag). |
Reconfigure a single service §2; Reconfigure Neutron §4 |
The run failed or a task failed¶
| What you see | What the code does | Read |
|---|---|---|
The check-containers task failed on a host; kolla_set_configs --check exit code 2 |
--check exits 2 when an optional file is rendered for the first time (e.g. nova-compute release) and no entry listed before it differs, and the check-containers task fails on that host instead of restarting CODE. If an entry listed before the optional one differs, the result is 1 and the restart also copies the new file. Exit 2 is also raised for a config.json that cannot be opened and for a uid or gid with no passwd or group entry. |
Kolla images §3.3; Kolla images §6 (item 3); Predict and inspect a run (row "See what will restart"); Controlled batches §1.1 |
| Every later run fails on the same check task | A newly added prometheus extras file makes --check exit 2: the check task fails on that host before the handlers run, and the container is not restarted. Later runs fail the same way, because the file is still not in the container, until prometheus_server restarts for another reason. |
Tags §3 |
docker exec <c> kolla_set_configs --check run by hand exits 2 |
In an image with a service USER (e.g. USER nova), a plain docker exec <c> kolla_set_configs --check cannot open the files in /etc/, which are owned by root with mode 0660, and exits 2. kolla_docker execs as root; by hand, use docker exec -u root. |
Kolla images §3.4; Predict and inspect a run (row "See what will restart") |
| After the restart, the container fails at every start | Exit 1 is also returned for a required source that is missing and for an invalid config.json; the restarted container then fails in kolla_set_configs at every start. Whether it then restarts in a loop is NOT AVAILABLE. |
Kolla images §3.3; Predict and inspect a run (row "See what will restart") |
deploy or reconfigure fails in the mariadb role after kolla-ansible stop, or while a node is not Synced |
Reconfigure aborts if the cluster is fully stopped (multi-node) or any alive node is not Synced (FACT). kolla-ansible stop stops every node at once, so the next deploy/reconfigure fails until mariadb_recovery is run. |
fact sheet B §1.8 (item 4); R15; Database operations |
| The MariaDB restart handler failed while it waited for WSREP sync | The WSREP-sync wait is only about 60 s (FACT); a restart that needs IST or SST longer than that fails the handler. The restart waves have no any_errors_fatal CODE: a node that fails its wait marks only its own host failed, so the next wave still restarts its nodes. NOT AVAILABLE: whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is. |
fact sheet B §1.8 (item 3); fact sheet B §1.5 (failure mode); R9; Known bugs and limitations (row mariadb); Q6 |
kolla-ansible pull with outward RabbitMQ enabled: outward_rabbitmq_services |
Bug (FACT): service-images-pull looks up outward_rabbitmq_services, and that variable is not defined anywhere. The vars lookup raises an undefined-variable error for a name it cannot find when no default is given, so "Pull images" fails on the outward-rabbitmq hosts. |
fact sheet B §2.8 (item 6); Known bugs and limitations (first row rabbitmq) |
pull for ovs-dpdk: ovs_services |
pull looks up a non-existent ovs_services variable CODE. |
Known bugs and limitations (row ovs-dpdk); fact sheet A §6 |
prechecks passed, but deploy failed in the keystone role at set_fact cron_jobs |
The precheck validates the wrong variable. The generator exits 0 with {"failed": true}, so the command task succeeds and set_fact cron_jobs then errors on a missing key (FACT). Any fernet_token_expiry above 5 days with the default window passes precheck but breaks deploy. |
fact sheet B §3.9 (item 5); Known bugs and limitations (row keystone) |
The swift reconfigure fails |
FACT: the swift reconfigure fails if any listed container is not running, so reconfigure cannot repair a crashed container. |
fact sheet D §3.4; fact sheet D §7 (item 4) |
upgrade: the task "Stopping all slave keepalived containers" with enable_keepalived: no, or with an api_interface whose name contains - |
The task has no enable_keepalived guard and uses raw interface names CODE; effect of the upgrade bugs NOT AVAILABLE. |
Known bugs and limitations (row haproxy); fact sheet A §1.8 (F) |
bootstrap-servers failed on every host at the /etc/hosts task |
FACT: the block loops over all groups['baremetal'] hosts, so every baremetal host's facts are required. A baremetal host outside bifrost without gathered facts (for example unreachable during fact gathering, with no fact cache) makes kolla_address raise on every host, which fails the task on every host and ends the play for them. |
fact sheet E §0 (item 4); fact sheet E §3 |
A run with --limit failed because a host outside the limit is unreachable |
FACT: an unreachable other host without cached facts makes its delegated item return unreachable; the limited host is then marked unreachable and removed from the play and from every later play. So --limit newnode can fail because an unrelated node is down. |
fact sheet E §4; fact sheet E §14 (H4) |
Templating failed in kolla_address: interface not present on host |
FACT: kolla_address raises FilterError if the interface or address family is missing on the host. Every octavia host must have octavia_network_interface with an address, or templating fails. The documented setup filter "ansible_[!qt]*" also hides real interfaces whose names start with q or t; kolla_address then fails for api_interface, tunnel_interface and so on if they are named that way. |
fact sheet D §2.6; fact sheet E §4; fact sheet E §14 (H16) |
Jinja errors on groups['ovn-…'] |
There is no group validation precheck: an old inventory without the ovn-* groups only surfaces as Jinja errors on groups['ovn-…']. |
fact sheet A §5.8 (H) |
-t nova failed at Keystone registration on a fresh cloud |
Nothing enforces a dependency when you use tags CODE: -t nova on a fresh cloud fails at Keystone registration if Keystone was never deployed. On an existing cloud it assumes MariaDB, RabbitMQ and Keystone are healthy. |
Component dependencies §6 |
| A bootstrap or registration task failed on the first host of a group | Bootstrap and registration tasks target groups[...][0] CODE: the first host of a group is a hidden dependency; it must be reachable and have kolla_toolbox running. |
Component dependencies §6 |
mariadb_recovery tried to start mariadb_clustercheck although it is disabled |
FACT: the recovery restart tasks loop over mariadb_services with no item.value.enabled or group check. With enable_mariadb_clustercheck: no, recovery still tries to start a mariadb_clustercheck container; if the image pull returns an error, kolla_docker fails the task on that host. |
fact sheet B §1.8 (item 6); Known bugs and limitations (row mariadb) |
docker.service is masked after a failed or interrupted bootstrap-servers run on Debian/Ubuntu |
FACT: on Debian/Ubuntu the run masks docker.service and unmasks it later; a failure or interruption between those points, unless it comes after the recovery block has run, leaves the symlink in place. NOT AVAILABLE: whether Docker starts after the next reboot. |
fact sheet E §2.3; fact sheet E §0 (item 3); Known bugs and limitations (row baremetal) |
| Prechecks: the port or the address that a precheck tests | The swift precheck probes rsync port 873, but rsyncd uses 10873 CODE. FACT: the octavia health-manager precheck uses the wrong address (api_interface_address, not octavia_network_interface_address), and the iscsi precheck tests port 3260 on iscsid hosts. |
Known bugs and limitations (row swift); fact sheet D §2.6; fact sheet D §7 (items 2, 5 and 11) |
| The RabbitMQ handler blocks after an uncontrolled full stop | After an uncontrolled full stop (kolla-ansible stop, power loss) followed by deploy, the handlers start the first node first and block on rabbitmqctl wait. NOT AVAILABLE (RabbitMQ semantics): whether, if that node was not the last to stop, it waits for its peers' Mnesia tables before giving up, so that the sequential handler stalls or fails on node 1 before nodes 2 and 3 are started. |
fact sheet B §2.8 (item 3) |
stop and the swift account-auditor |
The swift stop.yml uses the wrong container name CODE: the account-auditor stop checks 'swift_object_auditor' not in skip_stop_containers. |
Known bugs and limitations (row swift); fact sheet D §3.4 |
The run is slow¶
| What you see | What the code does | Read |
|---|---|---|
A run with --limit is slow on a large inventory: it spends a long time gathering facts |
With --limit, all hosts' facts are gathered sequentially by the limited hosts (delegated loop) CODE. With --limit compute001 on the 30-compute inventory (38 hosts, localhost included), compute001 runs 38 setup calls in sequence CODE; its duration is NOT AVAILABLE. |
F9; Compute scaling §6 (item 5); R23; fact sheet E §4 |
| A run with no change is still slow on a large fleet | Every run that includes the compute group executes the full per-host chain on every compute, even with no change: Kolla has no "only if changed" short-circuit at play level. Per compute, a no-change -t nova run makes 46 remote module invocations (53 on compute001) CODE. |
Compute scaling §5; Compute scaling §2; Why a small change creates a large workload |
The run is slower with ANSIBLE_SERIAL or kolla_serial |
kolla_serial bounds simultaneous restarts per play, but adds time: every batch re-runs run_once steps and the compute registration wait, and each batch waits for the slowest host. |
Compute scaling §7; Tags §2.5 |
| One slow host slows the whole run | A task runs on all hosts of the play (up to forks at a time), and the next task starts only when every host has finished the current one CODE: the slowest host sets the pace of every task. |
Compute scaling §1 |
The restart window of an upgrade includes image downloads |
Upgrade without a prior pull: hosts download lazily inside the handler phase (recreate_or_restart_container pulls before stopping), which lengthens and staggers the restart window. |
Update container images §4; Nova deep dive §7; Pre-execution checklist (step 4) |
| The run waits for compute registration (up to 20 × 10 s) | The compute registration wait runs for the first compute of each batch, delegated to the conductor, for up to 20 × 10 s per batch, multiplied by the number of batches when serial is used: it waits until all computes of the batch appear in compute service list. |
Compute scaling §3 |
| A 300 s wait on ovs-dpdk hosts | The 300 s wait handler has no port CODE: an ovsdpdk-db restart is followed by a mandatory 5-minute sleep. |
Known bugs and limitations (row ovs-dpdk); fact sheet A §6.8 |
bootstrap-servers is slow on a large inventory |
/etc/hosts lists every baremetal host: O(N²) templating CODE. |
Known bugs and limitations (row baremetal); fact sheet E §0 (item 4); Compute scaling §3 |
| A container is slow to start | kolla_set_configs makes one chown per file of the glance image store, of /var/lib/nova on computes, of the MariaDB datadir, of the RabbitMQ mnesia directory and of the cinder volume, on every COPY_ALWAYS start CODE. NOT AVAILABLE How start time grows with the number of files, especially on NFS-backed trees. |
Kolla images §6 (item 10) |
| Horizon is slow to start after a recreate or a settings change | FACT: collectstatic and compress run on the first start after a recreate (new image, plugin flag change, COPY_ONCE) or after a local_settings/custom_local_settings change, and not on a plain restart with unchanged settings and plugin flags. NOT AVAILABLE: how long they take. |
fact sheet C §4.8; Kolla images §5.2 |
| Every task costs several SSH round trips | Kolla ships no ansible.cfg, so pipelining stays off and each invocation still makes several round trips (tmp dir, transfer, exec, cleanup). |
Compute scaling §3 (row "SSH connection setup"); Compute scaling §8 |
| The deploy host is busy during the config tasks | template and merge_configs render on the deploy host for every host and item, then copy compares checksums remotely CODE: deploy-host CPU scales with hosts × templates. Whether, and at which forks value, the deploy host's CPU becomes the bottleneck is NOT AVAILABLE. |
Compute scaling §1; Compute scaling §6 (item 3) |
| How long a run should take | No timings are given. Work is counted as operations read from the source; task-result totals and wall times are NOT AVAILABLE. | Compute scaling; What is not known, "Timings, counts and scale"; Q7; Q8 |
--limit seems ignored¶
--limit does not confine actions (F8). The rows about --limit are in four tables of
this index:
- Scope surprises — tasks ran on a controller
although
--limitnamed only a compute - Nothing happened, or the change was not applied —
/etc/hostson the existing hosts does not list the new compute after a run with--limit - The run failed or a task failed — a run with
--limitfailed because a host outside the limit is unreachable - The run is slow — a run with
--limitis slow on a large inventory
Scope surprises: hosts or services touched outside what was asked¶
| What you see | What the code does | Read |
|---|---|---|
Tasks ran on a controller (control01) although --limit named only a compute |
--limit does not confine actions: delegate_to: groups[x][0] runs work on hosts outside the limit. With deploy --limit set to a compute host, control01 (the first neutron-server and nova-conductor host of the multinode inventory) runs the Neutron DB create/grant and the bootstrap_neutron container, plus cell discovery CODE. |
F8; Add a compute node §2; Reconfigure a single service §2 (row "Actions on hosts outside --limit") |
DB-sync (bootstrap) containers and Keystone registration ran on a reconfigure |
reconfigure = deploy for 70 of the 72 roles that have a tasks/: a reconfigure re-runs Keystone registration, DB creation/grants and the DB-sync bootstrap containers of most services CODE. |
F5; Execution flow §6; Repository architecture §4.2; Tags §4 (item 5) |
A service tag also ran common (fluentd, cron, kolla_toolbox) on every host of the service play |
Even with a narrow tag, the code guarantees that the run also touches common on every host of the service play (fluentd/cron/toolbox config and container check, restarts if drift): the meta dependency inherits the tag. |
Reconfigure a single service §2; Tags §4 (item 2); Repository architecture §5.1 |
| A service tag rendered the HAProxy frontend of the service on all haproxy hosts | -t <svc> runs that service's haproxy config on all haproxy hosts CODE. |
Tags §2.3; fact sheet A §1.9 |
| A tag ran tasks on every compute | nova, nova-cell, neutron, openvswitch, ovn, chrony and iscsi target compute hosts, as do baremetal (bootstrap-servers, all hosts) and nova-reload/nova-cell-reload (work only on upgrade); nova-api does not touch computes. A tag never restricts which hosts a play targets; use --limit for that. |
Tags §3; Tags §4 (item 2) |
-t keystone re-distributed the Fernet keys |
-t keystone runs distribute_fernet.yml: on a reconfigure the Fernet keys are re-distributed from the first running keystone_fernet node to the others. |
Reconfigure a single service §2; Execution flow §6; R19 |
-t glance changed a global MariaDB variable |
The glance role toggles the global MariaDB variable log_bin_trust_function_creators on every deploy/reconfigure CODE. |
Known bugs and limitations (row glance); R20; fact sheet C §1 |
nova_scheduler received SIGHUP on a run with no change |
Any deploy/reconfigure with the nova or nova-api tag, even with no change, sends SIGHUP to nova_scheduler (cell-cache refresh). It is the only signal Kolla-Ansible sends to a running service, and it loads no new config files. |
Nova deep dive §2.4; Repository architecture §4.3 |
-t haproxy also reconfigured fluentd, cron and kolla-toolbox on the haproxy hosts |
FACT: -t haproxy also runs common on haproxy hosts: common is a meta dependency, and its tasks inherit the play tag haproxy. |
fact sheet A §1.8 (E) |
genconfig with a service tag also changed files on the haproxy hosts |
genconfig writes /etc/ on the targets and also re-renders haproxy services.d for the tagged services. |
Predict and inspect a run (row "Render configs without restarting") |
genconfig ran NIC rebinding on ovs-dpdk hosts |
NIC rebinding and unit installation run on every deploy, reconfigure, upgrade and genconfig. They are idempotent only as long as the driver already matches. | fact sheet A §6.8 |
A bootstrap-servers re-run changed the host: firewall, SELinux, packages |
Re-running bootstrap-servers changes the host: it disables ufw/firewalld, sets SELinux permissive, unloads AppArmor profiles and removes the host packages open-iscsi/iscsi-initiator-utils, libvirt*, lxd/lxc and chrony (FACT). None of this restarts containers. |
fact sheet E §0 (item 6); R14 |
destroy also ran on the deploy host |
FACT: destroy acts on hosts: all, including the deploy host (localhost is in [deployment]). tools/ deletes every network namespace and every link whose line matches tap, brq or vxlan, and rm -rfs /etc/kolla/* except six names. |
fact sheet E §0 (item 10); fact sheet E §6; R16; Stop and destroy |
-t ovn overwrote or removed a manual ovn-cms-options value |
enable-chassis-as-gw overwrites ovn-cms-options on network nodes, and the option is removed entirely on non-gateway openvswitch hosts on every run (FACT). Any manual ovn-cms-options value is overwritten or removed on every reconfigure; the impact of that loss is NOT AVAILABLE. |
fact sheet A §5.8 (E); Tags §3.1 (row ovn) |
A manual net.ipv4.tcp_retries2 line disappeared from /etc/ on the haproxy hosts |
FACT: the sysctl task with KOLLA_UNSET removes any manual net.ipv4.tcp_retries2 line from /etc/ on every run. |
fact sheet A §1.8 (G) |
| A keystone reconfigure re-applied the admin credentials and the endpoints | Reconfigure re-asserts identity state: it re-applies the admin credentials and endpoints, and the DB user password, from passwords.yml and globals. NOT AVAILABLE: whether out-of-band changes are silently reverted. |
fact sheet B §3.9 (item 10) |
Enabling Swift, or glance_backend_ceph, changed the Glance backend |
Enabling Swift (or glance_backend_ceph) on a running cloud silently switches Glance's backend; NOT AVAILABLE whether existing file-store images stay readable. |
R21; Known bugs and limitations (row glance); fact sheet C §1.8 (item 1); Q12 |
| After the inventory was reordered: keepalived priorities, MariaDB restart waves, the glance host or the bootstrap target changed | Inventory reordering (haproxy, mariadb, glance, nova-conductor, keystone groups): keepalived priorities change; MariaDB restart waves and active backend move; glance moves (file backend); bootstrap/discovery target changes. | R22 |
| A task reports changed on every run although nothing changed | Set system-id (openvswitch) always reports changed CODE: it is a command task with no changed_when. FACT: the OVN chassis bootstrap runs 4 docker exec ovs-vsctl per compute (5 with computes_need_external_bridge) on every deploy, reconfigure and upgrade, always reported as changed. |
Known bugs and limitations (row openvswitch); Tags §3.1 (row openvswitch); fact sheet A §5.8 (D) |
An empty placement-api.cfg was written on the haproxy hosts although Placement is disabled |
The placement haproxy include lacks when: enable_placement CODE: under -t haproxy, placement-api.cfg is always written, and is empty if enable_placement is false. |
Known bugs and limitations (second row site.yml); fact sheet A §1.9 |
| A password rotation needed a reconfigure of the whole compute fleet | FACT: placement and cinder credentials are embedded in nova-compute config on every compute, so password rotation needs a reconfigure of the whole compute fleet. | fact sheet C §7; fact sheet C §2.8 |
On a new compute, nova_compute started before Open vSwitch and the Neutron agents |
The Nova deploy order puts nova-compute before OVS and the Neutron agents on new computes CODE. NOT AVAILABLE Whether instances can be scheduled on such a compute before the later plays finish. | F13; Execution flow §2.4; Component dependencies §6 |
Images¶
| What you see | What the code does | Read |
|---|---|---|
deploy or reconfigure did not pick up a rebuilt image pushed under the same tag |
deploy/reconfigure never pull an image that is already present locally, even if the registry has a newer build under the same tag CODE. Only absent images are pulled; an explicit pull is required CODE. |
Update container images §1.1; Update container images §2; F4; R13 |
An unrelated pull was followed by a mass recreate |
Mutable tags create silent drift between hosts. A later unrelated pull turns into a mass recreate. |
F4 ("Why it matters"); R13 |
A new openstack_tag applied with deploy recreated every container on every host |
New openstack_tag, deploy: every container on every host is recreated CODE. Changing the tag and running deploy (instead of upgrade) recreates the whole fleet in one go, without the upgrade-specific ordering: no nova-status upgrade check, no RPC-pin reload plays, and no online data migrations, except Placement's (and Ironic's only with ironic_enable_rolling_upgrade: no). |
Update container images §2; Update container images §4; R12 |
An OVS image change recreated openvswitch_vswitchd on every OVS host at the same time |
An OVS image change recreates openvswitch_vswitchd on every OVS host simultaneously (serial 0). NOT AVAILABLE Whether VM networking is interrupted on every compute at the same time. |
Update container images §4; R7 |
pull did not change the running containers |
pull never changes running containers CODE. After a pull that downloaded something, the next deploy, reconfigure or upgrade recreates the affected containers CODE. |
Update container images §1.1; R11 |
-t nova pull downloaded more than the nova images |
-t nova pull downloads on every host in the nova plays, for nova images and the 3 common images CODE. |
Update container images §2 |
genconfig pulled an image |
The fluentd label check still pulls the fluentd image if missing. | Execution flow §4 (row genconfig) |
The SSH host keys of nova_ssh and keystone_ssh changed after a recreate |
SSH host keys of nova_ssh and keystone_ssh change on every recreate, because /etc/ssh is on no volume. The clients set StrictHostKeyChecking no and keep no known-hosts file CODE; whether the SSH client then accepts the new keys without failing is NOT AVAILABLE (OpenSSH semantics). |
Kolla images §6 (item 18) |
prune-images did not remove the images of the previous release |
FACT: prune-images removes only dangling kolla-labelled images, so old release tags are not reclaimed. |
fact sheet E §0 (item 10); fact sheet E §7; R17 |
Settings that are ignored¶
| What you see | What the code does | Read |
|---|---|---|
docker_graceful_timeout (60 s) is ignored: containers get SIGKILL 10 s after the stop |
docker_graceful_timeout and docker_client_timeout are ignored. The kolla_docker argument defaults (10 s, 120 s) override common_options, so every stop sends SIGKILL to a container still running 10 s after SIGTERM CODE. The release note claiming 60 s has no effect. |
F3; Repository architecture §6; R24; Known bugs and limitations (row kolla_docker) |
-e kolla_serial=N is ignored by reconfigure and upgrade |
-e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade CODE: the wrapper appends -e kolla_serial=${ANSIBLE_SERIAL} after the user's -e options. |
Known bugs and limitations (first row CLI); Repository architecture §2; fact sheet E §11 |
ANSIBLE_SERIAL is ignored by deploy |
kolla_serial is not passed for deploy CODE. Batching is ANSIBLE_SERIAL for reconfigure/upgrade and -e kolla_serial=N for deploy, which ignores ANSIBLE_SERIAL. |
Known bugs and limitations (first row CLI); Ten things to know (item 4); Execution flow §5 |
mariadb, rabbitmq and cyborg are not batched by ANSIBLE_SERIAL |
Not batched: mariadb, rabbitmq and outward rabbitmq (no serial; their rolling logic is in the handlers), and cyborg, which uses serial instead of kolla_serial and so ignores ANSIBLE_SERIAL (bug). |
Controlled batches §1.2; Known bugs and limitations (first row site.yml); Execution flow §5 |
A per-host or per-group inventory value is ignored for a variable set in globals.yml |
globals.yml and passwords.yml are passed with -e @file, that is extra-vars, the highest precedence in Ansible: a value in globals.yml cannot be overridden per host or per group in the inventory CODE. |
Repository architecture §2.1 |
A restart_policy change is not applied |
restart_policy is never compared (FIXME :21) CODE. |
Known bugs and limitations (row kolla_docker); Repository architecture §6 |
A docker_custom_option change is not applied |
docker_custom_option is written to /etc/; adding, changing or removing it does not restart Docker (FACT). No Kolla task applies the new flags; when they take effect is NOT AVAILABLE (systemd behaviour). |
fact sheet E §0 (item 2); Known bugs and limitations (row baremetal) |
-v flags are lost with --ask-vault-pass |
--ask-vault-pass overwrites VERBOSITY with a copy of EXTRA_OPTS CODE, which duplicates all options seen so far and drops earlier -v flags. |
Known bugs and limitations (second row CLI); Repository architecture §2; fact sheet E §11 |
--include-dev and the next word of the command line |
--include-dev: is declared with an argument but used as a flag; the command line is executed unquoted CODE. |
Known bugs and limitations (third row CLI); fact sheet E §11 |
enable_swift_recon with its default "no" |
FACT: enable_swift_recon is tested without the bool filter, and its default is the string "no", not a boolean. NOT AVAILABLE: whether recon is then always on. |
fact sheet D §3.8; fact sheet D §7 (item 3) |
| A service variable overridden only on the backend hosts, and the HAProxy frontend | The haproxy-config variables come from the service role's defaults, evaluated on the haproxy host; inventory or group_vars overrides of service variables therefore apply when set for the haproxy hosts or for all. NOT AVAILABLE: whether per-host overrides set only on backend hosts are seen by haproxy-config, outside the places where the code explicitly uses hostvars[groups[…][0]] (nova-cell). |
fact sheet A §2.4 |
Listings, check mode and prechecks¶
| What you see | What the code does | Read |
|---|---|---|
--list-tasks shows only include_tasks placeholders |
ansible-playbook site.yml --list-tasks -t nova lists only include_tasks placeholders for each role: the listing walks only the statically compiled play, while dynamic includes are loaded at run time CODE. |
Tags §2.6; F7; Predict and inspect a run |
--list-hosts shows 0 hosts; Could not match supplied host pattern, ignoring: enable_nova_True |
Plays whose host pattern depends on the group_by groups (&enable_nova_True) resolve to 0 hosts at list time: those groups are created at run time CODE. |
Tags §2.6; F7; Predict and inspect a run |
--list-tags does not show a tag |
--list-tags lists only the tags it can see statically; tags that occur only inside dynamically included files, such as service-rabbitmq, do not appear. |
Tags §3.1 (paragraph "Validation") |
A --check --diff run skipped the kolla_docker tasks |
kolla_docker does not declare check-mode support, so Ansible skips it under --check CODE. NOT AVAILABLE Whether such a run fails or gives partial results. |
Predict and inspect a run (row "Check-mode run"); Tags §2.6; Q9 |
genconfig plus kolla_set_configs --check did not predict what the run did |
"genconfig + --check" under-reports some changes and over-reports others. |
F14 ("Why it matters"); Kolla images §3.2; Predict and inspect a run |
docker ps shows no health status |
No Docker HEALTHCHECK is wired in Ussuri: docker ps shows no health status; service health must be checked from outside CODE. |
Kolla images §6 (item 17) |
prechecks with -t <svc> skipped the general prechecks |
-t <svc> skips the prechecks role (untagged) but runs <svc>/. |
Predict and inspect a run (row "Pre-deployment checks") |
| The tag name did not predict what the run did | A tag name does not describe the work a run performs. What runs depends on which plays carry or inherit the tag, the common role reaching each host for the first time, the haproxy play's per-service includes, handlers, which are never filtered by tags, and Ansible 2.9's loop-notify semantics. |
Tags ("The single most important rule"); Tags §3 |
| The same command was a no-op one day and a fleet-wide restart another day | The same command can be a no-op or a fleet-wide restart, depending on whether a rendered file changed. That in turn depends on global variables (enable_*, group membership, passwords), not only on the tagged service CODE. |
Tags §4 (item 3); Component dependencies §5 |
Batching with serial¶
What serial does not reach is under Settings that are ignored; what it costs is under
The run is slow.
| What you see | What the code does | Read |
|---|---|---|
With serial, the DB sync, Keystone registration or discovery ran several times |
With kolla_serial > 0, a run_once task runs once per batch, not once per play CODE. This applies to bootstrap containers, Keystone registration and DB creation. With kolla_serial=1 and the three controllers of the multinode inventory, nova_api_bootstrap runs 3 times, once per controller batch CODE. |
Tags §2.5; F10; Controlled batches §1.2 |
With serial, the computes restarted before the conductors |
kolla_serial orders a play's restarts by host-pattern order (computes before conductors in the nova-cell play) CODE. |
F10 |
With serial, the neutron agents of the first hosts restarted only after every compute had finished the earlier plays |
A batch runs the complete play before the next batch starts, but each play is still traversed batch by batch for the whole fleet before the next play starts. The neutron agents on compute001 are restarted only after all computes have finished the nova-cell and openvswitch plays. | Execution flow §5 |
Upgrades and database migrations¶
| What you see | What the code does | Read |
|---|---|---|
A schema migration ran during a reconfigure |
A "reconfigure" can apply schema migrations if the image tag moved, for most services; for Neutron only with non-default flags: enable_neutron_sfc or neutron_enable_rolling_upgrade: "no". There is never a "config only" path except genconfig + deploy-containers. |
F5 ("Why it matters") |
Placement's online data migrations ran on a deploy or reconfigure |
Placement's online data migrations run on every deploy/reconfigure, inside bootstrap_placement, and twice on upgrade CODE. |
Kolla images §6 (item 6); F15 |
Nova's online data migrations did not run on deploy or reconfigure |
Nova's online data migrations never run outside upgrade. Deploy and reconfigure run the Nova bootstrap containers with KOLLA_UPGRADE (api_db sync + db sync, db sync --local_cell) CODE. |
Kolla images §6 (item 5); F15 |
neutron-dynamic-routing was not migrated by upgrade |
upgrade with the default neutron_enable_rolling_upgrade: "yes" expands/contracts only neutron, neutron-fwaas and neutron-vpnaas: neutron-dynamic-routing, which the KOLLA_BOOTSTRAP container migrates on deploy, is never migrated by upgrade, and the image's rolling neutron-db-manage calls pass no --config-file CODE. |
Known bugs and limitations (row neutron); Kolla images §6 (item 8); F15 |
An upgrade stopped neutron_server on every neutron-server host |
has_offline_migrations runs on the first neutron-server host with ignore_errors; any non-zero rc (a failed docker exec included) stops neutron_server on every neutron-server host in the play until the handler flush after the contract container CODE; impact on the Neutron API NOT AVAILABLE. |
Known bugs and limitations (row neutron); Kolla images §6 (item 8) |
The RabbitMQ upgrade stopped all brokers |
RabbitMQ image change (upgrade): all brokers stopped, full message-bus outage during upgrade. |
R10; fact sheet B §2.8 (item 2) |
On upgrade, keepalived was stopped on the hosts that do not hold the VIP |
On upgrade, keepalived is stopped on every non-VIP host and recreated in the handler phase. | fact sheet A §1.8 (F) |
The octavia upgrade removed the octavia admin role |
The upgrade removes the octavia admin role on admin without re-registering CODE. |
Known bugs and limitations (row octavia); fact sheet D §2.3 |
mysql_upgrade ran on all MariaDB nodes at once, with an unchanged image |
FACT: mysql_upgrade (upgrade_mariadb) runs on all nodes concurrently and on every upgrade, even when the image is unchanged. NOT AVAILABLE: whether concurrent mysql_upgrade on Galera nodes conflicts. |
fact sheet B §1.8 (item 8) |
Glance rolling upgrade with serial: the contract ran before the last batches were upgraded |
Answered from code: Glance rolling upgrade with ANSIBLE_SERIAL=1 runs the contract at the end of batch 1, before later batches are upgraded. |
fact sheet C §8 (item 2); fact sheet C §1.8 (item 4) |
Keystone upgrade with serial: log_bin_trust_function_creators |
With serial, log_bin_trust_function_creators is reset to 0 at the end of every batch, while Enable runs only in the batch whose first host is groups['keystone'][0] (FACT). The contract step in the last batch therefore runs with it disabled. The impact is NOT AVAILABLE. |
fact sheet B §3.9 (item 9) |
| The swift "rolling" upgrade stopped all storage containers at once | FACT: despite the name, the swift rolling upgrade stops all storage containers on all storage hosts at once. | fact sheet D §3.4; fact sheet D §7 (item 4) |
The cinder upgrade restarted every cinder-volume and cinder-backup at once |
FACT: upgrade restarts every cinder-volume and cinder-backup at once (serial 0 unless ANSIBLE_SERIAL is set). |
fact sheet C §3.8 (item 4) |
A reconfigure with a bumped image tag ran cinder's db sync without its online data migrations |
FACT: a reconfigure with a bumped image tag runs cinder's db sync without its OSM step, while placement's bootstrap container runs both db sync and db online_data_migrations. Whether this leaves cinder in a partially upgraded state is NOT AVAILABLE. |
fact sheet C §6 |
Services, VMs and traffic during and after a restart¶
Every statement about data-plane or VM impact is NOT AVAILABLE (Limits of this study). The rows say what the code shows and point to what is open.
| What you see | What the code does | Read |
|---|---|---|
Running VMs after a nova_libvirt restart or recreate |
NOT AVAILABLE Whether running VMs survive a nova_libvirt restart or recreate; what the container and its start path do is CODE. |
Nova deep dive §5; Q1; What is not known, "Running VMs and compute hosts" |
VM network traffic after a restart of the OVS agent, of openvswitch_vswitchd or of the L3 agent |
NOT AVAILABLE Whether VM network traffic is interrupted when the OVS agent, openvswitch_vswitchd or the L3 agent restarts; that the restarts happen is CODE. |
Reconfigure Neutron §4; R5; R6; R7; Q2; Q3; Q4; What is not known, "Network data plane" |
Open vSwitch flows after an openvswitch_vswitchd restart |
An openvswitch_vswitchd restart replaces only the userspace daemon. The kernel module stays loaded and conf.db keeps bridges and ports CODE. NOT AVAILABLE Whether the OpenFlow tables are lost and must be reprogrammed by the OVS agent or ovn-controller. |
Kolla images §6 (item 13); R7; Q3 |
Router namespaces after a neutron_l3_agent restart |
The L3 agent wrapper runs neutron-netns-cleanup --force --agent-type l3 at start when enable_neutron_agent_ha: yes CODE; its comment says this removes all L3 namespaces. What the command does is neutron code: NOT AVAILABLE. |
F12; R6; Reconfigure Neutron §4; Kolla images §5.3; Q4 |
| Work in progress was killed at a restart: SIGKILL after 10 s | On every restart, a service still running 10 s after SIGTERM is SIGKILLed with its in-flight work (heat-engine, cinder copies, glance uploads). NOT AVAILABLE Whether services started by a shell wrapper that does not exec them (e.g. neutron-l3-agent, haproxy) receive SIGTERM at all. |
F3 ("Why it matters"); R24; Kolla images §2.6 |
| The API of a service was unavailable during a run with its tag | The service API is unavailable while its containers restart (up to 10 s stop, then start) on all its hosts at once. Add the HAProxy restart window if the frontend changed. | Reconfigure a single service §3 ("Interruption") |
| Authentication while keystone restarts | All keystone containers restart on all hosts at once; NOT AVAILABLE whether authentication fails while keystone restarts. | R19; Reconfigure a single service §1.1 |
| APIs, DB traffic and the VIP while HAProxy restarts | haproxy is restarted (never reloaded) on all haproxy hosts at once; NOT AVAILABLE whether keepalived drops or moves the VIP, and for how long. Blast radius: all APIs and all DB traffic (the DB address is the VIP). |
R8; Q5; fact sheet A §1.8 (B); Kolla images §6 (item 15) |
| Token cache and Horizon sessions after a memcached restart | memcached has no data volume and no persistence option CODE. NOT AVAILABLE (memcached semantics) Whether every restart therefore empties the cache, with token-cache misses and lost Horizon sessions where they are kept in memcached. | Kolla images §6 (item 14); R4; fact sheet A §3.8 |
| RPC while RabbitMQ restarts | RabbitMQ restarts are one by one; NOT AVAILABLE whether nova-compute and the agents lose RPC while their broker restarts, and whether RPC-backed API calls time out meanwhile. | Tags §3.1 (row rabbitmq); fact sheet B §2.8 (item 1) |
| DB connections during a MariaDB rolling restart | MariaDB config change or image change: rolling restart in 4 waves by index % 4, one wave after another. HAProxy's single active backend is node 0; NOT AVAILABLE whether node 0's wave runs first. API impact: DB connection resets. |
R9; fact sheet B §1.8 (item 1) |
| MariaDB after all DB containers stopped (power loss, Docker restart on every DB host) | MariaDB has no automatic Galera recovery. NOT AVAILABLE Whether, after all DB containers stopped, they come back as plain mysqld_safe with the cluster non-primary until mariadb_recovery. A container left with BOOTSTRAP_ARGS=--wsrep-new-cluster by an interrupted run passes that option to mysqld_safe again on any manual docker start CODE; whether that bootstraps a new cluster (split-brain risk) is NOT AVAILABLE. |
Kolla images §6 (item 16); Kolla images §5.4; fact sheet E §2.4; R18 |
Image API outage when glance_api restarts |
With the file backend only glance-api[0] exists, so any restart is a full Image API outage; NOT AVAILABLE whether snapshots in progress fail. |
R20; fact sheet C §1.8 (item 3) |
Volumes and iSCSI sessions after an iscsid or tgtd restart |
iscsid: NOT AVAILABLE whether existing sessions are kernel-held and whether attach/detach during the window fails. tgtd: nothing in its start path runs tgt-admin to load /etc/; NOT AVAILABLE whether a tgtd restart cuts disk I/O for VMs using LVM volumes from that host until the targets are re-created. |
Tags §3.1 (row iscsi); What is not known, "Storage and images" |
The iSCSI initiator name (IQN) after a restart, a recreate or the removal of the iscsi_info volume |
The iSCSI initiator name is stable per iscsi_info volume across restart and recreate CODE. NOT AVAILABLE Whether removing the volume produces a new IQN at the next start, and whether that breaks backend ACLs keyed on the old one. |
Kolla images §6 (item 19) |
Heat stacks in progress when heat_engine restarted |
FACT: Docker sends SIGTERM, then SIGKILL if heat-engine is still running after 10 s. NOT AVAILABLE (upstream heat): whether stacks that a killed engine owned stay *_IN_PROGRESS until some engine starts again. |
fact sheet C §5.8 |
| Logs after a restart of the common containers | fluentd: log shipping gap. cron: log rotation is delayed. VM and API impact: NOT AVAILABLE. | Apply common changes §3 ("Impact") |
| The clock after a chrony restart or a VIP failover | NOT AVAILABLE: whether a chrony restart steps the clock (makestep 3 3 is set). The VIP is the internal time source: when the VIP fails over to another haproxy node, clients keep the same server IP but get a different clock source. |
fact sheet A §4.8 |
After kolla-ansible stop |
kolla-ansible stop stops all Kolla containers (incl. nova_libvirt, OVS, MariaDB) on targeted hosts; NOT AVAILABLE whether VM network traffic stops. MariaDB then needs mariadb_recovery. |
R15; Stop and destroy |
mariadb_recovery stopped healthy MariaDB nodes |
mariadb_recovery stops every MariaDB node, even healthy ones, and bootstraps from the highest seqno: full DB outage. |
R18; Database operations; fact sheet B §1.8 (item 5) |
| Anything else about VMs, traffic, storage or the control plane after a restart | What the sources do not establish is listed by topic, with the sections that mark each point NOT AVAILABLE. | What is not known |
Other bugs and oddities, by service¶
| What you see | What the code does | Read |
|---|---|---|
| Keystone Fernet keys: push, rotation times, expiry limits | Fernet cron generator hour calculation; precheck vs generator expiry limits; push script stops at the first unreachable peer CODE. | Known bugs and limitations (row keystone); fact sheet B §3.9 (items 3 to 6); Q11 |
Cinder: host, etcd coordination, backup driver |
No host/backend_host/cluster; etcd coordination uses only the first etcd host; Ceph backup driver default without a Ceph config CODE. |
Known bugs and limitations (row cinder); fact sheet C §3.8 |
| Glance with the file backend | File backend = single host CODE. FACT: with the file backend, glance runs on the first host of groups['glance-api'], so reordering the inventory moves glance-api to another host, and the HAProxy members follow it. |
Known bugs and limitations (row glance); fact sheet C §1.8 (item 2); R22 |
| Glance image cache | FACT: the image cache is never pruned; there is no cron/pruner container. | fact sheet C §1.8 (item 5) |
| Masakari instancemonitor and libvirt SASL | The instancemonitor uses qemu+tcp without SASL while nova-libvirt enables SASL CODE; effect on the instancemonitor NOT AVAILABLE. |
Known bugs and limitations (row masakari); fact sheet D §7 (item 7) |
Barbican secrets and barbican_crypto_key |
FACT: the crypto KEK lives in passwords.yml. NOT AVAILABLE: whether losing or changing barbican_crypto_key makes stored secrets undecryptable. |
fact sheet D §1.6 |
| MariaDB backup and the VIP | Backup runs only on groups['mariadb'][0] (reading that node's datadir), but my.cnf points the client at the VIP. NOT AVAILABLE: whether, if HAProxy has failed over away from node 0, the backup tool connects to a different server than the one whose files it copies. |
fact sheet B §1.8 (item 9) |
HAProxy on the backup nodes with set_sysctl: no |
ip_nonlocal_bind=1 is set only when set_sysctl is true, and backup nodes bind VIP addresses (FACT). NOT AVAILABLE: whether haproxy starts on non-VIP nodes with set_sysctl: no and no manual sysctl. |
fact sheet A §1.8 (H) |
| Two Kolla clouds on the same L2 segment: keepalived | FACT: keepalived_virtual_router_id defaults to 51 with multicast; two Kolla clouds on the same L2 segment collide unless it is changed. |
fact sheet A §1.8 (I) |
| OVN DB after the first host was replaced, or its volume lost | The DB on groups[...][0] gets no cluster-remote-addr. NOT AVAILABLE: what ovn-ctl does if that first host is replaced or its ovn_nb_db/ovn_sb_db volume is lost. There is no code for member removal or replacement, and none for backups. |
fact sheet A §5.8 (B) |
| OVN with DPDK | OVN + DPDK cannot work in this release (inference from code): with enable_ovs_dpdk the openvswitch play is skipped, and the OVN bootstrap execs into openvswitch_vswitchd. |
fact sheet A §5.8 (G) |
For any other role, the fact sheets list the notable behaviour in these sections:
- Fact sheet A: §1.8 (haproxy and keepalived), §2.4 (haproxy-config), §3.8 (memcached), §4.8 (chrony), §5.8 (ovn), §6.8 (ovs-dpdk), §7.3 (service-cert-copy), §8.3 (certificates), §9 (restart triggers and blast radius)
- Fact sheet B: §1.8 (MariaDB), §2.8 (RabbitMQ), §3.9 (Keystone)
- Fact sheet C: §1.8 (Glance),
§2.8
(Placement), §3.8 (Cinder),
§4.8 (Horizon),
§5.8 (Heat),
§6
(what
reconfiguredoes per role) - Fact sheet D: §1.6 (Barbican), §3.8 (Swift), §4.6 (Manila), §5.0 (the roles that place containers on compute hosts), §7 (consolidated bugs and oddities)
- Fact sheet E: §0 (most important findings), §15 (operational recommendations)
Investigate a component¶
Each of the 21 component pages says where the study covers the component and ends with its service card. Under "Where to look", the sections of the study are listed under these sub-headings when the study has something to list: Operate (the scenario and the command lines), Impact and risk (what the tag executes, and the risks), Dependencies, Evidence (the fact sheet, the image side), Before and after a run (the checks, prerequisites and precautions the study gives) and Known limits and open points (known bugs, open questions, findings, and what the sources do not establish) (How to read a component page).
In the table, "Risks" gives the risk numbers that the page lists in its "Risks" bullets, and "Per-role evidence" the fact sheet that the page names, or the page that it names in its place. A dash means that the page has no "Risks" bullet.
| Component | Role and tag | Risks | Per-role evidence |
|---|---|---|---|
| Keystone | role keystone, tag keystone |
R19, R8, R24 | fact sheet B §3 |
| Glance | role glance, tag glance |
R20, R21, R8, R24 | fact sheet C §1 |
| Placement | role placement, tag placement |
R8, R24 | fact sheet C §2 |
| Nova | roles nova and nova-cell; tags nova, nova-api(-*), nova-cell(-*) |
nova: R1, R2, R3, R24; nova-cell: R1 |
Nova deep dive |
| Neutron | role neutron, tag neutron |
R5, R6, R7; membership changes: R3, R4 | Reconfigure Neutron |
| Cinder | role cinder, tag cinder |
R8, R24 | fact sheet C §3 |
| Horizon | role horizon, tag horizon |
R8, R24 | fact sheet C §4 |
| Heat | role heat, tag heat |
R8, R24 | fact sheet C §5 |
| Barbican | role barbican, tag barbican |
– | fact sheet D §1 |
| Octavia | role octavia, tag octavia |
– | fact sheet D §2 |
| Swift | role swift, tag swift |
– | fact sheet D §3 |
| Manila | role manila, tag manila |
– | fact sheet D §4 |
| MariaDB/Galera | role mariadb, tag mariadb |
R9; mariadb_recovery: R18 |
fact sheet B §1 |
| RabbitMQ | role rabbitmq, tag rabbitmq |
R3, R10 | fact sheet B §2 |
| Memcached | role memcached, tags memcache, memcached |
R4 | fact sheet A §3 |
| HAProxy + Keepalived | role haproxy, tag haproxy |
R8 | fact sheet A §1, §2 |
| Common: fluentd, cron, kolla_toolbox | role common; -t common alone is a no-op |
R25, R2 | Repository architecture §5.1 |
| Open vSwitch | role openvswitch, tag openvswitch |
R7 | fact sheet A §6 for ovs-dpdk |
| OVN | role ovn, tag ovn |
– | fact sheet A §5 |
| Chrony | role chrony, tag chrony |
through common: R2, R25 |
fact sheet A §4 |
| iscsid / tgtd | role iscsi, tag iscsi |
– | fact sheet D §5.2 |
Two more pages cut across the components: Component dependencies, which maps what
depends on what and keeps four kinds of dependency apart (installation, configuration, runtime, operational), and Nova
deep dive. The roles without a component page are in fact sheet D
§5 (the roles
that place containers on compute hosts) and fact sheet E
(bootstrap-servers and the operational playbooks).
Cases the study does not solve¶
The study does not diagnose a runtime failure of OpenStack, such as a failed VM deployment: no container or VM behaviour is used as evidence (Limits of this study). For these classic cases it offers the dependencies and the restarts read from the code, and marks the rest as open.
| Case | What the study offers | What it does not establish |
|---|---|---|
| An instance fails to boot or to be scheduled (a VM deployment fails) | Nova deep dive §3 lists what must be available before Nova can operate, ordered by necessity: MariaDB, RabbitMQ, Keystone and memcached, Placement, cell mapping and host mapping, Neutron server and an L2 agent on the compute, Glance and Cinder, libvirt. Nova deep dive §2.2 gives what each Nova component requires at runtime. Component dependencies §1 and Component dependencies §3 give the shared substrate and the control-plane services. | Upstream Nova behaviour is not in these sources and is marked NOT AVAILABLE. Diagnosing the runtime failure itself is outside what the sources establish. |
| A new compute gets no instances | Nova deep dive §3 (item 5): the scheduler does not discover hosts by itself (discover_hosts_in_cells_interval = -1). Add a compute node §2 says what deploy --limit runs for a new compute. Reconfigure Nova §6 lists nova-manage cell_v2 list_hosts among the checks after a Nova run. |
NOT AVAILABLE Whether a compute that was not discovered, and so is not in a host mapping, gets no instances. |
| An instance has no network | Component dependencies §4 gives the networking containers, their placement rule, what they need and their data-plane role. Reconfigure Neutron §4 gives, per component, what a restart does in the code. | For neutron-server and the OVS, L3, DHCP and metadata agents, the effect of a restart on existing connectivity is marked NOT AVAILABLE (What is not known, "Network data plane"). |
| A live migration, a resize or a console fails | Nova deep dive §6 gives the mechanism and the dependency of live migration, cold migration and resize, the VNC console and scheduling. | NOT AVAILABLE Whether a nova_libvirt or nova_ssh restart interrupts operations in progress. NOT AVAILABLE Whether, with libvirt_tls: true, live migration to a new compute fails while the other computes' /etc/hosts lacks it (Add a compute node §2). |
| An API rejects requests: authentication | Component dependencies §1 says how every service reaches MariaDB, RabbitMQ, memcached, Keystone and HAProxy. fact sheet B §3.10 counts the roles whose templates point at Keystone and memcached. | NOT AVAILABLE: whether those API services fail to authenticate requests while Keystone or memcached is unavailable. |
| A volume cannot be attached | Component dependencies §2 (row "iscsid / tgtd"): nova-compute (volume attach) and cinder-volume (LVM) depend on it. Tags §3.1 (row iscsi) and fact sheet C §3.8 give what the code shows for iscsid, tgtd and cinder-volume. |
NOT AVAILABLE Whether existing iSCSI sessions are kernel-held and whether attach/detach fails during an iscsid restart. NOT AVAILABLE: whether existing attachments survive a cinder-volume restart. |
| The Galera cluster is down | Database operations gives the mariadb_recovery command line, with the comment "ONLY when the whole Galera cluster is down". R18 and fact sheet B §1.3 say what recovery does, and Kolla images §5.4 what the MariaDB image does not do. |
NOT AVAILABLE Whether the DB containers come back as a non-primary cluster after a full stop (What is not known, "Control plane"). |
Is it known?¶
- Known bugs and limitations — the implementation limitations and bugs of
ussuri-eol, by area: CLI,site.yml,kolla_docker, the kolla images and individual roles - Findings — fifteen headline findings (F1–F15), each with its evidence and why it matters
- Open questions — Q1–Q14: all are NOT AVAILABLE, except Q10 and Q14, which are answered from the source CODE, and the control-flow part of Q6, which is also answered from the source CODE
- What is not known — what the sources do not establish, grouped by topic, with the sections that mark each point NOT AVAILABLE
- Fact sheet corrections — eleven corrections to the fact sheets; the
one with the widest effect:
docker stopdoes not usedocker_graceful_timeout= 60 s, the effective timeout is 10 s. Their evidence is under Verification and errata - The lists that the fact sheets keep themselves: fact sheet C §8 and fact sheet D §8 (NOT AVAILABLE lists), fact sheet D §7 (consolidated bugs and oddities) and fact sheet E §14 (register of open questions: NOT AVAILABLE, or answered from code)
- Kolla images §7 — what is not available from the kolla image sources
- Risk register — 25 risks (R1–R25), each with what actually happens, its blast radius, VM and API impact, likelihood, severity, precautions and evidence
Checks the corpus names¶
The checks that the study names before and after a run, as links. This page adds none. No Docker HEALTHCHECK is wired in
Ussuri: docker ps shows no health status, and service health must be checked from outside CODE (Kolla images
§6, item 17).
| Page | Section | What it lists |
|---|---|---|
| Pre-execution checklist | step 5, "Health gates" | Before the run, and again after each batch: Galera (wsrep_cluster_size, wsrep_local_state_comment=Synced); RabbitMQ (cluster status, no partitions); openstack compute service list and openstack network agent list all up; HAProxy backends all UP. |
| Pre-execution checklist | step 6, "Workload gates" | No live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts. |
| Pre-execution checklist | step 1, "Know the diff before you run" | kolla-ansible genconfig with the same tags and --limit, a diff of /etc/ against the previous version, and kolla_set_configs --check on each container. |
| Predict and inspect a run | the table | The commands that show what a run would target, render and restart, among them "See what will restart" (the exit codes of kolla_set_configs --check) and "Pre-deployment checks" (prechecks, read-only). |
| Reconfigure a single service | §3, "Sequence, impact, risks and verification of -t <svc> reconfigure" |
Verification before (genconfig, the check in each <svc> container, a diff of /etc/) and after (openstack endpoint list --service <name>, the API through the VIP, the container start times, docker ps --filter name=haproxy). |
| Reconfigure Nova | §6, "Operational risks and verification" | Before (genconfig and a diff of nova.conf, the check in nova_compute on a few computes, no active live migrations) and after (openstack compute service list, docker ps --filter name=nova_ -a, virsh list --all inside nova_libvirt on a sample of computes, nova-manage cell_v2 list_hosts). |
| Reconfigure Neutron | §5, "Risks and verification" | Before (openstack network agent list, the network nodes that host HA router masters, a diff of openvswitch_agent.ini on a compute after genconfig -t neutron) and after (the agent list, the ip netns count on network nodes, a continuous ping of a VM's floating IP during each batch). |
Apply common changes (fluentd, cron, kolla_toolbox) |
§4, "Sequence, risks and verification" | Verification before (genconfig, then the check in each of fluentd, cron and kolla_toolbox) and after (docker ps start times, docker logs fluentd, logs still reaching the configured output). |
| Update container images | §5, "Verification" | Before (docker images on a sample of hosts after pull, image IDs against the registry) and after (docker ps --format shows the new tag everywhere, docker inspect of nova_compute equals the pulled image ID). |
| Risk register | the row "Precautions" of each risk | The precautions to take before running the operation. |
| Tags §3.1 | columns "Safe on its own?" and "Prerequisites" | Per tag: whether it is safe on its own, and its prerequisites. |
| Component pages | "Before and after a run" | The checks, prerequisites and precautions the study gives for the component, when it has some. |
Add a compute node and Change control-plane membership say that they give no verification steps.
Related¶
- Find it fast — typical operator questions and the section that answers each one
- Ten things to know before a production run — the mechanisms behind most rows of the symptom index
- Operations — the operation pages, with their commands, impact classes and risks
- Pre-execution checklist — the checks before every production run
- Tags — what a tag really executes, on which hosts, and which containers can restart
- Risk register — the 25 risks, with their likelihood, severity and precautions
- Method and limits of the study — how the sources were read, and what a source-only study cannot establish