Skip to content

Known bugs and limitations

The implementation limitations and bugs found in Kolla-Ansible 2024.1-eom, by area, with the evidence and the status of each row. The rows carried over from the 2023.1 study keep their area and their order; where an issue no longer holds in 2024.1, the row says so, with the code that shows it. Rows that are new in 2024.1 are marked so and come after the row of their area. Where a Status cell adds NOT AVAILABLE, it names what the sources do not establish about the issue.

Note

Key answers

  • Implementation limitations and bugs of 2024.1-eom, by area: CLI, site.yml, kolla_container (the module that replaced kolla_docker), the kolla images and individual roles (table below).
  • Still present in 2024.1: the kolla_serial override, the --ask-vault-pass and --include-dev oddities, the cyborg serial play, the placement loadbalancer include, the --check blind spots (F14), the HAProxy restart instead of reload and its stale files, the RabbitMQ cookie, the MariaDB, Keystone, Glance, Cinder, Heat and Swift items, the masakari qemu+tcp URI, the Docker restart of bootstrap-servers and the certificates that are never renewed.
  • No longer holding in 2024.1: the outward RabbitMQ pull bug (outward RabbitMQ is removed) and the ovs-dpdk handler name mismatch (the notify now matches). Still not holding, as in 2023.1: the ignored stop and client timeouts (F3), the -t common no-op (F2), the neutron-dynamic-routing upgrade gap, the haproxy upgrade task, the octavia items, the masakari SASL mismatch and docker_custom_option.
  • New in 2024.1: the Podman worker treats check exit codes differently from Docker; the first 2024.1 check of a container whose unit an earlier release wrote reports drift (F16); the cyborg endpoints use two variables that nothing defines; Octavia fails prechecks with the defaults unless Redis is enabled; stale single-external-frontend files; and, newly recorded, the genconfig/HAProxy interaction.
Area Issue Evidence Status
CLI -e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade; it is not passed for deploy (still) tools/kolla-ansible:296,490-493,522,531 CODE
CLI --ask-vault-pass overwrites VERBOSITY with a copy of EXTRA_OPTS (still) tools/kolla-ansible:391-392 CODE
CLI --include-dev: declared with an argument but used as a flag; the command line is executed unquoted (still) tools/kolla-ansible:193,275,355-358,471,604,146-148 CODE fact sheet E
site.yml cyborg play uses serial instead of kolla_serial (still) site.yml:869-883 (serial at :876) CODE
site.yml placement loadbalancer include lacks when: enable_placement (still) site.yml:248-251 CODE
kolla_container Changed in 2024.1: kolla_docker is replaced by kolla_container, which drives Docker or Podman. Still not holding (F3): graceful_timeout and client_timeout have no argument default, so the docker_common_options values are used (docker_graceful_timeout, 60 s, goes into each container's systemd unit as <engine> stop -t). The FIXME "restart_policy is not checked in the container" remains, but the policy is written into the unit, whose changes compare_container detects (Repository architecture §4.3). New in 2024.1 (F16): the unit template changed (it names the engine, StartLimitInterval=), and check_unit_change compares the whole unit text, so the first 2024.1 check of a container whose unit an earlier release wrote reports drift and restarts it library/kolla_container.py:19,313,335,361-373; group_vars/all.yml:139,142-153; module_utils/kolla_systemd_worker.py:20-39,154-167; module_utils/kolla_docker_worker.py:415-431,439-459 CODE
kolla_container (Podman) New in 2024.1: with kolla_container_engine: podman, the check handles exit codes differently from Docker. A container that is not running is drift before any exec; rc 0 is no drift and rc 1 is drift; any other rc, 137 included, raises "Failed to compare container configuration" and fails the task, where Docker reports 137 as drift (Kolla images §3.3) module_utils/kolla_podman_worker.py:294-301,428-453; module_utils/kolla_docker_worker.py:164-181 CODE; Podman's own behaviour on a host NOT AVAILABLE (Method and limits)
kolla images kolla_set_configs --check compares only config_files entries (F14): changes confined to config.json (command, permissions; all of memcached's settings) are never detected, so neither genconfig + deploy-containers nor a later run applies them until the container restarts for another reason; preserve_properties entries (prometheus extras) and directory sources with subdirectories report drift on every run; a newly added optional file is drift (exit 1); a restart deletes, or resets to the original, the destinations of entries dropped from config.json (glob sources excepted), which --check does not report. Still holds in 2024.1: kolla's set_configs.py is unchanged from 2023.1 (Kolla images §3, §6 items 1–4) kolla docker/base/set_configs.py:64-72,164-198,200-218,220-242,452-474,516-532,538-541,554-575; module_utils/kolla_docker_worker.py:146-181 CODE
common Still not holding (F2): common is its own play, tagged common, so -t common runs the role; roles/common/tasks/main.yml is a single include_tasks: "{{ kolla_action }}.yml" site.yml:88-103; roles/common/tasks/main.yml:2; Tags §2.1 CODE
haproxy The loadbalancer role. Restart instead of reload (still); stale services.d files never removed (still; the only removal is mariadb.cfg when ProxySQL is enabled); keepalived priority from inventory order (still); upgrade.yml still imports deploy.yml, now after it stops and removes prometheus_haproxy_exporter. New stale files in 2024.1: turning haproxy_single_external_frontend off leaves services.d/external-frontend.cfg in place, and a disabled service's block stays in external-frontend-map roles/loadbalancer/handlers/main.yml:1-345; roles/haproxy-config/tasks/main.yml:2-23,25-53; roles/loadbalancer/tasks/config.yml:60-72,162-175; roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15; roles/loadbalancer/tasks/upgrade.yml:4-21; HAProxy and keepalived; fact sheet A §1.8 CODE
haproxy (genconfig) Newly recorded in the 2024.1 study: genconfig re-renders services.d for the tagged services (the loadbalancer block runs for config), while every loadbalancer handler is skipped under config. A later -t <svc> run finds the file unchanged and does not restart HAProxy, and deploy-containers runs no loadbalancer work: the frontend change reaches HAProxy only through -t haproxy reconfigure or an untagged run, whose check compares services.d in the haproxy container (Predict and inspect a run, row "Render configs without restarting") site.yml:324-326; roles/haproxy-config/tasks/main.yml:2-23; roles/loadbalancer/handlers/main.yml:9-34; roles/loadbalancer/tasks/check-containers.yml:2-18; roles/loadbalancer/templates/haproxy/haproxy.json.j2:16-20 CODE
openvswitch Post-config sets external_ids:system-id, external_ids:hostname (new in 2024.1) and other_config:hw-offload through a kolla_toolbox call of the openvswitch_db module, whose result kolla_toolbox passes through, and notifies Restart openvswitch-vswitchd container roles/openvswitch/tasks/post-config.yml:3-23; library/kolla_toolbox.py:189-197,273; kolla docker/kolla-toolbox/requirements.yml:13 CODE; whether openvswitch_db (collection openvswitch.openvswitch, not in these sources) reports changed on an unchanged host, and so restarts openvswitch_vswitchd on every run, NOT AVAILABLE
neutron Still holds in 2024.1: upgrade with the default neutron_enable_rolling_upgrade: "yes" expands/contracts neutron plus every enabled subproject of neutron_subprojects, the same list the KOLLA_BOOTSTRAP container migrates on deploy; changed in 2024.1, the list also holds neutron-fwaas and tap-as-a-service. Still: the image's neutron-db-manage calls pass no --config-file; has_offline_migrations runs on the first neutron-server host with ignore_errors, and any non-zero rc (a failed exec included) stops neutron_server on every neutron-server host in the play until the handler flush after the contract container; the check loops over the enabled subprojects only, never neutron itself, so with no subproject enabled no server is stopped before upgrade --contract roles/neutron/defaults/main.yml:674-686,727,732-733; roles/neutron/tasks/upgrade.yml:4-5; roles/neutron/tasks/rolling_upgrade.yml:13-33,38-63,65-89; roles/neutron/tasks/bootstrap_service.yml:11-13; kolla docker/neutron/neutron-server/extend_start.sh:5-13,17-34 CODE; impact on the Neutron API, and whether a neutron contract migration is safe against running servers, NOT AVAILABLE
ovs-dpdk pull reads ovsdpdk_services through kolla_role_name: "ovsdpdk" (still fixed); the NIC mapping file is still written once; the 300 s wait handler still has no port. Changed in 2024.1: the ovsdpdk-db restart handler notifies Wait for dpdk tunnel ip, spelled like the handler, so the 2023.1 missing-handler mismatch is gone and the wait runs; ansible-core 2.16.20 wait_for with neither port nor path sleeps for the whole timeout, here 300 s roles/ovs-dpdk/tasks/pull.yml:2-3, roles/ovs-dpdk/vars/main.yml:4, roles/service-images-pull/tasks/main.yml:14; roles/ovs-dpdk/files/ovs-dpdkctl.sh:350-353; roles/ovs-dpdk/handlers/main.yml:2-22,88-92; ansible-core 2.16.20 lib/ansible/modules/wait_for.py:541-542; fact sheet A §6 CODE
rabbitmq No longer applies in 2024.1: the outward RabbitMQ pull bug (outward_rabbitmq_services looked up but never defined). Outward RabbitMQ is removed: rabbitmq.yml has three plays, all on the rabbitmq group, and no file of ansible/ names it; pull looks up rabbitmq_services through project_name rabbitmq.yml:3-70; roles/rabbitmq/vars/main.yml:2; roles/rabbitmq/tasks/pull.yml:2-3; roles/service-images-pull/tasks/main.yml:14; roles/rabbitmq/defaults/main.yml:2; RabbitMQ CODE
rabbitmq the image writes .erlang.cookie only in the bootstrap container, which runs only when the volume was just created; a cookie change recreates the brokers (env compare) but they keep the old cookie, while kolla_toolbox gets the new one (still) kolla docker/rabbitmq/rabbitmq/extend_start.sh:7-18; roles/rabbitmq/tasks/bootstrap.yml:2-26; roles/rabbitmq/defaults/main.yml:8-16; roles/rabbitmq/tasks/check-containers.yml:10; roles/common/tasks/config.yml:272-282, roles/common/templates/kolla-toolbox.json.j2:5-6 CODE; consequences, and whether rabbitmq-server uses RABBITMQ_CLUSTER_COOKIE or only the cookie file, NOT AVAILABLE
mariadb the restart batches have no any_errors_fatal (still); recovery stops healthy nodes (still); clustercheck started by recovery even when disabled (still: deploy and reconfigure notify the clustercheck handler only for enabled items). Changed in 2024.1: every play of mariadb.yml sets max_fail_percentage (mariadb_max_fail_percentage, then kolla_max_fail_percentage, then 100); with the default 100 it never stops a play early, because ansible-core counts the failures of the current batch only fact sheet B §1.5, §1.8; mariadb.yml:18-52; roles/mariadb/handlers/main.yml:56-73; roles/mariadb/tasks/restart_services.yml:18-45; roles/mariadb/tasks/config.yml:42-53, roles/mariadb/tasks/check-containers.yml:13-18; roles/mariadb/tasks/recover_cluster.yml:18-25,172-186,203-217; ansible-core 2.16.20 lib/ansible/executor/playbook_executor.py:204-209, lib/ansible/utils/helpers.py:25-32, lib/ansible/plugins/strategy/linear.py:336-348, lib/ansible/executor/task_queue_manager.py:330-338 CODE: a node that fails its port or WSREP wait marks only its own host failed; when that is every host of the batch (always so with three nodes, one host per batch) the playbook ends and later batches are not restarted; when other hosts of the batch succeed, the next batch still restarts its nodes (R9); whether the failed node is then still joining, and how large the quorum and availability risk is, NOT AVAILABLE
keystone fernet cron generator hour calculation; precheck vs generator expiry limits; push script stops at the first unreachable peer (all three unchanged in 2024.1) fact sheet B §3.9; roles/keystone/files/fernet_rotate_cron_generator.py:59-62,69-70; roles/keystone/tasks/precheck.yml:43-53, roles/keystone/tasks/config.yml:195-200; roles/keystone/templates/fernet-push.sh.j2:3-4,17-22 CODE
glance global log_bin_trust_function_creators toggle on every deploy/reconfigure; file backend = single host; backend switches silently when Swift, Ceph, S3 (new in 2024.1) or VMware is enabled (all still) roles/glance/tasks/bootstrap_service.yml:2-17,39-54, roles/glance/tasks/bootstrap.yml:38, roles/glance/tasks/deploy.yml:12, roles/glance/tasks/reconfigure.yml:2; group_vars/all.yml:1001,1006,1011; roles/glance/defaults/main.yml:247; fact sheet C §1 CODE
cinder no host/backend_host/cluster (still); etcd coordination uses only the first etcd host (still does not hold as written: the etcd backend_url points at kolla_internal_fqdn, the internal VIP, where the etcd role has a HAProxy frontend); Ceph backup driver default without a Ceph config (still) roles/cinder/templates/cinder.conf.j2:27-36,271-281; roles/etcd/defaults/main.yml:11-17; group_vars/all.yml:1047,1050; roles/cinder/tasks/config.yml:15-19; Cinder; fact sheet C §3 CODE
heat bootstrap_heat creates heat_user_domain/heat_domain_admin only when the domain is absent: a changed heat_domain_admin_password reaches heat.conf but never Keystone (still) kolla docker/heat/heat-api/extend_start.sh:5-17; roles/heat/tasks/bootstrap_service.yml:22; roles/heat/templates/heat.conf.j2:11-12 CODE; impact NOT AVAILABLE
swift no handler is ever notified (deploy does not apply config/ring changes; the only handler is a no-op debug); stop.yml wrong container name; precheck probes rsync port 873 vs 10873 (all still) roles/swift/handlers/main.yml:2-4 (no notify in roles/swift/tasks/); roles/swift/tasks/stop.yml:23-31; roles/swift/tasks/precheck.yml:86-95, group_vars/all.yml:628; fact sheet D §7 CODE; runtime effects NOT AVAILABLE
octavia Still not holding: upgrade.yml has no task that removes the octavia admin role, and with octavia_auto_configure (on by default: amphora is always in octavia_provider_drivers) deploy creates the amphora flavor, keypair, security groups and management network. New in 2024.1: upgrade.yml creates the persistence database and its user; the jobboard is on by default with the amphora provider (enable_octavia_jobboard) and its precheck asserts enable_redis, which defaults to no, so prechecks fails with the default settings roles/octavia/tasks/upgrade.yml:2-46; roles/octavia/tasks/deploy.yml:4-5, roles/octavia/tasks/prepare.yml:2-161; group_vars/all.yml:832,840,1323, roles/octavia/defaults/main.yml:386; roles/octavia/tasks/precheck.yml:67-73; Octavia; fact sheet D §7 (item 5) CODE; what Octavia does when Redis is unreachable NOT AVAILABLE
masakari The Ussuri SASL mismatch still does not hold: with libvirt_enable_sasl (default true in the masakari role's own defaults, as in nova-cell's) the role renders auth.conf with the libvirt SASL credentials into the instancemonitor. The connection_uri is still hard-coded qemu+tcp, while libvirt_tls: true makes libvirtd listen on TLS only (listen_tcp = 0) roles/masakari/defaults/main.yml:185,187; roles/masakari/tasks/config.yml:141-157, roles/masakari/templates/auth.conf.j2:1-6, roles/masakari/templates/masakari-instancemonitor.json.j2:9-15; roles/masakari/templates/masakari-monitors.conf.j2:18; roles/nova-cell/defaults/main.yml:552,569; roles/nova-cell/templates/libvirtd.conf.j2:1-13 CODE; effect on the instancemonitor with libvirt_tls NOT AVAILABLE
cyborg New in 2024.1: the endpoints are built as cyborg_internal_fqdn | kolla_url(...) and cyborg_external_fqdn | kolla_url(...), but no file of the 2024.1 kolla-ansible tree defines either variable (a search of the whole tree finds only these two uses). They feed cyborg_ks_services, which the register.yml step of deploy passes to service-ks-register. In 2023.1 the same endpoints were built from kolla_internal_fqdn/kolla_external_fqdn. Still: cyborg_services has no haproxy key, so no frontend serves port 6666 behind those names roles/cyborg/defaults/main.yml:2-27,147-148,169-175; roles/cyborg/tasks/register.yml:2-7; roles/cyborg/tasks/deploy.yml:2; group_vars/all.yml:369,762; kolla_ansible/kolla_url.py:20-30; fact sheet D §5.9 CODE for the missing definitions; whether deploy then fails at cyborg registration unless the operator defines both variables (as in the inventory or globals.yml) NOT AVAILABLE (not run)
baremetal ansible-collection-kolla roles. Changed in 2024.1: baremetal installs the engine named by kolla_container_engine (openstack.kolla.docker or, new, openstack.kolla.podman); the Docker items below apply to the Docker engine. Docker restarted on daemon.json change on all hosts at once (still: bootstrap-servers uses kolla_serial, default 0, and the handler restarts unless docker_systemd_reload); docker_custom_option still does not exist: docker_custom_config is merged into daemon.json, so its changes are applied (with that restart); Docker masked during re-runs on Debian/Ubuntu (still, when /etc/systemd/system/docker.service is absent; unmasked at the end of the docker config tasks); /etc/hosts lists every baremetal host (O(N²) templating, stale with --limit, still) kolla-host.yml:4-14; ansible-collection-kolla roles/baremetal/defaults/main.yml:37, roles/baremetal/tasks/install.yml:38, roles/docker/tasks/config.yml:9-53,119-125, roles/docker/handlers/main.yml:10-15, roles/docker/defaults/main.yml:29,65, roles/docker/tasks/install.yml:15-30, roles/etc_hosts/tasks/etc-hosts.yml:25-41; Risk register R14; fact sheet E §1–5 CODE; runtime effects, and whether the path is absent at a re-run (it depends on how systemd unmasks), NOT AVAILABLE; the Podman role was not studied
certificates test CA only; certs never renewed (every key, CSR and certificate task has creates:, 14 of 14); new backend hosts not added (all still). New in 2024.1: the external and internal certificates are not generated when enable_letsencrypt is set roles/certificates/tasks/generate-root.yml:20,33, roles/certificates/tasks/generate.yml:28,39,55,70,81,100,111,127,142, roles/certificates/tasks/generate-backend.yml:21,32,48, roles/certificates/tasks/generate-libvirt.yml:21,32,48; roles/certificates/templates/openssl-kolla-backend.cnf.j2:16; fact sheet A §8, E CODE
  • Troubleshooting — the symptom index, which points to these rows by what an operator sees
  • Findings — the headline findings; the rows kolla_container, kolla images and common cite F3, F16, F14 and F2
  • Per-role fact sheets — the fact sheets that the role rows cite as evidence, and their corrections
  • Fact sheet D §7 — the consolidated bugs and oddities of the roles of fact sheet D, cyborg included
  • Kolla images §3 — what kolla_set_configs --check compares, what it never compares, and how Docker and Podman read its exit codes
  • What is not known — the effects that the Status column marks NOT AVAILABLE, by topic
  • Open questions — the open questions Q1 to Q14

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • -e kolla_serial=N is still overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade, and still not passed for deploy; the cyborg play still reads serial instead of kolla_serial. CODE
  • kolla_set_configs --check still compares only config_files entries; new in 2024.1, with Podman any check exit code other than 0 or 1 fails the task instead of meaning drift. CODE
  • A frontend change written by genconfig reaches HAProxy only through -t haproxy reconfigure or an untagged run: the loadbalancer handlers are skipped under config. CODE
  • New in 2024.1: the cyborg endpoints use cyborg_internal_fqdn/cyborg_external_fqdn, which no file of the tree defines; whether cyborg registration then fails is NOT AVAILABLE.
  • No longer holding in 2024.1: the outward RabbitMQ pull bug (outward RabbitMQ is removed) and the ovs-dpdk handler name mismatch (the notify now matches the handler). CODE

Check yourself

Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. You run reconfigure with -e kolla_serial=N. Is N used as the batch size in 2024.1?
Show the answer

Answer: No: the wrapper appends -e kolla_serial=${ANSIBLE_SERIAL} (default 0) after your options. For reconfigure and upgrade the wrapper appends -e kolla_serial=${ANSIBLE_SERIAL} after the user's options, so ANSIBLE_SERIAL (default 0) wins CODE.

2. With kolla_container_engine: podman, the kolla_set_configs --check exec in a running container returns 137. What does kolla_container do?
Show the answer

Answer: It raises "Failed to compare container configuration": the Podman worker treats only 0 and 1 as answers. The Podman compare_config returns no drift for 0, drift for 1 and raises for any other code, 137 included; a container that is not running is reported as drift before any exec CODE.

3. You run genconfig -t keystone, then -t keystone reconfigure. The keystone frontend in services.d had changed. Is HAProxy restarted?
Show the answer

Answer: No: genconfig already wrote the file with the handlers skipped, so the reconfigure finds it unchanged. Every loadbalancer handler has when: kolla_action != "config"; the later service run finds the rendered file unchanged and notifies nothing. Only -t haproxy reconfigure or an untagged run compares services.d in the haproxy container CODE.

4. The Erlang cookie is changed. What happens to the RabbitMQ brokers?
Show the answer

Answer: They are recreated (env compare) but keep the old cookie file, while kolla_toolbox gets the new one. The image writes .erlang.cookie only in the bootstrap container (new volume); a cookie change recreates the brokers but they keep the old cookie file CODE. The consequences are NOT AVAILABLE.