Skip to content

Known bugs and limitations

The implementation limitations and bugs found in Kolla-Ansible 2023.1-eol, by area, with the evidence and the status of each row. The rows carried over from the Ussuri study keep their area; where an issue no longer holds in 2023.1, the row says so, with the code that shows it. Where a Status cell adds NOT AVAILABLE, it names what the sources do not establish about the issue.

Note

Key answers

  • Implementation limitations and bugs of 2023.1-eol, by area: CLI, site.yml, kolla_docker, the kolla images and individual roles (table below).
  • No longer holding in 2023.1: the ignored stop and client timeouts (F3), the -t common no-op (F2), the always-changed OVS Set system-id, the neutron-dynamic-routing upgrade gap, the haproxy upgrade bug, the octavia items, the masakari SASL mismatch, docker_custom_option and the ovs-dpdk pull bug.
Area Issue Evidence Status
CLI -e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade; it is not passed for deploy tools/kolla-ansible:290,476-479,508,517 CODE
CLI --ask-vault-pass overwrites VERBOSITY with a copy of EXTRA_OPTS tools/kolla-ansible:385-388 CODE
CLI --include-dev: declared with an argument but used as a flag; the command line is executed unquoted tools/kolla-ansible:188,270,349-352,458,587,140-142 CODE fact sheet E
site.yml cyborg play uses serial instead of kolla_serial site.yml:784 CODE
site.yml placement loadbalancer include lacks when: enable_placement site.yml:243-246 CODE
kolla_docker No longer holds in 2023.1 (F3): graceful_timeout and client_timeout from docker_common_options are now used. When the task does not set them, generate_module takes them from common_options (strings of digits become integers); client_timeout is the Docker client timeout, and graceful_timeout (docker_graceful_timeout, 60 s) goes into the container's systemd unit as docker stop -t. The FIXME "restart_policy is not checked in the container" remains, but the policy is written into the unit, whose changes compare_container detects (Repository architecture §4.3) library/kolla_docker.py:351-395; group_vars/all.yml:140,143-153; module_utils/kolla_docker_worker.py:15,35-38,436-452; module_utils/kolla_systemd_worker.py:20-40,50-74,149-162 CODE
kolla images kolla_set_configs --check compares only config_files entries (F14): changes confined to config.json (command, permissions; all of memcached's settings) are never detected, so neither genconfig + deploy-containers nor a later run applies them until the container restarts for another reason; preserve_properties entries (prometheus extras) and directory sources with subdirectories report drift on every run. Changed in 2023.1: a newly added optional file is drift (exit 1, the container restarts) instead of an exit 2 that failed check-containers; and a restart now deletes, or resets to the original, the destinations of entries dropped from config.json (glob sources excepted), which --check still does not report (Kolla images §3, §6 items 1–4) kolla docker/base/set_configs.py:64-72,164-198,200-218,220-242,452-474,516-532,538-541,554-575; module_utils/kolla_docker_worker.py:176-190 CODE
common No longer holds in 2023.1 (F2): common is its own play, tagged common, so -t common runs the role; roles/common/tasks/main.yml is now a single include_tasks: "{{ kolla_action }}.yml" site.yml:86-97; roles/common/tasks/main.yml:2; Tags §2.1 CODE
haproxy Now the loadbalancer role. Restart instead of reload (still); stale services.d files never removed (still; the only removal is mariadb.cfg when ProxySQL is enabled); keepalived priority from inventory order (still). The Ussuri upgrade "Stopping all slave keepalived" task is gone: upgrade.yml imports deploy.yml roles/loadbalancer/handlers/main.yml:1-118; roles/haproxy-config/tasks/main.yml:2-23; roles/loadbalancer/tasks/config.yml:60-71; roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15; roles/loadbalancer/tasks/upgrade.yml:2; HAProxy and keepalived; fact sheet A §1.8 CODE
openvswitch The Ussuri bug no longer holds as written: the command that always reported changed is replaced by a kolla_toolbox call of the openvswitch_db module (external_ids:system-id, other_config:hw-offload), whose changed result kolla_toolbox passes through and which notifies Restart openvswitch-vswitchd container roles/openvswitch/tasks/post-config.yml:3-22; library/kolla_toolbox.py:210-217,244; kolla docker/kolla-toolbox/requirements.yml:13-14 CODE; whether openvswitch_db (collection openvswitch.openvswitch, not in these sources) reports changed on an unchanged host, and so restarts openvswitch_vswitchd on every run, NOT AVAILABLE
neutron Changed in 2023.1: upgrade with the default neutron_enable_rolling_upgrade: "yes" expands/contracts neutron plus every enabled subproject of neutron_subprojects (networking-sfc, neutron-dynamic-routing, neutron-vpnaas, vmware-nsx), the same list the KOLLA_BOOTSTRAP container migrates on deploy, so the Ussuri neutron-dynamic-routing gap is closed. Still: the image's neutron-db-manage calls pass no --config-file; has_offline_migrations runs on the first neutron-server host with ignore_errors, and any non-zero rc (a failed docker exec included) stops neutron_server on every neutron-server host in the play until the handler flush after the contract container. New: the check loops over the enabled subprojects only, never neutron itself, so with no subproject enabled no server is stopped before upgrade --contract roles/neutron/defaults/main.yml:656-664,705,710-711; roles/neutron/tasks/upgrade.yml:4-5; roles/neutron/tasks/rolling_upgrade.yml:13-33,38-63,65-89; roles/neutron/tasks/bootstrap_service.yml:10-13; kolla docker/neutron/neutron-server/extend_start.sh:5-13,17-34 CODE; impact on the Neutron API, and whether a neutron contract migration is safe against running servers, NOT AVAILABLE
ovs-dpdk pull bug fixed in 2023.1 (pull imports service-images-pull, which reads ovsdpdk_services through kolla_role_name: "ovsdpdk"); the NIC mapping file is still written once; the 300 s wait handler still has no port roles/ovs-dpdk/tasks/pull.yml:2-3, roles/ovs-dpdk/vars/main.yml:4, roles/service-images-pull/tasks/main.yml:14; roles/ovs-dpdk/files/ovs-dpdkctl.sh:350-353; roles/ovs-dpdk/handlers/main.yml:88-92; fact sheet A §6 CODE
rabbitmq outward RabbitMQ pull looks up outward_rabbitmq_services fact sheet B §2.8 (item 6); rabbitmq.yml:60-80 (project_name: outward_rabbitmq, :71), roles/rabbitmq/tasks/pull.yml:2-3, roles/service-images-pull/tasks/main.yml:14; ansible-core 2.14.18 lib/ansible/plugins/lookup/vars.py:86-104 CODE: with outward RabbitMQ enabled, that variable is not defined anywhere (the role defines only rabbitmq_services, roles/rabbitmq/defaults/main.yml:2), the vars lookup raises an undefined-variable error and "Pull images" fails on the outward-rabbitmq hosts
rabbitmq the image writes .erlang.cookie only in the bootstrap container (new volume); a cookie change recreates the brokers (env compare) but they keep the old cookie, while kolla_toolbox gets the new one kolla docker/rabbitmq/rabbitmq/extend_start.sh:7-18; roles/rabbitmq/tasks/bootstrap.yml:2-26; roles/rabbitmq/defaults/main.yml:8-16; roles/common/tasks/config.yml:211-222, roles/common/templates/kolla-toolbox.json.j2:5-6 CODE; consequences, and whether rabbitmq-server uses RABBITMQ_CLUSTER_COOKIE or only the cookie file, NOT AVAILABLE
mariadb the restart batches have no any_errors_fatal; recovery stops healthy nodes; clustercheck started even when disabled. Changed in 2023.1: the restarts run in separate plays with serial: "33%"; deploy and reconfigure notify the clustercheck handler only for enabled items, so only recovery still starts it when disabled fact sheet B §1.5, §1.8 (items 5–6); mariadb.yml:14-40; roles/mariadb/handlers/main.yml:56-73; roles/mariadb/tasks/restart_services.yml:18-51; roles/mariadb/tasks/config.yml:42-53, check-containers.yml:13-18; roles/mariadb/tasks/recover_cluster.yml:19-23,171-185,202-216; ansible-core 2.14.18 lib/ansible/executor/playbook_executor.py:204-208, lib/ansible/utils/helpers.py:25-32 CODE: a node that fails its port or WSREP wait marks only its own host failed; when that is every host of the batch (always so with three nodes, one host per batch) the playbook ends and later batches are not restarted; when other hosts of the batch succeed, the next batch still restarts its nodes (R9); whether the failed node is then still joining, and how large the quorum and availability risk is, NOT AVAILABLE
keystone fernet cron generator hour calculation; precheck vs generator expiry limits; push script stops at the first unreachable peer (all three unchanged in 2023.1) fact sheet B §3.9 (items 3–5); roles/keystone/files/fernet_rotate_cron_generator.py:59-62,69-70; roles/keystone/tasks/precheck.yml:43-53, roles/keystone/tasks/config.yml:195-200; roles/keystone/templates/fernet-push.sh.j2:3-4,17-22 CODE
glance global log_bin_trust_function_creators toggle on every deploy/reconfigure; file backend = single host; backend switches silently when Swift/Ceph is enabled roles/glance/tasks/bootstrap_service.yml:2-16,39-53, bootstrap.yml:38, deploy.yml:12, reconfigure.yml:2; group_vars/all.yml:906-911,915; fact sheet C §1 CODE
cinder no host/backend_host/cluster (still); etcd coordination uses only the first etcd host (no longer holds as written: the etcd backend_url points at kolla_internal_fqdn, the internal VIP, where the etcd role has a HAProxy frontend); Ceph backup driver default without a Ceph config (still) roles/cinder/templates/cinder.conf.j2:27-30,245-253; roles/etcd/defaults/main.yml:26-32; group_vars/all.yml:951,954; roles/cinder/tasks/config.yml:15-19; Cinder; fact sheet C §3 CODE
heat bootstrap_heat creates heat_user_domain/heat_domain_admin only when the domain is absent: a changed heat_domain_admin_password reaches heat.conf but never Keystone kolla docker/heat/heat-api/extend_start.sh:5-17; roles/heat/tasks/bootstrap_service.yml:22; roles/heat/templates/heat.conf.j2:11-12 CODE; impact NOT AVAILABLE
swift no handler is ever notified (deploy does not apply config/ring changes; the only handler is a no-op debug); stop.yml wrong container name; precheck probes rsync port 873 vs 10873 roles/swift/handlers/main.yml:2-4 (no notify in roles/swift/tasks/); roles/swift/tasks/stop.yml:23-31; roles/swift/tasks/precheck.yml:89, group_vars/all.yml:544; fact sheet D CODE; runtime effects NOT AVAILABLE
octavia No longer holds in 2023.1: upgrade.yml has no task that removes the octavia admin role (it is config, check-containers, bootstrap_service, flush), and with octavia_auto_configure (on by default: amphora is always in octavia_provider_drivers) deploy creates the amphora flavor, keypair, security groups and management network roles/octavia/tasks/upgrade.yml:2-9; roles/octavia/tasks/deploy.yml:4-5, prepare.yml:2-146; group_vars/all.yml:1228, roles/octavia/defaults/main.yml:361; Octavia; fact sheet D CODE
masakari The Ussuri SASL mismatch no longer holds: with libvirt_enable_sasl (default true in the masakari role's own defaults, as in nova-cell's) the role renders auth.conf with the libvirt SASL credentials into the instancemonitor. The connection_uri is still hard-coded qemu+tcp, while libvirt_tls: true makes libvirtd listen on TLS only (listen_tcp = 0) roles/masakari/defaults/main.yml:178,180; roles/masakari/tasks/config.yml:141-157, roles/masakari/templates/auth.conf.j2:1-6, masakari-instancemonitor.json.j2:9-12; roles/masakari/templates/masakari-monitors.conf.j2:18; roles/nova-cell/defaults/main.yml:545,562; roles/nova-cell/templates/libvirtd.conf.j2:1-13 CODE; effect on the instancemonitor with libvirt_tls NOT AVAILABLE
baremetal Now ansible-collection-kolla roles. Docker restarted on daemon.json change on all hosts at once (still: bootstrap-servers uses kolla_serial, default 0, and the handler restarts unless docker_systemd_reload); docker_custom_option no longer exists: docker_custom_config is merged into daemon.json, so its changes are applied (with that restart); Docker masked during re-runs on Debian/Ubuntu (still, when /etc/systemd/system/docker.service is absent; unmasked at the end of the docker config tasks); /etc/hosts lists every baremetal host (O(N²) templating, stale with --limit, still) kolla-host.yml:4-10; ansible-collection-kolla roles/docker/tasks/config.yml:30-47,113-119, roles/docker/handlers/main.yml:10-15, roles/docker/defaults/main.yml:30,63, roles/docker/tasks/install.yml:13-29, roles/etc_hosts/tasks/etc-hosts.yml:25-41; Risk register R14; fact sheet E §1–5 CODE; runtime effects, and whether the path is absent at a re-run (it depends on how systemd unmasks), NOT AVAILABLE
certificates test CA only; certs never renewed (every key, CSR and certificate task has creates:); new backend hosts not added roles/certificates/tasks/generate-root.yml:20,33, generate.yml:28,39,55, generate-backend.yml:21,32,48; fact sheet A §8, E CODE
  • Troubleshooting — the symptom index, which points to these rows by what an operator sees
  • Findings — the headline findings; the rows kolla_docker, kolla images and common cite F3, F14 and F2
  • Per-role fact sheets — the fact sheets that the role rows cite as evidence, and their corrections
  • Fact sheet D §7 — the consolidated bugs and oddities of the roles of fact sheet D
  • Kolla images §3 — what kolla_set_configs --check compares and what it never compares (row kolla images)
  • What is not known — the effects that the Status column marks NOT AVAILABLE, by topic
  • Open questions — the open questions Q1 to Q14

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • -e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade; it is not passed for deploy. CODE
  • kolla_set_configs --check compares only config_files entries: changes confined to config.json are not applied until the container restarts for another reason. CODE
  • A changed heat_domain_admin_password reaches heat.conf but never Keystone. CODE
  • Certificates are never renewed (every key, CSR and certificate task has creates:), the CA is a test CA only, and new backend hosts are not added. CODE
  • No longer holding in 2023.1: graceful_timeout and client_timeout from docker_common_options are now used, and -t common runs the role. CODE

Check yourself

Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. You run reconfigure with -e kolla_serial=N. Is N used as the batch size?
Show the answer

Answer: No: it is overridden by ANSIBLE_SERIAL (default 0). -e kolla_serial=N is overridden by ANSIBLE_SERIAL (default 0) for reconfigure/upgrade; it is not passed for deploy CODE.

2. Does -t common run the common role in 2023.1?
Show the answer

Answer: Yes: common is its own play, tagged common. The -t common no-op no longer holds in 2023.1: common is its own play, tagged common, so -t common runs the role CODE.

3. The Erlang cookie is changed. What happens to the RabbitMQ brokers?
Show the answer

Answer: They are recreated (env compare) but keep the old cookie, while kolla_toolbox gets the new one. The image writes .erlang.cookie only in the bootstrap container (new volume); a cookie change recreates the brokers but they keep the old cookie CODE. The consequences are NOT AVAILABLE.