Skip to content

Predict and inspect a run

Impact Read only Writes config

The commands that show what a run would target, render and restart before it runs, read in Kolla-Ansible 2024.1-eom. Each row of the table gives one command, what it shows and its limits. §2 adds what the first 2024.1 run on a cloud deployed by 2023.1 restarts, and how to see it beforehand.

Note

Key answers

  • Predict a run: genconfig, then kolla_set_configs --check in each container; the meaning of exit codes 0, 1, 2 and 137, with Docker and with Podman, is in the same table CODE. --list-tasks is not useful (table below).
  • What the check does not show: a container also restarts when its image, volumes or other parameters differ, or when its systemd unit text differs CODE. After an upgrade from 2023.1, look at the unit files too, and at the services the upgrade has not reached, whose image changes on the next run (§2).
  • Changed in 2024.1: genconfig can pull the fluentd image; the container module is kolla_container, with a Podman path that reads the exit codes differently. set_configs.py itself is unchanged CODE.

1. Commands and what they show

Goal Command Notes
List tags as Ansible sees them EXTRA_OPTS="--list-tags" kolla-ansible -i multinode deploy Works. Shows the play tags and the tags inherited by the tasks of each play. common is tagged only on the common play (site.yml:88-103): no role has a meta dependency on it. list-tags-site.txt is the ansible-core 2.16.20 parse of the 2024.1 tree, where common appears on play #5 only. Tags set inside dynamic includes, such as rabbitmq-ha-precheck (roles/rabbitmq/tasks/precheck.yml:129), are not listed (Ansible listings)
List hosts a run would target EXTRA_OPTS="--list-hosts" kolla-ansible -i multinode -t nova reconfigure Plays with &enable_<svc>_True patterns show 0 hosts: the groups are created at run time by group_by (site.yml:6-74; Tags §2.6)
List tasks EXTRA_OPTS="--list-tasks" … Not useful: only include_tasks placeholders are shown (Tags §2.6)
Render configs without restarting kolla-ansible -i multinode genconfig [-t <svc>] [--limit <hosts>] Sets kolla_action=config (tools/kolla-ansible:557-560). Writes /etc/kolla/<svc>/ on the targets. Every container restart handler is skipped (when: kolla_action != "config", e.g. roles/nova-cell/handlers/main.yml:89-90). Also re-renders haproxy services.d (and, with the single external frontend, external-frontend-map) for the tagged services: the service includes of the loadbalancer play run for config, and each service's loadbalancer.yml imports loadbalancer-config with tags: always (site.yml:324-326; roles/nova/tasks/loadbalancer.yml:2-7; roles/loadbalancer-config/tasks/main.yml:2-4; roles/haproxy-config/tasks/main.yml:2-23, :25-53). Those tasks notify Restart haproxy container, but every loadbalancer handler is skipped under config (e.g. roles/loadbalancer/handlers/main.yml:16-17), so HAProxy keeps its old frontends. A later -t <svc> reconfigure then finds the file unchanged and does not restart HAProxy, and -t <svc> deploy-containers runs no loadbalancer work: the change reaches HAProxy only through -t haproxy reconfigure or an untagged run, whose loadbalancer check compares services.d (roles/loadbalancer/templates/haproxy/haproxy.json.j2:17, roles/loadbalancer/tasks/check-containers.yml:2-18); that restart is staged, backups first and the VIP holder last (roles/loadbalancer/handlers/main.yml:9-34). To diff the frontends, copy /etc/kolla/haproxy/services.d/ before genconfig (Controlled batches §2). New in 2024.1: when the common play runs (no tag, or -t common), its config step runs kolla_container ensure_image on the fluentd image, then reads its labels (roles/common/tasks/config.yml:42-91; enable_fluentd: "yes", group_vars/all.yml:765). ensure_image pulls the image when that tag is not on the host (module_utils/kolla_docker_worker.py:501-503, module_utils/kolla_podman_worker.py:675-677), so genconfig can need the registry, and fails on a host where the pull fails (module_utils/kolla_docker_worker.py:189-223). This is the only kolla_container task in a role's config.yml
See what will restart on each host: docker exec -u root <container> /usr/local/bin/kolla_set_configs --check; echo $? (Podman: podman exec -u root …) 0 = every file listed in config.json's config_files matches its live copy (content, mode, owner). command, permissions, removed entries and optional entries whose source is absent are not compared, so memcached ("config_files": []) always returns 0. 1 = a listed file differs or is missing from the container, optional files included (a newly rendered optional file is drift) → restart on the next deploy/reconfigure/deploy-containers. 1 is also returned for a required source missing from /var/lib/kolla/config_files or an invalid config.json; the restarted container then fails kolla_set_configs at start. preserve_properties entries (no perm, so the mode check never matches) and directory sources with subdirectories return 1 on every run (Kolla images §3.1). 2 = an unexpected exception, such as a config.json that cannot be opened. kolla's set_configs.py is byte-identical to 2023.1's, so these codes still hold CODE (kolla docker/base/set_configs.py:36-60, :64-72, :165-242, :245-298, :556-582, kolla docker/base/start.sh:3, :11). What Kolla does with the code depends on the engine (changed in 2024.1): with Docker, 0 = no drift, 1 and 137 = drift, a client error from the exec (container stopped or restarting) = drift, any other code fails the check task (module_utils/kolla_docker_worker.py:146-181, library/kolla_container.py:410-429); with Podman, a container that is not running is drift before any exec, a client error from the exec = drift, 0 = no drift, 1 = drift, and any other code, 137 included, fails the task (module_utils/kolla_podman_worker.py:428-453). The config check is reached only when the container exists and its parameters match: compare_container stops at the first difference (module_utils/kolla_container_worker.py:58-65), so a code that would fail the task does not when the image or parameters already differ. Details: Kolla images §3. Keep -u root: both workers run the check as root (module_utils/kolla_docker_worker.py:146-152, module_utils/kolla_podman_worker.py:435-436), and containers that run as a service user cannot read the root-owned 0660 files in /var/lib/kolla/config_files (Kolla images §3.4)
See what else will restart compare the image, volumes and other parameters of each container with the service definition; read /etc/systemd/system/kolla-<container>-container.service The check task (compare_container) also reports drift when the container is missing, when one of 16 container parameters differs (image, volumes, environment, dimensions, healthcheck, …), or when the unit file on disk differs from the rendered template, whole text compared (module_utils/kolla_container_worker.py:58-65, :67-89; module_utils/kolla_systemd_worker.py:20-39, :71, :84, :154-167). kolla_set_configs --check sees none of these. A changed docker_graceful_timeout, docker_client_timeout, docker_restart_policy or docker_restart_policy_retry changes the unit text of every long-running container (F16)
Check-mode run EXTRA_OPTS="--check --diff" kolla-ansible … reconfigure CODE kolla_container, kolla_container_facts and kolla_toolbox do not declare check-mode support (library/kolla_container.py:355-359, library/kolla_container_facts.py:120, library/kolla_toolbox.py:263), so Ansible skips them under --check (ansible-core 2.16.20 lib/ansible/module_utils/basic.py:506-507) unless the task sets check_mode: false, and many tasks register outputs used later. About 80 tasks set check_mode: false and run for real even under --check: 42 of them are kolla_container_facts calls, mostly in the prechecks, and others run commands, such as the hacluster authkey generation (roles/hacluster/tasks/bootstrap.yml:29-40). Two are kolla_container tasks that start one-shot containers: nova_list_cells (roles/nova-cell/tasks/get_cell_settings.yml:2-21) and nova_upgrade_checks (roles/nova/tasks/upgrade.yml:9-30). A third, rabbitmq_version_check (roles/rabbitmq/tasks/version-check.yml:18-36), also sets it but is not reached under --check: its block runs only when the preceding kolla_container_facts task found a rabbitmq container, and that task is skipped (roles/rabbitmq/tasks/version-check.yml:3-9, :75). NOT AVAILABLE Whether such a run fails or gives partial results, and which of those tasks it reaches; staging only
Pre-deployment checks kolla-ansible -i multinode prechecks [-t <svc>] Sets kolla_action=precheck (tools/kolla-ansible:444-447). Read-only (fact sheet E), apart from the commands it runs inside existing containers, e.g. rabbitmqctl list_queues (roles/rabbitmq/tasks/precheck.yml:131-137). -t <svc> skips the prechecks role (untagged play, site.yml:77-86) but runs <svc>/precheck.yml. Changed in 2024.1: the prechecks role fails when om_enable_rabbitmq_high_availability and om_enable_rabbitmq_quorum_queues are equal (roles/prechecks/tasks/service_checks.yml:67-72), and the rabbitmq precheck asserts that the existing queues are quorum queues when quorum is on, the new default (F17); the Docker version check runs only with Docker (roles/prechecks/tasks/service_checks.yml:9-19)

2. The first 2024.1 run on a cloud deployed by 2023.1

The first deploy, reconfigure or deploy-containers that the 2024.1 code runs on a host can restart containers whose operator config did not change. Three causes come from the code, and each can be seen before the run CODE. A fourth applies to every service that kolla-ansible upgrade has not reached yet, below the table:

Cause Containers How to see it before the run
Unit text (F16). The 2024.1 template writes StartLimitInterval= where 2023.1 wrote StartLimitIntervalSec= and a blank line; it names the engine in Description, After/Wants, ExecStart and ExecStop, which with Docker read as in 2023.1, so the start-limit line is what differs (module_utils/kolla_systemd_worker.py:20-39, :50-54); check_unit_change compares the whole file (:154-167). A container that the 2024.1 code has not yet created or restarted still has the unit that 2023.1 wrote every long-running container not recreated or restarted by 2024.1 code whose role runs a check task, for example one outside the tags of a partial upgrade; Swift deploy uses start_container, which does not compare the unit (roles/swift/tasks/deploy.yml:1-10). A container recreated or restarted by the upgrade got the new unit then grep -l StartLimitIntervalSec /etc/systemd/system/kolla-*-container.service: the 2024.1 template has no such key (module_utils/kolla_systemd_worker.py:27), so each file listed differs from it and its container reports drift. kolla_set_configs --check does not show it
hostnqn file (new in 2024.1). nova-cell and cinder render nqn.2014-08.org.nvmexpress:uuid:<hostname uuid> into hostnqn and notify the restart (roles/nova-cell/tasks/config.yml:203-216, roles/cinder/tasks/config.yml:95-108); the config.json of both copies it, not optional, to /etc/nvme/hostnqn (roles/nova-cell/templates/nova-compute.json.j2:59-63, roles/cinder/templates/cinder-volume.json.j2:23-27) nova_compute and cinder_volume, and by loop-notify the other containers of the same role on that host (Tags §2.4) after genconfig, kolla_set_configs --check in nova_compute and cinder_volume exits 1: the destination /etc/nvme/hostnqn is missing or differs from the rendered file in content, mode (0644) or owner (root) (kolla docker/base/set_configs.py:165-197); the images install nvme-cli, so whether the file already exists in the container is NOT AVAILABLE
Quorum queues (F17). om_enable_rabbitmq_quorum_queues now defaults to true (group_vars/all.yml:266-268), rendered as rabbit_quorum_queue = true in the [oslo_messaging_rabbit] section of the service configs (e.g. roles/nova-cell/templates/nova.conf.j2:207-216) every service whose config renders that section, if the 2023.1 default was kept after genconfig, a diff of /etc/kolla/<service>/ shows the new line and the check exits 1. kolla-ansible prechecks fails first while classic queues exist (roles/rabbitmq/tasks/precheck.yml:131-152, :185-189)

Image tag. openstack_release defaults to 2024.1 (group_vars/all.yml:679-681) and every image tag follows it (e.g. roles/keystone/defaults/main.yml:96-100). The check task compares the image (module_utils/kolla_container_worker.py:75), and the restart handler recreates a container whose image differs, pulling it first when absent (module_utils/kolla_docker_worker.py:353-361). So a deploy, reconfigure or deploy-containers moves every service that kolla-ansible upgrade has not reached (the whole cloud before the upgrade, the services outside the tags after upgrade -t <svc>) to its 2024.1 image through the deploy path, without the role's upgrade tasks (e.g. roles/keystone/tasks/upgrade.yml:23-31). kolla_set_configs --check does not show it; compare the image tag of each running container (docker ps) with the 2024.1 tag. Run kolla-ansible upgrade for those services first.

A container recreated or restarted by 2024.1 code gets the new unit (module_utils/kolla_systemd_worker.py:172-185, called on create and restart, module_utils/kolla_docker_worker.py:334-339, :439-459), so the unit cause appears once. NOT AVAILABLE What these restarts do to the running VMs and to existing classic queues: the sources do not establish it. The checklist for that run is in Pre-execution checklist §2.

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • Predict a run with genconfig, then kolla_set_configs --check in each container; --list-tasks is not useful. CODE
  • genconfig writes /etc/kolla/<svc>/, skips every container restart and re-renders haproxy services.d for the tagged services; new in 2024.1, an untagged or -t common genconfig pulls the fluentd image when it is absent. CODE
  • With Docker, exit 1 (or 137) means drift; with Podman, only 1 does, and 137 fails the task. Any other code fails the check task instead of restarting. Keep -u root. CODE
  • The check sees config files only: a changed image, volume or systemd unit text also restarts the container. On the first 2024.1 run, every unit still holding StartLimitIntervalSec= is drift, and every service that kolla-ansible upgrade has not reached gets its 2024.1 image. CODE
  • Whether a --check --diff run fails or gives partial results is unknown, and the tasks marked check_mode: false (about 80) still run, two of which start one-shot containers: staging only. NOT AVAILABLE

What would you do?

Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.

Scenario 1: The check returns 2

You changed a glance option and ran kolla-ansible -i multinode -t glance genconfig. On a controller, you run docker exec glance_api /usr/local/bin/kolla_set_configs --check; echo $? (without -u root) and it prints 2. Your colleague concludes that the check is broken and proposes to go ahead with the production reconfigure.

What would you do?

See every choice explained
  • Avoid: Read 2 as "no drift" and run the reconfigure.. 0 is the code for "every listed file matches". 2 = an unexpected exception, such as a config.json the user cannot open; when Kolla gets 2, the check task fails instead of restarting, if the comparison reaches the config check (it is skipped when the container is missing or its parameters already differ) CODE.
  • Best choice: Re-run the same check with docker exec -u root, as Kolla does, and read the exit code then.. Kolla runs the check as root, and containers that run as a service user cannot read the root-owned 0660 files in /var/lib/kolla/config_files CODE. Then 0 = no config drift, 1 = drift, restart on the next run.
  • Works, but risky: Skip the check and preview the run with EXTRA_OPTS="--check --diff" on production instead.. kolla_container does not declare check-mode support, so Ansible skips it under --check, while tasks marked check_mode: false still run, and two of them start one-shot containers (nova_list_cells, nova_upgrade_checks) CODE. Whether such a run fails or gives partial results is not known NOT AVAILABLE: staging only.

Read the section.

Scenario 2: Every check returns 0 after the upgrade

Last week you upgraded from 2023.1 to 2024.1 with -t nova,neutron,glance upgrade. Today you plan a reconfigure without tags for a small keystone change. After genconfig, docker exec -u root memcached /usr/local/bin/kolla_set_configs --check and the same check in the other containers outside those three services print 0. Your colleague says only keystone will restart.

What would you do?

See every choice explained
  • Avoid: Trust the checks: 0 everywhere means only keystone restarts.. kolla_set_configs --check compares only the files of config.json. compare_container also compares the image and the whole systemd unit text: the services outside the three upgraded ones still run 2023.1 images, and the 2024.1 template no longer writes StartLimitIntervalSec= CODE.
  • Best choice: First run kolla-ansible upgrade for the services the partial upgrade left out; then list the units still written by 2023.1 with grep -l StartLimitIntervalSec /etc/systemd/system/kolla-*-container.service: each one will restart on the run, with the rest of its role on that host.. An untagged reconfigure would move every service not yet upgraded to its 2024.1 image through the deploy path, without the role's upgrade tasks, since the image is compared too. A unit file that contains StartLimitIntervalSec cannot match the 2024.1 template, so the first 2024.1 check of that container reports drift and its handler restarts it; loop-notify adds the other containers of the role on that host CODE.
  • Works, but risky: Run the reconfigure with EXTRA_OPTS="--check" first to see the restarts.. kolla_container, which runs the comparison, is skipped under --check, so the restarts are not shown, and whether such a run is usable is not known NOT AVAILABLE.

Read the section.

Check yourself

Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. Is --list-tasks useful to predict a Kolla-Ansible run?
Show the answer

Answer: No: only include_tasks placeholders are shown. Row "List tasks" of the table: only include_tasks placeholders are shown, because the service roles are reached through dynamic includes. Read the section.

2. How can you see, on a host, whether a container's configuration will make it restart?
Show the answer

Answer: Run kolla_set_configs --check inside the container, as root, after genconfig. docker exec -u root <container> /usr/local/bin/kolla_set_configs --check (or podman exec -u root); the exit codes are in the table CODE. It covers config files only, not image, volume or unit changes. Read the section.

3. In 2024.1, can an untagged genconfig download anything?
Show the answer

Answer: Yes: the common config step pulls the fluentd image when it is not on the host. The common role's config.yml now runs kolla_container ensure_image on the fluentd image before reading its labels; ensure_image pulls when the image is absent CODE. Read the section.

4. The checks run with kolla_container_engine: podman. What does Kolla do when kolla_set_configs --check exits 137?
Show the answer

Answer: It fails the check task. The Podman worker treats only 1 as drift; any other non-zero code, 137 included, raises "Failed to compare container configuration" CODE. Read the section.