Predict and inspect a run¶
Impact Read only Writes config
The commands that show what a run would target, render and restart before it runs, read in Kolla-Ansible 2024.1-eom.
Each row of the table gives one command, what it shows and its limits.
§2
adds what the first 2024.1 run on a cloud deployed by 2023.1 restarts, and how to see it beforehand.
Note
Key answers
- Predict a run:
genconfig, thenkolla_set_configs --checkin each container; the meaning of exit codes 0, 1, 2 and 137, with Docker and with Podman, is in the same table CODE.--list-tasksis not useful (table below). - What the check does not show: a container also restarts when its image, volumes or other parameters differ, or when its systemd unit text differs CODE. After an upgrade from 2023.1, look at the unit files too, and at the services the upgrade has not reached, whose image changes on the next run (§2).
- Changed in 2024.1:
genconfigcan pull the fluentd image; the container module iskolla_container, with a Podman path that reads the exit codes differently.set_configs.pyitself is unchanged CODE.
1. Commands and what they show¶
| Goal | Command | Notes |
|---|---|---|
| List tags as Ansible sees them | EXTRA_OPTS="--list-tags" kolla-ansible -i multinode deploy |
Works. Shows the play tags and the tags inherited by the tasks of each play. common is tagged only on the common play (site.yml:88-103): no role has a meta dependency on it. list-tags-site.txt is the ansible-core 2.16.20 parse of the 2024.1 tree, where common appears on play #5 only. Tags set inside dynamic includes, such as rabbitmq-ha-precheck (roles/), are not listed (Ansible listings) |
| List hosts a run would target | EXTRA_OPTS="--list-hosts" kolla-ansible -i multinode -t nova reconfigure |
Plays with &enable_<svc>_True patterns show 0 hosts: the groups are created at run time by group_by (site.yml:6-74; Tags §2.6) |
| List tasks | EXTRA_OPTS="--list-tasks" … |
Not useful: only include_tasks placeholders are shown (Tags §2.6) |
| Render configs without restarting | kolla-ansible -i multinode genconfig [-t <svc>] [--limit <hosts>] |
Sets kolla_action=config (tools/). Writes /etc/ on the targets. Every container restart handler is skipped (when: kolla_action != "config", e.g. roles/). Also re-renders haproxy services.d (and, with the single external frontend, external-frontend-map) for the tagged services: the service includes of the loadbalancer play run for config, and each service's loadbalancer.yml imports loadbalancer-config with tags: always (site.yml:324-326; roles/; roles/; roles/). Those tasks notify Restart haproxy container, but every loadbalancer handler is skipped under config (e.g. roles/), so HAProxy keeps its old frontends. A later -t <svc> reconfigure then finds the file unchanged and does not restart HAProxy, and -t <svc> deploy-containers runs no loadbalancer work: the change reaches HAProxy only through -t haproxy reconfigure or an untagged run, whose loadbalancer check compares services.d (roles/, roles/); that restart is staged, backups first and the VIP holder last (roles/). To diff the frontends, copy /etc/ before genconfig (Controlled batches §2). New in 2024.1: when the common play runs (no tag, or -t common), its config step runs kolla_container ensure_image on the fluentd image, then reads its labels (roles/; enable_fluentd: "yes", group_vars/). ensure_image pulls the image when that tag is not on the host (module_utils/, module_utils/), so genconfig can need the registry, and fails on a host where the pull fails (module_utils/). This is the only kolla_container task in a role's config.yml |
| See what will restart | on each host: docker exec -u root <container> /usr/ (Podman: podman exec -u root …) |
0 = every file listed in config.json's config_files matches its live copy (content, mode, owner). command, permissions, removed entries and optional entries whose source is absent are not compared, so memcached ("config_files": []) always returns 0. 1 = a listed file differs or is missing from the container, optional files included (a newly rendered optional file is drift) → restart on the next deploy/reconfigure/deploy-containers. 1 is also returned for a required source missing from /var/ or an invalid config.json; the restarted container then fails kolla_set_configs at start. preserve_properties entries (no perm, so the mode check never matches) and directory sources with subdirectories return 1 on every run (Kolla images §3.1). 2 = an unexpected exception, such as a config.json that cannot be opened. kolla's set_configs.py is byte-identical to 2023.1's, so these codes still hold CODE (kolla docker/, kolla docker/). What Kolla does with the code depends on the engine (changed in 2024.1): with Docker, 0 = no drift, 1 and 137 = drift, a client error from the exec (container stopped or restarting) = drift, any other code fails the check task (module_utils/, library/); with Podman, a container that is not running is drift before any exec, a client error from the exec = drift, 0 = no drift, 1 = drift, and any other code, 137 included, fails the task (module_utils/). The config check is reached only when the container exists and its parameters match: compare_container stops at the first difference (module_utils/), so a code that would fail the task does not when the image or parameters already differ. Details: Kolla images §3. Keep -u root: both workers run the check as root (module_utils/, module_utils/), and containers that run as a service user cannot read the root-owned 0660 files in /var/ (Kolla images §3.4) |
| See what else will restart | compare the image, volumes and other parameters of each container with the service definition; read /etc/ |
The check task (compare_container) also reports drift when the container is missing, when one of 16 container parameters differs (image, volumes, environment, dimensions, healthcheck, …), or when the unit file on disk differs from the rendered template, whole text compared (module_utils/; module_utils/). kolla_set_configs --check sees none of these. A changed docker_graceful_timeout, docker_client_timeout, docker_restart_policy or docker_restart_policy_retry changes the unit text of every long-running container (F16) |
| Check-mode run | EXTRA_OPTS="--check --diff" kolla-ansible … reconfigure |
CODE kolla_container, kolla_container_facts and kolla_toolbox do not declare check-mode support (library/, library/, library/), so Ansible skips them under --check (ansible-core 2.16.20 lib/) unless the task sets check_mode: false, and many tasks register outputs used later. About 80 tasks set check_mode: false and run for real even under --check: 42 of them are kolla_container_facts calls, mostly in the prechecks, and others run commands, such as the hacluster authkey generation (roles/). Two are kolla_container tasks that start one-shot containers: nova_list_cells (roles/) and nova_upgrade_checks (roles/). A third, rabbitmq_version_check (roles/), also sets it but is not reached under --check: its block runs only when the preceding kolla_container_facts task found a rabbitmq container, and that task is skipped (roles/). NOT AVAILABLE Whether such a run fails or gives partial results, and which of those tasks it reaches; staging only |
| Pre-deployment checks | kolla-ansible -i multinode prechecks [-t <svc>] |
Sets kolla_action=precheck (tools/). Read-only (fact sheet E), apart from the commands it runs inside existing containers, e.g. rabbitmqctl list_queues (roles/). -t <svc> skips the prechecks role (untagged play, site.yml:77-86) but runs <svc>/. Changed in 2024.1: the prechecks role fails when om_enable_rabbitmq_high_availability and om_enable_rabbitmq_quorum_queues are equal (roles/), and the rabbitmq precheck asserts that the existing queues are quorum queues when quorum is on, the new default (F17); the Docker version check runs only with Docker (roles/) |
2. The first 2024.1 run on a cloud deployed by 2023.1¶
The first deploy, reconfigure or deploy-containers that the 2024.1 code runs on a host can restart containers
whose operator config did not change. Three causes come from the code, and each can be seen before the run CODE.
A fourth applies to every service that kolla-ansible upgrade has not reached yet, below the table:
| Cause | Containers | How to see it before the run |
|---|---|---|
Unit text (F16). The 2024.1 template writes StartLimitInterval= where 2023.1 wrote StartLimitIntervalSec= and a blank line; it names the engine in Description, After/Wants, ExecStart and ExecStop, which with Docker read as in 2023.1, so the start-limit line is what differs (module_utils/); check_unit_change compares the whole file (:154-167). A container that the 2024.1 code has not yet created or restarted still has the unit that 2023.1 wrote |
every long-running container not recreated or restarted by 2024.1 code whose role runs a check task, for example one outside the tags of a partial upgrade; Swift deploy uses start_container, which does not compare the unit (roles/). A container recreated or restarted by the upgrade got the new unit then |
grep -l StartLimitIntervalSec /etc/: the 2024.1 template has no such key (module_utils/), so each file listed differs from it and its container reports drift. kolla_set_configs --check does not show it |
hostnqn file (new in 2024.1). nova-cell and cinder render nqn.2014-08.org.nvmexpress:uuid:<hostname uuid> into hostnqn and notify the restart (roles/, roles/); the config.json of both copies it, not optional, to /etc/ (roles/, roles/) |
nova_compute and cinder_volume, and by loop-notify the other containers of the same role on that host (Tags §2.4) |
after genconfig, kolla_set_configs --check in nova_compute and cinder_volume exits 1: the destination /etc/ is missing or differs from the rendered file in content, mode (0644) or owner (root) (kolla docker/); the images install nvme-cli, so whether the file already exists in the container is NOT AVAILABLE |
Quorum queues (F17). om_enable_rabbitmq_quorum_queues now defaults to true (group_vars/), rendered as rabbit_quorum_queue = true in the [oslo_messaging_rabbit] section of the service configs (e.g. roles/) |
every service whose config renders that section, if the 2023.1 default was kept | after genconfig, a diff of /etc/ shows the new line and the check exits 1. kolla-ansible prechecks fails first while classic queues exist (roles/) |
Image tag. openstack_release defaults to 2024.1 (group_vars/all.yml:679-681) and every image tag follows it
(e.g. roles/keystone/defaults/main.yml:96-100). The check task compares the image
(module_utils/kolla_container_worker.py:75), and the restart handler recreates a container whose image differs,
pulling it first when absent (module_utils/kolla_docker_worker.py:353-361). So a deploy, reconfigure or
deploy-containers moves every service that kolla-ansible upgrade has not reached (the whole cloud before the
upgrade, the services outside the tags after upgrade -t <svc>) to its 2024.1 image through the deploy path, without
the role's upgrade tasks (e.g. roles/keystone/tasks/upgrade.yml:23-31). kolla_set_configs --check does not show it;
compare the image tag of each running container (docker ps) with the 2024.1 tag. Run kolla-ansible upgrade for those
services first.
A container recreated or restarted by 2024.1 code gets the new unit (module_utils/kolla_systemd_worker.py:172-185,
called on create and restart, module_utils/kolla_docker_worker.py:334-339, :439-459), so the unit cause appears once.
NOT AVAILABLE What these restarts do to the running VMs and to existing classic queues: the sources do not
establish it. The checklist for that run is in Pre-execution
checklist §2.
Related¶
- Kolla images §3 — what
kolla_set_configs --checkcompares, what it never compares, and its exit codes with Docker and Podman - Tags §2.6 — why
--list-tasksshows only placeholders and why some plays list 0 hosts - Pre-execution checklist — step 1 uses
genconfigand the check before a production run - Write config first, apply in controlled batches — applying what
genconfigwrote, batch by batch - Execution flow §4 — the rows
genconfigandprecheckof the action matrix - Apply
commonchanges — the fluentdensure_imagestep thatgenconfignow runs - Technical findings F14, F16, F17 — the blind spots of the check, the unit-text drift and the quorum-queue default
- Fact sheet E
§5 —
what
precheckschecks - Open questions Q9 — check mode (
--check --diff) usability: NOT AVAILABLE
To remember
Lumo: if you keep only 5 things from this page, keep these.
- Predict a run with
genconfig, thenkolla_set_configs --checkin each container;--list-tasksis not useful. CODE genconfigwrites/etc/kolla/<svc>/, skips every container restart and re-renders haproxyservices.dfor the tagged services; new in 2024.1, an untagged or-t commongenconfigpulls the fluentd image when it is absent. CODE- With Docker, exit 1 (or 137) means drift; with Podman, only 1 does, and 137 fails the task. Any other code fails the check task instead of restarting. Keep
-u root. CODE - The check sees config files only: a changed image, volume or systemd unit text also restarts the container. On the first 2024.1 run, every unit still holding
StartLimitIntervalSec=is drift, and every service thatkolla-ansible upgradehas not reached gets its 2024.1 image. CODE - Whether a
--check --diffrun fails or gives partial results is unknown, and the tasks markedcheck_mode: false(about 80) still run, two of which start one-shot containers: staging only. NOT AVAILABLE
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.