Fact sheet E¶
Source: kolla-ansible at tag 2024.1-eom (commit 890396f) and ansible-collection-kolla at tag 2024.1-eom
(commit 82a0f5c), which provides the roles that bootstrap-servers runs (ansible/kolla-host.yml:12-14;
requirements.yml:2-5 pins the collection to the git branch stable/2024.1). Kolla-Ansible paths starting with
tools/, doc/, tests/, etc/, zuul.d/, kolla_ansible/ or ansible/ are relative to the repo root;
roles/...,
group_vars/... and *.yml playbooks are relative to ansible/. In §1–§3, etc_hosts/…, baremetal/…, packages/…,
docker/…, podman/…, kolla_user/…, docker_sdk/…, podman_sdk/… and apparmor_libvirt/… are
ansible-collection-kolla roles/<role>/…. Ansible behaviour claims cite ansible-core 2.16.20 (as ansible-core
2.16.20 lib/ansible/...). 2024.1 accepts ansible-core 2.15 to 2.16 (tools/kolla-ansible:10-11,78-82,
roles/prechecks/vars/main.yml:4-5); only 2.16.20 is read here, so ansible-core 2.15 behaviour is NOT AVAILABLE.
Labels: FACT = CODE, read directly from the sources (cited): kolla-ansible, ansible-collection-kolla, the kolla
image repository (cited as kolla docker/...) or ansible-core. NOT AVAILABLE = NOT AVAILABLE, these sources do
not establish it, often because it depends on Docker, Podman, systemd, Galera or libvirt behaviour, or on a module that
lives in an Ansible collection outside the sources (ufw, selinux, sysctl, authorized_key, git_config,
docker_prune are redirected out of ansible-core: ansible-core 2.16.20
lib/ansible/config/ansible_builtin_runtime.yml,
e.g. :745-746 for docker_prune).
Note
Key answers
- New in 2024.1, Podman:
bootstrap-serversinstalls the engine named bykolla_container_engine(defaultdocker): the collection importsopenstack.kolla.{{ container_engine }}andopenstack.kolla.{{ container_engine }}_sdk(baremetal/tasks/install.yml:37-38,baremetal/tasks/post-install.yml:6-7,baremetal/defaults/main.yml:37). The newpodmanrole has no handlers and no Debian mask step, so with Podman nothing in bootstrap restarts an engine daemon (§2.5). - Still holds in 2024.1, Docker restart gate: Docker restarts only when the
daemon.jsonwrite or the registry CA copy reports changed (docker/tasks/config.yml:9-53,104-114), at the flush at:116-117. New keys can now enterdaemon.json(docker_registry_mirrors,docker_debug,:39,46), and the Zuncluster-storekey is gone (group_vars/all.yml:125), so a Zun compute host restarts Docker on its first 2024.1 run (§2.2). - Changed in 2024.1, apt repositories: the Docker repository is now a deb822 file
/etc/apt/sources.list.d/docker.sources, the olddocker.listis deleted, and the apt cache is refreshed on every run (docker/tasks/repo-Debian.yml:45-70). - Still holds in 2024.1:
/etc/hostslists every baremetal host and one fact-less host fails the task on every host (§3);--limitfact gathering is serial per limited host (§4); a user-e kolla_serialis overridden forupgradeandreconfigure(§11). - Changed in 2024.1, destroy: the CLI now passes the
-iinventories totools/cleanup-host, which skips an entry of/etc/kollaonly when its path is exactly an inventory path as typed (tools/cleanup-host:79-103);tools/cleanup-containersalso disables the container units (:43-49) (§6). - Changed in 2024.1, plays: every play of the playbooks on this sheet now sets
max_fail_percentage(default 100), which never stops a play early (Controlled batches).
0. Most important findings¶
- When Docker restarts. In 2024.1 the restart is still a handler, "Restart docker"
(
docker/handlers/main.yml:10-15:state: restarted, orreloadedwhendocker_systemd_reloadis true, defaultfalse,docker/defaults/main.yml:65; plusmasked: no). Only two tasks notify it: the/etc/docker/daemon.jsonwrite (docker/tasks/config.yml:9-53) and the private-registry CA copy (:104-114, only whendocker_registry_cais set). The systemd drop-in tasks notify "Reload docker service file" (docker/handlers/main.yml:2-8), adaemon_reload-only systemd call that never reports changed (ansible-core 2.16.20lib/ansible/modules/systemd.py:387-400;changedis set only at:484,530,562), and a handler notifies further handlers only when it reports changed (lib/ansible/plugins/strategy/__init__.py:656). So Docker restarts if and only if the daemon.json write (or the CA copy) reports changed: the rendered content differs, or the file mode is not0644(ansible-core 2.16.20lib/ansible/plugins/action/copy.py:285,341-380,lib/ansible/module_utils/basic.py:867,900,949). The handlers are flushed atdocker/tasks/config.yml:116-117, on every notified host of the batch at once, and the batch is all hosts by default:kolla_serialdefaults to0(kolla-host.yml:6) and the CLI never passes it forbootstrap-servers(tools/kolla-ansible:485-489). Default content:iptables: false,bridge: none,ip-forward: false(docker_disable_default_iptables_rules: "yes",group_vars/all.yml:108-110) and, on EL9 hosts,default-ulimitsnofile (docker/defaults/main.yml:46-48). New in 2024.1, the optional keysregistry-mirrorsanddebug(docker/tasks/config.yml:15-16,33-34,39,46; both off by default,docker/defaults/main.yml:58,68), and the Zuncluster-storekey is no longer rendered (docker_zun_config: {},group_vars/all.yml:125). A host that fails after the daemon.json write and before the flush (drop-in or CA tasks,config.yml:55-114) never runs the notified handler: the role has noforce_handlers(grep of ansible-collection-kollaroles/is empty), the default isFalse(ansible-core 2.16.20lib/ansible/config/base.yml:627-629), and a failed host leaves the play (lib/ansible/executor/play_iterator.py:506-517,lib/ansible/playbook/play.py:345-351). The next run finds daemon.json unchanged, so no Kolla task restarts Docker for that change (§2.2). - The systemd drop-in is never applied. Still holds in 2024.1. The drop-in
/etc/systemd/system/docker.service.d/kolla.confcarriesdocker_http_proxy/docker_https_proxy/docker_no_proxyand, on Zun compute hosts,docker_zun_options(docker/templates/docker_systemd_service.j2:1-13). Adding, changing or removing it does not restart Docker: it notifies only the reload handler, which never notifies the restart (§2.2;docker/tasks/config.yml:55-92). When dockerd picks the new settings up is NOT AVAILABLE (systemd behaviour). - Docker is masked on Debian/Ubuntu during a run. Still holds in 2024.1 for the Docker engine.
docker/tasks/install.yml:15-30symlinks/etc/systemd/system/docker.service -> /dev/nullwhenever that path does not exist at that point. The unit is unmasked by "Start and enable docker" (docker/tasks/config.yml:119-125), by the "Restart docker" handler when it runs (docker/handlers/main.yml:14), and earlier only by the recovery blockdocker/tasks/install.yml:44-72(after a package change with containers running). Whether that unmask removes the path again, so that every later run masks again, is NOT AVAILABLE (systemd behaviour). If the run fails or is interrupted in between (Docker package install, daemon.json, drop-in, CA file, handlers), the unmask does not run (§2.3). NOT AVAILABLE: whether docker then starts at the next reboot. Thepodmanrole has no such step (podman/tasks/install.yml:1-48). /etc/hostslists every host. Still holds in 2024.1. The block loops over allgroups['baremetal']hosts (etc_hosts/tasks/etc-hosts.yml:25-41), and it is the first thing the role does (baremetal/tasks/main.yml:2-3). So: (a) adding one host changes/etc/hostson every host that bootstrap-servers runs on; (b) every baremetal host's facts are required: a baremetal host outsidebifrostwithout gathered facts (for example unreachable during fact gathering, with no fact cache) makeskolla_addressraise on every host (kolla_ansible/kolla_address.py:84-91), which fails the task on every host and ends the play for them before any other bootstrap task (§3); (c) the per-run template work is O(N) per host, O(N²) in total.- Fact gathering with
--limitis serial per limited host. Still holds in 2024.1. With--limitcovering L of N hosts, the second play ofgather-facts.ymlmakes each limited host loop over about N/L other hosts in sequence (gather-facts.yml:34-62; loop items run one after another, ansible-core 2.16.20lib/ansible/executor/task_executor.py:288,336). With--limit one-computeon a 1000-host inventory, one worker runs about 999setupcalls one after another. Fact caching avoids this; the tuning guide documents filling the cache in parallel withkolla-ansible gather-facts(no limit) before a limited run (doc/source/user/ansible-tuning.rst:36-85). - Re-running
bootstrap-serverschanges the host. Still holds in 2024.1. It disables ufw/firewalld (baremetal/tasks/install.yml:3-32, defaultdisable_firewall: True,baremetal/defaults/main.yml:30), sets SELinux permissive (baremetal/tasks/post-install.yml:21-28,baremetal/defaults/main.yml:25-27), disables the libvirtd AppArmor profile on Ubuntu (apparmor_libvirt/tasks/remove-profile.yml:12-22), removes host packages (packages/tasks/main.yml:15-23,packages/defaults/main.yml:25-34): on Debian/Ubuntulxd,lxc,open-iscsiand, with the libvirt container (default for kvm/qemu,group_vars/all.yml:827),libvirt-bin; on RedHat, with the libvirt container,libvirt,libvirt-daemon,iscsi-initiator-utils. With Docker on Debian/Ubuntu it re-downloads the Docker apt key (docker/tasks/repo-Debian.yml:26-32,force: true) and, new in 2024.1, refreshes the apt cache on every run (:67-70;cache_valid_timedefaults to 0, ansible-core 2.16.20lib/ansible/modules/apt.py:1207,1365-1368). None of this restarts containers, but it is not a no-op on a hardened production host. kolla_serialreaches only some commands. Still holds in 2024.1. The CLI injects-e kolla_serial=${ANSIBLE_SERIAL}only forupgradeandreconfigure(tools/kolla-ansible:522,531). It does this after user-eoptions, so a user-e kolla_serial=30%is silently overridden to0for those two commands. You must set the environment variableANSIBLE_SERIAL. Forbootstrap-servers,deploy,prune-imagesand the others,-e kolla_serial=...does work (as the docs suggest:bootstrap-servers.rst:83).- CLI bugs. Still hold in 2024.1.
--ask-vault-passoverwritesVERBOSITYwith$EXTRA_OPTS --ask-vault-pass(tools/kolla-ansible:391-394). This drops earlier-vflags and duplicates the extra options given so far.--include-devis declared as taking an argument (include-dev:at:275) but is handled as a flag (:355-358).$CMDis run unquoted (:148), so it is word-split and glob-expanded. Without-i, no inventory is passed at all (INVENTORYset at:284is never passed; only-ivalues are appended,:605-607), so ansible-core's default inventory source applies (/etc/ansible/hostsunless configured, ansible-core 2.16.20lib/ansible/config/base.yml:722-724). - No
ansible.cfgis shipped for users. Still holds in 2024.1. The only ones are the CI templatetests/templates/ansible.cfg.j2and a vagrant heredoc. The docs (doc/source/user/ansible-tuning.rst) recommend pipelining,forks(20 as an example),gathering = smartwith jsonfile fact caching,inject_facts_as_vars = False,kolla_ansible_setup_filter: "ansible_[!qt]*",kolla_ansible_setup_gather_subsetand, new in 2024.1,kolla_max_fail_percentage(:143-162). They say nothing about ControlPersist, mitogen, strategy orserialfor scale (grep ofdoc/sourceis empty), although the CI template itself setsControlPersist=300(tests/templates/ansible.cfg.j2:13). destroyacts onhosts: all, including the deploy host (localhostis in[deployment],ansible/inventory/multinode:32-33).tools/cleanup-hostdeletes every network namespace and every link whose line matchestap,brqorvxlan, andrm -rfs/etc/kolla/*except six names and, new in 2024.1, the paths given with-iwhen they match an entry of/etc/kollaexactly (tools/cleanup-host:5-23,79-103).prune-imagesprunes kolla-labelled images withdocker_prune(roles/prune-images/tasks/prune_images.yml:2-9), a module that comes from thecommunity.dockercollection, which is outside the sources, or, new in 2024.1, withpodman image pruneunder Podman (:13-16); whether either removes only dangling images, so that old release tags are not reclaimed, is NOT AVAILABLE (§7).- No
kollauser by default. Still holds in 2024.1.create_kolla_userdefaults toFalsein ansible-collection-kolla (baremetal/defaults/main.yml:20,docker_sdk/defaults/main.yml:55,podman_sdk/defaults/main.yml:30) and kolla-ansible does not override it (grep ofansible/is empty). So by default bootstrap-servers creates nokollauser, group, authorized key or sudoers file (baremetal/tasks/post-install.yml:2-4), and/etc/kollais not chowned (:9-16). The kolla-ansible doc still says the opposite ("which istrueby default",doc/source/reference/deployment-and-bootstrapping/bootstrap-servers.rst:107-108).
1. ansible/kolla-host.yml (the bootstrap-servers command)¶
# ansible/kolla-host.yml:1-14
- import_playbook: gather-facts.yml
- name: Apply role baremetal
hosts: baremetal
serial: '{{ kolla_serial|default("0") }}'
gather_facts: false
max_fail_percentage: "{{ baremetal_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}"
roles:
- { role: openstack.kolla.baremetal, tags: baremetal }
- FACT: the CLI maps
bootstrap-serverstokolla-host.ymlwith-e kolla_action=bootstrap-servers(tools/kolla-ansible:485-489). The role isopenstack.kolla.baremetalfrom ansible-collection-kolla, installed bykolla-ansible install-deps(ansible-galaxy collection install -r requirements.yml --force,tools/kolla-ansible:131-144,440-443) from the git branch named inrequirements.yml:2-5. New in 2024.1, when theansiblepip package is not installed,install-depsalso installsrequirements-core.yml(pinned galaxy collections, among themcommunity.docker <4andcontainers.podman <2,tools/kolla-ansible:133-138). This sheet reads the2024.1-eomtag; what a later install of that branch contains is NOT AVAILABLE. - FACT:
max_fail_percentageis new in 2024.1 (kolla-host.yml:8-11). With the default 100 it never stops the play early: the check counts the failures of the current batch only (ansible-core 2.16.20lib/ansible/executor/playbook_executor.py:184-190,lib/ansible/executor/task_queue_manager.py:330-338). - FACT: the role does not dispatch on
kolla_action.baremetal/tasks/main.yml:1-15runs, in order: theetc_hostsrole,pre-install.yml,install.yml(which imports thepackagesrole and the engine roleopenstack.kolla.{{ container_engine }},baremetal/tasks/install.yml:34-38),post-install.yml(which importskolla_user,openstack.kolla.{{ container_engine }}_sdkandapparmor_libvirt), then optionallyconfigure-ceph-for-zun.yml(zun_configure_for_cinder_cephand host inzun-compute). - FACT, new in 2024.1:
container_engineis{{ kolla_container_engine | default('docker') }}(baremetal/defaults/main.yml:37), and kolla-ansible setskolla_container_engine: "docker"(group_vars/all.yml:92; the only other documented value ispodman,etc/kolla/globals.yml:84-85). So the default still installs Docker and the Docker SDK;podmanselects the newpodmanandpodman_sdkroles. 2023.1 hard-codeddockeranddocker_sdk. - FACT: the target is
hosts: baremetal, which in the sample inventories is control, network, compute, storage and monitoring (ansible/inventory/multinode:35-40). The deploy host ([deployment]) is not bootstrapped. - FACT: there are no task-level tags inside the collection roles (grep of
tags:in ansible-collection-kollaroles/is empty).--tagscan only selectbaremetalplus thealways-tagged gather-facts plays (gather-facts.yml:26,62). A subset of bootstrap tasks cannot be chosen with tags. - FACT: the default
serialis"0", meaning one batch of all hosts. Parallelism is limited only by--forks. - CI coverage:
bootstrap-serversruns once, on fresh hosts (tests/setup_gate.sh:133). Re-running it against a live deployment is not exercised in CI. New in 2024.1, CI has Podman jobs (zuul.d/jobs.yaml:21-28,41-47). The re-run hazards below are read from the code; runtime effects that the code does not show are marked NOT AVAILABLE.
1.1 Task-by-task: pre-install.yml¶
As in 2023.1, baremetal/tasks/pre-install.yml holds a single task; the /etc/hosts work is in the etc_hosts role,
which runs just before it, and the engine repository tasks are in the docker role, which runs during the install
phase (§1.2).
| Lines | Task | Change made | Scale | Safe to re-run on a prod host with VMs? |
|---|---|---|---|---|
etc_hosts/ |
Ensure 127.0.0.1 localhost (lineinfile, regexp ^127.0.0.1.*, create: true, mode 0644) |
Replaces the last 127.0.0.1 line with exactly 127.0.0.1 localhost |
per host, O(1) | Yes. It may strip extra aliases from a hand-edited 127.0.0.1 line. |
etc-hosts.yml:18-23 |
Remove 127.0.1.1 ... <hostname> line |
Deletes the Ubuntu-style hostname→127.0.1.1 mapping (RabbitMQ bug refs in comment, lines 12-17) | per host | Yes |
etc-hosts.yml:25-41 |
Generate /etc/hosts for all of the nodes (blockinfile # {mark} ANSIBLE GENERATED HOSTS) |
One line per every host in groups['baremetal']: <api addr> <nodename> <hostname> |
O(N) lines per host, O(N²) Jinja/hostvars work in total. Needs facts of all N hosts. | Yes (no restart is attached). But see §3. |
etc-hosts.yml:47-60 |
If /etc/ exists, write /etc/ = manage_etc_hosts: false (mode 0660) |
Stops cloud-init from rewriting /etc/hosts at boot | per host | Yes |
baremetal/ |
net.ipv4.ping_group_range=0 2147483647 (RedHat only) into kolla_sysctl_conf_path (/etc/, group_vars/) |
sysctl is ansible.posix.sysctl: whether it reloads sysctl, and when, is NOT AVAILABLE |
per host | Yes |
All four etc_hosts tasks run only when customize_etc_hosts (default true, etc_hosts/defaults/main.yml:3;
etc_hosts/tasks/main.yml:2-4). The sudo and kolla groups are created by the kolla_user role, which runs only
when create_kolla_user is true (default False, §1.3).
1.2 Task-by-task: install.yml¶
Rows marked Docker run only with kolla_container_engine: docker (the default); rows marked Podman only with
podman (baremetal/tasks/install.yml:37-38).
| Lines | Task | Notes, scale, re-run safety |
|---|---|---|
baremetal/ |
Firewall (when: disable_firewall, default True, baremetal/): ufw state=disabled policy=allow (Debian, ignore_errors), rpm -q firewalld, then stop and disable firewalld (RedHat) |
Silently removes any host firewall an operator added since the last run. Per host. Containers are not restarted. (ufw is community.general.ufw; its exact effect is NOT AVAILABLE.) |
packages/ |
Install ubuntu_pkg_install (Debian family: git, iputils-ping, sg3-utils-udev if enable_multipathd, tzdata), openeuler_pkg_install (python3-docker and, new in 2024.1, python3-dbus) or redhat_pkg_install (git, iputils, sudo) (packages/), state: present |
On Debian the apt cache is refreshed only when it is older than apt_cache_valid_time, 3600 s (packages/; ansible-core 2.16.20 lib/). |
packages/ |
Remove packages (state: absent): Debian family lxd, lxc, open-iscsi, plus libvirt-bin if enable_nova_libvirt_container; RedHat libvirt, libvirt-daemon, iscsi-initiator-utils, only if enable_nova_libvirt_container (packages/) |
enable_nova_libvirt_container is true for nova_compute_virt_type kvm or qemu (group_vars/; default kvm, :1127). A host that uses host-level iSCSI (for example boot-from-SAN, or a storage agent) loses open-iscsi (always on Debian/Ubuntu) or iscsi-initiator-utils (RedHat, with the libvirt container). Re-running is a no-op only if the packages are already gone. Same with both engines. |
Docker docker/ → repo-Debian.yml:2-70 |
When enable_docker_repo (true except openEuler, docker/): ca-certificates, gnupg state: present (2-10); /etc/, /etc/ (12-24); Docker apt key downloaded with force: true (26-32); optional apt pin from docker_apt_package_pin (34-43). Changed in 2024.1: the old /etc/ is deleted (45-49), the repository is written as the deb822 file /etc/ with Signed-by: /etc/ (53-65), then the apt cache is refreshed (67-70) |
Needs network access to the Docker repository and to the apt mirrors on every run: the key is fetched each time, and the cache refresh at 67-70 passes no cache_valid_time (default 0, ansible-core 2.16.20 lib/). |
Docker docker/ |
yum repo docker with module_hotfixes: true (9-20), rpm_key (22-27) |
Repo config |
Docker docker/ |
docker ps -f status=running -q → running_containers |
Snapshot taken before package changes |
Docker docker/ |
stat /etc/ |
stat does not follow symlinks by default (follow defaults to False, ansible-core 2.16.20 lib/) |
Docker docker/ |
Mask docker (symlink to /dev/null) on Debian/Ubuntu when that file does not exist | Intended to stop apt from auto-starting docker before daemon.json exists (comment 13-14). It also fires on any re-run where the path is absent. Whether the unmask removes the symlink, so that every re-run masks again, is NOT AVAILABLE. See §2.3. |
Docker docker/ |
Install docker_packages with state: present, update_cache: true (apt with cache_valid_time 3600) |
An already-installed docker-ce is NOT upgraded (present, not latest). List (docker/): docker-ce (or docker-ce-<docker_yum_package_pin> on RedHat, docker on openEuler, :19), apparmor on Debian, iptables only if not docker_disable_default_iptables_rules (default "yes", so not installed). Whether a changed pin up- or downgrades an installed engine is NOT AVAILABLE (package manager behaviour). Doc: bootstrap-servers.rst:64-67,. |
Docker docker/ |
If the install result changed and containers were running: systemd docker started/, wait for docker info (6×10 s), then docker start <previously running IDs> |
Recovery path after a package change (comment at 2-3: an engine upgrade "may cause containers to stop"). What docker start does to containers that are still running is NOT AVAILABLE. |
Docker docker/ |
/etc/docker, daemon.json (notifies "Restart docker"), drop-in removal or template (notify "Reload docker service file"), registry CA (notifies "Restart docker"), flush handlers (116-117), start and enable docker (masked: no, 119-125) |
See §2. |
Docker docker/ |
Zun containerd config (only containerd_configure_for_zun and host in zun-cni-daemon) |
See below |
Podman podman/ |
podman ps -f status=running -q → running_containers |
New in 2024.1. Snapshot taken before package changes |
Podman podman/ |
Install podman_packages (podman, plus catatonit on Debian, podman/), state: present, update_cache: true |
An installed podman is not upgraded (present). No repository task: the distribution packages are used (the role has no repo file, podman/). |
Podman podman/ |
podman.socket state: started, enabled: yes |
Every run. kolla-ansible's Podman worker talks to /run/ (ansible/). |
Podman podman/ |
If the install result changed and containers were running: wait for podman info (6×10 s), then podman start <previously running IDs> |
Same recovery pattern as Docker. What podman start does to running containers is NOT AVAILABLE. |
Podman podman/ |
/etc/ (2-11); registry, mirror and storage drop-ins only when podman_registry, podman_registry_mirror, podman_storage_driver or podman_runtime_directory is set (13-53); CA path and copy (55-77) |
All four variables default to empty (podman/) and kolla-ansible sets none of them (grep of ansible/ and etc/ is empty), so by default only the directories are created. docker_registry does not reach Podman's registry config. The CA copy task (67-77) reads private_registry_ca and private_registry, which neither the collection nor kolla-ansible defines (grep); it therefore cannot render unless the operator defines them. No task notifies anything: the role has no handlers. |
1.3 Task-by-task: post-install.yml¶
| Lines | Task | Notes |
|---|---|---|
baremetal/ → kolla_user/ |
groups docker, sudo, kolla_group |
Only when create_kolla_user, default False (baremetal/; not overridden in kolla-ansible). The docker group is created with either engine. |
kolla_user/ |
user kolla (group kolla, groups: [docker, sudo], append) |
Same condition |
kolla_user/ |
authorized_key from kolla_ssh_key.public_key (passwords.yml) |
Same condition. authorized_key is ansible.posix.authorized_key; whether an old key is removed when the key in passwords.yml is rotated is NOT AVAILABLE (no exclusive or state is passed) |
kolla_user/ |
/etc/: kolla ALL=(ALL) NOPASSWD: ALL, mode 0640 |
Same condition plus create_kolla_user_sudoers (default true, kolla_user/) |
Docker post-install.yml:6-7 → docker_sdk/ |
Changed in 2024.1. On Python ≥ 3.10, a stat of <stdlib>/ sets docker_sdk_python_externally_managed (2-19). Packages python3-setuptools, python3-pip, and python3-docker plus python3-dbus when the Python is externally managed and no virtualenv is used; with a virtualenv, the dbus build dependencies (gcc, python3-devel or build-essential, libdbus-1-dev, ...) (docker_sdk/; task 21-27). With a virtualenv, packaging is pip-installed into it first (29-45), then pip>19.3 (47-56). Docker SDK docker>=6.0.0,<7.0.0, requests<2.32 and dbus-python via pip3 or into the venv (docker_sdk/; task 58-68) |
The pip step is skipped when the Python is externally managed and no virtualenv is used (:68); 2023.1 tested the Debian release name bookworm instead. state defaults to present, so no change if matching versions are installed. Needs PyPI if not. Per host. |
Podman post-install.yml:6-7 → podman_sdk/ |
New in 2024.1. Same externally-managed check (2-19); on Debian with an externally managed Python and no virtualenv, an osbpo apt key (template), removal of osbpo.list, a deb822 repository bookworm-bobcat-backports-nochange written to /etc/, and a cache refresh (21-68); packages python3-setuptools, python3-pip, and python3-podman plus python3-dbus in the same case (podman_sdk/; task 70-76); pip podman>=4.7.0 and dbus-python otherwise (podman_sdk/; task 107-117) |
The osbpo repository file has the same path as the Docker role's repository file (docker/), so on a Debian host that ran the Docker role before, it replaces the Docker repository definition. The virtualenv tasks (79-94) use docker_sdk_virtualenv_owner, a variable that only docker_sdk/ defines and that role is not imported under Podman; whether it resolves at run time is NOT AVAILABLE. |
post-install.yml:9-16 |
node_config_directory (/etc/kolla), mode 0755, owned by kolla:kolla only when create_kolla_user |
|
post-install.yml:18-19 → apparmor_libvirt/ |
Ubuntu only, when apparmor_remove_libvirt_profile (= enable_nova_libvirt_container, apparmor_libvirt/; true by default through group_vars/): if /etc/ exists and /etc/ does not, apparmor_parser -v -R the profile and link it into disable/ |
changed_when: true. The disable link makes the task skip on later runs, so it runs once per host. |
post-install.yml:21-28 |
selinux: policy=targeted state={{ selinux_state }} (default permissive, change_selinux: True, baremetal/) on RedHat |
selinux is ansible.posix.selinux: whether it also changes the running mode is NOT AVAILABLE. No reboot is done by the role. |
post-install.yml:30-42 |
global git https.proxy / http.proxy from git_https_proxy / git_http_proxy (default empty) |
community.general.git_config |
baremetal/ → configure-ceph-for-zun.yml:2-81 |
Ceph repository, EPEL on RedHat, ceph-common (only zun_configure_for_cinder_ceph and host in zun-compute). Changed in 2024.1 on Debian: the key is downloaded to /etc/ with force: true, ceph.list is removed and a deb822 ceph.sources is written (:5-43) |
Needs network access to download.ceph.com |
There are still no enable_host_ntp tasks: no collection role names ntp (grep). Time sync is only checked, by
prechecks (§5).
configure-containerd-for-zun.yml (Docker only; only containerd_configure_for_zun and host in zun-cni-daemon,
docker/tasks/main.yml:9-12): it writes the CNI config and script and overwrites /etc/containerd/config.toml with
only [grpc] gid = ... (docker/templates/containerd_config.toml.j2:1-2). The CNI config and config.toml templates
notify "Restart containerd" (configure-containerd-for-zun.yml:11-20,38-45; docker/handlers/main.yml:17-21).
These tasks come after the explicit flush in config.yml, so the containerd restart runs at the play's implicit flush
after all roles, that is after post-install (ansible-core 2.16.20 lib/ansible/playbook/play.py:345-351).
NOT AVAILABLE: whether container shims survive a containerd restart.
Kernel modules: the bootstrap roles load none (grep for modprobe/module-load in ansible-collection-kolla
roles/
is empty). Modules and sysctls are applied during deploy/reconfigure by config-host.yml in service roles: the
module-load role from ironic, iscsi, loadbalancer, multipathd, neutron, nova-cell and openvswitch, and
sysctls from loadbalancer and neutron (through the new sysctl role, e.g.
roles/neutron/tasks/config-host.yml:20-22),
nova-cell and opensearch (grep of roles/*/tasks/config-host.yml).
2. Docker daemon configuration and restart: the exact trigger¶
This section applies to kolla_container_engine: docker, the default. For Podman see
§2.5.
2.1 What goes into /etc/docker/daemon.json¶
Base dict (docker/vars/main.yml:2-5): log-opts: {max-file: docker_log_max_file, max-size: docker_log_max_size}
(defaults "5" / "50m", group_vars/all.yml:113-114). Keys merged in by docker/tasks/config.yml:9-47:
| Key | Source / condition | Default | Line (config.yml) |
|---|---|---|---|
docker_zun_config (Zun) |
docker_configure_for_zun and host in zun-compute |
off (all.yml:123). Changed in 2024.1: docker_zun_config: {} (all.yml:125), so nothing is added; the 2023.1 cluster-store: etcd://... value is gone |
37 |
insecure-registries: [docker_registry] |
docker_registry_insecure true |
"no" (all.yml:102); docker_registry: "quay.io" (all.yml:98) |
12-14, 38 |
registry-mirrors |
docker_registry_mirrors non-empty. New in 2024.1 |
[] (docker/) |
15-16, 39 |
storage-driver |
docker_storage_driver non-empty |
"" (docker/) |
17-18, 40 |
data-root |
docker_runtime_directory non-empty |
"" (all.yml:103) |
19-20, 41 |
iptables: false |
docker_disable_default_iptables_rules |
"yes" (all.yml:108) |
21-22, 42 |
bridge: none |
docker_disable_default_network |
follows the previous (all.yml:109) |
23-24, 43 |
ip-forward: false |
docker_disable_ip_forward |
follows the previous (all.yml:110) |
25-26, 44 |
default-ulimits: {nofile: 1048576/ |
docker_ulimit_nofile |
true on RedHat-family major version 9 (docker/) |
27-32, 45 |
debug |
docker_debug. New in 2024.1 |
false (docker/) |
33-34, 46 |
| anything | docker_custom_config (dict) |
{}; {"exec-opts": ["native.umask=normal"]} on openEuler (docker/) |
47 |
FACT: the file is rendered with Ansible's to_nice_json, which uses sort_keys=True
(ansible-core 2.16.20 lib/ansible/plugins/filter/core.py:79-81), so key order is stable and does not cause spurious
changes.
FACT: copy content: overwrites the whole file. Any key added by hand (for example "live-restore": true added
outside docker_custom_config) is removed, and that removal itself triggers a restart.
FACT: when docker_registry_ca is set, the CA is copied to /etc/docker/certs.d/<docker_registry>/ca.crt and a change
also notifies "Restart docker" (config.yml:94-114).
2.2 Why docker_reloaded never triggers a restart¶
There is no docker_reloaded register in 2024.1; its successor is the "Reload docker service file" handler, and the
reason it never leads to a restart is the same.
# ansible-core 2.16.20 lib/ansible/modules/systemd.py:387-400
result = dict(name=unit, changed=False, status=dict())
# Run daemon-reload first, if requested
if module.params['daemon_reload'] and not module.check_mode:
(rc, out, err) = module.run_command("%s daemon-reload" % (systemctl))
if rc != 0: ... module.fail_json(...)
changed is set to True only by the masked, enabled and state branches (lines 484/530/562). The handler
"Reload docker service file" (docker/handlers/main.yml:2-8) passes only name and daemon_reload, so it always
reports unchanged, and its notify: Restart docker never fires: notifications are processed only for changed results
(ansible-core 2.16.20 lib/ansible/plugins/strategy/__init__.py:656). 2023.1 was studied with ansible-core 2.14.18,
which is not on disk, so no comparison between the two ansible-core versions is made here.
Conclusion (FACT for ansible-core 2.16.20): during bootstrap-servers, the Docker daemon restarts only when the
write of daemon.json, or of the registry CA file, reports changed (rendered content differs from the file on disk, or
its mode differs; lib/ansible/plugins/action/copy.py:285,341-380, lib/ansible/module_utils/basic.py:867,900,949),
and
only if the host reaches the flush at docker/tasks/config.yml:116-117. Triggers:
- first bootstrap of a host (the file does not exist yet), or any hand-edited daemon.json;
- a host whose daemon.json was rendered with other defaults (for example
iptables,bridge,ip-forward,group_vars/all.yml:108-110;default-ulimitson EL9); - new in 2024.1: the first 2024.1 run on a Zun compute host with
docker_configure_for_zun: the 2023.1 file carriedcluster-store, which 2024.1 no longer renders (group_vars/all.yml:125,config.yml:37), so the file changes and Docker restarts on those hosts; - changing
docker_registry_insecure(anddocker_registrywhile it is true),docker_registry_mirrors,docker_debug,docker_log_max_file,docker_log_max_size,docker_storage_driver,docker_runtime_directory,docker_disable_default_iptables_rules,docker_disable_default_network,docker_disable_ip_forward,docker_ulimit_nofile*ordocker_custom_config; - with
docker_registry_caset: a changed CA file, or a changeddocker_registry(new destination path); - no longer a trigger in 2024.1: adding or removing an etcd host on Zun compute hosts (2023.1 rendered every etcd
host into
cluster-store;docker_zun_configis now{},group_vars/all.yml:125); - a package change does not notify the handler by itself. What a docker-ce package upgrade does to the daemon
is NOT AVAILABLE; the role's comment says it "may cause containers to stop" (
docker/tasks/install.yml:2-3). Kolla then runsdocker starton previously running containers (docker/tasks/install.yml:44-72). The kolla-ansible doc says a package update "will cause the Docker engine to restart" (bootstrap-servers.rst:64-67); the sources do not show the mechanism (package manager behaviour).
With docker_systemd_reload: true the handler reloads instead of restarting (docker/handlers/main.yml:13); which
daemon.json changes dockerd applies on a reload is NOT AVAILABLE (Docker behaviour).
Not triggers (latent changes): adding, changing or removing docker_http_proxy, docker_https_proxy,
docker_no_proxy, or the Zun docker_zun_options drop-in (docker/tasks/config.yml:55-92). These are written to disk
and daemon-reloaded, and no Kolla task restarts Docker for them. When dockerd picks them up is NOT AVAILABLE (systemd
behaviour). Also latent: a daemon.json change on a host that failed before the flush
(§0 item 1).
NOT AVAILABLE: whether dockerd refuses to start if the same option ends up both as a dockerd flag in the drop-in and in
daemon.json.
2.3 The Debian/Ubuntu mask window¶
Sequence per run on Debian/Ubuntu with Docker (FACT):
docker/tasks/install.yml:21-29 mask (the symlink is created when the path is absent), then the Docker package install
(:32-39), then daemon.json, drop-in and CA tasks (docker/tasks/config.yml:9-114), then the handler flush
(:116-117; "Restart docker" carries masked: no when notified), then config.yml:119-125 masked: no, which runs
systemctl unmask docker when systemd reports the unit as masked (ansible-core 2.16.20
lib/ansible/modules/systemd.py:476-492). The recovery block docker/tasks/install.yml:44-72 also sets masked: no
(line 52), but only when the package install changed and containers were running. Pip installs, user creation and the
AppArmor and SELinux tasks come after the unmask (baremetal/tasks/main.yml:7-9).
FACT: a failure or interruption between those points, unless it comes after the recovery block or the handler has
run, leaves the symlink in place. The roles have no rescue or always (grep of ansible-collection-kolla roles/ is
empty), and a failed host runs no further task of the play (ansible-core 2.16.20
lib/ansible/executor/play_iterator.py:506-517).
NOT AVAILABLE: whether the running daemon is affected, and whether Docker, and therefore every Kolla container and VM
on that host, starts after the next reboot. One related statement in the sources is the Ansible systemd
module's note that "a masked unit is impossible to start" (ansible-core 2.16.20 lib/ansible/modules/systemd.py:44).
Also NOT AVAILABLE: whether systemctl unmask removes the symlink, which decides whether the next run masks again.
Check after any failed run: ls -l /etc/systemd/system/docker.service should not point to /dev/null.
2.4 Blast radius of a restart¶
- FACT: the restart is the "Restart docker" handler, run at
docker/tasks/config.yml:116-117on every notified host of the current batch. With defaultserial: 0, all baremetal hosts whose daemon.json changed restart Docker in the same step (subject to forks). That includes all controllers at once. - FACT: in 2024.1
kolla_container's Docker worker gives containers no Docker restart policy (inmodule_utils/kolla_docker_worker.pyrestart_policyappears only in the one-shot checks at:338,381,450). Each container runs under a systemd unitkolla-<name>-container.servicewithAfter=andWants=docker.service(for Docker;network-online.targetotherwise),ExecStart=/usr/bin/<engine> start -a <name>,ExecStop=/usr/bin/<engine> stop <name> -t <graceful_timeout>andRestart=alwaysforunless-stopped(module_utils/kolla_systemd_worker.py:20-39,50-58;docker_restart_policy,group_vars/all.yml:117). The unit text changed in 2024.1, so the first 2024.1 check of a container whose unit 2023.1 wrote reports a difference (F16). NOT AVAILABLE: whether those units bring the containers back after the daemon restarts (systemd and Docker behaviour). FACT: the Docker path sets no containerStopTimeout(grep forstop_timeout|StopTimeoutinlibrary/andmodule_utils/finds only the Podman worker,module_utils/kolla_podman_worker.py:60,125).docker_graceful_timeout: 60(all.yml:139) is used by the unit'sExecStopand by the Docker worker's stops (module_utils/kolla_docker_worker.py:415-430,439-459), not by the daemon. NOT AVAILABLE: the stop timeout Docker applies to containers on a daemon restart. - NOT AVAILABLE: whether restarting Docker on all MariaDB hosts at once stops the whole Galera cluster. FACT: the kolla
image has no recovery logic for it. On a normal start the container runs
mysqld_safe(plusBOOTSTRAP_ARGSonly if set) with every MariaDB host of the shard ingcomm://, and the image's start path has nograstate.datorsafe_to_bootstraphandling and no new-cluster decision (kolladocker/mariadb/mariadb-server/extend_start.sh:31-61;roles/mariadb/templates/galera.cnf.j2:29; Kolla images §5.4). NOT AVAILABLE: whether Galera re-forms the cluster by itself orkolla-ansible mariadb_recoveryis needed. With a single MariaDB host in the shard,gcomm://is empty (galera.cnf.j2:29); whether that host comes back without recovery is NOT AVAILABLE. - Compute nodes:
nova_libvirtruns withpid_mode: host,cgroupns_mode: hostand bind-mounts/sys/fs/cgroup(roles/nova-cell/defaults/main.yml:8-9,369-377). NOT AVAILABLE: whether running QEMU processes survive the container restart; the sources do not show whether the guests sit in machined slices or in the container cgroup. Notetools/cleanup-containers:16-22: "Container engine ($engine) will not be able to stop the nova_libvirt container with those [qemu processes] running". - Optional unit hazard:
tools/setup-compute-libvirt.yml(not run by any kolla-ansible command) installskolla-libvirt-guests-<engine>.service(:9-11,13-17). The docker variant hasRequires=docker.service, and itsExecStopstops nova_compute and runslibvirt-guests.sh shutdown(tools/kolla-libvirt-guests-docker.service:6,18-19). NOT AVAILABLE: whether systemd propagates a restart ofdocker.serviceto units thatRequires=it, and so whether, on hosts where an operator installed this unit, a Docker restart shuts down or managed-saves every guest. - Documented mitigations (
doc/source/reference/deployment-and-bootstrapping/bootstrap-servers.rst:56-88):--limitbatches keeping quorum,-e kolla_serial=30%, or Dockerlive-restore("issues reported ... use it at your own risk"). The way to set live-restore in a supported way isdocker_custom_config: {"live-restore": true}. FACT: adding it changes the rendered daemon.json, so that run restarts Docker (§2.2); if the file on disk already matches the rendered content, the handler is not notified. NOT AVAILABLE: whether containers survive that first restart, and whether pre-seeding daemon.json by hand and runningsystemctl reload dockerbefore bootstrap-servers activates live-restore. - Interaction with
config_strategy: COPY_ALWAYS(all.yml:43, passed asKOLLA_CONFIG_STRATEGYindocker_common_options,all.yml:147-148). FACT (kolla image): every Kolla service container exceptbifrost_deploy(which runs/sbin/init, kolladocker/bifrost/bifrost-deploy/Dockerfile.j2:29-30) runskolla_set_configswhen it starts, and underCOPY_ALWAYSthat re-copies the files listed in itsconfig.jsonfrom/etc/kolla/<svc>/and rewrites/run_command(kolladocker/base/start.sh:11,docker/base/set_configs.py:300-319,535-541; both files are identical to 2023.1's; Kolla images §2). A container that starts again after a Docker restart therefore activates any config that was rendered but not yet applied, for example bykolla-ansible genconfig(tools/kolla-ansible:557-560) or a failed reconfigure. Removals are not applied: destinations of entries dropped fromconfig.json, or of optional sources that are gone, stay in the container (Kolla images §3.5).
2.5 With kolla_container_engine: podman¶
New in 2024.1. FACT:
- The
podmanrole has nohandlers/directory and none of its tasks hasnotify(podman/tasks/install.yml:1-48,podman/tasks/config.yml:1-77). So no bootstrap task restarts a Podman service for a config change; changes to the/etc/containers/*.conf.ddrop-ins are written and nothing else is done. When Podman reads them is NOT AVAILABLE (Podman behaviour). - There is no mask step and no Debian/Ubuntu mask window (§2.3 is Docker only).
podman.socketis started and enabled on every run (podman/tasks/install.yml:24-29).- Container units depend on
network-online.target, not on an engine service (ansible/module_utils/kolla_systemd_worker.py:50-54). - NOT AVAILABLE: what a
podmanpackage upgrade does to running containers; the role takes the same snapshot and runspodman startafterwards (podman/tasks/install.yml:2-10,31-48).
3. /etc/hosts generation¶
# ansible-collection-kolla roles/etc_hosts/tasks/etc-hosts.yml:30-34
{% for host in groups['baremetal'] %}
{% set api_interface = (hostvars[host]['api_interface'] | replace('-', '_')) %}
{% if host not in groups['bifrost'] or api_interface in hostvars[host].ansible_facts %}
{% set hostnames = [hostvars[host].ansible_facts.nodename, hostvars[host].ansible_facts.hostname] %}
{{ 'api' | kolla_address(host) }} {{ hostnames | unique | join(' ') }}
The etc_hosts role is unchanged between 2023.1 and 2024.1 (a diff -ru of the two collection trees lists no
etc_hosts file).
- FACT: it loops over all baremetal hosts. Adding a host changes the managed block on every host bootstrap-servers runs against. No handler or restart is attached, so a change restarts nothing by itself.
- FACT: it needs facts for all baremetal hosts.
kolla_addressraisesFilterError("Interface ... not present on host ...")whenhostvars[host]['ansible_facts']lacks the interface (kolla_ansible/kolla_address.py:84-91). A host whose facts were never gathered (for example unreachable during fact gathering, with no fact cache) has an emptyansible_facts(ansible-core 2.16.20lib/ansible/vars/manager.py:308-309,lib/ansible/vars/clean.py:161-170). So one fact-less baremetal host outsidebifrostfails this task on every host:FilterErroris anAnsibleFilterError(kolla_ansible/exception.py:23), and anAnsibleErrorfails the task (ansible-core 2.16.20lib/ansible/executor/task_executor.py:198-199). Those hosts then run no further task of the play (lib/ansible/executor/play_iterator.py:506-517), and sinceetc_hostsruns first (baremetal/tasks/main.yml:2-3), no bootstrap task runs at all. In practice every baremetal host must be reachable, or have cached facts, for bootstrap-servers. - The template applies the same
-→_replacement as thewhen:clause (etc-hosts.yml:31,41). - With
--limit new-hosts, only the limited hosts get the full list. Existing hosts never learn the new names. The docs require including all controllers in the limit for RabbitMQ (doc/source/user/adding-and-removing-hosts.rst:22-29). For computes, withoutlibvirt_tlsthe libvirt connection and migration addresses are IPs (nova.conf.d/libvirt.conf.j2:7-8). Exception: withlibvirt_tls,live_migration_inbound_addr = migration_hostname = ansible_facts.nodename(roles/nova-cell/templates/nova.conf.d/libvirt.conf.j2:2-5,roles/nova-cell/defaults/main.yml:561). NOT AVAILABLE: whether live migration from old computes to a new compute then fails without name resolution. To make the new name resolvable on the old computes, re-run bootstrap-servers on all computes (or use DNS). - NOT AVAILABLE: whether running host-network containers see new
/etc/hostsentries before they are recreated (the task edits the host's/etc/hostsonly). - Disable the whole mechanism with
customize_etc_hosts: false(etc_hosts/defaults/main.yml:3,etc_hosts/tasks/main.yml:2-4) when DNS is authoritative. At large N, this also removes the O(N²) templating.
4. ansible/gather-facts.yml (confirmed) and its cost model¶
# gather-facts.yml:5-26 (play 1, hosts: all, serial: kolla_serial)
- group_by: key: "all_using_limit_{{ (ansible_play_batch | length) != (groups['all'] | length) }}"
- setup: {filter: kolla_ansible_setup_filter, gather_subset: kolla_ansible_setup_gather_subset}
when: not ansible_facts
# gather-facts.yml:34-62 (play 2, hosts: all_using_limit_True)
vars:
batch_index: "{{ ansible_play_batch.index(inventory_hostname) }}"
batch_size: "{{ ansible_play_batch | length }}"
delegate_hosts: "{{ groups['all'][batch_index | int::batch_size | int] }}"
setup: ... delegate_facts: True, delegate_to: "{{ item }}", with_items: "{{ delegate_hosts }}"
when: [item not in groups["all_using_limit_True"], not hostvars[item].ansible_facts]
The only change from 2023.1 is max_fail_percentage on both plays (gather-facts.yml:7-10,36-39), default 100.
Cost model, with N = |groups['all']| (this includes localhost from [deployment]) and L = hosts selected by
--limit:
- No limit, serial 0: play 1 runs N
setupcalls in parallel (forks). Play 2 is empty. --limitwith L hosts: play 1 runs L parallelsetupcalls. Play 2 runs on the same L hosts. Limited host i loops sequentially overgroups['all'][i::L], about N/L items, each a delegatedsetup(SSH from the deploy host to the item host, facts stored on the item); items that are themselves limited hosts are skipped (gather-facts.yml:59). The items of one host's loop run one after another (ansible-core 2.16.20lib/ansible/executor/task_executor.py:288,336). Total is N−L delegated setups, but only L run at the same time. The resulting wall time is NOT AVAILABLE. L=1 means fully serial fact gathering of the whole inventory.- With
kolla_serial(no limit): each batch is smaller than N, so every host joinsall_using_limit_True, and every item of play 2 is then skipped byitem not in groups["all_using_limit_True"](loop evaluation only, no SSH). - With fact caching (
gathering = smart+fact_caching,ansible-tuning.rst:36-66):when: not ansible_factsis false for cached hosts, so both plays skipsetup. The guide also documents populating the cache withkolla-ansible gather-factswithout a limit before a limited command (ansible-tuning.rst:68-85;tools/kolla-ansible:508-511). This is the only documented way to make--limitruns cheap at scale. Stale-cache risk: IP or interface changes will not be seen until the cache expires (fact_caching_timeout). - FACT: an unreachable other host without cached facts makes its delegated item in play 2 return
unreachable(ansible-core 2.16.20lib/ansible/executor/task_executor.py:648-649). One unreachable item makes the whole task result unreachable (lib/ansible/executor/task_executor.py:141-146,lib/ansible/executor/task_result.py:72-73,96-106), so the limited host is marked unreachable and removed from the play (lib/ansible/plugins/strategy/__init__.py:632-636) and from every later play (lib/ansible/executor/playbook_executor.py:174,217). So--limit newnodecan fail because an unrelated node is down. - FACT:
kolla_ansible_setup_filterandkolla_ansible_setup_gather_subsetapply only ingather-facts.yml. Playbooks that do not import it (destroy.yml,mariadb_backup.yml,mariadb_recovery.yml,post-deploy.yml,octavia-certificates.yml, andcertificates.ymlwhen none ofkolla_enable_tls_backend,rabbitmq_enable_tls,certificates_generate_libvirtis true,certificates.yml:2-6) use implicit full play-level fact gathering. - FACT: the documented filter
"ansible_[!qt]*"(ansible-tuning.rst:119, aimed at "virtual interfaces beginning with q or t", :114-115) also hides real interfaces whose names start withqort(for exampleteam0,trunk0,tun0).setupkeeps only theansible_-prefixed fact names that match the filter, and each interface's facts sit under its name (ansible-core 2.16.20lib/ansible/modules/setup.py:196,210-216,lib/ansible/module_utils/facts/ansible_collector.py:58-76,lib/ansible/module_utils/facts/network/linux.py:56-57).kolla_addressthen fails forapi_interface,tunnel_interfaceand so on if they are named that way (kolla_ansible/kolla_address.py:84-91).
5. ansible/roles/prechecks (kolla-ansible prechecks)¶
- FACT:
prechecksrunssite.ymlwith-e kolla_action=precheck(tools/kolla-ansible:444-447). The commonprechecksrole runs onhosts: kolla_action_precheck, which is all hosts (site.yml:7-18,77-86). Each service play then runs its own<role>/tasks/precheck.yml(54 files, about 2.9k lines in total). - The common role (
roles/prechecks/tasks/main.yml:1-23) checks:- inventory: the
loadbalancergroup exists and is non-empty whenenable_loadbalancer(inventory_checks.yml). - host OS (
host_os_checks.yml): CentOS Stream 9, Debian bullseye or bookworm, openEuler 22.03, Rocky 9, Ubuntu jammy and, new in 2024.1, Ubuntu noble (vars/main.yml:11-23). Can be skipped withprechecks_enable_host_os_checks: false. - time sync (skipped for
deploymenthosts): one of chrony, chronyd, ntp, ntpd or systemd-timesyncd must be active, andtimedatectl statusmust saysynchronized: yes(timesync_checks.yml:2-47). Can be skipped withprechecks_enable_host_ntp_checks: false(defaults/main.yml:6). /etc/localtimeexists, plus/etc/timezoneon Debian (datetime_checks.yml).api_interfacepresent, active, with an address (port_checks.yml; skipped fordeploymentgroup hosts,main.yml:13-15).- systemd is the service manager (
service_checks.yml:2-7);docker --version≥ 18.09, now only with Docker (service_checks.yml:9-19,vars/main.yml:2); Docker SDK ≥ 3.4.1, also only with Docker,dbusimportable (baremetal only, both engines) and Ansible 2.15–2.16 (package_checks.yml:2-32,vars/main.yml:3-5). - empty values in passwords.yml (
grep,delegate_to: localhost,run_once) (service_checks.yml:21-32; passes vacuously if the vault is encrypted, per the note at 21-23). - fails if
nscdis running (service_checks.yml:34-48). - TLS VIP consistency,
enable_cephremoved, and RabbitMQ settings (service_checks.yml:50-81). Changed in 2024.1: the RabbitMQ check now fails wheneverom_enable_rabbitmq_high_availabilityandom_enable_rabbitmq_quorum_queuesare equal, so both false fails too (:67-72), and a new check requiresom_enable_rabbitmq_transient_quorum_queuewhenom_enable_rabbitmq_stream_fanoutis on (:74-81) (F17). config_owner_userand group exist; passwordless sudo works (user_checks.yml).- external DB reachable when
enable_mariadbis false (database_checks.yml).
- inventory: the
- Service prechecks use
kolla_container_facts(now withcontainer_engine),wait_for ... state: stopped timeout: 1port checks (skipped when the owning container already exists, for exampleroles/nova-cell/tasks/precheck.yml:8-19,31-43),stat,getent,vgs,pingof the VIP,ip addr, a read-only HAProxyshow statvia{{ kolla_container_engine }} exec(roles/loadbalancer/tasks/precheck.yml:208-214), and read-onlyrabbitmqctl list_policies/list_queuesvia{{ kolla_container_engine }} exec(roles/rabbitmq/tasks/precheck.yml:107-189). - Does it change anything? FACT: no module that modifies state was found. Every
command/shellread ischanged_when: falseor read-only, and there are notemplate/copy/file/kolla_container/lineinfile/sysctl/service/systemdtasks in the precheck files (grep; the oneservice:hit,roles/letsencrypt/tasks/precheck.yml:12, is a variable name). It is safe on a running production cloud. - Cost: O(1) tasks per host per enabled service, plus full
gather-facts.yml. Loops over other hosts exist only forrabbitmqhostname resolution (roles/rabbitmq/tasks/precheck.yml:61-77, for the rabbitmq hosts only; the 2023.1outward-rabbitmqloop is gone with that service) and the external-database check (database_checks.yml:7). There is no N² behaviour. The dominant cost at scale is Ansible's per-task overhead across 71 plays (51 insite.ymlplus 20 imported fromgather-facts.yml,mariadb.yml,rabbitmq.yml,nova.yml; Generated inventory). Prechecks plays inheritserial: kolla_serialfrom site.yml where it is set.
6. destroy (ansible/destroy.yml, role destroy, tools/cleanup-*)¶
# ansible/destroy.yml:1-9
- name: Apply role destroy
hosts: all # includes localhost/[deployment] in the sample multinode inventory
max_fail_percentage: "{{ destroy_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}"
roles: [destroy] # no serial, implicit gather_facts, no gather-facts.yml import
- CLI: needs
--yes-i-really-really-mean-itspelled in full (tools/kolla-ansible:475-483;RAW_ARGScheck at 342-348).--include-imagessetsdestroy_include_images=yes.--include-devsetsdestroy_include_dev=yes, but see the bug in §11. New in 2024.1: the CLI joins the-ivalues with commas and always passes-e inventories_comma_separated=<list>, empty without-i(tools/kolla-ansible:462-474). - Role flow (
roles/destroy/tasks/main.yml:1-8):validate-docker-execute.sh(sudo $CONTAINER_ENGINE ps), thencleanup-containers <engine>, thencleanup-images --all -e <engine>(only ifdestroy_include_images, defaultenable_destroy_images: "no",all.yml:867), thencleanup-host(become: true, env vars passed incleanup_host.yml:5-17, now includingkolla_ansible_inventoriesat:17) and, for Octavia health-manager hosts with auto-configured tenant networking, removal of theoctavia-interfaceservice and dhclient config (cleanup_host.yml:19-40). The engine iskolla_container_engine(cleanup_containers.yml:3,cleanup_images.yml:3). The role ignoreskolla_actionand has no tags. tools/cleanup-containers:- Safety guard (16-22): aborts if
nova_libvirtis running and non-Xen-dom0 qemu processes exist. - If
openvswitch_vswitchdis among the containers andneutron_openvswitch_agentis running:neutron-ovs-cleanup --ovs_all_ports, then delete every OVS bridge (35-41). NOT AVAILABLE: whether theexec -itat line 40 fails without a TTY under Ansible'sscript, and so whether bridges survive. - For every kolla-labelled container, new in 2024.1:
systemctl disableits unit, thensystemctl stopit, thenreset-failedif the unit is failed (43-49); thenrm -v -fs them (52), disconnects them from the host network (54-57), thenvolume rmevery volume mounted by them (29-32, 60). That includesmariadb,rabbitmq,nova_compute(instance disks when the default volume is used),libvirtd,kolla_logsand so on. It also removes the/var/log/kollalink (63) and every/etc/systemd/system/kolla-*-container.serviceunit file, then runssystemctl daemon-reload(66-67).
- Safety guard (16-22): aborts if
tools/cleanup-host:- Deletes ALL network namespaces (
ip netns list, 5-8). This is not limited to kolla's. NOT AVAILABLE: whetherip netns listprints lines of the formname (id: N); the unquoted$line(line 7) would pass those extra words toip netns delete. - Deletes every link whose
ip -o linkline matches/tap/,/brq/or/vxlan/(10-23). - Removes the VIPs from interfaces when
enable_haproxy == yes(27-35). - Copies
/etc/fstabto/etc/fstab_backup(37-38), then for Swift:cat /etc/fstab | grep "/srv/node/d*" | xargs umount(42). This feeds whole fstab lines to umount (bug), then deletes those fstab lines (43). rm -rfvof custom datadirs when they differ from the default volume names (46-69): glance, nova_compute, gnocchi, influxdb, opensearch. For examplenova_instance_datadir_volumeset to a host path means instance storage is wiped.- Runs the OVS-DPDK uninstall script if present (73-76).
rm -rfv /etc/kolla/*exceptpasswords.yml,globals.yml,globals.d,kolla-build.conf,config,certificatesand, new in 2024.1, any entry whose path is string-equal to one of the inventories passed with-i(79-103). The comparison is[ "$dir" == "$inventory" ](83) against$dir=/etc/kolla/<name>, so it protects an inventory only when it was given as that absolute path: a relative-i multinode, or an inventory file inside a subdirectory such as/etc/kolla/inventory/hosts, matches no entry, and that entry is removed. The path is hard-coded (FOLDER_PATH="/etc/kolla", 71) and ignoresnode_config_directory. Becauselocalhostis inhosts: all, this also runs on the deploy host unless--limitexcludes it, which deletesadmin-openrc.sh, the other openrc files,clouds.yamland anything else stored under/etc/kolla.- FACT:
inventories_comma_separatedis set only by the CLI (tools/kolla-ansible:474); grep finds no default for it inansible/, so runningdestroy.ymlwithansible-playbookdirectly, without that variable, leaves the template atcleanup_host.yml:17undefined.
- Deletes ALL network namespaces (
tools/cleanup-images:--allrunsrmi -fon every kolla-labelled image (60-63, 94); the engine comes from-e(:42-51).- Scale and safety: per host, all hosts in parallel (forks), no serial and no batch protection. Never run it on
production
without
--limit. For decommissioning, the docs usestopplus inventory removal, notdestroy(doc/source/user/adding-and-removing-hosts.rst:144-150,211-217). - CI: no 2024.1 CI job runs
destroy(grep oftests/andzuul.d/).
7. prune-images (ansible/prune-images.yml, role prune-images)¶
# ansible/prune-images.yml:1-13
- import_playbook: gather-facts.yml
- name: Apply role prune-images
hosts: baremetal
serial: '{{ kolla_serial|default("0") }}'
gather_facts: false
max_fail_percentage: "{{ prune_images_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}"
roles: [prune-images]
# roles/prune-images/tasks/main.yml:2-4: validate_docker_execute.yml, then prune_images.yml
# roles/prune-images/tasks/prune_images.yml:2-9 (Docker)
docker_prune: {images: yes, images_filters: {label: kolla_version}, timeout: "{{ docker_image_prune_timeout }}"} # default 300
# roles/prune-images/tasks/prune_images.yml:13-16 (Podman, new in 2024.1)
command: podman image prune --force --filter 'label=kolla_version'
- CLI needs
--yes-i-really-really-mean-it(tools/kolla-ansible:565-576); the docs call it pruning "orphaned Docker images" (doc/source/user/operating-kolla.rst:256-257). - FACT:
docker_pruneis not an ansible-core module; ansible-core redirects it tocommunity.docker.docker_prune(ansible-core 2.16.20lib/ansible/config/ansible_builtin_runtime.yml:745-746). That collection is not in the sources. New in 2024.1,requirements-core.ymlpinscommunity.docker <4and the CLI installs it only when theansiblepip package is absent (tools/kolla-ansible:133-138); ansible-collection-kolla still declares no dependency (galaxy.yml:15,dependencies: {}). NOT AVAILABLE: whetherimages: yesremoves only dangling images, and so whether old-release images (for example2023.1after an upgrade to2024.1) are kept. - FACT, new in 2024.1: with Podman the role runs
podman image prune --force --filter 'label=kolla_version'(prune_images.yml:13-16), acommandtask withoutchanged_when, so it reports changed on every run (ansible-core 2.16.20lib/ansible/modules/command.py:338). NOT AVAILABLE: which images Podman removes with these flags (Podman behaviour). - To remove images of an old release explicitly,
tools/cleanup-images --image-version <value> -e <engine>removes images whosekolla_versionlabel equals that value (tools/cleanup-images:75-77,94); what value the images carry is NOT AVAILABLE here. Manualdocker rmi(orpodman rmi) also works. - NOT AVAILABLE: whether Docker would prune an image that a container uses (Docker behaviour). Cost is one engine call
per host, plus the full
gather-facts.yml(which is not needed by the role, but is imported and pays the --limit fact cost from §4). - CI: not covered (grep of
tests/andzuul.d/).
8. post-deploy (ansible/post-deploy.yml)¶
- Two plays on
hosts: localhost, implicit fact gathering, tasks withbecome: true(post-deploy.yml:1-65):- creates
/etc/openstack(mode0700, owned by the invoking user) and templatesroles/common/templates/clouds.yaml.j2to{{ node_config }}/clouds.yaml, mode0600(:2-21). Note the file goes tonode_config, not to/etc/openstack. - templates
admin-openrc.shand, new in 2024.1,admin-openrc-system.sh,public-openrc.shandpublic-openrc-system.shto{{ node_config }}/, owned by the invoking user's uid/gid, mode0600(:23-60), and, withenable_octavia, the Octavia openrc (:62-65).node_config=CONFIG_DIR, default/etc/kolla(all.yml:10).
- creates
- Contents:
OS_PASSWORD={{ keystone_admin_password }}in plaintext in every file.admin-openrc.shusesOS_AUTH_URL={{ keystone_internal_url }}(the internal endpoint),OS_INTERFACE=internal, and optionalOS_CACERTfromkolla_admin_openrc_cacert(admin-openrc.sh.j2:1-25);keystone_admin_urlno longer exists (grep ofansible/is empty). The-systemfiles useOS_SYSTEM_SCOPE=all(admin-openrc-system.sh.j2:6); thepublic-files usekeystone_public_url(public-openrc.sh.j2:11).clouds.yamlnow holds four clouds,kolla-admin,kolla-admin-system(public Keystone URL),kolla-admin-internalandkolla-admin-system-internal, all with the admin password in plaintext (clouds.yaml.j2:1-49). - No
kolla_actionis passed (tools/kolla-ansible:512-515). Cost is O(1) and it is safe to re-run.destroydeletes these files on the deploy host (§6).
9. mariadb_backup.yml and mariadb_recovery.yml (playbook level only)¶
# ansible/mariadb_backup.yml:1-11
- name: Backup MariaDB
hosts: mariadb
max_fail_percentage: ... # new in 2024.1, default 100
roles: [{ role: mariadb, tags: mariadb, when: enable_mariabackup | bool }]
# ansible/mariadb_recovery.yml:1-13
- name: Recover mariadb
hosts: mariadb
max_fail_percentage: ... # new in 2024.1, default 100
roles: [{ role: mariadb, tags: mariadb, when: enable_mariadb | bool }]
vars: { mariadb_recover: true }
- CLI:
mariadb_backuppasses-e kolla_action=backup -e mariadb_backup_type={full|incremental}.mariadb_recoverypasses-e kolla_action=deploy(tools/kolla-ansible:448-457).commonis its own play insite.yml(site.yml:88-103), and these two playbooks hold only the mariadb play, socommondoes not run. - Neither playbook imports
gather-facts.yml, and neither hasserial. Implicit full fact gathering covers the mariadb hosts only. - Backup runs only on
mariadb_backup_host=groups[mariadb_shard_group][0](roles/mariadb/defaults/main.yml:123,group_vars/all.yml:480), and only wheremariadb_backup_possible(roles/mariadb/defaults/main.yml:127;roles/mariadb/tasks/backup.yml:33-35). It requiresenable_mariabackup: yes(defaultno,all.yml:804). Otherwise every task is skipped silently. - Image-tag pitfall: still does not apply in 2024.1. The backup container uses the image of the running
mariadbcontainer, or the configured mariadb image (roles/mariadb/tasks/backup.yml:18-21), andopenstack_tagdoes not depend on facts registered only insite.yml(group_vars/all.yml:679-682). See Database operations for the backup and recovery tasks themselves. - CI:
mariadb_recoveryis tested with--tags mariadb --skip-tags common(tests/test-mariadb.sh:21-25).mariadb_backupis not tested in CI (grep oftests/andzuul.d/).
10. certificates.yml (playbook level)¶
# ansible/certificates.yml:1-11
- import_playbook: gather-facts.yml
when: kolla_enable_tls_backend | default(false) | bool or rabbitmq_enable_tls | default(false) | bool
or certificates_generate_libvirt | default(libvirt_tls) | default(false) | bool
- name: Apply role certificates
hosts: localhost
roles: [certificates]
- It runs on the deploy host only. Facts of all hosts are needed for the backend certificate's SAN list, which loops
over
groups['tls-backend']withkolla_address(roles/certificates/templates/openssl-kolla-backend.cnf.j2:16-18). Changed in 2024.1: the sample multinode inventory putsnetworkas well ascontrolintls-backend(ansible/inventory/multinode:42-44). - FACT: every openssl step is guarded with
creates:(for examplegenerate-backend.yml:21,32,48;generate.yml:28,39,55,100,111,127), so re-runs never regenerate. FACT: after adding atls-backendhost, the backend cert keeps its old SANs untilbackend.csr/backend.crtare deleted by hand. The SAN config is re-templated (generate-backend.yml:8-14), but the CSR and certificate steps are skipped while those files exist (:23-32,34-48; thecommandmodule exits without running when thecreatespath exists, ansible-core 2.16.20lib/ansible/modules/command.py:320-326). New in 2024.1, the external and internal certificate copies are skipped whenenable_letsencryptis on (generate.yml:69-71,80-84,141-144). The CLI marks this command "For Development Only" (tools/kolla-ansible:205).
11. tools/kolla-ansible CLI¶
--tags,--skip-tags,--limit,--forks,-e,--vault-*,--check/-Cand--diff/-Dare global options appended toEXTRA_OPTS(tools/kolla-ansible:312-399), so every subcommand accepts--tags. Whether they mean anything depends on the playbook:kolla-host.ymlonly has thebaremetaltag, anddestroy/prune-images/post-deployhave no role tags. The extra environment variableEXTRA_OPTSis honoured (:287).- Inventory:
-imay be given several times (:302-305); each value is appended as--inventory(:605-607). Without-i, no inventory is passed: the defaultINVENTORYset at:284is never passed (the name is reused as a loop variable at:463and:605), and ansible-core falls back to its configured default (/etc/ansible/hostsunless configured, ansible-core 2.16.20lib/ansible/config/base.yml:722-724). New in 2024.1,destroyalso passes the-ilist totools/cleanup-host(§6). kolla_serial:ANSIBLE_SERIAL=${ANSIBLE_SERIAL:-0}(:294-296, comment: "Serial is not recommended and disabled by default"). It is passed only byupgrade(:522) andreconfigure(:531) as-e kolla_serial=${ANSIBLE_SERIAL}. Because that-eis appended after the user's--extraoptions, Ansible's last-wins rule for extra-vars (ansible-core 2.16.20lib/ansible/utils/vars.py:202-225) meanskolla-ansible reconfigure -e kolla_serial=1silently runs with0. UseANSIBLE_SERIAL=1 kolla-ansible reconfigure.kolla-ansible upgrade -e kolla_serial=1is overridden the same way (:522); useANSIBLE_SERIAL=1there too. For every other command,-e kolla_serial=...works. Plays that ignorekolla_serialare listed below.site.yml: "Group hosts based on configuration" (:7), prechecks (:77), etcd, new in 2024.1 (:470, noserial), and cyborg, whose play readsserial: '{{ serial|default("0") }}'(site.yml:876). In the importedmariadb.ymlandrabbitmq.yml, the restart plays use a fixedserial: "33%"(mariadb.yml:24,42,rabbitmq.yml:36) and the other plays have noserial. The two imported gather-facts plays do usekolla_serial(gather-facts.yml:11,40), as do allnova.ymlplays andnova-libvirt-cleanup.yml:8.destroy.yml,mariadb_backup.yml,mariadb_recovery.yml,post-deploy.yml,certificates.yml,octavia-certificates.yml,bifrost.yml,rabbitmq-reset-state.yml,rabbitmq-upgrade.yml(grepserial:count 0).
-
Bug:
--ask-vault-pass(:391-394), still present:(--ask-vault-pass) VERBOSITY="$EXTRA_OPTS --ask-vault-pass"It should append to
EXTRA_OPTS. Effects: any-vgiven before it is lost, and allEXTRA_OPTScollected so far (including theEXTRA_OPTSenvironment variable) are repeated after the playbook path (:604). Usually harmless, because the duplicated-e,--tagsand--limitvalues are identical, but verbosity silently disappears. -
Bug:
--include-devis declaredinclude-dev:(requires an argument) inLONG_OPTS(:275) but handled as a flag withshift 1(:355-358). getopt swallows the next word as its argument.destroy --include-dev --yes-i-really-really-mean-itonly works by accident (the swallowed word is then re-parsed).destroy --include-dev -i invmis-parses, and a trailing--include-deverrors out. - Bug/limitation:
CMD="ansible-playbook $CONFIG_OPTS $EXTRA_OPTS $PLAYBOOK $VERBOSITY"runs unquoted (:148,604). It is word-split (no spaces are possible inside-eJSON values) and pathname-expanded (for example--limit compute[0-1]could glob-match a file in the current directory). - Environment check: refuses Ansible outside 2.15–2.16 (
:10-11,78-82; 2023.1 accepted 2.13–2.14) and requireskolla_ansibleimportable by Ansible's Python (:52-57). - Confirmation guard:
destroy,stopandprune-imagesrequire--yes-i-really-really-mean-it(:475,536,568).bootstrap-servers(which can restart Docker everywhere) has no confirmation, and neither hasnova-libvirt-cleanup(:577-580). - Commands, 2024.1 versus 2023.1:
monasca_cleanupis gone (no case in:438-598). New optionoctavia-certificates --check-expiry <days>(:206-207,411-414) passes-e octavia_certs_check_expiry=yes -e octavia_certs_expiry_limit=<days>(:550-556); the role then only reads the three certificates under{{ node_custom_config }}/octavia/withcommunity.crypto.x509_certificate_infoon the deploy host and fails if one expires within that many days, and skips every generation step (roles/octavia-certificates/tasks/main.yml:10-12,51;roles/octavia-certificates/tasks/check_expiry.yml:1-24).nova-libvirt-cleanuprunsnova-libvirt-cleanup.ymlonhosts: computewithkolla_serial(nova-libvirt-cleanup.yml:4-18).
12. Ansible configuration and documented tuning¶
- FACT: no
ansible.cfgships for users. Repo-wide search found only:tests/templates/ansible.cfg.j2:1-13(CI):callback_pluginsonly when the change message contains#ara(2-4), new in 2024.1callbacks_enabled = default,profile_tasks,timer(5),host_key_checking = False,inject_facts_as_vars = False;[ssh_connection] pipelining = True,retries = 3,ssh_args = -C -o ControlMaster=auto -o ControlPersist=300(10-13).contrib/dev/vagrant/bootstrap.sh:250-257(dev):forks=100,remote_user = root,scp_if_ssh=True.
- Documentation recommendations:
doc/source/user/quickstart.rstcontains noansible.cfg(grep foransible.cfg,pipelining,forksis empty); it points tokolla-ansible install-depsfor the collection (quickstart.rst:130-137).doc/source/user/ansible-tuning.rst:- SSH pipelining
[ssh_connection] pipelining = True(8-18). forks = 20as an example; "experiment to find out the CPU, memory and IO limits" (20-34).- Fact caching:
gathering = smart,fact_caching = jsonfile,fact_caching_connection = /tmp/ansible-facts, and an optionalfact_caching_timeout(36-66); populating the cache withkolla-ansible gather-factsbefore a--limitrun (68-85). inject_facts_as_vars = False(87-104). FACT: the 2024.1 playbooks, the collection roles and the filters reference facts only throughansible_facts(grep foransible_[a-z_]+other thanansible_factsreturns only magic and connection variables, kolla's ownansible_version_min/max/host, and Python names inkolla_ansible/andansible/action_plugins/), so this is safe for this tree.- Fact filtering
kolla_ansible_setup_filter: "ansible_[!qt]*", "reported to be as large as 18x" (106-129). kolla_ansible_setup_gather_subset: "all,!facter"(131-141).- New in 2024.1: "Max failure percentage",
kolla_max_fail_percentageand<service>_max_fail_percentage, undefined by default, "equivalent to a value of 100" (143-162).
- SSH pipelining
- Not documented anywhere in
doc/source: ControlPersist/ssh_args, mitogen,strategy(free/linear),ANSIBLE_SERIALorserialfor scale. The onlykolla_serialmention isbootstrap-servers.rst:83. Large-scale guidance amounts to--limitper cell (doc/source/reference/compute/nova-cells-guide.rst:376-394,426-449) and the tuning page above.
13. Day-2 and scale knobs: etc/kolla/globals.yml and ansible/group_vars/all.yml¶
Defaults live in all.yml and, for bootstrap knobs, in the ansible-collection-kolla role defaults. globals.yml shows
commented examples. Several Docker knobs are not listed in globals.yml at all (docker_log_max_*,
docker_disable_*,
docker_custom_config, docker_graceful_timeout, docker_restart_policy, docker_*_proxy, docker_registry_ca,
docker_registry_mirrors, docker_debug, and every podman_* variable; grep), but they can still be overridden there.
| Knob | Default (file:line) | Day-2 / scale relevance |
|---|---|---|
kolla_container_engine |
docker (all.yml:92; globals.yml:84-85) |
New in 2024.1. podman switches bootstrap to the podman/podman_sdk roles (§1) and every container to the Podman worker. What switching a deployed host does is NOT AVAILABLE. |
kolla_max_fail_percentage / <service>_max_fail_percentage |
undefined, read as 100 (globals.yml:31; e.g. kolla-host.yml:8-11) |
New in 2024.1. Below 100, a play stops when more than that share of the current batch fails (Controlled batches) |
kolla_ansible_setup_filter / kolla_ansible_setup_gather_subset |
omit (all.yml:31,37; globals.yml:16,) |
Fact size and time per host. Biggest win on computes with many tap/qvo interfaces. |
config_strategy |
COPY_ALWAYS (all.yml:43; globals.yml:37) |
Containers re-copy config at every start, so a container that starts again after a Docker restart or reboot applies pending rendered config (§2.4) |
openstack_release / openstack_tag / openstack_tag_suffix |
2024.1 / <release>-<distro>-<distro version><suffix> / "" (all.yml:679-682; globals.yml:43-49, whose commented openstack_tag example differs from the all.yml default) |
Image tag. Does not depend on site.yml-only facts (§9). |
docker_registry, docker_registry_insecure, docker_namespace |
quay.io / no / openstack.kolla (all.yml:98-102; globals.yml:93-100, whose commented docker_namespace example is kolla) |
insecure-registries does not follow the registry: changing docker_registry rewrites daemon.json (and restarts Docker on every host at the next bootstrap-servers) only when docker_registry_insecure is true or a docker_registry_ca is set. Not used by the podman role (§1.2). |
docker_registry_mirrors / docker_debug |
[] / false (docker/) |
New in 2024.1. In daemon.json, so setting either means a restart |
docker_log_max_file / docker_log_max_size |
"5" / "50m" (all.yml:113-114) |
daemon.json, so a change means a restart |
docker_disable_default_iptables_rules / _network / ip_forward |
"yes" x3 (the last two follow the first, all.yml:108-110) |
daemon.json, so a change means a restart. The first also drops iptables from the Docker package list (docker/). |
docker_storage_driver / docker_runtime_directory / docker_custom_config |
"" / "" / {} (docker/, all.yml:103) |
In daemon.json (restart). docker_runtime_directory also sets both volume paths, podman_volumes_path included (all.yml:156-158) |
docker_http_proxy / docker_https_proxy / docker_no_proxy |
"" (docker/) |
systemd drop-in, never applied by Kolla (§2.2) |
docker_systemd_reload |
false (docker/) |
true makes the handler reload instead of restart Docker |
docker_apt_package_pin / docker_yum_package_pin |
"" (docker/) |
Pins the Docker engine version; the package task stays state: present |
docker_restart_policy / docker_restart_policy_retry |
unless-stopped / 10 (all.yml:117,120) |
Written into each container's systemd unit (Restart=always for unless-stopped); whether containers return after a daemon restart or reboot is NOT AVAILABLE (§2.4) |
docker_graceful_timeout |
60 (all.yml:139) |
Applied: the unit's ExecStop and the Docker worker's stops use it (Repository architecture §6). The stop timeout of a daemon restart is NOT AVAILABLE (§2.4). |
docker_client_timeout |
120 (all.yml:105; globals.yml:103) |
Applied: passed in docker_common_options (all.yml:152) and used for the unit's StartLimitInterval and the wait for a unit to reach its state (module_utils/) |
docker_configure_for_zun / containerd_configure_for_zun |
no (all.yml:123,128) |
Zun hosts: no cluster-store in daemon.json any more (all.yml:125); the containerd config is overwritten and restarted |
customize_etc_hosts, create_kolla_user, disable_firewall, change_selinux/selinux_state, enable_docker_repo, virtualenv |
true, False, True, True/permissive, true (not openEuler), none (etc_hosts/, baremetal/, docker/, docker_sdk/) |
Host mutations made by bootstrap-servers (§1) |
enable_nova_libvirt_container |
true for kvm/qemu (all.yml:827) |
Removes host libvirt packages (and iscsi-initiator-utils on RedHat) and disables the libvirtd AppArmor profile during bootstrap. enable_chrony does not exist (grep). |
neutron_plugin_agent |
openvswitch (all.yml:325; globals.yml:164) |
Drives OVS and agent containers on every compute |
enable_neutron_dvr / neutron_compute_dvr_mode / computes_need_external_bridge |
no / dvr / derived (all.yml:813,; globals.yml:387) |
DVR puts L3 and metadata agents on every compute, and an external bridge per compute |
enable_neutron_agent_ha |
no (all.yml:816; globals.yml:390) |
L3/DHCP agent HA on network nodes |
enable_neutron_provider_networks |
no (all.yml:818; globals.yml:392) |
External bridge on computes |
nova_compute_virt_type |
kvm (all.yml:1127; globals.yml:605) |
nova_libvirt is enabled for kvm and qemu |
enable_nova_ssh |
yes (all.yml:829) |
nova_ssh container per compute (cold migration and resize) |
enable_cells |
no (all.yml:746; globals.yml:324) |
Cell sharding for scale, together with --limit cellNN (nova-cells guide) |
openstack_service_workers / openstack_service_rpc_workers |
min(vcpus,5) / min(vcpus,3) (all.yml:699-700) |
Caps API and RPC workers on large controllers (49 references in ansible/roles/, grep; 58 in 2023.1) |
database_max_pool_size / database_connection_recycle_time |
1 / 10 (all.yml:86-87) |
oslo.db pool per worker (used in the templates of 24 roles, grep; 29 in 2023.1) |
openstack_logging_debug |
False (all.yml:683; globals.yml:299) |
Log volume. Combine with docker log rotation. |
enable_fluentd / enable_central_logging |
yes / no (all.yml:765,747) |
Per-host fluentd container |
set_sysctl / kolla_sysctl_conf_path |
yes / /etc/ (all.yml:703,706) |
Host sysctl writes from service config-host.yml (and the one bootstrap sysctl on RedHat) |
skip_stop_containers |
[] (all.yml:876) |
Protects containers from kolla-ansible stop |
enable_destroy_images |
no (all.yml:867; globals.yml:342) |
Whether destroy also removes images |
enable_mariabackup |
no (all.yml:804; globals.yml:381) |
Must be yes or mariadb_backup silently does nothing |
kolla_enable_tls_backend |
no (all.yml:943) |
Pulls gather-facts.yml into certificates |
keepalived_virtual_router_id |
51 (all.yml:224; globals.yml:187) |
Must be unique per L2 segment in multi-region setups |
<container>_dimensions |
default_container_dimensions (all.yml:187-200: nofile 1048576 on RedHat family, now spelled RLIMIT_NOFILE/RLIMIT_NPROC under Podman, else {}), combined per role (e.g. roles/); globals.yml:188-199 |
CPU, memory and ulimit limits per container |
14. Register of open questions (NOT AVAILABLE, or answered from code)¶
| # | Question | Status | What the code shows |
|---|---|---|---|
| H1 | Does daemon-reload report changed? | Answered from code (FACT) for ansible-core 2.16.20: no | ansible-core 2.16.20 lib/ runs daemon-reload with changed=False; changed is set only at :484,530,562, so the reload handler never notifies the restart (§2.2). ansible-core 2.15 is NOT AVAILABLE. |
| H2 | Does an interrupted bootstrap leave docker masked on Debian/Ubuntu, so that docker does not start after reboot? | Symlink left in place unless the recovery block docker/ or the "Restart docker" handler (both masked: no) already ran: FACT. Effect at reboot: NOT AVAILABLE. Docker only. |
No rescue/always in the roles; a failed host runs no further task (ansible-core 2.16.20 lib/); systemd boot behaviour is not in the sources (§2.3) |
| H3 | Does one fact-less baremetal host abort bootstrap-servers on all hosts (/etc/hosts task)? |
Answered from code (FACT): yes, for a host outside bifrost with no gathered or cached facts |
Empty ansible_facts, kolla_address raises on that host's line, and every host renders every line (§3) |
| H4 | Does an unreachable non-limited host break --limit runs through delegated facts? |
Answered from code (FACT): yes, unless its facts are cached | Its delegated item returns unreachable, which removes the limited host from the run (§4) |
| H5 | What is the wall time of --limit fact gathering? |
NOT AVAILABLE | Each limited host runs about N/L delegated setup calls one after another, L hosts in parallel (§4) |
| H6 | What stop timeout does a Docker daemon restart apply to Kolla containers? | NOT AVAILABLE | The Docker path sets no StopTimeout; docker_graceful_timeout reaches the systemd units and the Docker worker's stops only (§2.4) |
| H7 | Does a simultaneous Docker restart on all controllers need mariadb_recovery? |
NOT AVAILABLE | All notified hosts restart Docker in the same handler flush; the image has no Galera recovery logic (§2.4) |
| H8 | Do QEMU guests survive a nova_libvirt container restart? | NOT AVAILABLE | pid_mode: host, cgroupns_mode: host and a /sys/fs/cgroup bind mount (§2.4) |
| H9 | Does kolla-libvirt-guests.service (optional) shut guests down on a Docker restart? |
NOT AVAILABLE | kolla-libvirt-guests-docker.service has Requires=docker.service; ExecStop runs libvirt-guests.sh shutdown (§2.4) |
| H10 | Does an option set both in the drop-in and in daemon.json make dockerd fail at the next restart? | NOT AVAILABLE | The role writes both files and restarts Docker only for daemon.json (or CA) changes (§2.2) |
| H11 | Does apparmor_parser -R fail on an already-unloaded profile (second run)? |
Does not arise (FACT): the task links the profile into /etc/ and is skipped once that link exists |
apparmor_libvirt/. Whether apparmor_parser -R fails on an unloaded profile remains NOT AVAILABLE, but the role does not run it twice (§1.3) |
| H12 | Do mariadb_backup and mariadb_recovery use a mismatched image tag? | Does not apply (FACT) | The backup uses the running mariadb container's image (roles/, group_vars/) (§9) |
| H13 | Is the backend TLS cert regenerated for new hosts? | Answered from code (FACT): no | creates: guards on the CSR and certificate steps (§10) |
| H14 | Does kolla-ansible bootstrap-servers --check --diff --limit X (or EXTRA_OPTS="--check --diff") preview daemon.json changes safely? |
NOT AVAILABLE | The CLI has --check/--diff (tools/). In check mode copy reports changed (with a diff) without writing (ansible-core 2.16.20 lib/), the systemd module runs no action (lib/), and command tasks without creates/removes are skipped (lib/); the collection modules used by the role (ufw, selinux, sysctl, authorized_key, git_config) are outside the sources |
| H15 | Do running containers see /etc/hosts updates? |
NOT AVAILABLE | The task edits the host's /etc/hosts only (§3) |
| H16 | Does the ansible_[!qt]* filter break hosts whose api/tunnel interface starts with q or t? |
Answered from code (FACT): yes | The filter is matched against fact names, and kolla_address raises for a missing interface (§4) |
| H17 | With Podman, does a change to the /etc/ drop-ins take effect, and when? |
NOT AVAILABLE. New in 2024.1 | The podman role writes them and notifies nothing (podman/) (§2.5) |
| H18 | Does destroy keep an inventory stored under /etc/kolla? |
Answered from code (FACT): only when it was passed with -i as the exact path of an entry of /etc/kolla |
tools/ (§6) |
15. Operational recommendations from this code¶
- Before any
bootstrap-serversre-run on production, render the target daemon.json and compare it with the hosts (kolla-ansible bootstrap-servers --check --diff --limit <one host>; whether check mode makes this safe is NOT AVAILABLE, H14). Expect a difference on the first 2024.1 run of a Zun compute host that hadcluster-store. If it differs, run with-e kolla_serial=1, or with--limitper controller and in compute batches, keeping Galera and RabbitMQ quorum. - Do not rely on the systemd drop-in (
docker_*_proxy, Zun options) being applied by Kolla. Put options indocker_custom_config(daemon.json) so changes are applied, and are applied visibly; schedule a controlled Docker restart after changing proxy settings. - After any failed or interrupted run on Debian/Ubuntu with Docker, check for a masked
docker.service, and compare daemon.json with the running daemon: a host that failed before the handler flush has a new daemon.json that no later run will apply. - Set
disable_firewall: false,change_selinux: falseand the package-removal lists (ubuntu_pkg_removals,redhat_pkg_removals) explicitly if the hosts are managed by other tooling. Checkopen-iscsiandiscsi-initiator-utilsbefore the first run on hosts that use host iSCSI. Setcreate_kolla_user: trueexplicitly if you rely on thekollauser. On Debian/Ubuntu with Docker, expect a deb822docker.sourcesin place ofdocker.listand an apt cache refresh on every run. - At scale, enable jsonfile fact caching with
gathering = smart,inject_facts_as_vars = Falseand a setup filter, and fill the cache withkolla-ansible gather-factsbefore--limitruns. Without them,--limitruns gather facts for the whole inventory, one host at a time per limited host. - Use
ANSIBLE_SERIAL(not-e kolla_serial) forreconfigureandupgrade. Always pass-i. - Never run
destroywithout--limit. Exclude the deploy host (--limit '!localhost') unless you want its/etc/kollawiped; an inventory under/etc/kollasurvives only when passed with-ias its exact absolute path. - With
kolla_container_engine: podman, setpodman_registry(and friends) yourself if you need a private registry configured on the hosts: kolla-ansible does not mapdocker_registryto them.
Related¶
- verification and errata for all sheets
- service index
- everything marked NOT AVAILABLE
- Database operations
- Findings F16 and F17