Repository architecture¶
This page explains how the machine is built: which entry points exist, how a CLI command becomes an
ansible-playbook run, how site.yml is laid out, what every service role has in common, and which custom modules
actually touch the hosts. The other pages build on the vocabulary defined here.
Note
Key answers
reconfigureisdeploy. Of the 59 roles that have areconfigure.yml, 56 only doimport_tasks: deploy.yml, soreconfigurere-runs Keystone registration, DB creation and the bootstrap (DB sync) container of most services CODE (§4.2).- A config change is applied by a full stop/start, never a reload. A changed file, a changed container definition,
a changed systemd unit or drift reported by
kolla_set_configs --checknotifies the restart handler, which recreates the container when the image or parameters differ and otherwise restarts its systemd unit CODE (§4.3). - The stop timeout is the 60 s of
docker_graceful_timeout. The Ussuri bug that forced 10 s is fixed: the value fromdocker_common_optionsnow reaches the module and is written into each container's systemd unit (docker stop -t 60) CODE (§6). deployandreconfigurepull an image only if it is absent locally (start_container). A rebuilt image pushed under the same tag is not fetched bydeployorreconfigure; it takes an explicitpullCODE (§6).- Batching is off by default.
ANSIBLE_SERIALdefaults to 0, which means all hosts of a play in one batch, and the wrapper passeskolla_serialonly forreconfigureandupgradeCODE (§2). commonis a play of its own, taggedcommon. No role has a meta dependency on it any more and thecommon_runguard is gone:-t common reconfigurenow renders and restarts the common containers, and-t novano longer runscommonCODE (§5.1).- Host bootstrap moved out of the repository.
bootstrap-serversrunsopenstack.kolla.baremetalfrom ansible-collection-kolla, whichkolla-ansible install-depsinstalls fromrequirements.ymlCODE (§1).
Scope: the
2023.1-eoltag ofopenstack/kolla-ansible(commit4891a21), with ansible-collection-kolla at theunmaintained/2023.1branch thatrequirements.ymlpins (studied at its2023.1-eoltag). Every path below is relative to the root of the kolla-ansible checkout; paths prefixedansible-collection-kolla,kollaoransible-core 2.14.18point into the collection, the kolla image repository or the Ansible source. The evidence labels CODE and NOT AVAILABLE are defined once, in Evidence labels.
1. Top-level layout¶
| Path | What it is | Why it matters for Day 2 |
|---|---|---|
tools/ |
Bash wrapper. Maps sub-commands to a playbook plus -e kolla_action=<action>; checks that ansible-core is between 2.13 and 2.14 (tools/) |
Decides which playbook runs and whether kolla_serial is passed (§2) |
requirements.yml |
Ansible Galaxy requirements: one collection, https://opendev.org/, type: git, version: unmaintained/ (requirements.yml:2-5) |
Installed by kolla-ansible install-deps (ansible-galaxy collection install -r requirements.yml --force, tools/). Without it bootstrap-servers cannot find its role. A branch, not a tag: what is installed is whatever the branch holds at install time |
ansible-collection-kolla (openstack.kolla, galaxy.yml:2-3) |
Separate repository, roles only (no modules or plugins): baremetal, docker, docker_sdk, etc_hosts, kolla_user, packages, apparmor_libvirt |
Holds the host-bootstrap code that was roles/baremetal in Ussuri. kolla-ansible references it in one place: kolla-host.yml:9 (role: openstack.kolla.baremetal); baremetal imports etc_hosts, packages, docker, kolla_user, docker_sdk and apparmor_libvirt (ansible-collection-kolla roles/, install.yml:34-38, post-install.yml:2-19) |
ansible/ |
The main playbook (79 plays once gather-facts.yml, mariadb.yml, rabbitmq.yml and nova.yml are expanded) |
Every deploy, reconfigure, upgrade, pull, stop, prechecks, genconfig, validate-config, deploy-containers run goes through it |
ansible/ |
Imported first by site.yml (and by kolla-host.yml) |
Fact gathering, including the special --limit path (see Compute scaling) |
ansible/, ansible/ |
Imported by site.yml:385 and :438 |
MariaDB in 5 plays, RabbitMQ and outward RabbitMQ in 3 plays each; the restart plays use serial: "33%" (mariadb.yml:20, :34; rabbitmq.yml:32, :88) |
ansible/ |
Imported by site.yml:544 |
Nova is split into 10 plays (see Nova deep dive) |
ansible/ |
bootstrap-servers: gather-facts.yml, then one play on baremetal with serial: kolla_serial applying openstack.kolla.baremetal (kolla-host.yml:1-10) |
Installs Docker and changes the host OS. Can restart the Docker daemon (ansible-collection-kolla roles/) |
ansible/, prune-images.yml, mariadb_backup.yml, mariadb_recovery.yml, post-deploy.yml, certificates.yml, octavia-certificates.yml, bifrost.yml, monasca_cleanup.yml, nova-libvirt-cleanup.yml, rabbitmq-reset-state.yml, rabbitmq-upgrade.yml |
Single-purpose playbooks | See Operations |
ansible/roles/ |
79 roles, none with a meta/ directory. 59 are applied by the plays of site.yml (56 service roles plus prechecks, common and loadbalancer); 5 by their own playbooks (bifrost, certificates, octavia-certificates, destroy, prune-images); kafka, storm, zookeeper and monasca only by monasca_cleanup.yml; 11 are helper roles imported by the others |
§4 and §5 |
ansible/ |
1,288 lines of defaults (every enable_*, ports, transport URLs, Docker options, kolla_container_engine: "docker" at :92) |
Lowest-priority defaults. globals.yml overrides them |
ansible/ |
Example inventories that define 222 groups each | Group membership drives where every container runs |
ansible/ |
Custom modules: kolla_docker, kolla_toolbox, kolla_container_facts, kolla_container_volume_facts |
Main container interface: the service roles' container lifecycle goes through kolla_docker. Many role tasks also run {{ kolla_container_engine }} exec/inspect directly (§6) |
ansible/ |
kolla_container_worker.py (shared comparison logic), kolla_docker_worker.py (Docker SDK calls), kolla_systemd_worker.py (one systemd unit per container) |
Where kolla_docker actually decides and acts (§6) |
ansible/ |
merge_configs (INI merge) and merge_yaml |
How /etc/ overrides are merged (§7) |
ansible/, kolla_ansible/ |
Jinja filters: service_enabled_and_mapped_to_host, select_services_enabled_and_mapped_to_host, kolla_address, extract_cell, customise_fluentd, database_shards_info, … |
Decide placement of a service on a host (§4.4) |
etc/, etc/ |
Templates copied to /etc/kolla/ by the operator |
globals.yml (and every *.yml in /etc/) is passed as extra-vars (§2.1) |
tools/cleanup-*, tools/, tools/ |
Host utilities | cleanup-host deletes network namespaces and links (tools/; see Risk register) |
doc/source/ |
Documentation (user/, reference/, …) |
Used only to cross-check, never as the source of truth |
No ansible.cfg is shipped for operators CODE (no ansible.cfg in the kolla-ansible tree; fact sheet E
§12). Forks,
pipelining, fact caching and SSH settings come from whatever ansible.cfg or environment the operator provides.
2. From CLI to ansible-playbook¶
tools/kolla-ansible builds a single command line (tools/kolla-ansible:583-591):
GLOBALS_DIR="${CONFIG_DIR}/globals.d"
EXTRA_GLOBALS=$([ -d "${GLOBALS_DIR}" ] && find ${GLOBALS_DIR} -maxdepth 1 -type f -name '*.yml' -printf ' -e @%p' || true 2>/dev/null)
CONFIG_OPTS="-e @${CONFIG_DIR}/globals.yml ${EXTRA_GLOBALS} -e @${PASSWORDS_FILE} -e CONFIG_DIR=${CONFIG_DIR}"
CMD="ansible-playbook $CONFIG_OPTS $EXTRA_OPTS $PLAYBOOK $VERBOSITY"
for INVENTORY in ${INVENTORIES[@]}; do
CMD="${CMD} --inventory $INVENTORY"
done
| Command | Playbook | Extra vars added | kolla_serial passed? |
Evidence |
|---|---|---|---|---|
prechecks |
site.yml |
kolla_action=precheck |
no | tools/ |
check |
– | the sub-command no longer exists; it falls to usage and exit 3 |
– | :578-580 |
bootstrap-servers |
kolla-host.yml |
kolla_action=bootstrap-servers |
no (the playbook has its own serial: kolla_serial) |
:471-475 |
deploy |
site.yml |
kolla_action=deploy |
no | :476-479 |
deploy-containers |
site.yml |
kolla_action=deploy-containers |
no | :485-488 |
pull |
site.yml |
kolla_action=pull |
no | :502-505 |
upgrade |
site.yml |
kolla_action=upgrade -e kolla_serial=${ANSIBLE_SERIAL} |
yes | :506-509 |
reconfigure |
site.yml |
kolla_action=reconfigure -e kolla_serial=${ANSIBLE_SERIAL} |
yes | :515-518 |
genconfig |
site.yml |
kolla_action=config |
no | :540-543 |
validate-config |
site.yml |
kolla_action=config_validate |
no | :544-547 |
stop |
site.yml |
kolla_action=stop (needs --yes-i-really-really-mean-it) |
no | :519-531 |
destroy |
destroy.yml |
(needs --yes-i-really-really-mean-it) |
no | :452-470 |
mariadb_recovery |
mariadb_recovery.yml |
kolla_action=deploy |
no | :437-441 |
mariadb_backup |
mariadb_backup.yml |
kolla_action=backup -e mariadb_backup_type=<full|incremental> |
no | :442-446 |
post-deploy |
post-deploy.yml |
– | no | :498-501 |
prune-images |
prune-images.yml |
(needs --yes-i-really-really-mean-it) |
no | :548-559 |
install-deps |
– | runs ansible-galaxy collection install -r requirements.yml --force and exits |
– | :131-138, :429-432 |
The 2023.1 wrapper also has deploy-bifrost, deploy-servers, upgrade-bifrost, gather-facts, certificates,
octavia-certificates, monasca_cleanup, nova-libvirt-cleanup, rabbitmq-reset-state and rabbitmq-upgrade
(:447-451, :480-497, :510-514, :532-539, :560-573), none of which passes kolla_serial.
Consequences CODE:
ANSIBLE_SERIALdefaults to 0 (tools/kolla-ansible:288-290), which means all hosts of a play in one batch. The comment there reads "Serial is not recommended and disabled by default".kolla_serialis overridden forreconfigureandupgrade. The wrapper appends-e kolla_serial=${ANSIBLE_SERIAL}after the user's-eoptions. For those two commands a user-supplied-e kolla_serial=Nis therefore silently replaced byANSIBLE_SERIAL. Use the environment variable instead (fact sheet E §11).- Other commands default to 0. For
deployand the rest,kolla_serialis not passed, and the plays fall back tokolla_serial|default("0"). --tags,--skip-tags,--limit,--forks,--checkand--diffpass straight through (:306-324,:370-378).EXTRA_OPTSfrom the environment is passed through as well (:282), which is the only way to reach--list-tasks,--list-tagsor--list-hoststhrough the wrapper.-ican be given several times, and without it no inventory is passed. Each-iadds a--inventory(:296-299, :588-590). TheINVENTORYdefault toansible/inventory/all-in-one(:279) is never used: the loop variable overwrites it, so a command without-iruns on whatever inventory the Ansible configuration names./etc/kolla/globals.d/*.ymlare extra-vars too, loaded afterglobals.ymland beforepasswords.yml(:583-586).- Bug:
--ask-vault-pass. The branch setsVERBOSITY="$EXTRA_OPTS --ask-vault-pass"(:385-388), which duplicates all options seen so far and drops earlier-vflags.
2.1 Variable precedence that matters¶
globals.yml and passwords.yml are passed with -e @file. That is extra-vars, the highest precedence in Ansible:
Ansible merges role defaults first, group vars and host vars later, and extra-vars last (ansible-core 2.14.18
lib/ansible/vars/manager.py:198-221, :290-302, :420). Consequences CODE:
- A value in
globals.ymlcannot be overridden per host or per group in the inventory. Per-host variation must come from variables that are not set inglobals.yml. Inventoryhost_varsoverride role defaults andgroup_vars/all.yml, but not extra-vars. - Anything placed in
globals.yml(orglobals.d/) applies to every play and every host. This is why "global" switches such asenable_barbican,openstack_tagornova_compute_virt_typehave fleet-wide effects (see Operations). - The same holds for the collection:
openstack.kolla.baremetaland the roles it imports read the variables of the run, so a Docker or host setting placed inglobals.ymloverrides the collection's role defaults.
3. Anatomy of site.yml¶
flowchart TB
subgraph pre["Always-run plays (tag: always)"]
GF1["gather-facts.yml play 1<br/>setup on all hosts<br/>(skipped if facts cached)"] --> GF2["gather-facts.yml play 2<br/>delegated setup when --limit is used"]
GF2 --> GB["Group hosts based on configuration<br/>group_by: kolla_action_*,<br/>enable_<svc>_True/False"]
end
GB --> PRE["Apply role prechecks<br/>(hosts: kolla_action_precheck)"]
PRE --> COM["common (cron, fluentd, kolla-logs, kolla-toolbox hosts; tag: common)"]
COM --> INFRA["loadbalancer (+44 loadbalancer includes) → collectd → influxdb → telegraf → redis<br/>→ mariadb.yml (5 plays) → memcached → prometheus → iscsi → multipathd<br/>→ rabbitmq.yml (6 plays) → etcd"]
INFRA --> CP1["keystone → opensearch → swift → ceph-rgw → glance → ironic → cinder → placement"]
CP1 --> NOVA["nova.yml (10 plays)"]
NOVA --> NET["openvswitch | ovs-dpdk → ovn-controller → ovn-db → neutron → kuryr → hacluster"]
NET --> CP2["heat → horizon → murano → solum → magnum → mistral → sahara → manila → gnocchi → ceilometer<br/>→ aodh → barbican → cyborg → designate → trove → watcher → grafana → cloudkitty → freezer<br/>→ senlin → tacker → octavia → zun → vitrage → blazar → masakari → venus → skyline"]
Facts about the structure CODE (ansible/site.yml):
- One play per service, in a fixed order. There is no dependency resolver. The deployment order is the order of
plays in
site.yml(see Execution flow). The only exception is the loadbalancer play, the "haproxy play" of Ussuri (site.yml:99-341): it runs before the services it fronts, and configures all of their frontends in one place through 44include_role … tasks_from: loadbalancer, run only fordeploy,reconfigure,upgradeandconfig(site.yml:339-341). - The "group by configuration" optimisation. The play at
site.yml:7-77puts every host into groups such asenable_nova_True. Every service play then intersects its host list with&enable_<svc>_True, for examplehosts: [compute, …, '&enable_nova_True']. If a service is disabled, the play matches no host and Ansible skips it entirely, which avoids evaluating hundreds ofwhen:clauses per host. The NOTE atsite.yml:4-6says this exists because "in large environments, even tasks that are skipped can take a significant amount of time". - Serial and tags differ between plays. Almost every play has
serial: '{{ kolla_serial|default("0") }}'.- Exceptions: the setup plays
Group hosts based on configurationandApply role prechecks(site.yml:7-84) have noserialkey. MariaDB and RabbitMQ are no longer single plays: their apply and post-configuration plays have noserialkey (all members in one batch), while their restart plays use a fixedserial: "33%"thatANSIBLE_SERIALdoes not change (mariadb.yml:14-40,rabbitmq.yml:26-41, :82-98). The handlers only sort hosts into themariadb_restart/rabbitmq_restartgroups; the restart itself happens in those plays (fact sheet B §1–2; Tags). The UssuriCheck CentOS variantplay no longer exists. - Bug:
cyborg(site.yml:784) usesserial|default("0"), notkolla_serial, soANSIBLE_SERIALhas no effect on cyborg. - Tag placement varies:
- Most plays tag the role, e.g.
roles: - { role: kuryr, tags: kuryr }. - The
commonand loadbalancer plays, themariadb.ymlandrabbitmq.ymlplays and allnova.ymlplays put tags at play level (site.yml:94-95, :105-108). memcachedhas two tags,[memcache, memcached](site.yml:394-395);ovn-controllerandovn-dbshareovn(site.yml:575-576, :587-588).- The
prechecksplay has no tag at all.
- Most plays tag the role, e.g.
- Exceptions: the setup plays
commonis a play, not a dependency. No role has ameta/directory, so nothing pullscommonin. It runs in its own play,Apply role common, on the hosts ofcron,fluentd,kolla-logsandkolla-toolbox(site.yml:86-97), which in the example inventories are children ofcommon= control, network, compute, storage, monitoring (ansible/inventory/multinode:48-53, :239-249) (§5.1).
The complete table of 79 plays (hosts, tags, serial) is generated from the source in Generated inventory §1.
4. Anatomy of a service role¶
Every service role follows the same skeleton. glance is a typical example (in 2023.1 nova keeps its bootstrap in
separate nova.yml plays, so its deploy.yml has no bootstrap step):
roles/<svc>/
defaults/main.yml <svc>_services dict (containers), images, volumes, healthchecks, ks users/endpoints, ports
vars/main.yml project_name (and, for nova-cell, kolla_role_name)
handlers/main.yml "Restart <service-key> container" handlers (kolla_docker recreate_or_restart_container)
tasks/main.yml - include_tasks: "{{ kolla_action }}.yml" <-- dynamic dispatch
tasks/deploy.yml register.yml → config.yml → check-containers.yml → bootstrap.yml → flush_handlers
tasks/reconfigure.yml - import_tasks: deploy.yml <-- reconfigure == deploy
tasks/upgrade.yml service-specific (stop/rolling, DB migrations, check-containers, flush)
tasks/deploy-containers.yml - import_tasks: check-containers.yml
tasks/pull.yml import_role: service-images-pull
tasks/stop.yml import_role: service-stop
tasks/precheck.yml import_role: service-precheck + port checks
tasks/check.yml (usually empty)
tasks/config.yml (also run alone for kolla_action=config, i.e. `genconfig`)
tasks/config_validate.yml import_role: service-config-validate (kolla_action=config_validate, `validate-config`)
tasks/copy-certs.yml import_role: service-cert-copy (from config.yml, when CA copy or backend TLS is enabled)
tasks/loadbalancer.yml import_role: loadbalancer-config (tags: always) <-- called from the loadbalancer play
templates/ <service>.json.j2 (kolla_set_configs manifest), <svc>.conf.j2, …
There is no meta/main.yml any more: no role in ansible/roles/ has a meta/ directory CODE. Evidence for glance:
roles/glance/tasks/main.yml, deploy.yml:2-15, reconfigure.yml:2, pull.yml, precheck.yml:2-6,
config.yml:38-40, loadbalancer.yml; for nova: roles/nova/tasks/deploy.yml, config_validate.yml,
vars/main.yml.
4.1 kolla_action dispatch¶
tasks/main.yml is a dynamic include_tasks: "{{ kolla_action }}.yml". Two consequences follow CODE:
--list-tasksis blind to role internals. It shows only theinclude_tasksplaceholder of every role: the listing walks the statically compiled play and does not load dynamically included files (ansible-core 2.14.18lib/ansible/cli/playbook.py:202-207; dynamic includes are loaded at run time,lib/ansible/plugins/strategy/linear.py:291). You cannot use--list-tasksto see what a tag will do in Kolla-Ansible. The linked list-tasks-nova.txt is the Ussuri parse; it shows the same placeholders.- Tags reach the role's tasks only through inheritance. The included tasks inherit the role and play tags (e.g.
nova), not tags written on theinclude_tasksstatement itself (see Tags §2). An include is not statically loaded, so the block of included tasks skips the include's tags and inherits those of the role, its dependency chain and the play (ansible-core 2.14.18lib/ansible/playbook/task_include.py:50,lib/ansible/playbook/block.py:313-361).
4.2 reconfigure is deploy¶
Of the 59 roles that have a reconfigure.yml, 56 only do import_tasks: deploy.yml. skyline does the same with
include_tasks. swift and bifrost have their own reconfigure logic: swift fails if a container is down, runs
kolla_set_configs --check and restarts only drifted containers (roles/swift/tasks/reconfigure.yml:32-50 and
following).
The bifrost copy cannot work and no kolla-ansible command runs it: it uses a group and a container named
bifrost-deploy (roles/bifrost/tasks/reconfigure.yml:2-23), but the container is bifrost_deploy
(roles/bifrost/tasks/start.yml:8), the example inventory has no bifrost-deploy group, roles/bifrost/templates/
holds no config.json template, and the wrapper sends reconfigure to site.yml, which has no bifrost play
(tools/kolla-ansible:515-518).
See the generated column "reconf→deploy" in Generated inventory
§3. So reconfigure re-runs
Keystone registration, DB creation and the bootstrap (DB sync) container of most services, exactly like deploy
CODE (e.g. roles/glance/tasks/reconfigure.yml:2, roles/glance/tasks/deploy.yml:2-15). The only built-in
differences between deploy and reconfigure are:
kolla_serialis passed fromANSIBLE_SERIAL(§2).- No
when: kolla_action == ...condition separates them any more. Every condition inansible/that namesdeployorreconfigurenames both (nova.yml:51, :87, :156;mariadb.yml:66;rabbitmq.yml:55, :111;site.yml:341). The Ussuri example is gone: Neutron's DB-sync container (bootstrap_neutron) is imported without any condition (roles/neutron/tasks/bootstrap.yml:38,roles/neutron/tasks/bootstrap_service.yml:2-21), so it runs on everydeployandreconfigure;neutron_enable_rolling_upgrade(roles/neutron/defaults/main.yml:710) is now read only byroles/neutron/tasks/upgrade.yml:5, :8. Nova's bootstrap plays run on bothdeployandreconfigure(ansible/nova.yml:47-51, :83-87).
4.3 The config → check → handler pattern¶
This is the heart of Day-2 behaviour:
sequenceDiagram
autonumber
participant DH as Deploy host (ansible-playbook)
participant H as Target host
participant C as Container
DH->>DH: render template / merge_configs locally (Jinja + INI merge)
DH->>H: copy module: compare checksum of /etc/kolla/<svc>/<file>
alt content differs
DH->>H: write file (changed) → notify "Restart <svc> container"
end
DH->>H: kolla_docker compare_container (check-containers.yml)
H->>C: docker inspect (image id/tag, volumes, env, privileged, pid/ipc/cgroupns, caps, dimensions, labels, command, healthcheck, state)
H->>C: docker exec kolla_set_configs --check (config drift between /var/lib/kolla/config_files and live files)
H->>H: compare /etc/systemd/system/kolla-<name>-container.service with the unit it would write
alt differs or drift
DH-->>DH: notify "Restart <svc> container"
end
Note over DH,H: meta: flush_handlers (end of deploy.yml) or end of play
DH->>H: handler: kolla_docker recreate_or_restart_container
alt image or container params differ
H->>C: pull image if absent → systemd stop (docker stop -t 60) → rm → create + unit → systemd start (RECREATE)
else only config changed (COPY_ALWAYS)
H->>C: rewrite unit if needed → systemd RestartUnit (docker stop -t 60, docker start) (RESTART)
end
- Files are rendered on the deploy host and copied. The
templateandmerge_configssteps render on the deploy host, then thecopylogic compares checksums on the target. So every config task costs one SSH round trip per host per item, even when nothing changes (see Compute scaling). -
Three independent triggers lead to the same handler. A changed file notifies directly. The
compare_containertask (module_utils/kolla_container_worker.py:59-66) also notifies when:- the container's definition differs (
check_container_differs,kolla_container_worker.py:68-90: capabilities, security options, image ID or tag, ipc/pid/cgroupns mode, labels, privileged, tmpfs, volumes, environment, state, dimensions, command, healthcheck); or kolla_set_configs --checkinside the running container reports drift (module_utils/kolla_docker_worker.py:155-190; a container that cannot be exec'd into counts as drifted,:164-171); or- the container's systemd unit file is missing or differs from the one the module would write
(
module_utils/kolla_systemd_worker.py:149-162), for example afterdocker_graceful_timeout,docker_restart_policyordocker_client_timeoutchanged, since all three are written into the unit (kolla_systemd_worker.py:20-40, :50-74).
The
--checkpath means a config file changed by an earliergenconfig, or by hand, will still restart the container on the nextdeploy/reconfigure/deploy-containersCODE. - the container's definition differs (
-
Restart and recreate are both full stop/start cycles. A config change is applied only by stopping and starting the container;
kolla_dockerhas no reload action (library/kolla_docker.py:263-279). The only signal Kolla-Ansible sends to a running containerised service is SIGHUP tonova_scheduler(cell-cache refresh), throughkill -HUPon the PID read with{{ kolla_container_engine }} inspect. It is sent only on adeploy/reconfigurethat runs thenova,nova-apiornova-refresh-scheduler-cell-cachetag (or no tag) and whose--limit, if any, includes thenova-schedulerhosts (roles/nova/tasks/refresh_scheduler_cell_cache.yml:10-12,ansible/nova.yml:140-156). It loads no new config files. (Outside containers, theReload libvirtdhandler reloads a host libvirtd when the external-Ceph tasks change its secrets andenable_nova_libvirt_containeris false,roles/nova-cell/handlers/main.yml:142-148,roles/nova-cell/tasks/external_ceph.yml:191-194.) The handler callsrecreate_or_restart_container(module_utils/kolla_docker_worker.py:362-384):- recreate when the image or parameters differ (or
COPY_ONCE): pull the image if absent, stop, remove, start; - otherwise restart: rewrite the unit if needed and ask systemd to restart it (
restart_container,:460-480;kolla_systemd_worker.py:101-108).
The stop timeout is
docker_graceful_timeout, 60 s by default (§6). - recreate when the image or parameters differ (or
-
Handlers run in definition order, on every notified host of the batch at once. Unless
kolla_serialis set, "Restart nova-compute container" restarts nova_compute on all notified computes in the same step CODE (ansible-core 2.14.18lib/ansible/executor/play_iterator.py:197copies the play's handlers in definition order and:430-453walks them per host; the linear strategy runs the handler phase in lockstep across the hosts of the batch,lib/ansible/plugins/strategy/linear.py:78-123).
4.4 Containers and placement¶
Each role's defaults/main.yml defines a <role>_services dict with an entry per container. The fields are
container_name, group or host_in_groups, enabled, image, volumes, dimensions, privileged,
pid_mode/ipc_mode/cgroupns_mode, healthcheck and haproxy. The filter service_enabled_and_mapped_to_host
(kolla_ansible/filters.py:59-96) decides placement:
host_in_groups(a per-host boolean expression) wins overgroup(filters.py:75-77).group: allmeans every host (filters.py:79-81).
select_services_enabled_and_mapped_to_host (filters.py:99-109) applies the same test to a whole dict; the helper
roles service-images-pull, service-config-validate and others loop over its result.
All 188 container definitions are listed in Generated inventory §2.
5. Shared helper roles¶
| Role | Used by | What it does | Scope |
|---|---|---|---|
common |
its own play (site.yml:86-97); no role depends on it |
Deploys fluentd, kolla_toolbox, cron on the hosts of their groups (children of common: every host of the example inventory). Renders td-agent.conf and logrotate.conf |
Once per host per run that executes the common play (§5.1) |
service-ks-register |
37 roles | Creates the Keystone service, endpoints, project, user, role and grants through kolla_toolbox |
run_once: once per play, or once per batch when kolla_serial is set (roles/) |
service-rabbitmq |
nova-cell only (roles/, imported from bootstrap.yml:37) |
Creates RabbitMQ vhost and user through kolla_toolbox, delegated to the first host of the cell's RPC (or notify) RabbitMQ group |
once per cell, on the cell's first nova-conductor host (the notify pass only when its group or users differ from RPC) |
service-images-pull |
the pull.yml of 55 roles |
kolla_docker pull_image for every enabled container mapped to the host, with retries (roles/) |
per host × per container |
service-stop |
the stop.yml of 56 roles |
kolla_docker stop_container, honouring skip_stop_containers |
per host × per container |
service-precheck |
the precheck.yml of 54 roles |
Validates that the inventory groups exist | per host |
service-cert-copy |
copy-certs.yml of 43 roles |
Copies the CA (and backend certs) into each service config dir when kolla_copy_ca_into_containers or the service's backend TLS is set (e.g. roles/) |
per host × per service |
service-config-validate |
config_validate.yml of 34 roles |
validate-config: runs the oslo-config validator for each enabled service mapped to the host (roles/) |
per host × per service |
loadbalancer-config |
every role's loadbalancer.yml (44 roles) |
Imports haproxy-config (when haproxy_config_install) and proxysql-config (when enable_proxysql) (roles/) |
per loadbalancer host |
haproxy-config |
loadbalancer-config only |
Renders /etc/ (custom templates under /etc/ win) and opens external ports in firewalld when enabled (roles/) |
per loadbalancer host |
proxysql-config |
loadbalancer-config only |
Renders the ProxySQL users and rules files for the service | per loadbalancer host |
module-load |
nova-cell (br_netfilter), neutron, openvswitch, loadbalancer, ironic, iscsi (configfs), multipathd, from each role's config-host.yml |
modprobe and persistence in /etc/ |
per host |
5.1 The common role and the common_run guard¶
The guard no longer exists. roles/common/tasks/main.yml:1-2 is now only:
- include_tasks: "{{ kolla_action }}.yml"
No file in ansible/ or tools/ mentions common_run, and roles/common/defaults/main.yml starts directly with
common_services CODE. What replaced it:
commonruns in one play of its own (site.yml:86-97: hostscron,fluentd,kolla-logs,kolla-toolbox;serial: kolla_serial; play tagcommon). It runs before the loadbalancer play and before any service play, on every host of those groups, once per run.-t commonselects that play and nothing else. The included tasks inherit the play tagcommon(ansible-core 2.14.18lib/ansible/playbook/block.py:313-361,lib/ansible/plugins/strategy/linear.py:311), so-t common reconfigurerenders the common configs and restarts what changed CODE. The Ussuri behaviour, where it rendered and restarted nothing, is gone.- Service tags no longer drag
commonin. No role has ameta/directory, so-t nova(or any service tag) skips the common play, which carries onlycommonCODE (Tags §2.1). The Ussuri behaviour, where-t novaran the wholecommonrole on every nova host, computes included, is gone. -
fluentdandcronconfigs are global.td-agent.confandlogrotate.confare rendered from the globalenable_*flags (roles/common/tasks/config.yml:76-209). Enabling a service listed influentd_input_openstack_services(roles/common/defaults/main.yml:138-202) and/orcron_logrotate_services(roles/common/tasks/config.yml:138-209), such as nova, neutron or barbican, changestd-agent.confand/orlogrotate.confon every host. Services only in the logrotate list (e.g. mariadb, rabbitmq, zun, vitrage, placement, prometheus) change onlylogrotate.conf, and some flags change neither file (e.g.enable_redis,enable_memcached,enable_ovn).Since
commonno longer runs with a service tag, enabling a service with-t <svc> deploydoes not update these files; they change on the next run that executes the common play (a full run or-t common). That run restarts the container whose file changed (fluentdand/orcron). Through loop-notify on the loopedCheck common containerstask (roles/common/tasks/check-containers.yml:2-17) it also restarts the other common containers (kolla_toolbox, plusfluentdorcronwhen only one of the two files changed), although their config did not change CODE (when any item of a looped task changes, Ansible notifies the handlers of every item that was not skipped, ansible-core 2.14.18lib/ansible/executor/task_executor.py:124-128, :804-805,lib/ansible/plugins/strategy/__init__.py:614-621; see Tags §2.4).
6. Custom modules — container lifecycle via kolla_docker¶
| Module | Purpose | Key behaviour (evidence) |
|---|---|---|
kolla_docker (ansible/, 416 lines, plus module_utils/ 524, kolla_container_worker.py 570, kolla_systemd_worker.py 211) |
Every container lifecycle operation of the service roles (390 kolla_docker: tasks under ansible/roles/) |
Actions: compare_container, compare_image, create_volume, ensure_image, get_container_env, get_container_state, pull_image, recreate_or_restart_container, remove_container, remove_image, remove_volume, restart_container, start_container, stop_container, stop_and_remove_container (kolla_docker.py:263-279). Uses host networking (kolla_docker_worker.py:286-288). Docker only: the module imports only DockerWorker (kolla_docker.py:22) and the unit template calls /usr/bin/docker (kolla_systemd_worker.py:32-33). Every non-oneshot container gets a systemd unit kolla-<name>-container.service that starts, stops and restarts it (below) |
kolla_toolbox (library/) |
Runs an Ansible module inside the kolla_toolbox container (exec_create of ansible localhost -m <module>, kolla_toolbox.py:115-116, :182, :219) |
Used for MySQL, Keystone (os_*) and RabbitMQ (rabbitmq_user/) operations. Takes a required container_engine parameter (:147). Needs a running kolla_toolbox on the target |
kolla_container_facts |
Returns the running containers by name; takes container_engine |
Used by prechecks and upgrade logic (e.g. roles/) |
kolla_container_volume_facts |
Returns container volume facts | New in 2023.1; used by roles/ and roles/ |
The Ussuri bslurp module is gone from ansible/library/. Swift rings are copied with copy from
{{ node_custom_config }}/swift (roles/swift/tasks/config.yml), and Fernet keys are distributed by fernet-push.sh,
run with {{ kolla_container_engine }} exec in the keystone_fernet container
(roles/keystone/tasks/distribute_fernet.yml:14).
These modules are not the only container interface. Many role tasks call the container CLI directly through
command/shell as {{ kolla_container_engine }} exec|inspect (20 roles, among them keystone, mariadb, rabbitmq,
openvswitch, ovn-db, nova-cell, swift and bifrost): fernet-push.sh, ovs-vsctl, kolla_set_configs --check
(swift, bifrost), the mysql/rabbitmqctl calls of the MariaDB and RabbitMQ roles (e.g.
roles/mariadb/tasks/restart_services.yml:34), and inspect plus kill -HUP for nova_scheduler. Outside the
service roles, destroy stops and removes containers with the tools/cleanup-containers script, passing the engine
(roles/destroy/tasks/cleanup_containers.yml:3; the script accepts docker or podman,
tools/cleanup-containers:4-10); prune-images uses the docker_prune module
(roles/prune-images/tasks/prune_images.yml:2-8), which is not part of ansible-core 2.14.18 (lib/ansible/modules/
has no docker module) and is not in requirements.yml, so it must come from the operator's Ansible installation
CODE; and bootstrap-servers runs docker start on the previously running containers after a Docker package
upgrade (ansible-collection-kolla roles/docker/tasks/install.yml:44-72) CODE.
How kolla_docker decides CODE:
start_containerpulls the image only if it is absent locally (module_utils/kolla_docker_worker.py:386-388). A rebuilt image pushed under the same tag is therefore not fetched bydeployorreconfigure. It takes an explicitpull(kolla-ansible pull) to download it, after whichcompare_imagesees a new image ID and recreates the container (kolla_docker_worker.py:116-131, :362-384). It then creates the container and its unit if needed and starts it through systemd, failing with "Container timed out" if the unit is notrunningwithinclient_timeout(120 s) (:386-410;kolla_systemd_worker.py:92-99, :200-211). One-shot containers (restart_policy: oneshot) get no unit and are started directly (:402-403).compare_imagecompares image ID and tag, using the local image for that reference (kolla_docker_worker.py:116-131).pull_imagereports changed only if the local image ID changed (kolla_docker_worker.py:198-232).docker_graceful_timeoutanddocker_client_timeoutnow take effect.generate_module()starts fromcommon_options(=docker_common_options, which carriesgraceful_timeout: "{{ docker_graceful_timeout }}"andclient_timeout: "{{ docker_client_timeout }}",group_vars/all.yml:140-153). It overwrites a key only when the task sets that parameter; a parameter the task leaves unset keeps thecommon_optionsvalue, and only when that is absent too does the module default (10 s and 120 s) apply (kolla_docker.py:351-382).graceful_timeoutandclient_timeouthave no default in the argument spec any more (kolla_docker.py:303, :325). Handlers passcommon_options(e.g.roles/nova-cell/handlers/main.yml:74-168), so the effective stop timeout isdocker_graceful_timeout, 60 s by default (group_vars/all.yml:140).- The value is written into the unit:
ExecStop=/usr/bin/docker stop <name> -t <graceful_timeout>(kolla_systemd_worker.py:33, :69).stop_containeruses systemd when the unit exists anddocker stopwith the same timeout otherwise (kolla_docker_worker.py:436-452);restart_containerasks systemd to restart the unit (:460-480). - Practical effect: a container still running 60 s after SIGTERM is killed. This holds for handler restarts,
recreates and
stopCODE (the SIGTERM-then-SIGKILL meaning of the timeout is kolla-ansible's own comment,ansible/group_vars/all.yml:139: "Timeout after Docker sends SIGTERM before sending SIGKILL."). The unit sets noTimeoutStopSec, so systemd's own stop timeout also applies; its value is host configuration NOT AVAILABLE.
- The restart policy lives in the unit, not in Docker.
build_host_configpasses no restart policy to Docker (kolla_docker_worker.py:286-321);docker_restart_policy: "unless-stopped"(group_vars/all.yml:117) becomesRestart=alwaysin the unit, withStartLimitBurst=docker_restart_policy_retry(kolla_systemd_worker.py:26-34, :50-74). The FIXME "restart_policy is not checked in the container" (kolla_docker_worker.py:15) remains, but a changed policy changes the unit, whichcompare_containerdoes detect (§4.3). kolla_container_engineis not an engine switch for the lifecycle. It defaults to"docker"(group_vars/all.yml:92) and is substituted into CLI calls and passed tokolla_toolboxand the facts modules, butkolla_dockeronly drives Docker and no file underansible/mentionspodmanCODE.
7. Configuration generation and overrides¶
- INI files go through
merge_configs.merge_configs(ansible/action_plugins/merge_configs.py) renders each listed source as a Jinja template on the deploy host, merges them as INI (later sources win), then calls thecopyaction. For nova-compute the sources are (roles/nova-cell/tasks/config.yml:69-89):templates/nova.conf.j2→/etc/kolla/config/global.conf→/etc/kolla/config/nova.conf→/etc/kolla/config/nova/nova-compute.conf→/etc/kolla/config/nova/<inventory_hostname>/nova.conf. - Overrides are scoped by file location. Putting a value into
/etc/kolla/config/nova.confaffects every nova container that has anova.confon every host (nova-api, scheduler, super-conductor, conductor, novncproxy and every nova-compute). The content of/etc/kolla/config/nova/nova-compute.confis merged only into nova-compute'snova.conf, so the control-plane nova services are untouched CODE (thenovaandnova-cellroles each mergenova/<service>.confonly into that service's ownnova.conf,roles/nova/tasks/config.yml:64-82,roles/nova-cell/tasks/config.yml:69-89,roles/nova/defaults/main.yml:2-57,roles/nova-cell/defaults/main.yml:478-484). Operationally,-t nova reconfigureafter changing it still restartsnova_compute,nova_libvirtandnova_sshon every compute host CODE. The changed file makeskolla_set_configs --checkreport drift innova_compute(module_utils/kolla_docker_worker.py:155-190). libvirt and ssh restart through loop-notify in the looped "Check nova-cell containers" task (roles/nova-cell/tasks/check-containers.yml:2-22, handlersroles/nova-cell/handlers/main.yml:74-168; Ansible lines in §5.1), not because their config changed. - Some overrides leak to other services. In Ussuri, Neutron's
openvswitch_agent.inimerged/etc/kolla/config/neutron/ml2_conf.ini. In 2023.1 it no longer does:ml2_conf.ini(template plusneutron/ml2_conf.inioverrides) is rendered only forneutron-serverandneutron-infoblox-ipam-agent(roles/neutron/tasks/config.yml:158-178), andopenvswitch_agent.inimerges only its own template andneutron/openvswitch_agent.inioverrides (:197-206). A change meant forneutron-servertherefore no longer restartsneutron_openvswitch_agentCODE (Reconfigure Neutron). The file-location scoping of point 2 is still the rule:/etc/kolla/config/neutron.conforglobal.confreach every container whose template merges them. -
The container receives its config through a manifest. The
config.json(rendered from<service>.json.j2) tellskolla_set_configs, inside the image, which files to copy from/var/lib/kolla/config_files(bind-mounted from/etc/kolla/<service>/) to their final location. It also names the command to start. Withconfig_strategy: COPY_ALWAYS(group_vars/all.yml:43) the copy happens at every container start. That is why a restart is enough to apply a changed or added config file, and why Kolla-Ansible never applies config by reloading a service.In 2023.1 a restart also applies most removals CODE. Before copying,
kolla_set_configs(kolla 2023.1-eol) reads a state file kept inside the container,/etc/kolla/defaults/state, and for every destination it has ever written either restores the image's original file (saved under/etc/kolla/defaults/) or deletes it when the image had none; it then records any new destinations (kolladocker/base/set_configs.py:32-33, :452-474, :477-513, :516-532, :535-553). So a file whose entry was dropped fromconfig.json(e.g. a removed Keystone policy file,roles/keystone/templates/keystone.json.j2:31-37), or whose optional source is gone (the copy then skips it,set_configs.py:143-145), returns to the image default or disappears at the next start. Limits:- The state lives in the container's own filesystem, so a recreate starts with an empty state. A stale file on a
named volume (e.g. nova-compute's SASL
auth.confunder/var/lib/nova,roles/nova-cell/templates/nova-compute.json.j2:58-61) whose entry was dropped is then neither restored nor deleted. kolla_set_configs --checkcompares only the entries of the currentconfig.json(set_configs.py:556-559, :220-243), so a dropped entry alone does not report drift; the restart comes from the changedconfig.jsonfile itself.- Deleting a custom file such as
/etc/kolla/config/keystone/keystone-paste.inileaves its rendered copy in/etc/kolla/keystone/(roles/keystone/tasks/config.yml:169-193), and the optional manifest entry (keystone.json.j2:18-24) keeps copying it in, even after a recreate.
The image-side start path and
--checkare described in Kolla images §2-§3. - The state lives in the container's own filesystem, so a recreate starts with an empty state. A stale file on a
named volume (e.g. nova-compute's SASL
8. Inventory model¶
- Groups are mostly aliases of a few physical groups. The physical groups are
control,network,compute,storage,monitoringanddeployment(ansible/inventory/multinode:3-33). Service groups are defined as children. Examples:[nova-api:children] nova,[nova:children] control(multinode:263-264, :117-118).[loadbalancer:children] network(multinode:86-87). In the 2023.1 example inventory, HAProxy and keepalived sit on the network nodes; there is nohaproxygroup any more.[openvswitch:children] network, compute, manila-share(multinode:123-126).[common:children]= control, network, compute, storage, monitoring, andcron,fluentd,kolla-logs,kolla-toolboxare children ofcommon(multinode:48-53, :239-249).
- Groups that include
computein the example inventory (33 in total, counted fromansible/inventory/multinode):computeitself and the groups that list it directly as a child:baremetal,common,collectd,telegraf,hacluster-remote,monasca-agent,openvswitch,cyborg,neutron-ovn-metadata-agent,neutron-ovn-agent,iscsid,ceilometer-compute,ceilometer-ipmi,cyborg-agent,multipathd,zun-compute,zun-cni-daemon,prometheus-node-exporter,prometheus-cadvisor,prometheus-libvirt-exporter,masakari-instancemonitor,ovn-controller-compute.- Through nesting:
cron,fluentd,kolla-logsandkolla-toolbox(viacommon),cyborg-apiandcyborg-conductor(children ofcyborg, which is control + compute, so they also land on compute hosts),monasca-agent-collector,monasca-agent-forwarderandmonasca-agent-statsd(viamonasca-agent), andovn-controller(viaovn-controller-compute).
- The neutron play also targets every compute host. It lists
computeandmanila-shareexplicitly (site.yml:590-608); the agents'host_in_groupsdecides what lands there. - Inventory order is semantically significant CODE.
groups['<grp>'][0]is the target of almost everyrun_once/delegate_tobootstrap step. Examples:groups[nova_cell_conductor_group][0]for cell creation and compute discovery (roles/nova-cell/tasks/bootstrap.yml:15, :34); the firstglance-apihost as the only glance host with the default file backend (group_vars/all.yml:915; fact sheet C). Other index-sensitive logic:- keepalived priority is the index in
groups['loadbalancer']plus 1 (roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15; fact sheet A §1.8 C); - HAProxy's single active MariaDB backend is the first host of the default shard, the others being
backup(roles/mariadb/defaults/main.yml:64). The Ussuri modulo-4 restart waves are gone: MariaDB restarts in theserial: "33%"batches of theRestart mariadb servicesplay (mariadb.yml:14-26); in which batch the active backend restarts is NOT AVAILABLE (R9; fact sheet B §1.5).
- keepalived priority is the index in
Related¶
- Tags (how tags select this work)
- Execution flow (execution order and the action matrix)
- Generated inventory (every play, container, role and handler)
- Tag index, Components and Glossary (tag index, service index, glossary)
To remember
Lumo: if you keep only 5 things from this page, keep these.
reconfigureisdeploy: it re-runs Keystone registration, DB creation and the bootstrap (DB sync) container of most services. CODE- A config change is applied by a full stop/start, never a reload: the restart handler recreates or restarts the container. CODE
- The stop timeout is the 60 s of
docker_graceful_timeout, written into each container's systemd unit (docker stop -t 60). CODE deployandreconfigurepull an image only if it is absent locally; a rebuilt image pushed under the same tag takes an explicitpull. CODE- Batching is off by default:
ANSIBLE_SERIALdefaults to 0, which means all hosts of a play in one batch. CODE
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.