Kolla images: start path and one-shot containers¶
This page answers open question Q14 of the Open
questions: what the kolla images do when a container starts, and what the
one-shot containers that Kolla-Ansible starts with KOLLA_BOOTSTRAP, KOLLA_UPGRADE, KOLLA_OSM and similar variables
actually run. It was read from the kolla image repository (openstack/kolla, tag 2023.1-eol, commit de82941) and
matched against Kolla-Ansible 2023.1-eol (4891a21). The images were read, not built or run.
Note
Key answers
- What every container start does:
kolla_startrunskolla_set_configs,kolla_copy_cacertsand the image'skolla_extend_start, then execs the service command (§2.2). In 2023.1 it runs witherrexit,pipefailandnounset. kolla_set_configs --checkcompares only theconfig_filesentries ofconfig.json(content, mode, owner) CODE (§3.1). It never comparescommand,permissionsor removed entries (§3.2). Exit codes 0, 1, 2 and 137, and whatkolla_dockerdoes with each: §3.3. A newly renderedoptionalfile is now drift (exit 1), not an error. Run by hand, it needsdocker exec -u root(§3.4).- What a
COPY_ALWAYSrestart applies: changed or added files, mode and owner changes and a changed command CODE. New in 2023.1,kolla_set_configsrecords the state of each destination at the container's first start and, at every later start, deletes or restores those destinations before copying: most removals now apply too, with exceptions (§3.5). - One-shot containers: what
KOLLA_BOOTSTRAP,KOLLA_UPGRADE,KOLLA_OSMand the other variables run in each image, and which Kolla-Ansible task sets them (§4). - Per-image start behaviour: the table in §5. The nova-libvirt start path is
passive towards running guests (§5.1); with
enable_neutron_agent_ha, the L3 agent wrapper runsneutron-netns-cleanup --force --agent-type l3on every container start (§5.3). - Day-2 consequences: 19 items (§6). What is not available from the sources: §7.
Citation convention: kolla image paths carry a kolla prefix (kolla docker/base/start.sh:11). Kolla-Ansible paths
have
no prefix and are relative to its ansible/ directory (roles/nova/tasks/bootstrap_service.yml:11,
module_utils/kolla_docker_worker.py:155-190, nova.yml:45-61), except tools/.... Ansible source paths carry an
ansible-core 2.14.18 prefix. Evidence labels are those of Conventions and evidence
labels: CODE (a kolla or Ansible path counts as CODE) and NOT AVAILABLE
(the sources do not establish it).
1. Scope and method¶
- Read in full CODE: kolla
docker/base(Dockerfile.j2,start.sh,set_configs.py,copy_cacerts.sh,httpd_setup.sh,sudoers,healthcheck_*), and theDockerfile.j2and start scripts of the images in §5: Keystone, Glance, Placement, Cinder, Heat, Horizon, Barbican, Octavia, Manila, Swift, Nova, Neutron, Open vSwitch, OVN, MariaDB (server and clustercheck), RabbitMQ, memcached, HAProxy, keepalived (withcheck_alive.sh), fluentd, cron, kolla-toolbox, iscsid and tgtd. The chrony image and role of Ussuri do not exist in 2023.1 (nodocker/chronyin kolla, noroles/chronyin Kolla-Ansible). Of the other images only theironic-apibootstrap branch (§4) and thebifrost-deployENTRYPOINT/CMD(§2.1) were read, and every Dockerfile was searched forHEALTHCHECK(§2.7) and forENTRYPOINT/CMDoverrides. The rest were not. - Matched to Kolla-Ansible. Each one-shot variable was traced to the task that sets it
(§4) CODE. In 2023.1
kolla_dockerdrives long-running containers through systemd units (module_utils/kolla_systemd_worker.py:20-74); this page cites that path where it changes what a start or stop does. kolla_set_configs --checkwas read, not run: exit codes 0, 1 and 2 of §3.3 are read from kolladocker/base/set_configs.py; 137 and the exec client error arekolla_dockeroutcomes (module_utils/kolla_docker_worker.py:155-190).- Not covered: behaviour against real services and data, signal delivery, timing. These are NOT AVAILABLE and are listed in §7.
2. Start path common to every image¶
2.1 Image wiring¶
Every kolla image inherits from kolla docker/base/Dockerfile.j2 CODE:
ENTRYPOINT ["dumb-init", "--single-child", "--"]whenuse_dumb_initis on (:352-354; default True, kollakolla/common/config.py:151), andCMD ["kolla_start"](:382). The MariaDB server image drops--single-child(§2.6). bifrost-deploy resets both (ENTRYPOINT [],CMD [ "/sbin/init" ], kolladocker/bifrost/bifrost-deploy/Dockerfile.j2:29-30), sobifrost_deploynever runskolla_startorkolla_set_configs.- The base scripts are installed as
kolla_start(start.sh),kolla_set_configs(set_configs.py),kolla_copy_cacerts(copy_cacerts.sh) andkolla_httpd_setup(httpd_setup.sh);sudoersbecomes/etc/sudoers, mode 0440 (:346-350,366-368). /usr/local/bin/kolla_extend_startis created empty bytouch(:365). An image gets start logic only by overwriting it./var/log/kollagets groupkollaand mode 2775 (:369-371).- Service users are added to group
kolla(kolladocker/macros.j2:50). The basesudoerslets members of that group run two commands as root without a password, keeping their environment:kolla_set_configsandkolla_copy_cacerts(kolladocker/base/sudoers:7,18,21). New in 2023.1, it sets asecure_paththat puts/var/lib/kolla/venv/binfirst (:9). It also includes/etc/sudoers.d(:23), where many images add entries of their own, e.g. cinder-volume'schown -Randchmod(kolladocker/cinder/cinder-volume/cinder_sudoers:2-5) and mariadb'skolla_security_reset(kolladocker/mariadb/mariadb-server/mariadb_sudoers:1). Thekolla_set_configsentry restricts no arguments, sosudo kolla_set_configs --checkis allowed. - Kolla-Ansible passes
KOLLA_CONFIG_STRATEGYto every container throughdocker_common_options(group_vars/all.yml:43,143-149; defaultCOPY_ALWAYS).
2.2 kolla_start step by step¶
flowchart TB
D["dumb-init --single-child (PID 1)"] --> KS["kolla_start (errexit, pipefail, nounset, xtrace)"]
KS --> SC["sudo -E kolla_set_configs"]
SC --> ST{"KOLLA_CONFIG_STRATEGY"}
ST -- "COPY_ALWAYS" --> CA["restore defaults + copy + permissions + /run_command (every start)"]
ST -- "COPY_ONCE, first start" --> CO["restore defaults + copy + permissions + /run_command, create /configured"]
ST -- "COPY_ONCE, /configured exists" --> SK["copy nothing, exit 0"]
ST -- "invalid strategy or copy error" --> F1["exit 1 or 2: start aborts"]
CA --> RC["CMD = cat /run_command"]
CO --> RC
SK --> RC
RC --> CC["sudo kolla_copy_cacerts"]
CC --> SE{"KOLLA_SKIP_EXTEND_START set?"}
SE -- "yes" --> EX
SE -- "no" --> XS["source kolla_extend_start (image USER)"]
XS --> OS{"one-shot variable set?"}
OS -- "yes" --> Z["run DB or setup commands, exit 0"]
OS -- "no" --> EX["exec CMD ARGS"]
-
set -o errexit; set -o pipefail; set -o nounset; set -o xtrace(kolladocker/base/start.sh:3-6). Any failing command ends the container, including commands in the sourcedkolla_extend_start;pipefailandnounsetare new in 2023.1, so a failing command inside a pipeline, or a reference to an unset variable, also ends it. The 2023.1 start scripts give defaults to the variables Kolla-Ansible does not always set (e.g. kolladocker/keystone/keystone/extend_start.sh:21,docker/neutron/neutron-base/extend_start.sh:19-22,docker/horizon/extend_start.sh:5) CODE. xtrace is on in that script, and in thekolla_<service>_extend_startit sources (§2.5), where one-shots put a secret on a command line: heat's--password ${HEAT_DOMAIN_ADMIN_PASSWORD}(kolladocker/heat/heat-api/extend_start.sh:12) and the MariaDB upgrademysql_upgrade ... --password="${DB_ROOT_PASSWORD}"(kolladocker/mariadb/mariadb-server/extend_start.sh:55). The MariaDB bootstrap now turns xtrace off around its root-password commands (set +o xtrace...set -o xtrace,:24-28) CODE. NOT AVAILABLE (bash semantics, not in these sources) Whether the trace on stderr shows the remaining commands with their variables expanded, and so writes the secret there.kolla_dockerreturns the container's stderr in the task result when it waits for the container (module_utils/kolla_docker_worker.py:412-434,library/kolla_docker.py:408-412). Ansible's default callback prints a failed result,stderrincluded, and an ok or changed one only with-v(ansible-core 2.14.18lib/ansible/plugins/callback/default.py:50-68,73-106,lib/ansible/plugins/callback/__init__.py:200-202); a task withno_logreaches the callback with its result replaced by acensoredmessage (ansible-core 2.14.18lib/ansible/executor/task_queue_manager.py:436,lib/ansible/executor/task_result.py:131-139).bootstrap_heatsets nono_log(roles/heat/tasks/bootstrap_service.yml:2-30);upgrade_mariadbdoes (roles/mariadb/tasks/post-upgrade.yml:2-24) CODE. -
sudo -E kolla_set_configs(:11) runs as root whatever the imageUSER, copies the configuration (§2.3) and writesconfig.json'scommandto/run_command. CMD=$(cat /run_command)andARGS=""(:12-13).sudo kolla_copy_cacerts(:16, §2.4), on every start whatever the strategy.- Unless
KOLLA_SKIP_EXTEND_STARTis set,. kolla_extend_start(:18-21). The script is sourced: it runs in the same shell, as the imageUSER, with errexit on. It can setARGSorCMD, orexit. Anexit 0ends the container before the service starts, which is how every one-shot branch in §4 works. The test is${!KOLLA_SKIP_EXTEND_START[@]}, the idiom that kolla's own comments describe as catching "all cases of the ... variable being set, including empty" (kolladocker/keystone/keystone/extend_start.sh:41-42), so any value, even empty orfalse, skips the script and every one-shot branch with it CODE. Kolla-Ansible never sets this variable (grep ofansible/). echo "Running command: ...", thenexec ${CMD} ${ARGS}, unquoted (:23-24). The service replaces the shell and becomes dumb-init's only child.
2.3 kolla_set_configs: copy and permissions¶
Without arguments, kolla_set_configs loads /var/lib/kolla/config_files/config.json (the Ussuri KOLLA_CONFIG and
KOLLA_CONFIG_FILE overrides are gone in 2023.1), validates it (command is required; each config_files entry needs
source and dest, plus owner and perm unless preserve_properties) and applies the strategy (kolla
docker/base/set_configs.py:245-260,279-299,535-553) CODE:
COPY_ALWAYS(:538-541): defaults handling, copy and permissions on every start.COPY_ONCE(:542-551): on the first start, defaults handling, copy, permissions andmknod /configured. On any later start of the same containerImmutableConfigis raised with exit code 0: nothing is copied, permissions are not re-applied and/run_commandkeeps its first value.kolla_startcontinues normally.- Unset or any other value:
InvalidConfig, exit 1 (:552-553).
Defaults handling (handle_defaults, new in 2023.1, :389-532), before the copy:
- The state lives in
/etc/kolla/defaults/state, a JSON file in the container layer (:32-33,429-434,437-449). - Restore (
remove_or_restore_configs,:452-474): for every destination recorded in the state, the file or tree is deleted if it did not exist when it was recorded, or copied back from its saved original under/etc/kolla/defaults/if it did. - Record (
backup_configs,:477-513): everyconfig_filesentry not yet in the state is added, but only if itssourcepath exists as written (os.path.exists, so a glob source such asextras/*is never recorded, and an optional entry is recorded only once its source exists). A destination that exists at that moment, from the image or from a volume, is first saved under/etc/kolla/defaults/<dest>; one that does not exist is recorded as absent. A destination already in the state is never re-recorded.
Copy, per config_files entry (ConfigFile.copy, :136-163):
- The source is a glob. A non-optional entry that matches nothing raises
MissingRequiredSource(exit 1). An optional one is skipped silently (:138-144). - A
destending in/meansdest/<basename of source>(:150-151). - Unless
mergeis set, the destination is deleted first, a whole tree for a directory, then parent directories are created (:152-154). In 2023.1 two Kolla-Ansible manifests usemerge: Keystone's OIDC metadata and certificate directories (roles/keystone/templates/keystone.json.j2:56-70) and nova-libvirt's/etc/libvirt/secrets(roles/nova-cell/templates/nova-libvirt.json.j2:46-52) (grep ofroles/*/templates/*.json.j2). - Directory sources are copied recursively. Every directory and file gets the entry's
ownerandperm(so directories get the file mode, plus the execute bit wherever the read bit is set, new in 2023.1), or the source's mode and owner withpreserve_properties(:78-103,119-134,364-371). - An
OSErrorduring the copy is logged and re-raised: exit 2 (:155-163,579-581). - Finally
commandis written to/run_command, mode 0644 (:311-321).
Permissions, per permissions entry (handle_permissions, :334-386):
pathis a glob. A path that matches nothing is skipped without a warning (:379).ownerisuser[:group], the group defaulting to the user (:324-331). An unknown user or group raisesKeyError: exit 2 (:341-343,579-581).chownalways,chmodonly ifpermis given. A failing chown or chmod is only logged (:351-377).- With
recurse, the same is applied to every directory and file below, one unconditionalos.chownper entry (:381-386). Several manifests do this on data volumes (§6 item 10).
A failure here (missing required source, bad JSON, unknown user) aborts kolla_start through errexit. The container
then exits non-zero. In 2023.1 the default unless-stopped policy (group_vars/all.yml:117) is not given to Docker
(module_utils/kolla_docker_worker.py:286-321 sets no restart policy) but written into the container's systemd unit as
Restart=always, with RestartSec 11 s, StartLimitIntervalSec 120 and StartLimitBurst 10 from
docker_client_timeout 120 and docker_restart_policy_retry 10 (module_utils/kolla_systemd_worker.py:20-74,
group_vars/all.yml:105,120,150-153) CODE. NOT AVAILABLE (systemd semantics) Whether the unit then restarts the
container in a loop until the input is fixed, or stops after the start limit is reached.
2.4 kolla_copy_cacerts and kolla_httpd_setup¶
kolla_copy_cacertsdeletes everykolla-customca-*file from the system trust store, copies each file of/var/lib/kolla/config_files/ca-certificates/back under that prefix, and runsupdate-ca-certificates(Debian family,/etc/debian_version) orupdate-ca-trust(Red Hat family,/etc/redhat-release) (kolladocker/base/copy_cacerts.sh:7-27) CODE. The update command runs only when that directory exists and is not empty (:10-11). NOT AVAILABLE Whether, once the last custom CA is removed, the generated trust bundle in the container layer keeps it until the container is recreated.kolla_httpd_setupacts on httpd only when the shell runs as root (whoami, kolladocker/base/httpd_setup.sh:7). It empties/var/run/apache2/(Debian/Ubuntu) or removes/var/run/httpd/*,/run/httpd/*and/tmp/httpd*, so a restarted httpd does not trip on stale run files. On CentOS/Rocky it also writes a/server-statushandler limited to local requests (99-server-status.conf) and createslocalhost.crtif missing (:10-40). Whatever the user, it then setsLANG=C.UTF-8and sources the distribution's locale file if present (:43-51). In images that run as a service user (barbican-api, manila-api) the httpd part does nothing. nova-api and octavia-api, which ran as a service user in Ussuri, have noUSERin 2023.1 and run it as root (§5).
2.5 The extend_start chain¶
Most service families layer two scripts CODE. The service base image installs its extend_start.sh as
kolla_extend_start, creates an empty /usr/local/bin/kolla_<service>_extend_start, and its last line sources that
file (e.g. kolla docker/cinder/cinder-base/Dockerfile.j2:41,50-51, docker/cinder/cinder-base/extend_start.sh:10).
The API image overwrites kolla_<service>_extend_start with the one-shot branches. On those images
kolla_extend_start prepares the log directory (neutron-base also selects the iptables backend,
§5), and the DB commands are in kolla_<service>_extend_start. Keystone,
keystone-fernet, keystone-ssh, horizon, mariadb-server, mariadb-clustercheck, rabbitmq, swift-rsyncd, keepalived,
iscsid, fluentd, cron, kolla-toolbox, ovn-base, nova-libvirt and nova-ssh put everything in kolla_extend_start
instead (mariadb-clustercheck, cron and kolla-toolbox have a start script only since 2023.1).
2.6 Signals and PID 1¶
- PID 1 is
dumb-init --single-child, andkolla_startexecs the service, so the service is dumb-init's direct child (kolladocker/base/Dockerfile.j2:353,docker/base/start.sh:24) CODE. NOT AVAILABLE (dumb-init semantics, not in these sources) Whether in this mode dumb-init forwardsdocker stop's SIGTERM only to that child. A comment in the mariadb image describes the other mode: without--single-child, "the TERM signal is sent to all subprocesses" (kolladocker/mariadb/mariadb-server/Dockerfile.j2:64-68). - MariaDB overrides the entrypoint to
dumb-init --without--single-child(kolladocker/mariadb/mariadb-server/Dockerfile.j2:64-71) CODE. Its comment gives the reason:mysqld_safeignores SIGTERM, and without--single-childTERM also reachesmysqldCODE (for the comment; NOT AVAILABLE: dumb-init and mysqld_safe semantics). - Some Kolla-Ansible commands are shell wrappers that do not
execthe service, so dumb-init's child is bash:neutron-l3-agent-wrapper.sh(roles/neutron/templates/neutron-l3-agent-wrapper.sh.j2:13-16),start-ovsandstart-ovsdb-server(roles/openvswitch/templates/start-ovs.j2:2,start-ovsdb-server.j2:11) andhaproxy_run.sh(roles/loadbalancer/templates/haproxy/haproxy_run.sh.j2:7-9) CODE. NOT AVAILABLE Whether the service then gets any SIGTERM before the container stops. A long-running container is stopped through its systemd unit, whoseExecStopisdocker stop <name> -t <graceful_timeout>,docker_graceful_timeout60 s by default;kolla_dockercalls the Docker client'sstopdirectly only for a container without a unit file, such as a one-shot (module_utils/kolla_systemd_worker.py:31-36,module_utils/kolla_docker_worker.py:436-452,460-480,group_vars/all.yml:140,152, Repository architecture §6) CODE.
2.7 Healthcheck scripts¶
The base image installs healthcheck_curl (curl --fail, 10 s maximum), healthcheck_filemod,
healthcheck_listen, healthcheck_port and healthcheck_socket when docker_healthchecks is on (kolla
docker/base/Dockerfile.j2:356-363, docker/base/healthcheck_curl:2,8-12; default True, kolla
kolla/common/config.py:237). No kolla Dockerfile has a HEALTHCHECK instruction (grep of docker/) CODE.
Unlike Ussuri, Kolla-Ansible 2023.1 wires Docker healthchecks itself: enable_container_healthchecks defaults to yes
with interval 30 s, timeout 30 s, 3 retries and a 5 s start period (group_vars/all.yml:171-177); service definitions
carry a healthcheck dict, e.g. memcached's healthcheck_listen memcached <port>
(roles/memcached/defaults/main.yml:10,32-43) and MariaDB's /usr/bin/clustercheck
(roles/mariadb/defaults/main.yml:100-107). kolla_docker passes it at container creation and compares it, so a
changed healthcheck recreates the container (module_utils/kolla_docker_worker.py:349-351,
module_utils/kolla_container_worker.py:89,479-525) CODE. No Kolla-Ansible task or handler reads the resulting
health status (grep of roles/*/tasks, roles/*/handlers, library/, module_utils/ for healthy) CODE.
NOT AVAILABLE (Docker semantics) What Docker does with an unhealthy container beyond reporting it. The mariadb image's
own healthcheck_mariadb now exits 1 when the node is not synced (kolla
docker/mariadb/mariadb-server/healthcheck_mariadb:10-19);
Kolla-Ansible does not use it.
3. kolla_set_configs --check: what it compares and its exit codes¶
kolla_docker's compare_container reports drift when the container is missing or differs, when the systemd unit file
would change, or when /usr/local/bin/kolla_set_configs --check, run inside the running container with docker exec
as root, says so (module_utils/kolla_container_worker.py:20,59-66, module_utils/kolla_docker_worker.py:155-190;
Repository architecture §4.3). --check loads and
validates config.json like a normal start, then calls ConfigFile.check for each config_files entry (kolla
docker/base/set_configs.py:556-575). It does no defaults handling.
3.1 What is compared¶
For each entry whose source glob matches something (:220-242) CODE:
- File source (
_cmp_file,:165-198): destination missing (drift), foroptionalentries too: the Ussuri exemption of optional entries is gone (:166-169); byte-for-byte content; mode, as the last four octal digits ofst_modeagainst the entry'spermstring; owner user and group names. - Directory source (
_cmp_dir,:200-218): every file below, as above. For subdirectories therelpatharguments are reversed (:204-205), so the mode checked is that of the destination's parent against the fileperm. A directory source that contains a subdirectory therefore reports drift on every run. preserve_propertiesentries carry noperm(:64-72), so the mode test always fails (:182) and they report drift on every run. Example:prometheus_server'sextras/*entry whenever extras exist (roles/prometheus/templates/prometheus-server.json.j2:10-15).
3.2 What is never compared¶
config.json'scommandandpermissions:--checkiterates onlyconfig_files(:556-559). A manifest with"config_files": [], like memcached's, whose settings are all in its command, always returns 0 (roles/memcached/templates/memcached.json.j2:2-3).- Entries removed from
config.json, and optional entries whose source no longer exists (:226-227), even while their old destination is still in the container. - Extra files in a destination directory, and the mode and owner of the destination directories themselves (for
subdirectories
_cmp_dirchecks the mode of a directory above the destination instead, §3.1). A file destination's mode and owner are compared (§3.1).
3.3 Exit codes¶
| Exit code | Raised when | What kolla_docker does |
Day-2 meaning |
|---|---|---|---|
| 0 | Every listed entry matches. Also a manifest with empty config_files, and any difference confined to command, permissions, removed entries or optional entries without a source |
no drift (module_utils/) |
"No drift" covers the listed files only. A config.json-only change written by genconfig makes neither deploy-containers nor a later run restart the container (§6 item 1) |
| 1 | ConfigFileBadState: a listed file differs in content, mode or owner, or a destination is missing, optional entries included (e.g. a newly rendered nova-compute release file, roles/). Also MissingRequiredSource (a non-optional source matches nothing, set_configs.py:224-225) and InvalidConfig (bad JSON, no command, an entry without owner/perm; :245-260,) |
drift: the handler restarts the container (module_utils/) |
Usually real drift; the restart copies the new file. For a missing source or an invalid config.json the restarted container then fails in kolla_set_configs at every start (errexit; what systemd then does is NOT AVAILABLE, §2.3). preserve_properties entries and directory sources with subdirectories return 1 on every run |
| 2 | Any other exception (set_configs.py:579-581): a config.json that cannot be opened (:285, outside the try, e.g. run as a service user); a uid or gid with no passwd or group entry (:188,193) |
raises: the check task fails on that host (module_utils/, library/) |
An environment problem, not drift. Entries are checked in order and the first difference ends the check (set_configs.py:556-559,) |
| 137 | the exec was killed because the container exited | drift (module_utils/) |
the container is unstable; Kolla restarts it |
| none | docker exec fails with a client error (container stopped or restarting) |
drift (module_utils/) |
a stopped container is restarted by the next run |
Read from main (kolla docker/base/set_configs.py:562-582) CODE: rc 0 for a match, for empty config_files
with a changed command (:556-559), for an optional entry without a source (:226-227), and for a flat directory
with an extra file in the destination (_cmp_dir walks the source only, :200-218); rc 1 for a content, mode or
missing destination difference, optional or not (:166-197), a missing required source (:224-225), a
preserve_properties entry (:182), a directory with a subdirectory (:204-205), a missing command (:248-249)
and bad JSON (:286-290); rc 2 for a missing or unreadable config.json (:285).
3.4 Running it by hand needs root¶
Kolla-Ansible writes the files in /etc/kolla/<service>/ with mode 0660 into directories with mode 0770, owned by
config_owner_user root (group_vars/all.yml:20-21; e.g. roles/nova/tasks/config.yml:2-9,
roles/keystone/tasks/config.yml:45-50), and mounts them read-only (e.g. roles/nova-cell/defaults/main.yml:416).
In an image with a service USER (e.g. USER nova, kolla docker/nova/nova-compute/Dockerfile.j2:116) a plain
docker exec <c> kolla_set_configs --check cannot open them and exits 2. kolla_docker execs as root
(module_utils/kolla_docker_worker.py:157-161). By hand, use docker exec -u root as the swift reconfigure does
(roles/swift/tasks/reconfigure.yml:47), or sudo kolla_set_configs --check inside the container
(§2.1). The bifrost reconfigure omits -u root (roles/bifrost/tasks/reconfigure.yml:17), but its
check cannot work anyway: it execs into bifrost-deploy, while the container is bifrost_deploy
(roles/bifrost/tasks/start.yml:8), which runs /sbin/init and mounts no config.json (§2.1). No
kolla-ansible command runs that file: the bifrost role includes <kolla_action>.yml
(roles/bifrost/tasks/main.yml:2)
and is reached only through deploy-bifrost (tools/kolla-ansible:480).
3.5 What a restart applies¶
- A
COPY_ALWAYSrestart (systemdRestartUnitof the same container, or stop and start for a unit-less one-shot,module_utils/kolla_docker_worker.py:460-480) re-copies the listed entries and rewrites/run_command. Changed or added files, mode and owner changes and a changed command apply CODE. Stale files inside a listed directory destination go too, because the destination is deleted first (except formergeentries, §2.3). -
Removals now apply in most cases (new in 2023.1). Before copying, every start deletes each recorded destination that did not exist at the container's first start, and restores each one that did (from the image or a volume) to that original (kolla
docker/base/set_configs.py:452-474,516-532,538-541) CODE. So the destination of an entry dropped fromconfig.json, or of an optional source that is gone, is removed (or reset to the original) at the next start, provided it was recorded. Exceptions CODE:- Glob sources are never recorded (
:502), so their destinations still stay, e.g. prometheusextras/*(roles/prometheus/templates/prometheus-server.json.j2:10-15). - The state and the saved originals are in
/etc/kolla/defaults, in the container layer (:32-33). A recreate starts a new state. A destination on a volume that already holds a file at the new container's first start is saved as the "original" and restored on later starts. Example: nova-compute's SASLauth.conf(roles/nova-cell/templates/nova-compute.json.j2:58-64) is under/var/lib/nova, thenova_computevolume when the libvirt container is used (roles/nova-cell/defaults/main.yml:379,group_vars/all.yml:731,1032). Turned off bylibvirt_enable_saslafter a restart-only history, it is deleted at the next start; turned off after the container was recreated while SASL was on, the copy found on the volume at that recreate is restored instead. - Under
COPY_ONCEnothing happens after the first start (§2.3).
--checkstill does not flag dropped entries (§3.2), so only a restart for another reason applies the removal. NOT AVAILABLE Whether Keystone still reads a removed policy file (roles/keystone/templates/keystone.json.j2:31-37) when the restart has not happened. - Glob sources are never recorded (
-
CA certificates are purged and re-added on every start (§2.4), from
/etc/kolla/<svc>/ca-certificates/. A CA deleted fromkolla_certificates_dir/ca/, or left there afterkolla_copy_ca_into_containersis turned off, stays in that directory, because thecopytask never deletes files (roles/service-cert-copy/tasks/main.yml:2-12; the copy action only walks the source tree, ansible-core 2.14.18lib/ansible/plugins/action/copy.py:485-599) CODE, so it is re-added at every start until it is deleted there by hand. NOT AVAILABLE Whether, once the last custom CA is removed there, the generated trust bundle keeps it (§2.4). COPY_ONCE:/configuredlives in the container layer, so only a recreate re-applies config.kolla_dockeralways recreates underCOPY_ONCE(module_utils/kolla_docker_worker.py:374-382).
4. One-shot containers: what the environment variables run¶
Kolla-Ansible starts one-shot containers with kolla_docker start_container, detach: False,
restart_policy: oneshot, a label such as BOOTSTRAP or UPGRADE, and a variable that selects a branch of the
image's start script. A oneshot container gets no systemd unit and is started with the Docker client directly; the
module waits for it and returns its rc, stdout and stderr (module_utils/kolla_docker_worker.py:355-360,400-434). The
variables are sent with an empty value (module_utils/kolla_docker_worker.py:330-332) and every branch tests only
whether they are set (${!VAR[@]}). The one-shot runs the whole of kolla_start first: kolla_set_configs with the
config directory mounted into it, kolla_copy_cacerts, then the branch, which ends in exit 0 before the service
command (§2.2) CODE. That directory is the service's own, except for Nova: its
bootstrap and online-migration one-shots mount a dedicated nova-api-bootstrap or nova-cell-bootstrap directory
whose command is "false" (roles/nova/defaults/main.yml:181-182, roles/nova-cell/defaults/main.yml:435-436,
roles/nova/templates/nova-api-bootstrap.json.j2:2).
| Image | KOLLA_BOOTSTRAP |
KOLLA_UPGRADE / other one-shot variables |
Set by Kolla-Ansible (task, action) | Day-2 note |
|---|---|---|---|---|
| keystone | keystone-manage db_sync, as user keystone (kolla docker/) |
KOLLA_UPGRADE: db_sync --expand, then --migrate (:24-31). KOLLA_FINISH_UPGRADE: db_sync --contract (:36-39) |
bootstrap_keystone, deploy/reconfigure, run_once (roles/). Handlers init_upgrade_database and finish_upgrade_database, upgrade only (roles/) |
Schema sync on every deploy/reconfigure. EXTRA_KEYSTONE_MANAGE_ARGS (:21) is never set by Kolla-Ansible |
| glance-api | glance-manage db_sync, then db_load_metadefs (kolla docker/) |
KOLLA_UPGRADE with GLANCE_DB_EXPAND, GLANCE_DB_MIGRATE or GLANCE_DB_CONTRACT: glance-manage db expand, migrate or contract (:13-27) |
bootstrap_glance, deploy/reconfigure and legacy upgrade (roles/, legacy_upgrade.yml:6). Rolling containers only with glance_enable_rolling_upgrade (default no, group_vars/; roles/, roles/) |
glance-manage db_load_metadefs runs on every deploy/reconfigure; whether, without --merge, it only adds namespaces not yet in the DB is NOT AVAILABLE |
| placement-api | placement-manage db sync, then db online_data_migrations (kolla docker/) |
KOLLA_UPGRADE: db sync only (:12-15), never used. KOLLA_OSM: db online_data_migrations (:18-21) |
bootstrap_placement, deploy/reconfigure and upgrade (roles/, deploy.yml:11, reconfigure.yml:2, upgrade.yml:11). KOLLA_OSM after the restart, upgrade only (roles/) |
Online data migrations on every deploy/reconfigure, and twice on upgrade |
| cinder-api | cinder-manage db sync (kolla docker/) |
No KOLLA_UPGRADE. KOLLA_OSM: db online_data_migrations, unbounded: the Ussuri MAX_NUMBER/--max_count option is gone (:11-14) |
bootstrap_cinder, deploy/reconfigure and upgrade (roles/, upgrade.yml:7). KOLLA_OSM, upgrade only (roles/) |
A tag change applied by deploy/reconfigure syncs the schema without the online migrations |
| heat-api | heat-manage db_sync, then, only if heat_user_domain is absent, creates that domain, user heat_domain_admin (password HEAT_DOMAIN_ADMIN_PASSWORD) and its admin role (kolla docker/) |
none; upgrade re-runs the bootstrap | bootstrap_heat with admin OS_* credentials, deploy/reconfigure and upgrade (roles/, upgrade.yml:6) |
Needs a reachable Keystone. A changed heat_domain_admin_password never reaches Keystone (§6 item 7) |
| horizon | manage.py migrate --noinput, after the settings symlinks (kolla docker/) |
none; FORCE_GENERATE is a start-time switch (§5.2) |
bootstrap_horizon only when horizon_backend_database (= enable_murano, group_vars/; roles/, upgrade.yml:10-11) |
Usually never runs |
| barbican-api | barbican-db-manage upgrade (kolla docker/) |
none | bootstrap_barbican, deploy/reconfigure and upgrade (roles/, upgrade.yml:6) |
Schema upgrade on every deploy/reconfigure |
| octavia-api | octavia-db-manage upgrade head (kolla docker/) |
none | bootstrap_octavia, deploy/reconfigure and upgrade (roles/, upgrade.yml:6) |
Schema upgrade on every deploy/reconfigure |
| manila-api | manila-manage db sync (kolla docker/) |
none | bootstrap_manila, deploy/reconfigure and upgrade (roles/, upgrade.yml:6) |
Schema sync on every deploy/reconfigure |
| nova-api | nova-manage api_db sync, db sync --local_cell, db online_data_migrations (kolla docker/). Unreachable: Kolla-Ansible never sets it for Nova |
KOLLA_UPGRADE: api_db sync, db sync --local_cell (:16-20). KOLLA_OSM: db online_data_migrations (:22-25). New: KOLLA_UPGRADE_CHECK: nova-status upgrade check, exiting with its code (:27-30) |
nova_api_bootstrap with KOLLA_UPGRADE, run_once, deploy/reconfigure and upgrade (roles/, nova.yml:45-61), again when map_cell0 reports a change (roles/). KOLLA_OSM, upgrade only (nova.yml:230-247, roles/). nova_upgrade_checks with KOLLA_UPGRADE_CHECK and the normal nova-api config, upgrade only; the run fails unless rc is 0 or 1 (roles/) |
db sync --local_cell targets cell0: the bootstrap nova.conf has [database] = cell0 (roles/, roles/). The bootstrap command is "false" (nova-api-bootstrap.json.j2:2), so success relies on the exit 0 |
| nova-conductor | nova-manage db sync --local_cell, db online_data_migrations (kolla docker/). Unreachable |
KOLLA_UPGRADE: db sync --local_cell (:9-12). KOLLA_OSM: db online_data_migrations (:14-17) |
nova_cell_bootstrap with KOLLA_UPGRADE on the first conductor, deploy/reconfigure and upgrade (roles/, nova.yml:81-96). KOLLA_OSM, upgrade only (nova.yml:253-272) |
The same image runs nova_super_conductor (roles/) |
| neutron-server | neutron-db-manage --subproject neutron upgrade head, then upgrade head for each NEUTRON_BOOTSTRAP_SERVICES entry (kolla docker/); no --config-file |
KOLLA_UPGRADE with NEUTRON_DB_EXPAND or NEUTRON_DB_CONTRACT: upgrade --expand or --contract for neutron, then for each NEUTRON_ROLLING_UPGRADE_SERVICES entry (:17-34). The Ussuri NEUTRON_SFC_BOOTSTRAP branch is gone |
bootstrap_neutron on every deploy/reconfigure, and on upgrade only when neutron_enable_rolling_upgrade is off (default yes) (roles/, deploy.yml:20, bootstrap.yml:38, legacy_upgrade.yml:13, upgrade.yml:4-8, roles/). Both lists are the enabled neutron_subprojects: networking-sfc, neutron-dynamic-routing, neutron-vpnaas, vmware-nsx (roles/). Expand and contract containers in rolling_upgrade.yml:13-33, |
Only enabled subprojects are migrated, on deploy and on upgrade alike (§6 item 8) |
| mariadb | mysql_install_db, a temporary mysqld_safe --wsrep-new-cluster --skip-networking --wsrep-on=OFF, kolla_security_reset (root password), grants to root@localhost and root@'%', shutdown; both steps' output is appended to mariadb-bootstrap.log (kolla docker/) |
KOLLA_UPGRADE: removes /var/, runs mysql_upgrade over TCP against the running local server (:47-57). BOOTSTRAP_ARGS: appended to mysqld_safe on any start (:59-61) |
bootstrap_mariadb when a new cluster is bootstrapped (roles/), then a handler starts mariadb with BOOTSTRAP_ARGS=--wsrep-new-cluster (roles/). upgrade_mariadb on every MariaDB host, upgrade only, after the restarts (roles/, mariadb.yml:68-72); roles/ now only imports deploy.yml. --wsrep-recover and --wsrep-new-cluster in mariadb_recovery (roles/) |
The image forms no cluster itself and has no Galera recovery logic (§5.4) |
| rabbitmq | writes RABBITMQ_CLUSTER_COOKIE to /var/, mode 400; nothing else: no rabbitmqctl, no join, no users (kolla docker/; the image moved from docker/, and Kolla-Ansible uses rabbitmq, roles/) |
none | <project_name>_bootstrap, only when the volume was just created on that host (roles/) |
A normal start never rewrites the cookie (§6 item 9) |
No one-shot branch exists in: keystone-fernet, keystone-ssh, cinder-volume, cinder-scheduler, cinder-backup,
heat-api-cfn, heat-engine, the barbican, octavia and manila worker images, all swift images, nova-scheduler,
nova-compute, nova-compute-ironic, nova-libvirt, nova-ssh, the Nova console proxies, every neutron agent,
openvswitch, ovn, mariadb-clustercheck, memcached, haproxy, keepalived, fluentd, cron, kolla-toolbox, iscsid and tgtd
(grep of their start scripts for ${!...[@]}) CODE. The only DB step in the OVS and OVN images is at start
(§5).
Note on ironic (only this branch was read): ironic-api's KOLLA_BOOTSTRAP runs ironic-dbsync upgrade and
online_data_migrations, KOLLA_UPGRADE only upgrade, KOLLA_OSM only online_data_migrations (kolla
docker/ironic/ironic-api/extend_start.sh:5-19). bootstrap_ironic sends KOLLA_UPGRADE unless
ironic_enable_rolling_upgrade (default yes) is turned off (roles/ironic/tasks/bootstrap_service.yml:2-25,
roles/ironic/defaults/main.yml:286).
4.1 One-shots that bypass kolla_start¶
A few one-shots replace the container command with bash -c 'sudo -E kolla_set_configs && ...' CODE. The config
is copied, but kolla_copy_cacerts and kolla_extend_start do not run, and the config.json command is written to
/run_command but never executed. Custom CA certificates are therefore not installed in these containers.
| Container | Image | What runs after kolla_set_configs |
Kolla-Ansible task |
|---|---|---|---|
bootstrap_keystone_fernet |
keystone-fernet | keystone-manage fernet_setup |
roles/ |
nova_api_map_cell0 |
nova-api | nova-manage cell_v2 map_cell0 |
roles/ |
nova_list_cells, cell create and update |
nova-conductor | nova-manage cell_v2 list_cells, create_cell, update_cell |
roles/, create_cells.yml:12, |
mariabackup |
mariadb-server, the image of the running mariadb container when known |
/usr/ (§5); the container is removed on exit |
roles/ |
The keystone-fernet image ships its own kolla_keystone_bootstrap, a fernet_setup wrapper, which Kolla-Ansible does
not use. Its changed detection is inverted: it reports changed when "Key repository is already initialized"
appears (kolla docker/keystone/keystone-fernet/keystone_bootstrap.sh:35-41). In this pairing that is dead code.
discover_hosts is a docker exec into the running nova_conductor, not a one-shot
(roles/nova-cell/tasks/discover_computes.yml:14-18).
5. Per-image start behaviour¶
"Runs as" is the image USER; root when the image and its parents set none. Every row starts with the common path of
§2. "Log dir" means: create /var/log/kolla/<service> if missing and set
mode 755.
| Image | Runs as | What its start script does on every start | Day-2 note |
|---|---|---|---|
| base, openstack-base | root | Nothing: empty kolla_extend_start (kolla docker/) |
No USER in base, openstack-base or nova-base |
| keystone | root | Log dir and keystone.log with owners fixed, kolla_httpd_setup, ARGS=-DFOREGROUND (kolla docker/). The command keystone-startup.sh waits up to 36 × 5 s for fernet key 0, then execs httpd (roles/) |
Stale httpd run files are removed, so a restart is safe |
| keystone-fernet | root | Runs /usr/ if present, then chowns the key directory (kolla docker/). Kolla-Ansible installs that script; it waits up to 10 × 60 s for key 0, then exits 1 (roles/) |
Without key 0 on the node the start waits up to 10 × 60 s and the script exits 1 before the cron command is exec'd (§2.2), so no rotation cron runs there meanwhile CODE; what systemd then does is NOT AVAILABLE (§2.3) |
| keystone-ssh | root | Generates any missing SSH host key (rsa, dsa, ecdsa, ed25519); creates /var/ (kolla docker/) |
/etc/ssh is on no volume (roles/): new host keys on every recreate (§6 item 18) |
| glance-api | glance | Log dir (kolla docker/); no httpd step. kolla_set_configs recursively chowns /var/lib/glance and /var/ (roles/) |
With the file backend /var/lib/glance is the image store: one chown per file on every start (§6 item 10) |
| placement-api | root | Log dir and placement-api.log; chown -R only if the top-level owner differs (kolla docker/); kolla_httpd_setup (placement-api/) |
– |
| cinder-api | root | Log dir (kolla docker/); kolla_httpd_setup (cinder-api/) |
– |
| cinder-volume | cinder | Log dir; sudo chown -R cinder:kolla of /var/lib/cinder and /etc/iscsi only if the top-level owner differs, chmod 2775 if needed (kolla docker/) |
Not only on a fresh volume: kolla_set_configs first chowns /var/lib/cinder recursively to cinder:cinder (roles/), so the cinder:kolla test always fails and chown -R walks the cinder volume a second time on every start |
| cinder-scheduler, cinder-backup | cinder | Log dir only (empty kolla_cinder_extend_start, kolla docker/) |
– |
| heat-api, heat-api-cfn, heat-engine | root (APIs), heat (engine) | Log dir (kolla docker/); the two API images run kolla_httpd_setup (heat-api/, heat-api-cfn/) |
– |
| horizon | root | Settings symlinks, plugin files from ENABLE_*, conditional static-asset regeneration, log dir, secret-key-store owner, kolla_httpd_setup (kolla docker/; §5.2) |
Assets are regenerated after a recreate or a settings change (§5.2); how long that start takes is NOT AVAILABLE |
| barbican, octavia and manila images | barbican, manila (all images); octavia workers; octavia-api root | Log dir (barbican chowns it, kolla docker/; octavia chowns it recursively when the owner differs, docker/). The API images call kolla_httpd_setup (barbican-api/, octavia-api/, manila-api/): a no-op for httpd as a service user (barbican-api/, manila-api/); octavia-api sets no USER in 2023.1 (octavia-api/), so it runs the httpd cleanup as root |
Worker images do the log-dir step only |
| swift-rsyncd | swift | sudo find /srv/ (not recursive) and mkdir -p /var/ (kolla docker/) |
Only rsyncd fixes the ownership of the disk mount points |
| swift account, container, object, object-expirer, proxy-server | swift | Nothing of their own | Config and rings are copied only at container start, and the swift role notifies no handler (fact sheet D) |
| nova-base | – | Log dir, then sources kolla_nova_extend_start (kolla docker/) |
Parent of every nova image except nova-libvirt |
| nova-api | root | nova-base step; kolla_httpd_setup (kolla docker/), which now acts: the 2023.1 Dockerfile sets no USER (docker/), where Ussuri had USER nova |
No DB action on a normal start |
| nova-conductor, nova-scheduler | nova | nova-base step only | Also nova_super_conductor. The scheduler SIGHUP comes from Kolla-Ansible (nova.yml:140-156, roles/) |
| nova-compute | nova | kolla_set_configs recursively chowns /var/ and /var/lib/nova (roles/); nova-base step; mkdir -p /var/ (kolla docker/) |
/var/lib/nova is the instance datadir: one chown per file on every start (§6 item 10) |
| nova-compute-ironic | nova | nova-base step; permissions only on /var/ (roles/) |
– |
| nova-libvirt | root | xenfs mounted if loaded, log file created (kolla docker/); exec libvirtd --listen. The Ussuri /dev/kvm chown/chmod is gone from the image; Kolla-Ansible installs a host udev rule instead (§5.1) |
Passive towards running guests (§5.1) |
| nova-ssh | root | Replaces the nova-base script: generates missing SSH host keys, creates and chowns /var/ (kolla docker/); exec sshd -D |
New host keys on every recreate (§6 item 18) |
| nova-novncproxy, nova-serialproxy, nova-spicehtml5proxy | nova | nova-base step only | – |
| neutron-base | – | Log dir; iptables backend, skipped on CentOS and Rocky and when update-alternatives --display iptables fails: update-alternatives --set to the -legacy binaries when KOLLA_LEGACY_IPTABLES is "true", otherwise --auto (unset counts as false) (kolla docker/) |
Only the OVS, linuxbridge and L3 agents get KOLLA_LEGACY_IPTABLES (roles/). Changing neutron_legacy_iptables recreates them; whether rules in the old backend are migrated is NOT AVAILABLE |
| neutron-server | neutron | neutron-base step; no branch taken; exec neutron-server |
– |
| neutron-openvswitch-agent | neutron | neutron-base step; nothing to OVS bridges or flows | NOT AVAILABLE Whether and how the agent resyncs flows after a restart: agent behaviour, not in these sources |
| neutron-dhcp-agent | neutron | neutron-base step; no dnsmasq or namespace handling | NOT AVAILABLE Whether dnsmasq children die with the container and whether qdhcp- namespaces persist on the shared /run/netns |
| neutron-l3-agent | neutron | neutron-base step; the command is Kolla-Ansible's wrapper (§5.3) | Namespace cleanup on every start with enable_neutron_agent_ha (Risk register R6) |
| neutron-metadata-agent (also the OVN metadata agent) | neutron | neutron-base step; recursive chown of /var/, the socket volume shared with the L3 and DHCP agents (roles/) |
2023.1 runs the OVN metadata agent from this image (roles/) |
| neutron-sriov-agent, neutron-linuxbridge-agent | neutron | neutron-base step | – |
| openvswitch-base | root | Log dir, then kolla_openvswitch_extend_start (kolla docker/) |
– |
| openvswitch-db-server | root | mkdir -p /run/; ovsdb-tool create if conf.db is absent (kolla docker/). start-ovsdb-server converts the schema when needed, then runs ovsdb-server without exec (roles/) |
conf.db is on the openvswitch_db volume: bridges and ports survive restart and recreate |
| openvswitch-vswitchd | root | modprobe openvswitch (kolla docker/). start-ovs runs ovs-vswitchd without exec and without flow save or restore (roles/) |
The kernel module stays loaded; whether flows are lost is NOT AVAILABLE (§6 item 13) |
| ovn-base | root | Replaces the OVS script: creates /run/ovn and the log dir (kolla docker/) |
/run/ovn is on no volume |
| ovn-controller | root | ovn-base step; exec ovn-controller (roles/) |
Chassis settings and br-int are set by Kolla-Ansible through kolla_toolbox (openvswitch_bridge, openvswitch_db; roles/), not by the image. The Ussuri ovn role is split into ovn-controller and ovn-db |
| ovn-northd | root | ovn-base step; exec ovn-northd |
Stateless |
| ovn-nb-db-server, ovn-sb-db-server | root | ovn-base step. Kolla-Ansible calls /usr/ (or run_sb_ovsdb) directly, not the image's start-nb-db-server wrapper (roles/, ovn-sb-db.json.j2:2, roles/) |
The wrapper's comment says ovn-ctl creates the DB file when absent and updates it when the schema changes (kolla docker/); whether the direct call does so is NOT AVAILABLE |
| mariadb | mysql | kolla_set_configs recursively chowns /var/ and /var/lib/mysql (roles/); log dir; exec mysqld_safe plus BOOTSTRAP_ARGS if set (kolla docker/) |
No Galera recovery in the image (§5.4) |
| mariadb (backup) | mysql | §4.1. backup.sh: mariabackup --backup --stream=xbstream, gzipped into /backup, full or incremental from BACKUP_TYPE (kolla docker/) |
No retention or rotation in the image |
| mariadb-clustercheck | root | Log dir (kolla docker/, new in 2023.1). The command is socat, listening on the clustercheck port and running clustercheck per connection (roles/); the Ussuri xinetd setup is gone |
Stateless. The clustercheck script is not in the kolla source: it is downloaded from a Fedora package repository at build time (kolla docker/) |
| rabbitmq | rabbitmq | Recursive chown of /var/ and /var/ (roles/); log dir (kolla docker/); cookie untouched |
Clustering is classic_config peer discovery (roles/), not the image |
| memcached | memcached | Nothing: no start script, empty config_files, all settings in the command (roles/) |
No volume; whether a restart empties the cache is NOT AVAILABLE (§6 item 14) |
| haproxy | root | No start script. kolla_set_configs replaces haproxy.cfg and the whole services.d/ (roles/); haproxy_run.sh starts haproxy -W -db (roles/) |
The image's reload helper kolla_ensure_haproxy_latest_config (kolla docker/) is never called (grep of ansible/): config applies by restart only (Risk register R8) |
| keepalived | root | Removes a stale /run/ (kolla docker/); exec keepalived -nld |
VIP health is check_alive.sh, which now runs the scripts Kolla-Ansible puts in /checks (§6 item 15) |
| fluentd | td-agent | Sets owner and mode 2775 on /var/log/kolla, the kolla_logs volume every container shares, and chowns /var/, through sudo (kolla docker/); recursive chowns of its log dirs and /var/ (roles/) |
NOT AVAILABLE Whether read positions under /var/ are lost on recreate but kept on a restart |
| cron | root | New in 2023.1: moves the logrotate cron file to /etc/ when the schedule is hourly, daily, weekly or monthly (kolla docker/; Kolla-Ansible sets cron_logrotate_schedule, default daily, roles/); copies logrotate.conf, exec crond -s -n or cron -f (roles/) |
A restart only delays rotation |
| kolla-toolbox | ansible | New in 2023.1: creates /var/, owner ansible:kolla, mode 664 (kolla docker/). When RabbitMQ is enabled, copies the Erlang cookie from passwords.yml (roles/); command sleep infinity |
Gets the current cookie on every start, unlike the brokers (§6 item 9) |
| chrony | – | Gone in 2023.1: no docker/chrony image in kolla and no chrony role in Kolla-Ansible (ls of kolla docker/ and of roles/) |
– |
| iscsid | root | Writes /etc/ with a new IQN only if the file is missing (kolla docker/) |
/etc/iscsi is the iscsi_info volume (roles/): the IQN is stable (§6 item 19) |
| tgtd | root | No start script, empty config_files; exec tgtd (roles/) |
NOT AVAILABLE Whether the target table survives a restart, or exported targets drop until re-created |
5.1 nova-libvirt: the start path is passive¶
The image is built from base, not nova-base (kolla docker/nova/nova-libvirt/Dockerfile.j2:1), and runs as root
CODE. On every start kolla_set_configs deletes and re-copies only the files listed in
roles/nova-cell/templates/nova-libvirt.json.j2:3-70 (libvirtd.conf, qemu.conf, TLS files, Ceph and SASL files),
except /etc/libvirt/secrets, which is merged rather than deleted first (:46-52); before that, the defaults
handling of §2.3 deletes or restores each recorded destination. The
manifest has no permissions block. extend_start.sh then mounts xenfs only if lsmod shows it and creates the log
file (kolla docker/nova/nova-libvirt/extend_start.sh:3-16). The 2023.1 image no longer touches /dev/kvm:
Kolla-Ansible
writes a host udev rule (KERNEL=="kvm", group qemu_user_gid, mode 0660) and, on Ubuntu, masks qemu-kvm.service
(roles/nova-cell/tasks/config-host.yml:30-53, roles/nova-cell/templates/99-kolla-kvm.rules.j2:4). Nothing touches
cgroups, running QEMU processes, /var/run/libvirt, /var/lib/libvirt or /etc/libvirt/qemu, and no virtlogd or
virtlockd is started. qemu.conf.j2:1-7 and libvirtd.conf.j2:1-27 set no cgroup option. The container is deployed
only while enable_nova_libvirt_container is true (default: nova_compute_virt_type is kvm or qemu,
group_vars/all.yml:731). Whether guests survive a restart is therefore decided by the stop path, where libvirt placed
QEMU, and libvirtd's reconnect on start (Nova deep dive
§5,
Q1). The SASL DB /etc/libvirt/passwd.db is on no volume; only
/etc/libvirt/qemu is (roles/nova-cell/defaults/main.yml:381). It survives a restart and is lost on every recreate,
and the Create libvirt SASL user handler re-creates it (roles/nova-cell/handlers/main.yml:96,131-141).
NOT AVAILABLE Whether the image ships /etc/libvirt/secrets; if it does not, the defaults handling deletes the
merged directory at every start before re-copying it, which makes merge moot.
5.2 horizon: static assets¶
On every start the script copies or removes each plugin's enabled/*.py file according to its ENABLE_* variable
(unset counts as no); any change sets FORCE_GENERATE=yes (kolla docker/horizon/extend_start.sh:32-46,267-287)
CODE. It runs manage.py collectstatic --noinput --clear and compress --force only when
/var/lib/kolla/.settings.md5sum.txt is missing, the md5 of local_settings, custom_local_settings and
local_settings.d differs from it, or FORCE_GENERATE is yes; then it stores the new hash (:248-265,289-293).
Kolla-Ansible sets FORCE_GENERATE=yes only with horizon_dev_mode (roles/horizon/defaults/main.yml:30).
/var/lib/kolla is not a volume (:122-129), so the hash file lives in the container layer; only the start script
writes it (HASH_PATH, kolla docker/horizon/extend_start.sh:6,292). A recreate (new image, changed parameters) or a
settings change therefore makes the next start regenerate the assets; a plain restart with unchanged settings does not
CODE. NOT AVAILABLE How long such a start takes.
5.3 neutron-l3-agent: the wrapper¶
The command is Kolla-Ansible's /usr/local/bin/neutron-l3-agent-wrapper.sh
(roles/neutron/templates/neutron-l3-agent.json.j2:2-9) CODE. It sets errexit. With enable_neutron_agent_ha
(default no, group_vars/all.yml:721) it first runs neutron-netns-cleanup --force --agent-type l3 (the wrapper's
comment: "Remove all l3 related netns"); it then runs neutron-l3-agent without exec
(neutron-l3-agent-wrapper.sh.j2:3-16). The cleanup runs on every container start: handler restart, recreate, systemd
restart, reboot. A cleanup failure aborts the start. NOT AVAILABLE What the command deletes: it is neutron code, which
is not in these sources (F12).
Because the wrapper does not exec the agent, bash stays dumb-init's only child (§2.6) CODE.
NOT AVAILABLE Whether docker stop's SIGTERM then reaches only bash, so that the agent gets no graceful shutdown, and
whether its keepalived, haproxy and radvd children die with the container.
5.4 MariaDB: what the image does not do¶
The image has no Galera recovery logic: no grastate.dat or safe_to_bootstrap handling, no seqno comparison, no
new-cluster decision CODE (kolla docker/mariadb/mariadb-server/extend_start.sh has none). All of that is in
Kolla-Ansible (roles/mariadb/tasks/recover_cluster.yml, lookup_cluster.yml). A normal start is mysqld_safe with
the rendered galera.cnf, whose gcomm:// lists every host of the MariaDB shard group, or is empty on a single node
(roles/mariadb/templates/galera.cnf.j2:29). BOOTSTRAP_ARGS is read from the container environment on every start
(extend_start.sh:59-61), and the containers started with --wsrep-new-cluster have restart_policy: oneshot, so
they get no systemd unit (roles/mariadb/handlers/main.yml:2-20, roles/mariadb/tasks/recover_cluster.yml:113-129,
module_utils/kolla_docker_worker.py:355-360,400-403). The mariadb bootstrap, upgrade and --wsrep-recover one-shots
also run the recursive /var/lib/mysql chown of §5, since they use the same manifest.
6. Day-2 consequences¶
config.json-only changes written bygenconfignever cause a restart. A normal deploy or reconfigure applies them, because the changedconfig.jsontemplate task notifies the restart handler (e.g.roles/memcached/tasks/config.yml:13-26). After a separategenconfig,--checkignorescommandandpermissions, sodeploy-containerssees no drift, and later runs do not restart for them either because the template task is then unchanged. Memcached is the extreme case. The change applies at the next restart for another reason CODE (Predict and inspect a run, Execution flow §4, Write config first, apply in controlled batches §2).- Some containers restart on every deploy/reconfigure. A
preserve_propertiesentry, such as prometheusextras, or a directory source with subdirectories makes--checkexit 1 on every run (§3.1, §3.3) CODE (Tags §3). - A new optional file is drift, no longer a failure. In 2023.1
--checkexits 1 when an optional file is rendered for the first time (e.g. nova-computerelease), and the container is restarted with the new file; in Ussuri the same case exited 2 and failed the check-containers task (kolladocker/base/set_configs.py:166-169) CODE (Pre-execution checklist, Predict and inspect a run). - A restart now removes most dropped files. Destinations of entries removed from
config.jsonare deleted, or reset to the original found at the container's first start, at the next start; glob sources are not covered, and a file found on a volume when the container was created is restored rather than deleted (§3.5).--checkdoes not report them, so the removal waits for a restart. CA certificate files are purged and re-added on every start, but only from/etc/kolla/<svc>/ca-certificates/, where a CA deleted fromkolla_certificates_dir/ca/stays (thecopytask never deletes, §3.5) and is re-added; whether the generated bundle keeps the last custom CA once it is removed there by hand is not known (§2.4) CODE (bundle and service effect NOT AVAILABLE) (Repository architecture §7). - Nova's online data migrations never run outside
upgrade. Deploy and reconfigure run the Nova bootstrap containers withKOLLA_UPGRADE(api_db sync+db sync --local_cell,db sync --local_cell); the images'KOLLA_BOOTSTRAPbranches are unreachable (§4) CODE. Upgrade also runsnova-status upgrade checkfirst and stops on a result other than 0 or 1 (roles/nova/tasks/upgrade.yml:10-40) CODE (Nova deep dive §4, Findings F15). - Placement's online data migrations run on every deploy/reconfigure, inside
bootstrap_placement, and twice on upgrade. A tag change applied withdeploytherefore runs them; cinder's bootstrap does not (§4) CODE (Update container images). - A changed
heat_domain_admin_passwordnever reaches Keystone. The bootstrap only creates the user whenheat_user_domainis absent, and a domain left without its user by a partial run is never repaired, whileheat.confgets the new value (roles/heat/templates/heat.conf.j2:11-12) CODE. NOT AVAILABLE Whether stack-domain operations then fail authentication (Known bugs and limitations). - Neutron upgrade gaps. The Ussuri gap on neutron-dynamic-routing is closed: deploy and the rolling
upgradeboth migrate neutron plus every enabled subproject (§4) CODE. The contract step still has two traps: ifhas_offline_migrationsreturns non-zero for any enabled subproject, or itsdocker execfails (ignore_errors), everyneutron_serveris stopped until the final flush, a Neutron API outage; and the check loops over the enabled subprojects only, never neutron itself, so with no subproject enabled no server is ever stopped beforeupgrade --contract(roles/neutron/tasks/rolling_upgrade.yml:38-89) CODE. NOT AVAILABLE Whether a neutron contract migration is then safe against running servers (Known bugs and limitations). - The RabbitMQ cookie is fixed at bootstrap. The image writes
.erlang.cookieonly in the bootstrap container on a new volume. A changedrabbitmq_cluster_cookierecreates the brokers (env compare,roles/rabbitmq/defaults/main.yml:13-15,module_utils/kolla_container_worker.py:85) but their cookie file keeps the old value, whilekolla_toolboxgets the new one CODE (consequences NOT AVAILABLE) (fact sheet B §2). - Recursive chown on data volumes at every start.
kolla_set_configsmakes onechownper file of the glance image store, of/var/lib/novaon computes, of the MariaDB datadir, of the RabbitMQ mnesia directory and of thecindervolume (twice for cinder-volume, §5), on everyCOPY_ALWAYSstart CODE. NOT AVAILABLE Whether, withenable_shared_var_lib_nova_mnt, the walk also enters the shares still mounted under/var/lib/nova/mnt, and how start time grows with the number of files, especially on NFS-backed trees (fact sheet C, Nova deep dive §5). - nova-libvirt's start path is passive; VM survival is decided on the stop path (§5.1) CODE (for the start path; NOT AVAILABLE for survival) (Nova deep dive §5, Q1).
- The L3 agent wrapper does not
execthe agent. NOT AVAILABLE Whether SIGTERM then reaches only bash, so that the agent is killed without a graceful shutdown. Withenable_neutron_agent_haevery start also runsneutron-netns-cleanup --force --agent-type l3before the agent starts CODE; what that command deletes is NOT AVAILABLE (Risk register R6, Component dependencies §4). - An
openvswitch_vswitchdrestart replaces only the userspace daemon. The kernel module stays loaded andconf.dbkeeps bridges and ports CODE. NOT AVAILABLE Whether the OpenFlow tables are lost and must be reprogrammed by the OVS agent orovn-controller(Risk register R7, Q3). - memcached has no data volume and no persistence option. The container mounts no data volume
(
roles/memcached/defaults/main.yml:45-48) and its command has no persistence option (roles/memcached/templates/memcached.json.j2:2) CODE. NOT AVAILABLE (memcached semantics) Whether every restart therefore empties the cache, with token-cache misses and lost Horizon sessions where they are kept in memcached (Risk register R4). - The VIP follows the HAProxy stats socket, and the ProxySQL admin socket when ProxySQL is enabled. keepalived's
check_alive.shruns every script under/checksand fails if one fails, falling back to the HAProxy socket when there is none (kolladocker/keepalived/check_alive.sh:9-38). Kolla-Ansible puts there one script per enabled load-balancer service on the host,check_alive_haproxy.shandcheck_alive_proxysql.sh, each ashow infoon the service's socket (roles/loadbalancer/tasks/config.yml:74-105,roles/loadbalancer/templates/keepalived/check_alive_haproxy.sh.j2:6,check_alive_proxysql.sh.j2:6,keepalived.json.j2:11-12); withinterval 2,fall 2,rise 10whenkeepalived_track_script_enabled(default True) (roles/loadbalancer/templates/keepalived/keepalived.conf.j2:1-8,roles/loadbalancer/defaults/main.yml:152) CODE. NOT AVAILABLE (keepalived semantics) Whether an HAProxy or ProxySQL restart puts the node in FAULT, and for how long (Risk register R8). - MariaDB has no automatic Galera recovery (§5.4). NOT AVAILABLE
Whether, after all DB containers stopped (power loss, Docker restart on every DB host), they come back as plain
mysqld_safewith the cluster non-primary untilmariadb_recovery. A container left withBOOTSTRAP_ARGS=--wsrep-new-clusterby an interrupted run passes that option tomysqld_safeagain on any manualdocker start(§5.4) CODE; whether that bootstraps a new cluster (split-brain risk) is NOT AVAILABLE (fact sheet B §1, Risk register R18). - Docker healthchecks are wired in 2023.1, but nothing waits on them (§2.7). With the
default
enable_container_healthchecks: yes,docker psshows a health status for the services that define a test; no Kolla-Ansible task gates on it, so service health must still be checked from outside CODE (Pre-execution checklist item 5, health gates). - SSH host keys of
nova_sshandkeystone_sshchange on every recreate, because/etc/sshis on no volume. The clients setStrictHostKeyChecking noand keep no known-hosts file (UserKnownHostsFile /dev/null,roles/nova-cell/templates/ssh_config.j2:2-3,roles/keystone/templates/ssh_config.j2:2-3) CODE; whether the SSH client then accepts the new keys without failing is NOT AVAILABLE (OpenSSH semantics) (Nova deep dive §6). - The iSCSI initiator name is stable per
iscsi_infovolume across restart and recreate CODE. NOT AVAILABLE Whether removing the volume produces a new IQN at the next start (the start script writes one when the file is missing, §5), and whether that breaks backend ACLs keyed on the old one (Component dependencies §5).
7. What is not available from the sources¶
- dumb-init signal forwarding. Whether
--single-childforwards SIGTERM to the direct child only, and what happens to grandchildren (dnsmasq, keepalived and haproxy under the agents, services behind non-execwrappers). - Whether the runtime kills the container's cgroup when its init exits, and how that applies under
pid_mode: host(nova_libvirt). - QEMU survival across a
nova_libvirtrestart or recreate (Q1): placement inmachine.slicethrough the host D-Bus and systemd-machined, reconnect from the runtime state, the old QEMU binary kept after a recreate (Nova deep dive §5). - Whether OVS flows are lost on an
openvswitch_vswitchdrestart and, if so, how long reprogramming takes (Q3). - Horizon regeneration time after a recreate or a settings change (§5.2).
- Cost of the recursive chown on large trees: glance store, NFS instance directory, MariaDB datadir, RabbitMQ mnesia.
- Upstream semantics of
keystone-manage bootstrap: whether it resets the admin password tokeystone_admin_password. The 2023.1 wrapper, which Kolla-Ansible runs withdocker exec keystone kolla_keystone_bootstrap(roles/keystone/tasks/register.yml:5), counts eight "already exists"/"skipping" patterns but reports unchanged only when the count is nine (kolladocker/keystone/keystone/keystone_bootstrap.sh:51-63; the admin-endpoint pattern was dropped); whether any output line matches twice, so that it can ever report unchanged, depends on keystone's output. ovn-ctlschema creation and upgrade with Kolla-Ansible's direct call, and its hard-coded/usr/share/ovn/scripts/path on distributions that shipovn-ctlelsewhere./var/runas a symlink to/runin the images, which would put libvirt's runtime state on the host/runbind mount. The Dockerfiles do not show it.- Whether a failing
kolla_set_configscauses a restart loop under the container's systemd unit (§2.3), what Docker does with an unhealthy container (§2.7), and the CA bundle after the last custom CA is removed (§2.4). - Service-side effects of the one-shot findings: heat stack-domain authentication after a changed
heat_domain_admin_password(§6 item 7); whetherrabbitmq-serverignoresRABBITMQ_CLUSTER_COOKIEand uses only the cookie file (§6 item 9); a neutron contract migration against running servers (§6 item 8); Galera after a full stop, or a manual start left with--wsrep-new-cluster(§6 item 16).
Related¶
- Open questions (Q14)
- Repository architecture §4.3 (how Kolla-Ansible
uses
--check) - Predict and inspect a run (running
--checkby hand) - Nova deep dive §5 (nova-libvirt restarts)
- Glossary and Components (glossary, service index)
To remember
Lumo: if you keep only 5 things from this page, keep these.
kolla_set_configs --checkcompares only theconfig_filesentries ofconfig.json(content, mode, owner). CODE- The check never compares
command,permissionsor removed entries. - A
COPY_ALWAYSrestart applies changed or added files, mode and owner changes and a changed command. CODE - A newly rendered
optionalfile is now drift (exit 1), not an error. - Run by hand,
kolla_set_configs --checkneedsdocker exec -u root.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.