Fact sheet C¶
All paths are relative to the repo root ($SRC). Every claim comes from reading source code only: kolla-ansible at the
tag 2024.1-eom, the kolla images repo at the tag 2024.1-eom, and, for Ansible semantics, the Ansible source (cited
as ansible-core 2.16.20 lib/ansible/...:N). Modules run inside the kolla-toolbox image come from that image's
ansible-core==2.16.* plus Ansible collections (kolla docker/kolla-toolbox/Dockerfile.j2:76,
docker/kolla-toolbox/requirements.yml:2-13); the collection sources installed in that image are not among the studied
trees.
- FACT (the default, not written out): shown by the cited source lines.
- NOT AVAILABLE: the sources do not establish it, for example upstream OpenStack runtime behaviour, durations, or what a restart does to work in progress.
Behaviour inside the images (what KOLLA_BOOTSTRAP, KOLLA_UPGRADE, KOLLA_OSM, GLANCE_DB_EXPAND and similar
variables actually run) is defined in the separate kolla images repo, whose 2024.1-eom tag was read (see
Kolla images). A statement that cites a kolla path, written as kolla docker/... (for
example kolla docker/glance/glance-api/extend_start.sh or kolla docker/base/set_configs.py), is read from that
source like any other code claim. Image-side behaviour without such a citation is NOT AVAILABLE.
Note
Key answers
- Reconfigure runs the bootstrap (schema sync) container in all five roles (for Horizon only with its DB backend), exactly as deploy does; this still holds in 2024.1 (§0, §6).
- Changed in 2024.1, External Ceph layout: Glance and Cinder now copy one
<cluster>.confand<cluster>.<keyring>per Ceph backend into aceph/sub-directory, andconfig.jsoncopies that directory to/etc/ceph; the first 2024.1 run therefore changes the files ofglance-api,cinder-volumeandcinder-backup(§1.4, §3.4). - Changed in 2024.1, Horizon settings files: kolla-ansible no longer writes
local_settingsandcustom_local_settings; it writes_9998-kolla-settings.pyand_9999-custom-settings.pyintolocal_settings.d, and an override kept under the old names is no longer looked up (§4.4). - New in 2024.1: an S3 backend for Glance and for Cinder backup, a
hostnqnfile forcinder-volume, and RabbitMQ quorum queues on by default in the Glance, Cinder and Heat configs (§1.7, §3.4, F17). - Still holds: the Glance file backend runs a single
glance-apiwith no HA, Cinder's upgrade restarts every Cinder container twice, andbootstrap_heatnever updates theheat_domain_adminpassword (§1.8, §3.8, §5.6).
0. Cross-cutting facts (apply to all five roles)¶
| Topic | Fact | Evidence |
|---|---|---|
| Task dispatch | Every role's tasks/main.yml is - include_tasks: "{{ kolla_action }}.yml". |
e.g. ansible/ (identical in placement/cinder/horizon/heat) |
reconfigure |
reconfigure.yml is just - import_tasks: deploy.yml in all five roles, so reconfigure performs keystone registration, DB create/grant and the bootstrap (db sync) container, exactly as deploy does. Still holds in 2024.1. |
ansible/ |
common |
In 2024.1 none of the five roles has a meta/ directory (each holds only defaults, handlers, tasks, templates, vars), so there is no role dependency on common. common is its own play, tagged common, on the cron, fluentd, kolla-logs and kolla-toolbox groups. |
ansible/ (no meta/); ansible/ |
| Serial | Every play uses serial: '{{ kolla_serial|default("0") }}'. The CLI passes kolla_serial=${ANSIBLE_SERIAL} (default 0) only for upgrade and reconfigure. |
ansible/; tools/ |
max_fail_percentage |
New in 2024.1: each of the five plays sets max_fail_percentage: {{ <svc>_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}. With the default 100 it never stops a play early: the ratio counts only the failures of the current batch (see Controlled batches). |
ansible/; ansible-core 2.16.20 lib/, lib/ |
| run_once + serial | Bootstrap, DB and keystone tasks are run_once: True. When kolla_serial > 0, Ansible runs the whole play once per batch and sends a run_once task only to the first host of the current batch, so run_once tasks run once per batch. So db sync and keystone registration repeat for each batch. This matters for glance rolling upgrade (§1.8). |
e.g. ansible/; ansible-core 2.16.20 lib/, lib/ |
| Bootstrap container | kolla_container action=start_container, detach: False, restart_policy: oneshot (the module was kolla_docker in 2023.1). With Docker, a one-shot container gets no systemd unit and is started directly; the module waits for exit, then stops and removes it (remove_on_exit defaults to True), before it reports a non-zero exit. Nothing guards it with "already bootstrapped", so it runs on every deploy/reconfigure. |
ansible/; ansible/; e.g. ansible/ |
| Restart semantics | Handlers use recreate_or_restart_container and pass common_options: docker_common_options, which carries graceful_timeout: docker_graceful_timeout (60 s) and, new in 2024.1, container_engine: kolla_container_engine. Long-running containers run under a systemd unit kolla-<name>-container.service whose ExecStop is /usr/; restarts and stops go through systemd RestartUnit/StopUnit. With Docker, SIGTERM is sent, then SIGKILL if the container is still running after 60 s. No role here overrides it. The 10 s value is only the module default when no graceful_timeout is passed. The unit template text changed in 2024.1, so the first 2024.1 check of a container whose unit 2023.1 wrote reports drift (F16). |
ansible/; ansible/; ansible/; ansible/; R24 |
| Handler skip | All handlers have when: kolla_action != "config", so genconfig never restarts anything. |
e.g. ansible/ |
| Network | With Docker, all containers run with network_mode: host, so the container hostname is the node hostname (this matters for cinder host). |
ansible/ |
| Empty volume strings | Volume lists contain '' entries (timezone, dev-mode, and in 2024.1 the /dev/shm entry of the queue manager). generate_volumes skips empty entries, so the places that pass lists without reject('equalto','') (placement/cinder upgrade OSM, horizon) are harmless. |
ansible/ |
| Shared endpoints | database_address: "{{ kolla_internal_fqdn }}" (DB is reached through the VIP). rpc_transport_url / notify_transport_url enumerate every host in the rabbit groups. |
ansible/ |
| memcached | Every service template (glance, placement, cinder, heat, horizon) enumerates all groups['memcached'] hosts. Adding or removing a memcached or rabbitmq node therefore changes the config of all five services, and they restart on the next deploy/reconfigure. |
e.g. ansible/ |
| RabbitMQ quorum queues | New default in 2024.1: om_enable_rabbitmq_quorum_queues: true, rendered as rabbit_quorum_queue = true in the glance, cinder and heat configs. A cloud that kept the 2023.1 default gets a changed config, so a restart, on its first 2024.1 deploy/reconfigure (F17). Placement and Horizon have no [oslo_messaging_rabbit] section. |
ansible/; ansible/; ansible/; ansible/ |
| Workers | openstack_service_workers: "{{ [ansible_facts.processor_vcpus, 5] | min }}". The value is per host. |
ansible/ |
| Sanity checks | 2024.1 has no sanity checks: kolla_enable_sanity_* is defined nowhere under ansible/, glance and cinder have no check.yml, and the check.yml of placement, horizon and heat is empty. The CLI has no check action. |
grep -rn sanity ansible/ → nothing; ansible/; tools/ |
| HAProxy | Each role's loadbalancer.yml is applied from the loadbalancer play (site.yml:105-327, tags haproxy, keepalived, loadbalancer), whose task block only runs when enable_haproxy and kolla_action in ['deploy','reconfigure','upgrade','config']. |
ansible/ |
| TLS backend certs | service-cert-copy copies <project>-cert.pem/-key.pem into a service's config dir only when one of its haproxy entries is enabled and one has tls_backend true, and kolla_externally_managed_cert is false. It notifies the container restart. |
ansible/ |
| Service-to-host mapping | host_in_groups (used only by glance) takes precedence over group in the filter select_services_enabled_and_mapped_to_host. That filter drives pull/stop/cert-copy. |
kolla_ansible/ |
| Ansible forks | No ansible.cfg ships in the repo, so "all hosts restart together" really means up to forks hosts at once (Ansible default 5): Ansible starts at most min(forks, batch size) workers and queues each host's task on a free one. The forks value in use depends on the operator's config (ansible.cfg, ANSIBLE_FORKS or kolla-ansible --forks) and is NOT AVAILABLE here. |
find . -name ansible.cfg → none; tools/; ansible-core 2.16.20 lib/, lib/, lib/ |
| Container engine | New in 2024.1: the module dispatches to a Podman worker when kolla_container_engine is not docker (default docker). Cinder-volume and cinder-backup then mount /run without :shared plus the run_default_volumes_podman sub-mounts. This sheet reads the Docker path; Podman behaviour on a host is NOT AVAILABLE. |
ansible/; ansible/ |
site.yml order (relevant slice)¶
loadbalancer play (105) → … → glance (544) → ironic (558) → cinder (576) → placement (593) →
import_playbook: nova.yml (609) → … → heat (724) → horizon (740). Placement is therefore (re)deployed before
nova, and glance before cinder. Still holds in 2024.1 (ansible/site.yml:105,544,558,576,593,609,724,740).
1. GLANCE¶
1.1 Play¶
# ansible/site.yml:544-556
- name: Apply role glance
gather_facts: false
hosts:
- glance-api
- '&enable_glance_True'
serial: '{{ kolla_serial|default("0") }}'
max_fail_percentage: >-
{{ glance_max_fail_percentage |
default(kolla_max_fail_percentage) |
default(100) }}
roles:
- { role: glance,
tags: glance }
- The role line has no
when: enable_glance; the host pattern&enable_glance_True(group built atsite.yml:36) does the selection. - The LB include is at
ansible/site.yml:168-172(tagglance,when: enable_glance). enable_glance: "{{ enable_openstack_core | bool }}"(ansible/group_vars/all.yml:722).
1.2 glance_services (ansible/roles/glance/defaults/main.yml:2-64)¶
In 2024.1 the dict is at ansible/roles/glance/defaults/main.yml:2-68.
| key | container | group / mapping | enabled | flags |
|---|---|---|---|---|
| glance-api | glance_api |
group: glance-api, host_in_groups: inventory_hostname in glance_api_hosts (:6) |
true |
privileged: enable_cinder and (enable_cinder_backend_iscsi or cinder_backend_ceph) (:10); environment: glance_api_container_proxy = container_proxy (:9, :215). This is the only service in scope that receives http(s)_proxy. Volumes (:193-203): glance_file_datadir_volume:/ (default named volume glance), /dev/shm when om_enable_queue_manager (new, :200), and iscsi_info:/ plus /dev:/dev when cinder iSCSI is enabled (:202-203). |
| glance-tls-proxy | glance_tls_proxy |
same host_in_groups (:39) |
glance_enable_tls_backend (= kolla_enable_tls_backend, all.yml:1013) |
HAProxy image (.../haproxy, :160-162). It listens on api_interface_address:9292 with TLS and forwards to glance-api on 127.0.0.1, port 9292 (templates/; glance-api bind_host = 127.0.0.1 when TLS, templates/). Changed in 2024.1: the proxy uses nbthread {{ glance_tls_proxy_threads }} instead of nbproc, so a 2023.1 override of glance_tls_proxy_processes is no longer read (defaults/; templates/). |
Critical mapping rule:
# ansible/group_vars/all.yml:1001,1006-1007,1011
glance_backend_file: "{{ not (glance_backend_ceph | bool or glance_backend_s3 | bool or glance_backend_swift | bool or glance_backend_vmware | bool) }}"
glance_backend_swift: "{{ enable_swift | bool }}"
glance_file_datadir_volume: "glance"
glance_api_hosts: "{{ [groups['glance-api'] | first] if glance_backend_file | bool and glance_file_datadir_volume == 'glance' else groups['glance-api'] }}"
With the default file backend on a local named volume, glance-api runs only on the first host of glance-api.
HAProxy members are built from glance_api_hosts too (defaults/main.yml:86-87), so glance has no HA. A glance-api
restart is a full Image API outage. The docs confirm this: "By default when using file backend only one
glance-api container can be running" (doc/source/reference/shared-services/glance-guide.rst:25-27). Changed in
2024.1: glance_backend_s3 (default "no", all.yml:1004) is a new way to leave the file backend.
HAProxy: client and server timeouts are 6h (defaults/main.yml:139-140). In TLS mode the HAProxy members are the
custom_member_list haproxy_tls_members, which always carries ssl verify required ca-file ...
(defaults/main.yml:55,67,87), and ignore kolla_verify_tls_backend: the template prints a custom member list as
is (haproxy-config/templates/haproxy_single_service_split.cfg.j2:85-88). Other services honour it in
haproxy-config/templates/haproxy_single_service_split.cfg.j2:91-97.
1.3 Task flow per action¶
| action | flow |
|---|---|
| deploy / reconfigure | deploy.yml: register.yml → config.yml → check-containers.yml → (clone.yml if dev) → bootstrap.yml → meta: flush_handlers (tasks/; reconfigure.yml:2). There is no check.yml step. |
| upgrade | upgrade.yml:1-6: rolling_upgrade.yml if glance_enable_rolling_upgrade (default no, all.yml:1008), else legacy_upgrade.yml. No register.yml in either path, so endpoints are not refreshed on upgrade. |
| legacy upgrade | config.yml → check-containers.yml → bootstrap_service.yml (db sync, new image) → flush (tasks/). It does not stop glance-api first. The docs claim "This upgrade method will stop APIs during database schema migrations" (glance-guide.rst:143), but the code does not do that: the old glance-api keeps serving while the new-image bootstrap migrates the schema. |
| rolling upgrade | see §1.8 |
| deploy-containers | check-containers.yml only, then handlers (tasks/) |
| pull | service-images-pull. Only hosts where host_in_groups is true pull (roles/). |
| stop | service-stop with glance_services (tasks/) |
| precheck | service-precheck (groups exist) plus a free-port check on 9292 when glance_api is not running (tasks/). New in 2024.1: when glance_backend_s3, an assert that the four glance_backend_s3_* variables are defined (tasks/). |
| check | Not available in 2024.1: the role has no check.yml, and the CLI has no check action (§0). |
| config | config.yml only. Handlers are notified but skipped. |
1.4 Config files (ansible/roles/glance/tasks/config.yml)¶
dest (under /etc/kolla/) |
source / merge order | when | notifies |
|---|---|---|---|
glance-api/ and glance-tls-proxy/ dirs |
file |
enabled and mapped (:2-13) | – |
glance-api/ dir |
file, mode 0770 (external_ceph.yml:2-14) |
glance_backend_ceph (config.yml:15-17) |
– |
glance-api/ |
changed in 2024.1: merge_configs of {{node_custom_config}}/ + glance/, once per entry of glance_ceph_backends (default one entry, cluster ceph, defaults/). In 2023.1 it was a single template of glance/ to glance-api/. |
glance_backend_ceph (external_ceph.yml:16-27) |
Restart glance-api |
glance-api/ |
template {{node_custom_config}}/; the keyring default is now client.glance.keyring (all.yml:1184), so the default operator file name stays ceph.client.glance.keyring |
ceph (external_ceph.yml:29-38) |
Restart glance-api |
| policy file | first found of supported_policy_format_list in {{node_custom_config}}/ (:19-36, :156-169) |
if found | Restart glance-api |
| CA/TLS certs | copy-certs.yml → service-cert-copy (:38-40) |
kolla_copy_ca_into_containers or glance_enable_tls_backend |
Restart <svc> |
glance-tls-proxy/ |
assemble of glance-(cert|key).pem (:42-54) |
TLS proxy enabled | no notify (the cert copy notifies instead) |
<svc>/ |
glance-api.json.j2, glance-tls-proxy.json.j2 (:56-67) |
– | Restart <svc> |
glance-api/ |
merge: templates/ → global.conf → glance.conf → glance/ → glance/ (:69-86) |
– | Restart glance-api |
glance-api/ |
merge: template → glance-cache.conf → glance/ → glance/ (:88-105). Note that no global.conf is in this merge. |
enable_glance_image_cache |
Restart glance-api |
glance-api/ |
merge: template → glance-swift.conf → glance/ → glance/ (:107-124) |
glance_backend_swift |
Restart glance-api |
glance-api/ |
copy {{node_custom_config}}/ (:126-139) |
glance_enable_interoperable_image_import (default no, all.yml:1010) |
Restart glance-api |
glance-api/ |
copy {{node_custom_config}}/ (:141-154) |
glance_enable_property_protection (default no, all.yml:1009) |
Restart glance-api |
glance-tls-proxy/ |
first_found: glance/ → glance/ → role template (:171-187) |
TLS proxy | Restart glance-tls-proxy |
Changed in 2024.1: with Ceph, glance-api.json.j2 copies the whole ceph directory to /etc/ceph
(templates/glance-api.json.j2:15-21) instead of two separate files. Because the files sit at a new path and the
config.json content changes, the first 2024.1 deploy/reconfigure with glance_backend_ceph notifies the glance-api
restart.
glance-api.json.j2:47-52 has
"permissions": [{"path": "/var/lib/glance", "owner": "glance:glance", "recurse": true}]. /var/lib/glance is the
glance_file_datadir_volume mount (defaults/main.yml:197), which with the file backend is the entire image store
(filesystem_store_datadir = /var/lib/glance/images/, templates/glance-api.conf.j2:65). kolla_set_configs walks it
and calls os.chown on every directory and file, unconditionally (kolla docker/base/set_configs.py:345-355,378-385).
It does this at every container start under the default COPY_ALWAYS (kolla docker/base/start.sh:11,
set_configs.py:538-541; ansible/group_vars/all.yml:43), and only on a container's first start under COPY_ONCE
(kolla set_configs.py:542-551); see
Kolla images §2. NOT AVAILABLE: how long this
recursive chown takes at container start with a big NFS-backed glance_file_datadir_volume.
1.5 Handlers (ansible/roles/glance/handlers/main.yml)¶
Restart glance-api container(:2-18): recreate_or_restart. It passesprivileged,environment(the proxy), volumes, dimensions and healthcheck.Restart glance-tls-proxy container(:20-34): recreate_or_restart.- Neither has special logic beyond
kolla_action != "config". There is no rolling restart and no wait for the API to come back.
1.6 Bootstrap¶
| step | where | re-runs on reconfigure? |
|---|---|---|
mysql_db glance |
tasks/, run_once, delegate groups['glance-api'][0], unless use_preconfigured_databases |
yes (idempotent) |
mysql_user glance with password and ALL on glance.* |
bootstrap.yml:18-36 |
yes. The task runs mysql_user through kolla_toolbox, which runs ansible localhost -m mysql_user in the toolbox container (ansible/), whose image installs ansible-core==2.16.* (kolla docker/); ansible-core redirects mysql_user to community.mysql.mysql_user (ansible-core 2.16.20 lib/), installed in the image from community.mysql <3 (kolla docker/). That collection's source is not among the studied trees, so whether each run resets the stored password when it differs is NOT AVAILABLE. |
Set GLOBAL log_bin_trust_function_creators=1 |
bootstrap_service.yml:2-17, logged in as glance_database_shard_root_user (the DB root user without ProxySQL, defaults/), unless use_preconfigured_databases |
yes, on every deploy/reconfigure |
bootstrap_glance container, KOLLA_BOOTSTRAP: glance-manage db_sync then glance-manage db_load_metadefs (kolla docker/) |
bootstrap_service.yml:19-37, delegate glance_api_hosts[0] |
yes, so a schema sync runs on every reconfigure |
Set log_bin_trust_function_creators=0 |
bootstrap_service.yml:39-54 |
yes |
Keystone: service glance/image, 2 endpoints (internal, public), user glance (admin in service) |
tasks/; defaults/ |
yes (deploy/reconfigure); not on upgrade |
Oddity / risk: the enable/db_sync/disable sequence is still not wrapped in block/always. If the bootstrap
container fails, the global MariaDB variable stays at 1. Whether this leftover state has security consequences is NOT
AVAILABLE. A related upstream fix for keystone appears in
releasenotes/notes/bug-1987982-fix-keystone-log_bin_trust_function_creators-variables-8628098891513dac.yaml.
Reconfigure as uncontrolled upgrade: if an operator bumps openstack_tag/glance_tag and runs reconfigure, the
new image runs the KOLLA_BOOTSTRAP branch (glance-manage db_sync and db_load_metadefs, kolla
docker/glance/glance-api/extend_start.sh:5-9) and the handler recreates glance-api on the new image. For the glance
role that is the default legacy upgrade (glance_enable_rolling_upgrade: "no", all.yml:1008):
legacy_upgrade.yml:2-9 runs the same config, check-containers, bootstrap_service.yml and flush steps, and
reconfigure only adds registration, DB/user creation and the dev-mode clone (deploy.yml:2-15).
1.7 Dependencies (from templates/glance-api.conf.j2)¶
- MariaDB via the VIP (:34-38); RabbitMQ:
transport_url = rpc_transport_url(:27) and notifications (:118-125; topicnotificationsonly if ceilometer, otherwisedriver = noop,defaults/main.yml:109-113).[oslo_messaging_rabbit]gainsrabbit_quorum_queue = trueby default and, with the off-by-default queue manager,use_queue_manager(:127-148). - Keystone:
www_authenticate_uriandauth_urlare bothkeystone_internal_url(:40-51). memcached: all hosts (:53-55). - Cinder:
cinder_catalog_info = volume:cinder:internalURL(:25), andenabled_backendscontainscinder:cinderwheneverenable_cinder(defaults/main.yml:233-235; template :17-19). Oddity, still holds in 2024.1: cinder registers onlycinderv3/volumev3(roles/cinder/defaults/main.yml:335-341). Novolume/cindercatalog entry exists, and the template has no[cinder]store section, so the glance cinder store is enabled without a matching catalog entry or store section. Whether it works if used is NOT AVAILABLE. Nova, by contrast, usescatalog_info = volumev3:cinderv3:internalURL(roles/nova-cell/templates/nova.conf.j2:113). - Ceph: changed in 2024.1, one
[<backend name>]section per entry ofglance_ceph_backends(default[rbd]), withrbd_store_user/pooland the newrbd_store_ceph_conf = /etc/ceph/<cluster>.conf(:68-75). The template still does not setshow_multiple_locations(grep -rn show_multiple_locations ansible/roles/glance→ nothing). - Default store: changed in 2024.1,
default_backend = "{{ glance_default_backend }}"(:60-61), whereglance_default_backendpicks vmware, then the first Ceph backend, then swift, then s3, then file (defaults/main.yml:247). The value is rendered between double quotes; how glance's config parser reads the quotes is NOT AVAILABLE. - Swift:
[swift]section (:77-87) withswift_store_auth_insecure = Truehard-coded.glance-swift.confauthenticates againstopenstack_auth.auth_url(=keystone_internal_url,all.yml:990-991) asservice:glance(templates/glance-swift.conf.j2:1-8). - S3 (new in 2024.1):
[s3]withs3_store_host/access_key/secret_key/bucketfromglance_backend_s3_*, which default to the globals3_url,s3_bucket,s3_access_key,s3_secret_key(:89-95;defaults/main.yml:145-148;all.yml:1282-1285). - Barbican (:163-168), osprofiler (:155-161), VMware (:97-104). New in 2024.1:
[cors] allowed_origin = {{ horizon_public_endpoint }}(:170-171). - Cinder iSCSI flag toggles glance-api
/devandiscsi_info, and cinder iSCSI orcinder_backend_cephtogglesprivileged(defaults/main.yml:10,202-203). The code comment says the mounts are "to support Cinder iSCSI backends" (defaults/main.yml:201). How glance uses these mounts is NOT AVAILABLE.
1.8 Operational notes¶
-
Backend switch via unrelated flags:
glance_backend_swiftdefaults toenable_swift(all.yml:1006), andglance_backend_fileis the negation of ceph/s3/swift/vmware (all.yml:1001). Enabling Swift (orglance_backend_ceph, or the newglance_backend_s3) on a running cloud does three things:- removes
filefromenabled_backendsand drops the[file]section (defaults/main.yml:224-226; template :17-19,63-66); - changes
default_backend(defaults/main.yml:247; template :60-61); - expands
glance_api_hoststo everyglance-apihost (all.yml:1011).
NOT AVAILABLE: whether existing images in the file store stay readable after the switch.
A related doc mismatch, still present: the guide says ceph is enabled "automatically" (
glance-guide.rst:43), but the default isglance_backend_ceph: "no"(all.yml:1002). - removes
-
Inventory order matters: with the file backend, glance runs on
groups['glance-api']|first. The INI parser reads the file line by line and adds each host in that order, and Ansible lists a group's hosts in the order they were added (ansible-core 2.16.20lib/ansible/plugins/inventory/ini.py:160,215-217,lib/ansible/plugins/inventory/__init__.py:205-210,lib/ansible/inventory/group.py:231-237,280-292,lib/ansible/inventory/data.py:258-268), so reordering the inventory moves glance-api to another host, and the HAProxy members follow it (all.yml:1011;defaults/main.yml:86-87). NOT AVAILABLE: what theglancevolume on the new host contains. - Restart impact: the HAProxy client and server timeouts are 6h (
defaults/main.yml:139-140). Restarting glance-api stops it with the 60 s graceful timeout (§0). With the file backend this is a total Image API outage, because glance-api is the only backend (§1.2). NOT AVAILABLE: what the restart does to in-flight uploads, downloads, nova snapshots and cinder upload-to-image operations, whether images are left insaving/queued, and whether nova boots that need an uncached image fail during the window. - Rolling upgrade (
tasks/rolling_upgrade.yml), unchanged in 2024.1:- Sequence: config on host[0] (:7-8) → stop glance-api on host[0] (:10-11,
stop_service.yml) → enable log_bin (:13-28) →GLANCE_DB_EXPAND(:30-49) →GLANCE_DB_MIGRATE, which notifies restart (:51-73) → config on the other hosts (:75-76) → check-containers (:78) → flush handlers, so all glance-api containers restart in the same handler run (:80-81) →GLANCE_DB_CONTRACT(:83-102) → disable log_bin (:104-119). - The
glance_upgradingfact (:2-4, :121-123) is set but never read anywhere (dead code;grep -rn glance_upgrading ansible/finds only this file). - With
kolla_serial> 0, each batch re-runs the run_once expand/migrate/contract onglance_api_hosts[0](:48-49, :69-70, :101-102), because Ansible runs the play once per batch and sends arun_oncetask to the first host of each batch (§0). The contract therefore runs at the end of batch 1, while hosts in later batches still run the old glance-api containers. - The docs warn that rolling upgrade with a non-shared file backend leads to "a corrupt state"
(
glance-guide.rst:134-136).
- Sequence: config on host[0] (:7-8) → stop glance-api on host[0] (:10-11,
- Image cache is never pruned. The docs say "Glance caches are not cleaned up automatically"
(
glance-guide.rst:173); there is no cron/pruner container. - Sanity check (if enabled) creates and deletes an image on every deploy/reconfigure. Not available in 2024.1: the
role has no
check.yml,tasks/deploy.yml:1-15has no check step, andkolla_enable_sanity_glanceis not defined (§0). - Glance does not touch compute hosts. Nova-compute reaches glance through
[glance] api_servers=glance_internal_endpoint(internal FQDN, port 9292,all.yml:1015;roles/nova-cell/templates/nova.conf.j2:105-109), and cinder throughglance_api_servers(roles/cinder/templates/cinder.conf.j2:18).
2. PLACEMENT¶
2.1 Play¶
# ansible/site.yml:593-605
- name: Apply role placement
gather_facts: false
hosts: [placement-api, '&enable_placement_True']
serial: '{{ kolla_serial|default("0") }}'
max_fail_percentage: '{{ placement_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}'
roles: [{ role: placement, tags: placement }]
enable_placement: "{{ enable_nova | bool or enable_zun | bool }}"(all.yml:837).- Oddity, still holds in 2024.1: the LB include for placement has no
when: enable_placement(site.yml:248-251), unlike every other service. When placement is disabled it still writes an (effectively empty)services.d/placement-api.cfg, becauseplacement-api.enabled: True(roles/placement/defaults/main.yml:7) and the haproxy-config task checks onlyservice.enabled,service.haproxyandenable_haproxy(roles/haproxy-config/tasks/main.yml:17-20). Only the inner haproxy entries carryenable_placement(roles/placement/defaults/main.yml:13,20), and the template skips disabled entries (roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:121). - The play sits right before
import_playbook: nova.yml(site.yml:609).
2.2 placement_services (ansible/roles/placement/defaults/main.yml:2-24)¶
In 2024.1 the dict is at ansible/roles/placement/defaults/main.yml:2-26.
| key | container | group | enabled | flags |
|---|---|---|---|---|
| placement-api | placement_api |
placement-api |
True |
Plain Apache/mod_wsgi. No privileged, no host paths beyond config, localtime, timezone (Debian) and logs, plus the dev-mode source (:80-85). Unlike glance, cinder and heat it gets no /dev/shm entry. HAProxy internal and external on 8780, tls_backend: placement_enable_tls_backend (:11-26); the external entry now names external_fqdn and placement_api_public_port (:23-24). |
2.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | register.yml → (clone) → config.yml → check-containers.yml → bootstrap.yml → flush (tasks/) |
| upgrade | register.yml → (clone) → config.yml → check-containers.yml → bootstrap.yml (KOLLA_BOOTSTRAP with the new image: placement-manage db sync and db online_data_migrations, kolla docker/, while the old placement-api is still running; the image's KOLLA_UPGRADE branch, db sync only, :12-15, is not used) → flush (all placement-api restart) → KOLLA_OSM online data migrations container, which runs them a second time (tasks/) |
| deploy-containers | check-containers.yml |
| pull / stop | service-images-pull / service-stop |
| precheck | groups exist, port 8780 free if not running (tasks/) |
| check | empty file (tasks/) |
| config | config.yml |
2.4 Config files (ansible/roles/placement/tasks/config.yml)¶
| dest | sources | notifies |
|---|---|---|
placement-api/ |
placement-api.json.j2 (:38-49) |
Restart placement-api |
placement-api/ |
merge: template → global.conf → placement.conf → placement/ → placement/ (:51-69) |
Restart |
placement-api/ |
first_found: placement/ → placement/ → template (:71-87). Changed in 2024.1: the template's ErrorLog is placement-api-error.log (templates/), so the file changes on the first 2024.1 run. |
Restart |
placement-api/ |
migrate-db.rc.j2 (:89-101) |
Restart |
| policy | {{node_custom_config}}/ (:15-32, :103-115) |
Restart |
| certs | service-cert-copy (:34-36) | Restart |
migrate-db.rc still contains the nova_api DB user and password together with the placement DB creds
(templates/migrate-db.rc.j2:1-8; vars at defaults/main.yml:122-125). The defaults file files these vars under
"Upgrade" (defaults/main.yml:119-121); no task of the role uses it, and the container receives it as
/etc/placement/migrate-db.rc (templates/placement-api.json.j2:25-26). It spreads a nova secret into the placement
container and is notified on change.
2.5 Handlers¶
Restart placement-api container: recreate_or_restart, no special logic (handlers/main.yml:2-16).
2.6 Bootstrap¶
mysql_db placementandmysql_user(tasks/bootstrap.yml:2-36) re-run on reconfigure.bootstrap_placementwithKOLLA_BOOTSTRAP(kolla image:placement-manage db syncthenplacement-manage db online_data_migrations, kolladocker/placement/placement-api/extend_start.sh:5-9) runs once, delegated togroups['placement-api'][0](bootstrap_service.yml:2-20). It re-runs on every reconfigure.- Keystone registration (service
placement, 2 endpoints, internal and public, userplacement,defaults/main.yml:130-142) happens in deploy, reconfigure and upgrade (upgrade.yml:2), unlike glance and heat. - Online data migrations: the separate
KOLLA_OSMcontainer runs only in upgrade (upgrade.yml:16-34). Because the bootstrap container above also runsdb online_data_migrations, placement OSM also runs on every deploy/reconfigure (deploy.yml:11,reconfigure.yml:2) and twice in upgrade (bootstrap atupgrade.yml:11, thenKOLLA_OSM).
2.7 Dependencies (templates/placement.conf.j2)¶
- MariaDB
[placement_database]withmax_overflow = 1000hard-coded (:14-19). - memcached
[cache]andkeystone_authtoken(:21-42); Keystone (:27-38). - The template sets no
transport_url(placement does no RPC). [DEFAULT]holds only debug, log and state paths (:1-6).- Dead variables:
placement_notification_topics/placement_enabled_notification_topics(defaults/main.yml:104-108) are not referenced by any template. There is no[oslo_messaging_notifications]section.
2.8 Operational notes and the relation to nova¶
- Consumers authenticate as the placement user. nova-api/conductor/scheduler
(
roles/nova/templates/nova.conf.j2:192-202) and nova-compute on every compute host (roles/nova-cell/templates/nova.conf.j2:234-244) useusername = {{ placement_keystone_user }}/password = {{ placement_keystone_password }},valid_interfaces = internal. Neutron does the same withendpoint_type = internal(roles/neutron/templates/neutron.conf.j2:196-206), as does cyborg (roles/cyborg/templates/cyborg.conf.j2:34-38); zun does not (its[placement_client]has no credentials,roles/zun/templates/zun.conf.j2:92-95). Rotatingplacement_keystone_passwordtherefore needs a reconfigure of placement and nova (all computes) and neutron. This dependency scales with the number of compute hosts. - NOT AVAILABLE (upstream nova): how scheduling and the nova-compute resource tracker periodic task behave during a
placement-api outage or restart. With serial=0 every placement-api restarts in the same handler flush, up to
forkshosts at once (§0); whether and for how long no placement-api is serving is NOT AVAILABLE. - The upgrade db sync and online data migrations (bootstrap container) run while the old placement-api is still serving (no stop).
3. CINDER¶
3.1 Play¶
# ansible/site.yml:576-591
- name: Apply role cinder
gather_facts: false
hosts: [cinder-api, cinder-backup, cinder-scheduler, cinder-volume, '&enable_cinder_True']
serial: '{{ kolla_serial|default("0") }}'
max_fail_percentage: '{{ cinder_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}'
roles: [{ role: cinder, tags: cinder }]
- LB include:
site.yml:143-147. - Defaults:
enable_cinder: "no",enable_cinder_backup: "yes",enable_cinder_backend_iscsi: "{{ enable_cinder_backend_lvm | bool }}",enable_cinder_backend_lvm: "no"(all.yml:750-754).cinder_backend_ceph: "no",cinder_backup_driver: "ceph"(all.yml:1041,1050). - Sample inventory:
cinder-api/cinder-schedulersit under[cinder:children] control.cinder-volume/cinder-backupsit understorage(ansible/inventory/multinode:110-111,285-295).
3.2 cinder_services (ansible/roles/cinder/defaults/main.yml:2-48)¶
In 2024.1 the dict is at ansible/roles/cinder/defaults/main.yml:2-54.
| key | container | group | enabled | flags / host volumes |
|---|---|---|---|---|
| cinder-api | cinder_api |
cinder-api | true | Apache/mod_wsgi on 8776; HAProxy internal and external (:11-26) |
| cinder-scheduler | cinder_scheduler |
cinder-scheduler | true | – |
| cinder-volume | cinder_volume |
cinder-volume | true | privileged: True, ipc_mode: host (:40-41), tmpfs for /var/ only if cinder_enable_conversion_tmpfs (default false, :42,206-210). Volumes (:187-198, plus run_default_volumes_<engine>, empty for Docker, :43): /dev/:/dev/, /lib/modules:ro, /run:/ (:shared only with Docker), cinder:/, iscsi_info:/ (if enable_iscsid), target_config:/ (LVM and lioadm). |
| cinder-backup | cinder_backup |
cinder-backup | enable_cinder_backup |
privileged: True (no ipc host). Volumes (:169-179, plus run_default_volumes_<engine>, :52): /dev/, /lib/modules, /run (:shared with Docker), cinder:/, iscsi_info. |
Related: cinder_target_helper is lioadm on RedHat and tgtadm otherwise (all.yml:1045). With tgtadm the targets
are served by a separate tgtd container from the iscsi role, which also deploys iscsid on compute,
cinder-volume and cinder-backup hosts (roles/iscsi/defaults/main.yml:2-20,62-73; play site.yml:439-452).
3.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | register.yml → config.yml → check-containers.yml → (clone) → bootstrap.yml → flush (tasks/). There is no check.yml step. |
| upgrade | config.yml only on groups['cinder-api'][0] (:4-5) → bootstrap_service.yml (db sync, new image, old services still running) (:7) → config.yml on all (:9) → check-containers.yml (:11) → service-ks-register for the cinder user's service role grant only (:13-18; changed in 2024.1: it no longer creates the service role, and a comment says to remove the task in the next cycle) → flush, so every cinder container on every host restarts in this flush (serial 0), cinder-volume and cinder-backup included; on upgrade a handler then pauses cinder_rpc_version_startup_delay (30 s) once (:20-21; handlers/; defaults/) → reload.yml: restart_container of every enabled cinder container on the host again, "to remove RPC version pin" (:23; tasks/) → KOLLA_OSM online migrations (:25-43). No register.yml, so services and endpoints are not refreshed. |
| deploy-containers | check-containers |
| pull / stop | helpers |
| precheck | groups; API port free; at least one backend enabled unless skip_cinder_backend_check (:28-44); vgs cinder-volumes on cinder-volume hosts if LVM (:46-56); new in 2024.1, an assert on the four cinder_backup_s3_* variables when cinder_backup_driver == "s3" (tasks/) |
| check | Not available in 2024.1: the role has no check.yml, and kolla_enable_sanity_cinder is not defined (§0). |
| config | config.yml |
3.4 Config files (ansible/roles/cinder/tasks/config.yml)¶
| dest | sources | when | notifies |
|---|---|---|---|
cinder-volume/, cinder-backup/ dirs |
file, mode 0770 (external_ceph.yml:2-17) |
cinder_backend_ceph and host in volume/backup (config.yml:15-19) |
– |
<svc>/ (cinder-volume, cinder-backup) |
changed in 2024.1: merge cinder/ + cinder/, once per service and per entry of cinder_ceph_backends (default one entry rbd-1, cluster ceph, defaults/) (external_ceph.yml:19-42). In 2023.1 the sources were cinder/ + cinder/ and the destination <svc>/. |
cinder_backend_ceph and host in volume/backup |
Restart <svc> |
cinder-volume/ |
cinder/ (external_ceph.yml:44-58; all.yml:1182) |
cephx | Restart cinder-volume |
cinder-backup/ keyrings (cinder plus cinder-backup) |
cinder/ and .client.cinder-backup.keyring, cluster of cinder_backup_ceph_backend (external_ceph.yml:60-75; defaults/; all.yml:1182-1183) |
cephx | Restart cinder-backup |
| policy | {{node_custom_config}}/ (:21-38, :110-122) |
Restart <svc> |
|
| certs | service-cert-copy (:40-42) | ||
<svc>/ |
<svc>.json.j2 (:44-55) |
Restart <svc> |
|
cinder-api/ |
first_found: cinder/ → cinder/ → template (:57-73) |
Restart cinder-api | |
<svc>/ (one per service, service_name var) |
merge: template → global.conf → cinder.conf → cinder/ → cinder/ (:75-93) |
Restart <svc> |
|
cinder-volume/ (new in 2024.1) |
template hostnqn.j2, value nqn.2014-08.org.nvmexpress:uuid:<to_uuid of the host name> (:95-108); copied into the container as /etc/, owner root (templates/) |
host in cinder-volume | Restart cinder-volume |
cinder-volume/ |
first_found among 8 paths (nfs_shares.j2 variants, then plain), skip unless NFS or HNAS (:124-143) |
host in cinder-volume | Restart cinder-volume |
Because the Ceph files move into ceph/ (copied to /etc/ceph by templates/cinder-volume.json.j2:9-15 and
templates/cinder-backup.json.j2:15-21) and hostnqn is a new file, the first 2024.1 deploy/reconfigure changes the
config of every cinder-volume (and, with Ceph, cinder-backup) and notifies its restart.
3.5 Handlers (ansible/roles/cinder/handlers/main.yml)¶
Four restart handlers (api :2-16, scheduler :18-32, volume :34-51, backup :53-68), all recreate_or_restart. The volume
handler passes privileged, ipc_mode and tmpfs; the backup handler passes privileged. A fifth handler,
Wait for cinder services to update service versions (:78-88), listens to all four and pauses 30 s once, only when
kolla_action == 'upgrade'. There is no drain logic. Unchanged in 2024.1 apart from the module name.
3.6 Bootstrap¶
mysql_db cinderandmysql_userrun every deploy/reconfigure (bootstrap.yml:2-36).bootstrap_cinder(KOLLA_BOOTSTRAP:cinder-manage db sync, kolladocker/cinder/cinder-api/extend_start.sh:6-9), run_once ongroups['cinder-api'][0](bootstrap_service.yml:2-20). It re-runs on every reconfigure.- Online data migrations (
KOLLA_OSM) run only in upgrade. - Keystone: only
cinderv3/volumev3, with 2/v3/%(tenant_id)sendpoints (internal, public); usercinder(admin inservice); theservicerole granted to the cinder user in theserviceproject (defaults/main.yml:218-219,335-352;tasks/register.yml:2-8). Changed in 2024.1: the role no longer passesservice_ks_register_roles, so it no longer creates theservicerole itself (service_ks_register_rolesdefaults to[],roles/service-ks-register/defaults/main.yml:29; roles created atroles/service-ks-register/tasks/main.yml:87-97). Which component creates theservicerole before the grant in 2024.1 is NOT AVAILABLE from these roles. Services and endpoints are registered on deploy/reconfigure, not on upgrade.
3.7 Dependencies (templates/cinder.conf.j2)¶
- Glance:
glance_api_servers=glance_internal_endpoint(internal FQDN, port 9292) (:18).glance_num_retries = {{ groups['glance-api'] | length }}(:20) scales with the number of glance hosts. With the glance file backend only one glance-api exists, yet retries equal the group size. - Nova:
[nova]section authenticates withnova_keystone_user/nova_keystone_password(:109-119), so rotating the nova password needs a cinder reconfigure. In the other direction, nova-api and nova-compute usecinder_keystone_user/password(roles/nova-cell/templates/nova.conf.j2:112-121,roles/nova/templates/nova.conf.j2:66-75), so rotating the cinder password needs a reconfigure of all computes. Manila also uses the cinder creds (roles/manila/templates/manila-share.conf.j2:29-30). - MariaDB (:121-125); RabbitMQ RPC (:64) and notify (:66-73; topic
notificationsfor ceilometer only,defaults/main.yml:285-289; the 2023.1 vitrage topic is gone with the vitrage role);[oslo_messaging_rabbit]with quorum queues by default (:75-99); Keystone and memcached (:127-146); Barbican (:264-269); osprofiler (:256-262). - Ceph: changed in 2024.1, one section per entry of
cinder_ceph_backends(default[rbd-1]) withrbd_ceph_conf = /etc/ceph/<cluster>.conf, the newrbd_cluster_nameandrbd_keyring_conf, an optionalbackend_availability_zone, andrbd_secret_uuid = cinder_rbd_secret_uuid(:161-178). The same UUID is installed into libvirt on computes byroles/nova-cell/tasks/external_ceph.yml:169,194, so the two must stay in sync. - Backup drivers: ceph (
backup_ceph_conf = /etc/ceph/<cluster of cinder_backup_ceph_backend>.conf), nfs, swift (backup_swift_urlinternal) and, new in 2024.1, s3 (:27-58). - Coordination: redis or etcd (:271-281). The etcd URL is
etcd3+<internal_protocol>://{{ kolla_internal_fqdn }}:{{ etcd_client_port }}?api_version=v3(:280; the?api_version=v3query is new in 2024.1), and the etcd role defines an internal HAProxy entry onetcd_client_port(roles/etcd/defaults/main.yml:11-17), so cinder reaches etcd through the internal VIP. Defaultcinder_coordination_backendis''unless redis or etcd is enabled (all.yml:1047;enable_etcd: "no",enable_redis: "no",all.yml:764,840). - LVM/iSCSI:
target_helper,target_protocol = iscsi(:152-159).
3.8 Operational notes¶
- No
host/backend_host/clusterset in cinder.conf (templates/cinder.conf.j2has none;grep backend_host ansible/roles/cinderreturns nothing). Because ofnetwork_mode: host, each cinder-volume container has the node hostname (§0). Whether cinder then registers it as<node-hostname>@<backend>is upstream cinder behaviour and NOT AVAILABLE here.- NOT AVAILABLE (upstream cinder): whether, with a shared backend (ceph
rbd-1) on several cinder-volume hosts, every volume is owned by the host that created it. - NOT AVAILABLE: whether, if that node is down, its volumes cannot be attached, detached, deleted or snapshotted
until the node returns or
cinder-manage volume update_hostis run. - The same question applies to renaming or replacing a storage node. The new
hostnqnis also derived from the host name (tasks/config.yml:98), so a renamed node gets a new NQN. The docs also note LVM issues in a multi-controller setup (doc/source/reference/storage/cinder-guide.rst:52-55).
- NOT AVAILABLE (upstream cinder): whether, with a shared backend (ceph
- cinder-volume restart impact (upstream cinder and os-brick behaviour, NOT AVAILABLE beyond the code facts):
- Code facts: with LVM, the LIO config is kept in the
target_configvolume (defaults/main.yml:196) or the targets are served by the separatetgtdcontainer (§3.2); with Ceph, the RBD secret is installed into libvirt on computes (§3.7). NOT AVAILABLE: whether the data path for attached volumes bypasses cinder-volume, and so whether existing attachments survive a cinder-volume restart. - In-flight operations: the stop sends SIGKILL if cinder-volume is still running after the 60 s graceful timeout
(§0). NOT AVAILABLE: whether this kills
qemu-img/ddcopies (image→volume, volume→image, migrate, retype) and what state they leave. - NOT AVAILABLE: what
init_hostdoes on start with resources left increating,downloadingoruploading, and whether it re-runsensure_exportfor in-use volumes.
- Code facts: with LVM, the LIO config is kept in the
- cinder-backup restart: NOT AVAILABLE: whether backups and restores in progress are killed and marked
erroron the next start. Because of thecinder_backup_driverdefault, see oddity 5. - Upgrade restarts every cinder-volume and cinder-backup at once, twice (flush at
upgrade.yml:20-21, thenreload.ymlatupgrade.yml:23; serial 0 unlessANSIBLE_SERIALis set). There is no[upgrade_levels]pinning in the template. The code comment explains the second restart: after an upgrade, services keep an RPC version cap until all of them are restarted, and the role waits 30 s for the services to update their version in the database (handlers/main.yml:70-76). Thereload.ymlrestart is a plainrestart_container, withoutwhen: kolla_action != "config"or any other condition, on every enabled cinder service mapped to the host (tasks/reload.yml:2-10). NOT AVAILABLE: whether 30 s is enough on a given cloud, and whether in-flight operations survive two restarts in a row. - Oddity: backup driver vs. ceph files, still holds in 2024.1.
cinder_backup_driver: "ceph"is the default even whencinder_backend_cephisno. cinder.conf then setsbackup_driver = ...CephBackupDriver,backup_ceph_conf=/etc/ceph/ceph.conf(:28-36; the cluster ofcinder_backup_ceph_backendiscepheven when the backend is disabled,defaults/main.yml:270-276). Butcinder-backup.json.j2only copies thecephdirectory{% if cinder_backend_ceph %}(:15-21), andexternal_ceph.ymlonly runs forcinder_backend_ceph(config.yml:15-19). With an LVM or NFS-only deployment and default backup settings, kolla-ansible therefore gives cinder-backup no ceph config. Whether its driver then fails is NOT AVAILABLE. - Multipath is never configured on the cinder side: no
use_multipath_for_image_xfer/volume_use_multipathin the cinder templates (grep multipath ansible/roles/cinderreturns nothing). On the nova side,volume_use_multipath = Trueis set in nova-compute's libvirt config whenenable_multipathdis on (roles/nova-cell/templates/nova.conf.d/libvirt.conf.j2:27-28). Themultipathdcontainer is off by default (all.yml:809). - Compute-host touchpoints: the cinder role itself does not target computes. However,
iscsidon computes (iscsi role) sharesiscsi_info, nova-compute uses the cinder creds, and the RBD secret is shared with nova-libvirt. Enablingenable_cinder_backend_iscsi/LVM also flips glance-api to privileged with/dev, so glance-api is recreated (roles/glance/defaults/main.yml:10,202-203).cinder_backend_ceph(with Cinder enabled) also makes glance-api privileged, without/dev(roles/glance/defaults/main.yml:10). - External Ceph needs cinder-volume and cinder-backup hosts in the
storagegroup, otherwise "Cinder and related services relying on this group [will] fail" (doc/source/reference/storage/external-ceph-guide.rst:146-155). - The sanity check (if enabled) creates a 1 GB volume on every deploy/reconfigure. Not available in 2024.1: the role
has no
check.yml(§3.3).
4. HORIZON¶
4.1 Play¶
# ansible/site.yml:740-752
- name: Apply role horizon
gather_facts: false
hosts: [horizon, '&enable_horizon_True']
serial: '{{ kolla_serial|default("0") }}'
max_fail_percentage: '{{ horizon_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}'
roles: [{ role: horizon, tags: horizon }]
- LB include:
site.yml:188-192.enable_horizon: "{{ enable_openstack_core | bool }}"(all.yml:773). - Horizon is the last of these plays.
4.2 horizon_services (ansible/roles/horizon/defaults/main.yml:2-69)¶
In 2024.1 the dict is at ansible/roles/horizon/defaults/main.yml:2-73.
| key | container | group | enabled | flags |
|---|---|---|---|---|
| horizon | horizon |
horizon | true | environment: 16 ENABLE_<PLUGIN> flags plus FORCE_GENERATE (:8-25). Changed in 2024.1: the freezer, murano, sahara, senlin, solum and vitrage flags are gone with their roles, and ENABLE_FWAAS is new (:12). There is no host /tmp bind: the volumes are config, dev-mode sources, localtime, timezone and logs (:123-129). Settings use LOCAL_PATH = '/ (templates/), which is therefore the container's own /tmp. HAProxy uses balance roundrobin when enable_memcached or horizon_backend_database, balance source otherwise (:39, :59; enable_memcached: "yes", all.yml:729, so roundrobin by default). Redirect frontends (mode: redirect, port 80) are added when internal or external TLS is on (:41-48, :61-69), and an acme_client backend is defined (:70-73); new in 2024.1, every frontend routes /.well-known/ to it (:37, :48, :57, :69). |
Because the plugin flags live in the container environment, and check-containers compares environment
(tasks/check-containers.yml:11), enabling or disabling any service that has a horizon plugin (designate, octavia,
magnum, …) changes the env. The horizon container is then recreated on the next deploy/reconfigure even if no
horizon config changed. On the first 2024.1 run the removed flags alone change the environment of a container created by
2023.1.
4.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | config.yml → check-containers.yml → (clone) → bootstrap.yml only if horizon_backend_database → flush (tasks/). There is no keystone registration. |
| upgrade | config.yml → check-containers.yml → (clone) → bootstrap_service.yml if DB backend → flush (tasks/) |
| precheck | groups; horizon_listen_port free (80, or 443 when horizon_enable_tls_backend; all.yml:423-425) (tasks/) |
| check | empty (tasks/) |
| pull / stop / deploy-containers / config | standard |
4.4 Config files (ansible/roles/horizon/tasks/config.yml)¶
| dest | source | notifies |
|---|---|---|
per-service policy files (<svc>_policy.json/) |
for each enabled service in the list (:19-41), policy_item.yml looks for {{node_custom_config}}/ (tasks/) and copies the matches (:111-124) |
Restart horizon |
horizon/ |
horizon.json.j2 (:43-55) |
Restart horizon |
horizon/ (Apache) |
first_found: horizon/ → horizon/ → template (:57-73). This is a whole-file override, not a merge. |
Restart horizon |
horizon/ |
changed in 2024.1 (was local_settings): first_found: horizon/ → horizon/ → _9998-kolla-settings.py.j2 (:75-91). Whole-file override. |
Restart horizon |
horizon/ |
changed in 2024.1 (was custom_local_settings): first_found, same pattern; the default template is empty (:93-109; templates/ has 0 lines). |
Restart horizon |
| custom themes | {{node_custom_config}}/ for each horizon_custom_themes entry (default [], defaults/) (:126-140) |
Restart horizon |
| certs | service-cert-copy (:142-144) |
Changed in 2024.1, the settings layout: horizon.json.j2 copies both files into
/etc/openstack-dashboard/local_settings.d/ (templates/horizon.json.j2:21-32) instead of replacing
/etc/openstack-dashboard/local_settings. The base local_settings.py is now the one the image ships: the kolla image
copies Horizon's local_settings.py.example to /etc/openstack-dashboard/local_settings.py and links the dashboard's
local_settings.d to /etc/openstack-dashboard/local_settings.d (kolla docker/horizon/Dockerfile.j2:56-65).
Consequences read from the code:
- An operator override kept under the old names (
/etc/kolla/config/horizon/local_settingsorcustom_local_settings) is no longer looked up (tasks/config.yml:83-86,101-104): it is silently ignored on the first 2024.1 run. _9998-kolla-settings.pyis a fragment, not a full settings file: it assigns intoCACHES['default'](templates/_9998-kolla-settings.py.j2:21-22), so it relies on the base settings definingCACHES. How Horizon loadslocal_settings.d(order, error handling) is upstream Horizon behaviour and NOT AVAILABLE.
4.5 Handlers¶
Restart horizon container (handlers/main.yml:2-17): recreate_or_restart. It passes environment; there is no other
logic.
4.6 Bootstrap¶
- Changed in 2024.1:
horizon_backend_database: false(all.yml:1141); in 2023.1 it followedenable_murano, a role that no longer exists. Only when an operator sets it does deploy/reconfigure create the DB and user and runbootstrap_horizon(KOLLA_BOOTSTRAP: Djangomanage.py migrate --noinput, then exits; kolladocker/horizon/extend_start.sh:27-30) (tasks/bootstrap.yml:2-38,bootstrap_service.yml:2-20). In that case the bootstrap re-runs on every reconfigure. - There is no keystone registration at all.
4.7 Dependencies (templates/_9998-kolla-settings.py.j2)¶
In 2024.1 the dependencies are read in _9998-kolla-settings.py.j2; local_settings.j2 no longer exists.
- Keystone:
OPENSTACK_KEYSTONE_URL = "{{ horizon_keystone_url }}/v3"(template :58), withhorizon_keystone_url=keystone_internal_url, orkeystone_public_urlwhenhorizon_use_keystone_public_url(default False) (defaults/main.yml:137,160).OPENSTACK_HOST = kolla_internal_fqdn(:56).OPENSTACK_ENDPOINT_TYPE = "internalURL"(:131), so Horizon talks to every service's internal endpoint from its own container. - memcached:
SESSION_ENGINE = cachewith all memcached hosts,ignore_exc: True, unless the DB backend is used (:19-23). With the DB backend:SESSION_ENGINE = dbplus MariaDB (:7-18). Changed in 2024.1: the cache branch is chosen whengroups['memcached']is not empty, not fromenable_memcached(:19), while the HAProxy balance still followsenable_memcached(§4.2). - Cinder:
enable_backupfollowsenable_cinder_backup(:98-100). The 2023.1 Murano/BarbicanKEY_MANAGERblock is gone with the murano role. SECRET_KEY = horizon_secret_key(:44), which is the same on all nodes.- Multi-region
AVAILABLE_REGIONS(:48-54). - TLS:
SECURE_PROXY_SSL_HEADERand secure cookies when either internal or external TLS is on (:25-29).horizon.conf.j2only handleshorizon_enable_tls_backend(templates/horizon.conf.j2:3,43), and the http→https redirects are HAProxy entries (§4.2).
4.8 Operational notes¶
- Sessions live in memcached unless the DB backend is used (
_9998-kolla-settings.py.j2:7-23), HAProxy usesbalance roundrobinby default andbalance sourceonly when memcached is disabled and the DB backend is not used (§4.2), andLOCAL_PATH=/tmpis the container's/tmp, so it does not survive a recreate (§4.2). NOT AVAILABLE: whether a horizon restart or a memcached restart logs users out, what restarting a node does to in-flight requests (including image uploads,HORIZON_IMAGES_UPLOAD_MODE = 'direct'at_9998-kolla-settings.py.j2:130), and whether anything is staged inLOCAL_PATH. - Image side (kolla
docker/horizon/extend_start.sh): every start syncs eachENABLE_<PLUGIN>dashboard file intoopenstack_dashboard/local/enabled(:32-46,225-241).manage.py collectstatic --noinput --clearandcompress --forcerun only when/var/lib/kolla/.settings.md5sum.txtis missing, the md5 oflocal_settings+custom_local_settings+local_settings.ddiffers from it, a plugin file was just copied in or removed, orFORCE_GENERATE=yes(:38-45,206-223,243-259). The kolla-ansible files now sit inlocal_settings.d, which the hash covers (:209-212). New in 2024.1:compressis tried up to 5 times and the start script exits 1 if all attempts fail (:245-257). kolla-ansible setsFORCE_GENERATE=yesonly whenhorizon_dev_modeis on, andnootherwise (defaults/main.yml:25,153). The settings setCOMPRESS_OFFLINE = True(_9998-kolla-settings.py.j2:3)./var/lib/kollais not a volume (defaults/main.yml:123-129), so the hash file lives in the container layer. The handler restarts the container underCOPY_ALWAYS(default,all.yml:43) and recreates it when image, env or volumes differ, or underCOPY_ONCE(module_utils/kolla_docker_worker.py:341-363). Only this start script writes the hash file (:6,258), so a new container starts without one.collectstaticandcompresstherefore run on the first start after a recreate (new image, plugin flag change,COPY_ONCE) or after a settings change, and not on a plain restart with unchanged settings and plugin flags. NOT AVAILABLE: how long they take. See Kolla images §5. - Horizon config is independent of host count except for the memcached list.
5. HEAT¶
5.1 Play¶
# ansible/site.yml:724-738
- name: Apply role heat
gather_facts: false
hosts: [heat-api, heat-api-cfn, heat-engine, '&enable_heat_True']
serial: '{{ kolla_serial|default("0") }}'
max_fail_percentage: '{{ heat_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}'
roles: [{ role: heat, tags: heat }]
- LB include:
site.yml:183-187.enable_heat: "{{ enable_openstack_core | bool }}"(all.yml:772).
5.2 heat_services (ansible/roles/heat/defaults/main.yml:2-53)¶
In 2024.1 the dict is at ansible/roles/heat/defaults/main.yml:2-58.
| key | container | group | enabled | flags |
|---|---|---|---|---|
| heat-api | heat_api |
heat-api | true | Apache/mod_wsgi on 8004, HAProxy internal and external (:11-26) |
| heat-api-cfn | heat_api_cfn |
heat-api-cfn | true | Apache/mod_wsgi on 8000, HAProxy internal and external (:35-50) |
| heat-engine | heat_engine |
heat-engine | true | No haproxy, not privileged. Volumes are config, localtime, timezone, dev-mode source and logs, plus /dev/shm when om_enable_queue_manager (new in 2024.1) (:162-168). |
5.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | register.yml → config.yml → check-containers.yml → (clone) → bootstrap.yml → flush (tasks/) |
| upgrade | config.yml → check-containers.yml → bootstrap_service.yml (db sync with new image while the old engine and APIs still run) → flush, which restarts all heat-api, heat-api-cfn and heat-engine containers together (tasks/). No register.yml. |
| precheck | ports 8004/8000 free on api/cfn hosts (tasks/). heat-engine is not checked. |
| check | empty (tasks/) |
| pull / stop / deploy-containers / config | standard |
5.4 Config files (ansible/roles/heat/tasks/config.yml)¶
| dest | sources | notifies |
|---|---|---|
<svc>/ |
<svc>.json.j2 (:38-49) |
Restart <svc> |
<svc>/ (service_name per container) |
merge: template → global.conf → heat.conf → heat/ → heat/ (:51-69) |
Restart <svc> |
| policy | {{node_custom_config}}/ (:15-32, :71-83) |
Restart <svc> |
heat-api/, heat-api-cfn/ |
src: "{{ role_path }}/ (:85-111). No override path exists, unlike cinder/placement wsgi. |
Restart heat-api / heat-api-cfn |
| certs | service-cert-copy (:34-36) |
The wsgi template sets no python-path; both WSGIScriptAlias and the <Directory> block use
binary_path = /var/lib/kolla/venv/bin (templates/wsgi-heat-api.conf.j2:2,18,34), as placement does with its
wsgi_directory (roles/placement/templates/placement-api-wsgi.conf.j2:2). Still holds in 2024.1.
5.5 Handlers¶
Three handlers: heat-api (:2-16), heat-api-cfn (:18-32), heat-engine (:34-48). All recreate_or_restart, with no special logic.
5.6 Bootstrap¶
mysql_db heatandmysql_user(tasks/bootstrap.yml:2-36) run every deploy/reconfigure.bootstrap_heatcontainer (bootstrap_service.yml:2-30) getsKOLLA_BOOTSTRAPplus the cloud admin credentials as env vars (OS_AUTH_URL,OS_USERNAME,OS_PASSWORD,OS_PROJECT_NAME, …) andHEAT_DOMAIN_ADMIN_PASSWORD(:10-22). The image runsheat-manage db_sync, thenopenstack domain list. Only ifheat_user_domainis absent does it create the domain, theheat_domain_adminuser (passwordHEAT_DOMAIN_ADMIN_PASSWORD) and that user'sadminrole on the domain (kolladocker/heat/heat-api/extend_start.sh:5-17). It re-runs on every reconfigure (and on upgrade,tasks/upgrade.yml:6), but once the domain exists it never touches the user. A changedheat_domain_admin_password(etc/kolla/passwords.yml:108) is therefore not pushed to Keystone, whileheat.confgets the new value (templates/heat.conf.j2:11-12). NOT AVAILABLE: whether stack-domain operations then fail authentication. The check looks only at the domain name, so if a run fails partway and leaves the domain without the user or its role, no later run repairs it. With Docker, the container is removed on exit, after a failure too (module_utils/kolla_docker_worker.py:405-413), so the passwords do not linger indocker inspect. Butkolla_startruns with xtrace: when the bootstrap creates the domain, theopenstack user create … --passwordline goes to stderr withHEAT_DOMAIN_ADMIN_PASSWORDexpanded. The module returns that stderr in the task result (module_utils/kolla_docker_worker.py:400-404), which Ansible prints on failure or with-v(ansible-core 2.16.20lib/ansible/plugins/callback/default.py:50-71,104-105,lib/ansible/plugins/callback/__init__.py:200-202), and the task sets nono_log(kolladocker/base/start.sh:6,docker/heat/heat-api/extend_start.sh:12; Kolla images §2.2).- Keystone: services
heat/orchestration(/v1/%(tenant_id)s) andheat-cfn/cloudformation(/v1), each with 2 endpoints (internal, public); userheat; rolesheat_stack_ownerandheat_stack_user;heat_stack_ownergranted to the cloud admin user in the admin project (defaults/main.yml:178-184,218-245;tasks/register.yml:2-9). This runs on deploy/reconfigure, not on upgrade.
5.7 Dependencies (templates/heat.conf.j2)¶
- Public endpoints reached from inside VMs:
heat_metadata_server_urlandheat_waitcondition_server_urluseheat_cfn_public_base_endpoint(:6-7), i.e.public_protocol,heat_cfn_external_fqdnandheat_api_cfn_public_port(defaults/main.yml:181; the port follows the single external frontend option,all.yml:419).server_keystone_endpoint_type = public(:23). Instances that use wait conditions or software deployments must be able to reach the external VIP on that port and the public keystone endpoint. - All other services:
[clients] endpoint_type = internalURL(:105-107). heat-engine calls nova, neutron, cinder, glance, swift, etc. through their internal catalog endpoints. The real dependency set is whatever the users' templates use. - Keystone:
keystone_authtoken(:31-46), plus[trustee]with the heat service user (:55-60), plus[ec2authtoken](:62-63).stack_domain_admin=heat_domain_admin/heat_domain_admin_password(:11-13). - MariaDB (:25-29); RabbitMQ RPC (:19), notifications (ceilometer only, otherwise
driver = noop; :65-72,defaults/main.yml:209-213) and[oslo_messaging_rabbit]with quorum queues by default (:74-98); memcached for authtoken and[cache](:44-52). New in 2024.1: with the queue manager, an[oslo_concurrency] lock_path(:120-123). num_engine_workers = heat_engine_workers=openstack_service_workers(≤5) per engine host (:15-17;defaults/main.yml:196).
5.8 Operational notes¶
- heat-engine restarts with stacks in progress: Docker sends SIGTERM, then SIGKILL if heat-engine is still running
after 60 s (
module_utils/kolla_systemd_worker.py:32;group_vars/all.yml:139,151; §0). NOT AVAILABLE (upstream heat): how that compares with the duration of long resource creates; whether stacks that a killed engine owned stay*_IN_PROGRESSuntil some engine starts again; whether heat-engine'sreset_stack_statusthen marks themFAILED("Engine went down during stack …"); and whether users must then runstack updateor a check/resume. - With serial 0, every heat-engine restarts in the same handler flush on any heat.conf change. A memcached or rabbit
membership change, a password rotation, an
openstack_service_workerschange, or the first 2024.1 run of a cloud that kept classic queues (§0) is enough. Every in-progress stack in the cloud is therefore affected, not just a fraction. - On upgrade the db sync runs while old engines are still active, and all engines restart together after it.
- Heat does not touch compute hosts. Its dependency on instance reachability to the external VIP (above) is the main network coupling.
6. Summary matrix: what reconfigure does per role¶
| role | keystone register | DB create/grant | bootstrap container (schema sync) on reconfigure | extra side effects on reconfigure | restarts (serial 0) |
|---|---|---|---|---|---|
| glance | yes | yes | yes (KOLLA_BOOTSTRAP on glance_api_hosts[0]) |
toggles GLOBAL log_bin_trust_function_creators 1→0 (unless use_preconfigured_databases); no sanity image in 2024.1 |
glance-api (and tls-proxy) on glance_api_hosts only (1 host with the file backend) |
| placement | yes | yes | yes (also runs online data migrations) | – | all placement-api |
| cinder | yes | yes | yes | – (no sanity volume in 2024.1) | api, scheduler, volume, backup on all storage hosts at once if their config changed (the first 2024.1 run changes cinder-volume through hostnqn) |
| horizon | no | only if horizon_backend_database (default false) |
only if horizon_backend_database |
env-driven recreate when plugin flags change | all horizon |
| heat | yes (also roles and a grant to admin) | yes | yes (with admin creds in env) | – | api, api-cfn, all engines |
The separate online-data-migration container (KOLLA_OSM) runs only on upgrade, for placement
(roles/placement/tasks/upgrade.yml:16-34) and cinder (roles/cinder/tasks/upgrade.yml:25-43). Glance's
expand/migrate/contract runs only in the rolling path. A reconfigure with a bumped image tag therefore runs cinder's
db sync without its OSM step (kolla docker/cinder/cinder-api/extend_start.sh:6-14), while placement's bootstrap
container runs both db sync and db online_data_migrations (kolla
docker/placement/placement-api/extend_start.sh:5-9). A reconfigure also skips cinder's second "remove RPC version pin"
restart, which only upgrade.yml:23 runs. Whether this leaves cinder in a partially upgraded state is NOT AVAILABLE.
See Kolla images §4.
7. Things that scale with host count¶
- memcached and rabbit host lists in every config (§0). Any membership change rewrites the config of all 5 services and restarts every container of every service.
glance_num_retries = len(groups['glance-api'])(roles/cinder/templates/cinder.conf.j2:20).- HAProxy backend lists: glance uses
glance_api_hosts(roles/glance/defaults/main.yml:86-87); the others usegroups[host_group](roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:101). - Placement and cinder credentials are embedded in nova-compute config on every compute. Password rotation therefore needs a reconfigure of the whole compute fleet (§2.8, §3.7).
run_oncetasks repeat per batch whenANSIBLE_SERIALis set (§0).
8. NOT AVAILABLE list¶
- NOT AVAILABLE: whether an existing glance file-store image stays readable after
enable_swift: yes,glance_backend_ceph: yesorglance_backend_s3: yesis applied by reconfigure. - Answered from code: Glance rolling upgrade with
ANSIBLE_SERIAL=1runs the contract at the end of batch 1, before later batches are upgraded (§1.8 note 4; §0 run_once + serial;roles/glance/tasks/rolling_upgrade.yml:80-102). - NOT AVAILABLE: Glance-api start time with a large NFS
glance_file_datadir_volume(recursive chown,glance-api.json.j2:47-52). - NOT AVAILABLE: whether the glance cinder store works with
cinder_catalog_info = volume:cinder:internalURLand no[cinder]store section. - NOT AVAILABLE: whether cinder-backup works with default
cinder_backup_driver=cephandcinder_backend_ceph=no(kolla-ansible gives it no ceph config, §3.8 oddity 5). - NOT AVAILABLE: whether the volumes owned by one of several RBD cinder-volume nodes (
host) can still be managed while that node is down, and what state an in-flight create-from-image is in after a cinder-volume restart. - Cinder RPC pinning after
kolla-ansible upgrade(is a second restart needed?): answered from code for 2024.1: the role does a second restart of every cinder service after the upgrade flush and a 30 s pause, to remove the RPC version pin (roles/cinder/tasks/upgrade.yml:20-23,tasks/reload.yml:2-10,handlers/main.yml:70-88; §3.8 note 4). Remaining NOT AVAILABLE: whether 30 s is always enough for all services to report their new version. - NOT AVAILABLE: Heat-engine restart with
CREATE_IN_PROGRESSstacks: final state and recovery steps. - NOT AVAILABLE: Horizon restart: session survival (memcached) and start-up duration. When collectstatic/compress run is answered from kolla source (§4.8).
- Answered from kolla source:
bootstrap_heatdoes not update theheat_domain_adminpassword onceheat_user_domainexists (kolladocker/heat/heat-api/extend_start.sh:10-14; §5.6). Remaining NOT AVAILABLE: how heat behaves afterheat_domain_admin_passwordchanges, whenheat.confgets the new value (roles/heat/templates/heat.conf.j2:12) but Keystone keeps the old one. - NOT AVAILABLE: whether the kolla-toolbox
mysql_usertask (fromcommunity.mysql <3, whose source is not among the studied trees) resets the DB user's password on every deploy/reconfigure (§1.6). - New in 2024.1, NOT AVAILABLE: how Horizon loads the
_9998/_9999fragments oflocal_settings.don top of the image's baselocal_settings.py(§4.4), and which component creates the Keystoneservicerole that the cinder role now only grants (§3.6).