Fact sheet C¶
All paths are relative to the repo root ($SRC). Every claim comes from reading source code only: kolla-ansible at the
tag 2023.1-eol, the kolla images repo at the tag 2023.1-eol, and, for Ansible semantics, the Ansible source (cited
as ansible-core 2.14.18 lib/ansible/...:N). Modules run inside the kolla-toolbox image come from that image's
ansible-core==2.14.* plus Ansible collections (kolla docker/kolla-toolbox/Dockerfile.j2:73,93-95,
docker/kolla-toolbox/requirements.yml); the collection sources are not among the studied trees.
- FACT (the default, not written out): shown by the cited source lines.
- NOT AVAILABLE: the sources do not establish it, for example upstream OpenStack runtime behaviour, durations, or what a restart does to work in progress.
Behaviour inside the images (what KOLLA_BOOTSTRAP, KOLLA_UPGRADE, KOLLA_OSM, GLANCE_DB_EXPAND and similar
variables actually run) is defined in the separate kolla images repo. That repo is not in this checkout, but its
2023.1-eol tag was read separately (see Kolla images). A statement that cites a
kolla path, written as kolla docker/... (for example kolla docker/glance/glance-api/extend_start.sh or kolla
docker/base/set_configs.py), is read from that source like any other code claim. Image-side behaviour without such a
citation is NOT AVAILABLE.
0. Cross-cutting facts (apply to all five roles)¶
| Topic | Fact | Evidence |
|---|---|---|
| Task dispatch | Every role's tasks/main.yml is - include_tasks: "{{ kolla_action }}.yml". |
e.g. ansible/ (identical in placement/cinder/horizon/heat) |
reconfigure |
reconfigure.yml is just - import_tasks: deploy.yml in all five roles, so reconfigure performs keystone registration, DB create/grant and the bootstrap (db sync) container, exactly as deploy does. |
ansible/ |
common |
In 2023.1 none of the five roles has a meta/ directory, so there is no role dependency on common. common is its own play, tagged common, on the cron, fluentd, kolla-logs and kolla-toolbox groups. |
ansible/ (no meta/); ansible/ |
| Serial | Every play uses serial: '{{ kolla_serial|default("0") }}'. The CLI passes kolla_serial=${ANSIBLE_SERIAL} (default 0) only for upgrade and reconfigure. |
ansible/; tools/ |
| run_once + serial | Bootstrap, DB and keystone tasks are run_once: True. When kolla_serial > 0, Ansible runs the whole play once per batch and sends a run_once task only to the first host of the current batch, so run_once tasks run once per batch. So db sync and keystone registration repeat for each batch. This matters for glance rolling upgrade (§1.8). |
e.g. ansible/; ansible-core 2.14.18 lib/, lib/ |
| Bootstrap container | kolla_docker action=start_container, detach: False, restart_policy: oneshot. A one-shot container gets no systemd unit and is started directly; the module waits for exit and then stops and removes it (remove_on_exit defaults to True), before it reports a non-zero exit. Nothing guards it with "already bootstrapped", so it runs on every deploy/reconfigure. |
ansible/; ansible/; e.g. ansible/ |
| Restart semantics | Handlers use recreate_or_restart_container and pass common_options: docker_common_options, which carries graceful_timeout: docker_graceful_timeout (60 s). Long-running containers run under a systemd unit kolla-<name>-container.service whose ExecStop is docker stop <name> -t <graceful_timeout>; restarts and stops go through systemd RestartUnit/StopUnit. Docker sends SIGTERM, then SIGKILL if the container is still running after 60 s. No role here overrides it. The 10 s value is only the module default when no graceful_timeout is passed. |
ansible/; ansible/; ansible/; R24 |
| Handler skip | All handlers have when: kolla_action != "config", so genconfig never restarts anything. |
e.g. ansible/ |
| Network | All containers run with network_mode: host, so the container hostname is the node hostname (this matters for cinder host). |
ansible/ |
| Empty volume strings | Volume lists contain '' entries (timezone, dev-mode). generate_volumes skips empty entries, so the places that pass lists without reject('equalto','') (placement/cinder upgrade OSM, horizon) are harmless. |
ansible/ |
| Shared endpoints | database_address: "{{ kolla_internal_fqdn }}" (DB is reached through the VIP). rpc_transport_url / notify_transport_url enumerate every host in the rabbit groups. |
ansible/ |
| memcached | Every service template (glance, placement, cinder, heat, horizon) enumerates all groups['memcached'] hosts. Adding or removing a memcached or rabbitmq node therefore changes the config of all five services, and they restart on the next deploy/reconfigure. |
e.g. ansible/ |
| Workers | openstack_service_workers: "{{ [ansible_facts.processor_vcpus, 5] | min }}". The value is per host. |
ansible/ |
| Sanity checks | 2023.1 has no sanity checks: kolla_enable_sanity_checks and kolla_enable_sanity_* are defined nowhere under ansible/, glance and cinder have no check.yml, and the check.yml of placement, horizon and heat is empty. The CLI has no check action. |
grep -rn sanity ansible/ → nothing; ansible/; tools/ |
| HAProxy | Each role's loadbalancer.yml is applied from the loadbalancer play (site.yml:99-341, tags haproxy, keepalived, loadbalancer), whose task block only runs when enable_haproxy and kolla_action in ['deploy','reconfigure','upgrade','config']. |
ansible/ |
| TLS backend certs | service-cert-copy copies <project>-cert.pem/-key.pem into a service's config dir only when one of its haproxy entries is enabled and one has tls_backend true, and kolla_externally_managed_cert is false. It notifies the container restart. |
ansible/ |
| Service-to-host mapping | host_in_groups (used only by glance) takes precedence over group in the filter select_services_enabled_and_mapped_to_host. That filter drives pull/stop/cert-copy. |
kolla_ansible/ |
| Ansible forks | No ansible.cfg ships in the repo, so "all hosts restart together" really means up to forks hosts at once (Ansible default 5): Ansible starts at most min(forks, batch size) workers and queues each host's task on a free one. The forks value in use depends on the operator's config (ansible.cfg, ANSIBLE_FORKS or kolla-ansible --forks) and is NOT AVAILABLE here. |
find . -name ansible.cfg → none; tools/; ansible-core 2.14.18 lib/, lib/, lib/ |
site.yml order (relevant slice)¶
loadbalancer play (99) → … → glance (495) → ironic (505) → cinder (519) → placement (532) →
import_playbook: nova.yml (544) → … → heat (631) → horizon (643). Placement is therefore (re)deployed before
nova, and glance before cinder.
1. GLANCE¶
1.1 Play¶
# ansible/site.yml:495-503
- name: Apply role glance
gather_facts: false
hosts:
- glance-api
- '&enable_glance_True'
serial: '{{ kolla_serial|default("0") }}'
roles:
- { role: glance,
tags: glance }
- In 2023.1 the role line has no
when: enable_glance; the host pattern&enable_glance_True(group built atsite.yml:33) does the selection. - The LB include is at
ansible/site.yml:163-167(tagglance,when: enable_glance). enable_glance: "{{ enable_openstack_core | bool }}"(ansible/group_vars/all.yml:621).
1.2 glance_services (ansible/roles/glance/defaults/main.yml:2-64)¶
In 2023.1 the dict is at ansible/roles/glance/defaults/main.yml:2-66.
| key | container | group / mapping | enabled | flags |
|---|---|---|---|---|
| glance-api | glance_api |
group: glance-api, host_in_groups: inventory_hostname in glance_api_hosts (:6) |
true |
privileged: enable_cinder and (enable_cinder_backend_iscsi or cinder_backend_ceph) (:10); environment: glance_api_container_proxy = container_proxy (:9, :204). This is the only service in scope that receives http(s)_proxy. Volumes (:184-193): glance_file_datadir_volume:/ (default named volume glance), and iscsi_info:/ plus /dev:/dev when cinder iSCSI is enabled (:192-193). |
| glance-tls-proxy | glance_tls_proxy |
same host_in_groups (:38) |
glance_enable_tls_backend (= kolla_enable_tls_backend, all.yml:917) |
HAProxy image (.../haproxy, :151-153). It listens on api_interface_address:9292 with TLS and forwards to glance-api on 127.0.0.1:9292 (templates/; glance-api bind_host = 127.0.0.1 when TLS, templates/). |
Critical mapping rule:
# ansible/group_vars/all.yml:906,910-911,915
glance_backend_file: "{{ not (glance_backend_ceph | bool or glance_backend_swift | bool or glance_backend_vmware | bool) }}"
glance_backend_swift: "{{ enable_swift | bool }}"
glance_file_datadir_volume: "glance"
glance_api_hosts: "{{ [groups['glance-api'] | first] if glance_backend_file | bool and glance_file_datadir_volume == 'glance' else groups['glance-api'] }}"
With the default file backend on a local named volume, glance-api runs only on the first host of glance-api.
HAProxy members are built from glance_api_hosts too (defaults/main.yml:84-85), so glance has no HA. A glance-api
restart is a full Image API outage. The docs confirm this: "By default when using file backend only one
glance-api container can be running" (doc/source/reference/shared-services/glance-guide.rst:25-26).
HAProxy: client and server timeouts are 6h (defaults/main.yml:137-138). In TLS mode the HAProxy members are the
custom_member_list haproxy_tls_members, which always carries ssl verify required ca-file ...
(defaults/main.yml:54,65,85), and ignore kolla_verify_tls_backend: the template prints a custom member list as
is (haproxy-config/templates/haproxy_single_service_split.cfg.j2:80-83). Other services honour it in
haproxy-config/templates/haproxy_single_service_split.cfg.j2:85-92.
1.3 Task flow per action¶
| action | flow |
|---|---|
| deploy / reconfigure | deploy.yml: register.yml → config.yml → check-containers.yml → (clone.yml if dev) → bootstrap.yml → meta: flush_handlers (tasks/; reconfigure.yml:2). In 2023.1 there is no check.yml step. |
| upgrade | upgrade.yml:1-6: rolling_upgrade.yml if glance_enable_rolling_upgrade (default no, all.yml:912), else legacy_upgrade.yml. No register.yml in either path, so endpoints are not refreshed on upgrade. |
| legacy upgrade | config.yml → check-containers.yml → bootstrap_service.yml (db sync, new image) → flush (tasks/). It does not stop glance-api first. The docs claim "This upgrade method will stop APIs during database schema migrations" (glance-guide.rst:115), but the code does not do that: the old glance-api keeps serving while the new-image bootstrap migrates the schema. |
| rolling upgrade | see §1.8 |
| deploy-containers | check-containers.yml only, then handlers (tasks/) |
| pull | service-images-pull. Only hosts where host_in_groups is true pull (roles/). |
| stop | service-stop with glance_services (tasks/) |
| precheck | service-precheck (groups exist) plus a free-port check on 9292 when glance_api is not running (tasks/) |
| check | Not available in 2023.1: the role has no check.yml, and the CLI has no check action (§0). |
| config | config.yml only. Handlers are notified but skipped. |
1.4 Config files (ansible/roles/glance/tasks/config.yml)¶
dest (under /etc/kolla/) |
source / merge order | when | notifies |
|---|---|---|---|
glance-api/ and glance-tls-proxy/ dirs |
file |
enabled and mapped (:2-13) | – |
glance-api/ |
template: {{node_custom_config}}/ (a single file, no merge) |
glance_backend_ceph (config.yml:14-16; external_ceph.yml:2-10) |
Restart glance-api |
glance-api/ |
template {{node_custom_config}}/ (default ceph.client.glance.keyring, all.yml:1095) |
ceph (external_ceph.yml:12-20) |
Restart glance-api |
| policy file | first found of supported_policy_format_list in {{node_custom_config}}/ (:18-35, :156-169) |
if found | Restart glance-api |
| CA/TLS certs | copy-certs.yml → service-cert-copy (:37-39) |
kolla_copy_ca_into_containers or glance_enable_tls_backend |
Restart <svc> |
glance-tls-proxy/ |
assemble of glance-(cert|key).pem (:41-54) |
TLS proxy enabled | no notify (the cert copy notifies instead) |
<svc>/ |
glance-api.json.j2, glance-tls-proxy.json.j2 (:56-67) |
– | Restart <svc> |
glance-api/ |
merge: templates/ → global.conf → glance.conf → glance/ → glance/ (:69-86) |
– | Restart glance-api |
glance-api/ |
merge: template → glance-cache.conf → glance/ → glance/ (:88-105). Note that no global.conf is in this merge. |
enable_glance_image_cache |
Restart glance-api |
glance-api/ |
merge: template → glance-swift.conf → glance/ → glance/ (:107-124) |
glance_backend_swift |
Restart glance-api |
glance-api/ (new in 2023.1) |
copy {{node_custom_config}}/ (:126-139) |
glance_enable_interoperable_image_import (default no, all.yml:914) |
Restart glance-api |
glance-api/ (new in 2023.1) |
copy {{node_custom_config}}/ (:141-154) |
glance_enable_property_protection (default no, all.yml:913) |
Restart glance-api |
glance-tls-proxy/ |
first_found: glance/ → glance/ → role template (:171-187) |
TLS proxy | Restart glance-tls-proxy |
glance-api.json.j2:53-58 has
"permissions": [{"path": "/var/lib/glance", "owner": "glance:glance", "recurse": true}]. /var/lib/glance is the
glance_file_datadir_volume mount (defaults/main.yml:188), which with the file backend is the entire image store
(filesystem_store_datadir = /var/lib/glance/images/, templates/glance-api.conf.j2:73). kolla_set_configs walks it
and calls os.chown on every directory and file, unconditionally (kolla docker/base/set_configs.py:345-355,378-385).
It does this at every container start under the default COPY_ALWAYS (kolla docker/base/start.sh:11,
set_configs.py:538-541; ansible/group_vars/all.yml:43), and only on a container's first start under COPY_ONCE
(kolla set_configs.py:542-551); see
Kolla images §2. NOT AVAILABLE: how long this
recursive chown takes at container start with a big NFS-backed glance_file_datadir_volume.
1.5 Handlers (ansible/roles/glance/handlers/main.yml)¶
Restart glance-api container(:2-18): recreate_or_restart. It passesprivileged,environment(the proxy), volumes, dimensions and healthcheck.Restart glance-tls-proxy container(:20-34): recreate_or_restart.- Neither has special logic beyond
kolla_action != "config". There is no rolling restart and no wait for the API to come back.
1.6 Bootstrap¶
| step | where | re-runs on reconfigure? |
|---|---|---|
mysql_db glance |
tasks/, run_once, delegate groups['glance-api'][0], unless use_preconfigured_databases |
yes (idempotent) |
mysql_user glance with password and ALL on glance.* |
bootstrap.yml:18-36 |
yes. The task runs mysql_user through kolla_toolbox (ansible/), whose image installs ansible-core==2.14.* (kolla docker/); ansible-core redirects mysql_user to community.mysql.mysql_user (ansible-core 2.14.18 lib/), installed in the image from community.mysql <3 (kolla docker/). That collection's source is not among the studied trees, so whether each run resets the stored password when it differs is NOT AVAILABLE. |
Set GLOBAL log_bin_trust_function_creators=1 |
bootstrap_service.yml:2-17, logged in as glance_database_shard_root_user (the DB root user without ProxySQL, defaults/), unless use_preconfigured_databases |
yes, on every deploy/reconfigure |
bootstrap_glance container, KOLLA_BOOTSTRAP: glance-manage db_sync then glance-manage db_load_metadefs (kolla docker/) |
bootstrap_service.yml:19-37, delegate glance_api_hosts[0] |
yes, so a schema sync runs on every reconfigure |
Set log_bin_trust_function_creators=0 |
bootstrap_service.yml:39-54 |
yes |
Keystone: service glance/image, 2 endpoints (internal, public), user glance (admin in service) |
tasks/; defaults/ |
yes (deploy/reconfigure); not on upgrade |
Oddity / risk: the enable/db_sync/disable sequence is not wrapped in block/always. If the bootstrap container
fails, the global MariaDB variable stays at 1. Whether this leftover state has security consequences is NOT AVAILABLE.
A related upstream fix for keystone appears in
releasenotes/notes/bug-1987982-fix-keystone-log_bin_trust_function_creators-variables-8628098891513dac.yaml.
Reconfigure as uncontrolled upgrade: if an operator bumps openstack_tag/glance_tag and runs reconfigure, the
new image runs the KOLLA_BOOTSTRAP branch (glance-manage db_sync and db_load_metadefs, kolla
docker/glance/glance-api/extend_start.sh:5-9) and the handler recreates glance-api on the new image. For the glance
role that is the default legacy upgrade (glance_enable_rolling_upgrade: "no", all.yml:912): legacy_upgrade.yml:2-9
runs the same config, check-containers, bootstrap_service.yml and flush steps, and reconfigure only adds registration,
DB/user creation and the dev-mode clone (deploy.yml:2-15).
1.7 Dependencies (from templates/glance-api.conf.j2)¶
- MariaDB via the VIP (:34-38); RabbitMQ:
transport_url = rpc_transport_url(:27) and notifications (:115-122; topicnotificationsonly if ceilometer, otherwisedriver = noop,defaults/main.yml:107-111). - Keystone:
www_authenticate_uriandauth_urlare bothkeystone_internal_url(:40-51). memcached: all hosts (:53-55). - Cinder:
cinder_catalog_info = volume:cinder:internalURL(:25), andenabled_backendscontainscinder:cinderwheneverenable_cinder(defaults/main.yml:222-224; template :17-19). Oddity: in 2023.1 cinder registers onlycinderv3/volumev3(roles/cinder/defaults/main.yml:320-326). Novolume/cindercatalog entry exists, and the template has no[cinder]store section, so the glance cinder store is enabled without a matching catalog entry or store section. Whether it works if used is NOT AVAILABLE. Nova, by contrast, usescatalog_info = volumev3:cinderv3:internalURL(roles/nova-cell/templates/nova.conf.j2:113). - Ceph:
[rbd] rbd_store_user/pool(:76-80). In 2023.1 the template no longer setsshow_multiple_locations(grep -rn show_multiple_locations ansible/roles/glance→ nothing). - Swift:
[swift]section (:82-92) withswift_store_auth_insecure = Truehard-coded.glance-swift.confauthenticates againstopenstack_auth.auth_url(=keystone_internal_url,all.yml:895-896) asservice:glance(templates/glance-swift.conf.j2:1-8). - Barbican (:150-155), osprofiler (:142-148), VMware (:94-101).
- Cinder iSCSI flag toggles glance-api
/devandiscsi_info, and cinder iSCSI orcinder_backend_cephtogglesprivileged(defaults/main.yml:10,192-193). The code comment says the mounts are "to support Cinder iSCSI backends" (defaults/main.yml:191). How glance uses these mounts is NOT AVAILABLE.
1.8 Operational notes¶
-
Backend switch via unrelated flags:
glance_backend_swiftdefaults toenable_swift(all.yml:910), andglance_backend_fileis the negation of ceph/swift/vmware (all.yml:906). Enabling Swift (orglance_backend_ceph) on a running cloud does three things:- removes
filefromenabled_backendsand drops the[file]section (defaults/main.yml:209-212; template :17-19,71-74); - changes
default_backend(:60-69); - expands
glance_api_hoststo everyglance-apihost (all.yml:915).
NOT AVAILABLE: whether existing images in the file store stay readable after the switch.
A related doc mismatch: the guide says ceph is enabled "automatically" (
glance-guide.rst:43), but the default isglance_backend_ceph: "no"(all.yml:907). - removes
-
Inventory order matters: with the file backend, glance runs on
groups['glance-api']|first. The INI parser reads the file line by line and adds each host in that order, and Ansible lists a group's hosts in the order they were added (ansible-core 2.14.18lib/ansible/plugins/inventory/ini.py:159,216,lib/ansible/plugins/inventory/__init__.py:209-210,lib/ansible/inventory/group.py:224-228,266-278,lib/ansible/inventory/data.py:258-268), so reordering the inventory moves glance-api to another host, and the HAProxy members follow it (all.yml:915;defaults/main.yml:84-85). NOT AVAILABLE: what theglancevolume on the new host contains. - Restart impact: the HAProxy client and server timeouts are 6h (
defaults/main.yml:137-138). Restarting glance-api stops it with the 60 s graceful timeout (§0). With the file backend this is a total Image API outage, because glance-api is the only backend (§1.2). NOT AVAILABLE: what the restart does to in-flight uploads, downloads, nova snapshots and cinder upload-to-image operations, whether images are left insaving/queued, and whether nova boots that need an uncached image fail during the window. - Rolling upgrade (
tasks/rolling_upgrade.yml):- Sequence: config on host[0] (:7-8) → stop glance-api on host[0] (:10-11,
stop_service.yml) → enable log_bin (:13-28) →GLANCE_DB_EXPAND(:30-49) →GLANCE_DB_MIGRATE, which notifies restart (:51-72) → config on the other hosts (:75-76) → check-containers (:78) → flush handlers, so all glance-api containers restart in the same handler run (:80-81) →GLANCE_DB_CONTRACT(:83-102) → disable log_bin (:104-119). - The
glance_upgradingfact (:2-4, :121-123) is set but never read anywhere (dead code). - With
kolla_serial> 0, each batch re-runs the run_once expand/migrate/contract onglance_api_hosts[0](:48-49, :69-70, :101-102), because Ansible runs the play once per batch and sends arun_oncetask to the first host of each batch (§0). The contract therefore runs at the end of batch 1, while hosts in later batches still run the old glance-api containers. - The docs warn that rolling upgrade with a non-shared file backend leads to "a corrupt state"
(
glance-guide.rst:108).
- Sequence: config on host[0] (:7-8) → stop glance-api on host[0] (:10-11,
- Image cache is never pruned. The docs say "Glance caches are not cleaned up automatically"
(
glance-guide.rst:145); there is no cron/pruner container. - Sanity check (if enabled) creates and deletes an image on every deploy/reconfigure. Not available in 2023.1: the
role has no
check.yml,tasks/deploy.yml:1-15has no check step, andkolla_enable_sanity_glanceis not defined (§0). - Glance does not touch compute hosts. Nova-compute reaches glance through
[glance] api_servers=glance_internal_endpoint(internal FQDN, port 9292,all.yml:919;roles/nova-cell/templates/nova.conf.j2:105-109), and cinder throughglance_api_servers(roles/cinder/templates/cinder.conf.j2:18).
2. PLACEMENT¶
2.1 Play¶
# ansible/site.yml:532-541
- name: Apply role placement
gather_facts: false
hosts: [placement-api, '&enable_placement_True']
serial: '{{ kolla_serial|default("0") }}'
roles: [{ role: placement, tags: placement }]
enable_placement: "{{ enable_nova | bool or enable_zun | bool }}"(all.yml:740).- Oddity: the LB include for placement has no
when: enable_placement(site.yml:243-246), unlike every other service. When placement is disabled it still writes an (effectively empty)services.d/placement-api.cfg, becauseplacement-api.enabled: True(roles/placement/defaults/main.yml:7) and the haproxy-config task checks onlyservice.enabled,service.haproxyandenable_haproxy(roles/haproxy-config/tasks/main.yml:17-20). Only the inner haproxy entries carryenable_placement(roles/placement/defaults/main.yml:13,20), and the template skips disabled entries (roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:116). - The play sits right before
import_playbook: nova.yml(site.yml:544).
2.2 placement_services (ansible/roles/placement/defaults/main.yml:2-24)¶
In 2023.1 the dict is at ansible/roles/placement/defaults/main.yml:2-25.
| key | container | group | enabled | flags |
|---|---|---|---|---|
| placement-api | placement_api |
placement-api |
True |
Plain Apache/mod_wsgi. No privileged, no host paths beyond config, localtime, timezone (Debian) and logs, plus the dev-mode source (:79-84). HAProxy internal and external on 8780, tls_backend: placement_enable_tls_backend (:11-25). |
2.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | register.yml → (clone) → config.yml → check-containers.yml → bootstrap.yml → flush (tasks/) |
| upgrade | register.yml → (clone) → config.yml → check-containers.yml → bootstrap.yml (KOLLA_BOOTSTRAP with the new image: placement-manage db sync and db online_data_migrations, kolla docker/, while the old placement-api is still running; the image's KOLLA_UPGRADE branch, db sync only, :12-15, is not used) → flush (all placement-api restart) → KOLLA_OSM online data migrations container, which runs them a second time (tasks/) |
| deploy-containers | check-containers.yml |
| pull / stop | service-images-pull / service-stop |
| precheck | groups exist, port 8780 free if not running (tasks/) |
| check | empty file (tasks/) |
| config | config.yml |
2.4 Config files (ansible/roles/placement/tasks/config.yml)¶
| dest | sources | notifies |
|---|---|---|
placement-api/ |
placement-api.json.j2 (:38-49) |
Restart placement-api |
placement-api/ |
merge: template → global.conf → placement.conf → placement/ → placement/ (:51-69) |
Restart |
placement-api/ |
first_found: placement/ → placement/ → template (:71-87) |
Restart |
placement-api/ |
migrate-db.rc.j2 (:89-101) |
Restart |
| policy | {{node_custom_config}}/ (:15-32, :103-115) |
Restart |
| certs | service-cert-copy (:34-36) | Restart |
migrate-db.rc contains the nova_api DB user and password together with the placement DB creds
(templates/migrate-db.rc.j2:1-8; vars at defaults/main.yml:121-124). The defaults file files these vars under
"Upgrade" (defaults/main.yml:118-120); no task of the role uses it, and the container receives it as
/etc/placement/migrate-db.rc (templates/placement-api.json.j2:25-26). It spreads a nova secret into the placement
container and is notified on change.
2.5 Handlers¶
Restart placement-api container: recreate_or_restart, no special logic (handlers/main.yml:2-16).
2.6 Bootstrap¶
mysql_db placementandmysql_user(tasks/bootstrap.yml:2-36) re-run on reconfigure.bootstrap_placementwithKOLLA_BOOTSTRAP(kolla image:placement-manage db syncthenplacement-manage db online_data_migrations, kolladocker/placement/placement-api/extend_start.sh:5-9) runs once, delegated togroups['placement-api'][0](bootstrap_service.yml:2-20). It re-runs on every reconfigure.- Keystone registration (service
placement, 2 endpoints, internal and public, userplacement,defaults/main.yml:129-140) happens in deploy, reconfigure and upgrade (upgrade.yml:2), unlike glance and heat. - Online data migrations: the separate
KOLLA_OSMcontainer runs only in upgrade (upgrade.yml:16-34). Because the bootstrap container above also runsdb online_data_migrations, placement OSM also runs on every deploy/reconfigure (deploy.yml:11,reconfigure.yml:2) and twice in upgrade (bootstrap atupgrade.yml:11, thenKOLLA_OSM).
2.7 Dependencies (templates/placement.conf.j2)¶
- MariaDB
[placement_database]withmax_overflow = 1000hard-coded (:14-19). - memcached
[cache]andkeystone_authtoken(:21-42); Keystone (:27-38). - In 2023.1 the template sets no
transport_url(placement does no RPC). - Leftovers: the Ussuri
osapi_compute_listenandmy_iplines are gone in 2023.1;[DEFAULT]holds only debug, log and state paths (:1-6). - Dead variables:
placement_notification_topics/placement_enabled_notification_topics(defaults/main.yml:103-107) are not referenced by any template. There is no[oslo_messaging_notifications]section.
2.8 Operational notes and the relation to nova¶
- Consumers authenticate as the placement user. nova-api/conductor/scheduler
(
roles/nova/templates/nova.conf.j2:179-189) and nova-compute on every compute host (roles/nova-cell/templates/nova.conf.j2:221-231) useusername = {{ placement_keystone_user }}/password = {{ placement_keystone_password }},valid_interfaces = internal. Neutron does the same withendpoint_type = internal(roles/neutron/templates/neutron.conf.j2:188-198), as does cyborg (roles/cyborg/templates/cyborg.conf.j2); in 2023.1 zun does not. Rotatingplacement_keystone_passwordtherefore needs a reconfigure of placement and nova (all computes) and neutron. This dependency scales with the number of compute hosts. - NOT AVAILABLE (upstream nova): how scheduling and the nova-compute resource tracker periodic task behave during a
placement-api outage or restart. With serial=0 every placement-api restarts in the same handler flush, up to
forkshosts at once (§0); whether and for how long no placement-api is serving is NOT AVAILABLE. - The upgrade db sync and online data migrations (bootstrap container) run while the old placement-api is still serving (no stop).
3. CINDER¶
3.1 Play¶
# ansible/site.yml:519-530
- name: Apply role cinder
gather_facts: false
hosts: [cinder-api, cinder-backup, cinder-scheduler, cinder-volume, '&enable_cinder_True']
serial: '{{ kolla_serial|default("0") }}'
roles: [{ role: cinder, tags: cinder }]
- LB include:
site.yml:133-137. - Defaults:
enable_cinder: "no",enable_cinder_backup: "yes",enable_cinder_backend_iscsi: "{{ enable_cinder_backend_lvm | bool }}",enable_cinder_backend_lvm: "no"(all.yml:650-654).cinder_backend_ceph: "no",cinder_backup_driver: "ceph"(all.yml:945,954). - Sample inventory:
cinder-api/cinder-schedulersit under[cinder:children] control.cinder-volume/cinder-backupsit understorage(ansible/inventory/multinode:128-129,321-331).
3.2 cinder_services (ansible/roles/cinder/defaults/main.yml:2-48)¶
In 2023.1 the dict is at ansible/roles/cinder/defaults/main.yml:2-53.
| key | container | group | enabled | flags / host volumes |
|---|---|---|---|---|
| cinder-api | cinder_api |
cinder-api | true | Apache/mod_wsgi on 8776; HAProxy internal and external (:11-25) |
| cinder-scheduler | cinder_scheduler |
cinder-scheduler | true | – |
| cinder-volume | cinder_volume |
cinder-volume | true | privileged: True, ipc_mode: host (:39-40), tmpfs for /var/ only if cinder_enable_conversion_tmpfs (default false, :41,203-207). Volumes (:184-195): /dev/:/dev/, /lib/modules:ro, /run/:/, cinder:/, iscsi_info:/ (if enable_iscsid), target_config:/ (LVM and lioadm). |
| cinder-backup | cinder_backup |
cinder-backup | enable_cinder_backup |
privileged: True (no ipc host). Volumes (:167-177): /dev/, /lib/modules, /run/:shared, cinder:/, iscsi_info. |
Related: cinder_target_helper is lioadm on RedHat and tgtadm otherwise (all.yml:949). With tgtadm the targets
are served by a separate tgtd container from the iscsi role, which also deploys iscsid on compute,
cinder-volume and cinder-backup hosts (roles/iscsi/defaults/main.yml:2-20,61-73; play site.yml:417-426).
3.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | register.yml → config.yml → check-containers.yml → (clone) → bootstrap.yml → flush (tasks/). In 2023.1 there is no check.yml step. |
| upgrade | config.yml only on groups['cinder-api'][0] (:4-5) → bootstrap_service.yml (db sync, new image, old services still running) (:7) → config.yml on all (:9) → check-containers.yml (:11) → service-ks-register for the service role and the cinder user's service role only (:13-19) → flush, so every cinder container on every host restarts in this flush (serial 0), cinder-volume and cinder-backup included; on upgrade a handler then pauses cinder_rpc_version_startup_delay (30 s) once (:21-22; handlers/; defaults/) → reload.yml: restart_container of every enabled cinder container on the host again, "to remove RPC version pin" (:24; tasks/) → KOLLA_OSM online migrations (:26-44). No register.yml, so services and endpoints are not refreshed. |
| deploy-containers | check-containers |
| pull / stop | helpers |
| precheck | groups; API port free; at least one backend enabled unless skip_cinder_backend_check (:28-44); vgs cinder-volumes on cinder-volume hosts if LVM (tasks/) |
| check | Not available in 2023.1: the role has no check.yml, and kolla_enable_sanity_cinder is not defined (§0). |
| config | config.yml |
3.4 Config files (ansible/roles/cinder/tasks/config.yml)¶
| dest | sources | when | notifies |
|---|---|---|---|
<svc>/ (cinder-volume, cinder-backup) |
merge cinder/ + cinder/ (external_ceph.yml:2-20) |
cinder_backend_ceph and host in volume/backup (config.yml:15-19) |
Restart <svc> |
cinder-volume/ |
cinder/ (external_ceph.yml:22-33) |
cephx | Restart cinder-volume |
cinder-backup/ keyrings (cinder plus cinder-backup) |
cinder/ (external_ceph.yml:35-50) |
cephx | Restart cinder-backup |
| policy | {{node_custom_config}}/ (:21-38, :95-107) |
Restart <svc> |
|
| certs | service-cert-copy (:40-42) | ||
<svc>/ |
<svc>.json.j2 (:44-55) |
Restart <svc> |
|
cinder-api/ |
first_found: cinder/ → cinder/ → template (:57-73) |
Restart cinder-api | |
<svc>/ (one per service, service_name var) |
merge: template → global.conf → cinder.conf → cinder/ → cinder/ (:75-93) |
Restart <svc> |
|
cinder-volume/ |
first_found among 8 paths (nfs_shares.j2 variants, then plain), skip unless NFS or HNAS (:109-128) |
host in cinder-volume | Restart cinder-volume |
3.5 Handlers (ansible/roles/cinder/handlers/main.yml)¶
Four restart handlers (api :2-16, scheduler :18-32, volume :34-51, backup :53-68), all recreate_or_restart. The volume
handler passes privileged, ipc_mode and tmpfs; the backup handler passes privileged. A fifth handler,
Wait for cinder services to update service versions (:78-88), listens to all four and pauses 30 s once, only when
kolla_action == 'upgrade'. There is no drain logic.
3.6 Bootstrap¶
mysql_db cinderandmysql_userrun every deploy/reconfigure (bootstrap.yml:2-36).bootstrap_cinder(KOLLA_BOOTSTRAP:cinder-manage db sync, kolladocker/cinder/cinder-api/extend_start.sh:6-9), run_once ongroups['cinder-api'][0](bootstrap_service.yml:2-20). It re-runs on every reconfigure.- Online data migrations (
KOLLA_OSM) run only in upgrade. - Keystone: in 2023.1 only
cinderv3/volumev3, with 2/v3/%(tenant_id)sendpoints (internal, public); usercinder(admin inservice); aservicerole, granted to the cinder user in theserviceproject (defaults/main.yml:212-216,320-341;tasks/register.yml:2-9). Services and endpoints are registered on deploy/reconfigure, not on upgrade.
3.7 Dependencies (templates/cinder.conf.j2)¶
- Glance:
glance_api_servers=glance_internal_endpoint(internal FQDN, port 9292) (:18).glance_num_retries = {{ groups['glance-api'] | length }}(:20) scales with the number of glance hosts. With the glance file backend only one glance-api exists, yet retries equal the group size. - Nova:
[nova]section authenticates withnova_keystone_user/nova_keystone_password(:90-100), so rotating the nova password needs a cinder reconfigure. In the other direction, nova-api and nova-compute usecinder_keystone_user/password(roles/nova-cell/templates/nova.conf.j2:112-122,roles/nova/templates/nova.conf.j2:66-76), so rotating the cinder password needs a reconfigure of all computes. Manila also uses the cinder creds (roles/manila/templates/manila-share.conf.j2:29-30). - MariaDB (:102-106); RabbitMQ RPC (:58) and notify (:60-67; topics for ceilometer/vitrage,
defaults/main.yml:276-282); Keystone and memcached (:108-127); Barbican (:238-243); osprofiler (:230-236). - Ceph:
[rbd-1]withrbd_secret_uuid = cinder_rbd_secret_uuid(:142-152). The same UUID is installed into libvirt on computes byroles/nova-cell/tasks/external_ceph.yml:152,177, so the two must stay in sync. - Swift backup:
backup_swift_urlinternal (:43-50). - Coordination: redis or etcd (:245-253). In 2023.1 the etcd URL is
etcd3+<internal_protocol>://{{ kolla_internal_fqdn }}:{{ etcd_client_port }}(:252), and the etcd role defines an internal HAProxy entry onetcd_client_port(roles/etcd/defaults/main.yml:26-31), so cinder reaches etcd through the internal VIP rather thangroups['etcd'][0]as in Ussuri. Defaultcinder_coordination_backendis''unless redis or etcd is enabled (all.yml:951;enable_etcd: "no",enable_redis: "no",all.yml:664,743). - LVM/iSCSI:
target_helper,target_protocol = iscsi(:133-140).
3.8 Operational notes¶
- No
host/backend_host/clusterset in cinder.conf (templates/cinder.conf.j2has none;grep backend_host ansible/roles/cinderreturns nothing). Because ofnetwork_mode: host, each cinder-volume container has the node hostname (§0). Whether cinder then registers it as<node-hostname>@<backend>is upstream cinder behaviour and NOT AVAILABLE here.- NOT AVAILABLE (upstream cinder): whether, with a shared backend (ceph
rbd-1) on several cinder-volume hosts, every volume is owned by the host that created it. - NOT AVAILABLE: whether, if that node is down, its volumes cannot be attached, detached, deleted or snapshotted
until the node returns or
cinder-manage volume update_hostis run. - The same question applies to renaming or replacing a storage node. The docs also note LVM issues in a
multi-controller setup (
doc/source/reference/storage/cinder-guide.rst:52-55).
- NOT AVAILABLE (upstream cinder): whether, with a shared backend (ceph
- cinder-volume restart impact (upstream cinder and os-brick behaviour, NOT AVAILABLE beyond the code facts):
- Code facts: with LVM, the LIO config is kept in the
target_configvolume (defaults/main.yml:193) or the targets are served by the separatetgtdcontainer (§3.2); with Ceph, the RBD secret is installed into libvirt on computes (§3.7). NOT AVAILABLE: whether the data path for attached volumes bypasses cinder-volume, and so whether existing attachments survive a cinder-volume restart. - In-flight operations: the stop sends SIGKILL if cinder-volume is still running after the 60 s graceful timeout
(§0). NOT AVAILABLE: whether this kills
qemu-img/ddcopies (image→volume, volume→image, migrate, retype) and what state they leave. - NOT AVAILABLE: what
init_hostdoes on start with resources left increating,downloadingoruploading, and whether it re-runsensure_exportfor in-use volumes.
- Code facts: with LVM, the LIO config is kept in the
- cinder-backup restart: NOT AVAILABLE: whether backups and restores in progress are killed and marked
erroron the next start. Because of thecinder_backup_driverdefault, see oddity 5. - Upgrade restarts every cinder-volume and cinder-backup at once, twice (flush at
upgrade.yml:21-22, thenreload.ymlatupgrade.yml:24; serial 0 unlessANSIBLE_SERIALis set). There is no[upgrade_levels]pinning in the template. The code comment explains the second restart: after an upgrade, services keep an RPC version cap until all of them are restarted, and the role waits 30 s for the services to update their version in the database (handlers/main.yml:70-76). Thereload.ymlrestart is a plainrestart_container, withoutwhen: kolla_action != "config"or any other condition, on every enabled cinder service mapped to the host (tasks/reload.yml:2-10). NOT AVAILABLE: whether 30 s is enough on a given cloud, and whether in-flight operations survive two restarts in a row. - Oddity: backup driver vs. ceph files.
cinder_backup_driver: "ceph"is the default even whencinder_backend_cephisno. cinder.conf then setsbackup_driver = ...CephBackupDriver,backup_ceph_conf=/etc/ceph/ceph.conf(:27-36). Butcinder-backup.json.j2only lists ceph.conf and the keyrings{% if cinder_backend_ceph %}(:15-36), andexternal_ceph.ymlonly runs forcinder_backend_ceph(config.yml:15-19). With an LVM or NFS-only deployment and default backup settings, kolla-ansible therefore gives cinder-backup no ceph config. Whether its driver then fails is NOT AVAILABLE. - Multipath is never configured on the cinder side: no
use_multipath_for_image_xfer/volume_use_multipathin the cinder templates (grep multipath ansible/roles/cinderreturns nothing). On the nova side, 2023.1 setsvolume_use_multipath = Truein nova-compute's libvirt config whenenable_multipathdis on (roles/nova-cell/templates/nova.conf.d/libvirt.conf.j2:27-28). Themultipathdcontainer is off by default (all.yml:714). - Compute-host touchpoints: the cinder role itself does not target computes. However,
iscsidon computes (iscsi role) sharesiscsi_info, nova-compute uses the cinder creds, and the RBD secret is shared with nova-libvirt. Enablingenable_cinder_backend_iscsi/LVM also flips glance-api to privileged with/dev, so glance-api is recreated (roles/glance/defaults/main.yml:10,192-193). In 2023.1cinder_backend_ceph(with Cinder enabled) also makes glance-api privileged, without/dev(roles/glance/defaults/main.yml:10). - External Ceph needs cinder-volume and cinder-backup hosts in the
storagegroup, otherwise "Cinder and related services relying on this group [will] fail" (doc/source/reference/storage/external-ceph-guide.rst:94-101). - The sanity check (if enabled) creates a 1 GB volume on every deploy/reconfigure. Not available in 2023.1: the role
has no
check.yml(§3.3).
4. HORIZON¶
4.1 Play¶
# ansible/site.yml:643-651
- name: Apply role horizon
gather_facts: false
hosts: [horizon, '&enable_horizon_True']
serial: '{{ kolla_serial|default("0") }}'
roles: [{ role: horizon, tags: horizon }]
- LB include:
site.yml:183-187.enable_horizon: "{{ enable_openstack_core | bool }}"(all.yml:673). - Horizon is the last of these plays.
4.2 horizon_services (ansible/roles/horizon/defaults/main.yml:2-69)¶
In 2023.1 the dict is at ansible/roles/horizon/defaults/main.yml:2-72.
| key | container | group | enabled | flags |
|---|---|---|---|---|
| horizon | horizon |
horizon | true | environment: 21 ENABLE_<PLUGIN> flags plus FORCE_GENERATE (:8-30). In 2023.1 there is no host /tmp bind: the volumes are config, dev-mode sources, localtime, timezone and logs (:122-129). Settings use LOCAL_PATH = '/ (templates/), which is therefore the container's own /tmp. HAProxy uses balance roundrobin when enable_memcached or horizon_backend_database, balance source otherwise (:44, :61; enable_memcached: "yes", all.yml:628, so roundrobin by default). Redirect frontends (mode: redirect, port 80) are added when internal or external TLS is on (:46-51, :63-68), and an acme_client backend is defined (:69-72). |
Because the plugin flags live in the container environment, and check-containers compares environment
(tasks/check-containers.yml:11), enabling or disabling any service that has a horizon plugin (designate, octavia,
magnum, …) changes the env. The horizon container is then recreated on the next deploy/reconfigure even if no
horizon config changed.
4.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | config.yml → check-containers.yml → (clone) → bootstrap.yml only if horizon_backend_database → flush (tasks/). There is no keystone registration. |
| upgrade | config.yml → check-containers.yml → (clone) → bootstrap_service.yml if DB backend → flush (tasks/) |
| precheck | groups; horizon_listen_port free (80, or 443 when horizon_enable_tls_backend; all.yml:353-355) (tasks/) |
| check | empty (tasks/) |
| pull / stop / deploy-containers / config | standard |
4.4 Config files (ansible/roles/horizon/tasks/config.yml)¶
| dest | source | notifies |
|---|---|---|
per-service policy files (<svc>_policy.json/) |
for each enabled service in the list (:19-46), policy_item.yml looks for {{node_custom_config}}/ (tasks/) and copies the matches (:116-129) |
Restart horizon |
horizon/ |
horizon.json.j2 (:48-60) |
Restart horizon |
horizon/ (Apache) |
first_found: horizon/ → horizon/ → template (:62-78). This is a whole-file override, not a merge. |
Restart horizon |
horizon/ |
first_found: horizon/ → horizon/ → local_settings.j2 (:80-96). Whole-file override. |
Restart horizon |
horizon/ |
first_found, same pattern; the default template is empty (:98-114). It is imported last by local_settings (local_settings.j2:869-870). |
Restart horizon |
| custom themes (new in 2023.1) | {{node_custom_config}}/ for each horizon_custom_themes entry (default [], defaults/) (:131-145) |
Restart horizon |
| certs | service-cert-copy (:147-149) |
4.5 Handlers¶
Restart horizon container (handlers/main.yml:2-17): recreate_or_restart. It passes environment; there is no other
logic.
4.6 Bootstrap¶
horizon_backend_database: "{{ enable_murano | bool }}"(all.yml:1052). Only then does deploy/reconfigure create the DB and user and runbootstrap_horizon(KOLLA_BOOTSTRAP: Djangomanage.py migrate --noinput, then exits; kolladocker/horizon/extend_start.sh:27-30) (tasks/bootstrap.yml:2-38,bootstrap_service.yml:2-20). In that case the bootstrap re-runs on every reconfigure.- There is no keystone registration at all.
4.7 Dependencies (templates/local_settings.j2)¶
- Keystone:
OPENSTACK_KEYSTONE_URL = "{{ horizon_keystone_url }}/v3"(template :210), withhorizon_keystone_url=keystone_internal_url, orkeystone_public_urlwhenhorizon_use_keystone_public_url(default False) (defaults/main.yml:137,163).OPENSTACK_HOST = kolla_internal_fqdn(:207).OPENSTACK_ENDPOINT_TYPE = "internalURL"(:424), so Horizon talks to every service's internal endpoint from its own container. - memcached:
SESSION_ENGINE = cachewith all memcached hosts unless the DB backend is used (:177-185). With the DB backend:SESSION_ENGINE = dbplus MariaDB (:32-44). - Cinder:
enable_backupfollowsenable_cinder_backup(:314). Murano + Barbican:KEY_MANAGERwith murano creds (:858-866). SECRET_KEY = horizon_secret_key(:165), which is the same on all nodes.- Multi-region
AVAILABLE_REGIONS(:198-205). - TLS:
SECURE_PROXY_SSL_HEADERand secure cookies when either internal or external TLS is on (:58-62). The Ussuri FIXME inhorizon.conf.j2("enabling of either tls will break the other if not enabled too") is gone in 2023.1: the template only handleshorizon_enable_tls_backend(templates/horizon.conf.j2:3,43), and the http→https redirects are HAProxy entries (§4.2).
4.8 Operational notes¶
- Sessions live in memcached unless the DB backend is used (
local_settings.j2:177-185), HAProxy usesbalance roundrobinby default andbalance sourceonly when memcached is disabled and the DB backend is not used (§4.2), andLOCAL_PATH=/tmpis the container's/tmp, so it does not survive a recreate (§4.2). NOT AVAILABLE: whether a horizon restart or a memcached restart logs users out, what restarting a node does to in-flight requests (including legacy image uploads that go through Horizon), and whether those uploads are staged inLOCAL_PATH. - Image side (kolla
docker/horizon/extend_start.sh): every start syncs eachENABLE_<PLUGIN>dashboard file intoopenstack_dashboard/local/enabled(:32-46,267-287).manage.py collectstatic --noinput --clearandcompress --forcerun only when/var/lib/kolla/.settings.md5sum.txtis missing, the md5 oflocal_settings+custom_local_settings+local_settings.ddiffers from it, a plugin file was just copied in or removed, orFORCE_GENERATE=yes(:38-45,248-265,289-293). kolla-ansible setsFORCE_GENERATE=yesonly whenhorizon_dev_modeis on, andnootherwise (defaults/main.yml:30,154). The settings setCOMPRESS_OFFLINE = True(local_settings.j2:13)./var/lib/kollais not a volume (defaults/main.yml:122-129), so the hash file lives in the container layer. The handler restarts the container underCOPY_ALWAYS(default,all.yml:43) and recreates it when image, env or volumes differ, or underCOPY_ONCE(module_utils/kolla_docker_worker.py:362-384). Only this start script writes the hash file (:6,292; no other file under kolladocker/names it), so a new container starts without one.collectstaticandcompresstherefore run on the first start after a recreate (new image, plugin flag change,COPY_ONCE) or after alocal_settings/custom_local_settingschange, and not on a plain restart with unchanged settings and plugin flags. NOT AVAILABLE: how long they take. See Kolla images §5. - Horizon config is independent of host count except for the memcached list.
5. HEAT¶
5.1 Play¶
# ansible/site.yml:631-641
- name: Apply role heat
gather_facts: false
hosts: [heat-api, heat-api-cfn, heat-engine, '&enable_heat_True']
serial: '{{ kolla_serial|default("0") }}'
roles: [{ role: heat, tags: heat }]
- LB include:
site.yml:178-182.enable_heat: "{{ enable_openstack_core | bool }}"(all.yml:672).
5.2 heat_services (ansible/roles/heat/defaults/main.yml:2-53)¶
In 2023.1 the dict is at ansible/roles/heat/defaults/main.yml:2-56.
| key | container | group | enabled | flags |
|---|---|---|---|---|
| heat-api | heat_api |
heat-api | true | Apache/mod_wsgi on 8004, HAProxy internal and external (:11-25) |
| heat-api-cfn | heat_api_cfn |
heat-api-cfn | true | Apache/mod_wsgi on 8000, HAProxy internal and external (:34-48) |
| heat-engine | heat_engine |
heat-engine | true | No haproxy, not privileged. Volumes are config, localtime, timezone, dev-mode source and logs only (:158-163). |
5.3 Task flow¶
| action | flow |
|---|---|
| deploy / reconfigure | register.yml → config.yml → check-containers.yml → (clone) → bootstrap.yml → flush (tasks/) |
| upgrade | config.yml → check-containers.yml → bootstrap_service.yml (db sync with new image while the old engine and APIs still run) → flush, which restarts all heat-api, heat-api-cfn and heat-engine containers together (tasks/). The Ussuri step that removed heat-engine/ is gone in 2023.1. No register.yml. |
| precheck | ports 8004/8000 free on api/cfn hosts (tasks/). heat-engine is not checked. |
| check | empty (tasks/) |
| pull / stop / deploy-containers / config | standard |
5.4 Config files (ansible/roles/heat/tasks/config.yml)¶
| dest | sources | notifies |
|---|---|---|
<svc>/ |
<svc>.json.j2 (:38-49) |
Restart <svc> |
<svc>/ (service_name per container) |
merge: template → global.conf → heat.conf → heat/ → heat/ (:51-69) |
Restart <svc> |
| policy | {{node_custom_config}}/ (:15-32, :71-83) |
Restart <svc> |
heat-api/, heat-api-cfn/ |
src: "{{ role_path }}/ (:85-111). No override path exists, unlike cinder/placement wsgi. |
Restart heat-api / heat-api-cfn |
| certs | service-cert-copy (:34-36) |
Oddity: in Ussuri templates/wsgi-heat-api.conf.j2 (and the cfn twin) set a python-path that was not the venv
path. This no longer holds in 2023.1: the template sets no python-path, and both WSGIScriptAlias and the
<Directory> block use binary_path = /var/lib/kolla/venv/bin (templates/wsgi-heat-api.conf.j2:2,18,34), as
placement does with its wsgi_directory (roles/placement/templates/placement-api-wsgi.conf.j2:2).
5.5 Handlers¶
Three handlers: heat-api (:2-16), heat-api-cfn (:18-32), heat-engine (:34-48). All recreate_or_restart, with no special logic.
5.6 Bootstrap¶
mysql_db heatandmysql_user(tasks/bootstrap.yml:2-36) run every deploy/reconfigure.bootstrap_heatcontainer (bootstrap_service.yml:2-30) getsKOLLA_BOOTSTRAPplus the cloud admin credentials as env vars (OS_AUTH_URL,OS_USERNAME,OS_PASSWORD,OS_PROJECT_NAME, …) andHEAT_DOMAIN_ADMIN_PASSWORD(:10-22). The image runsheat-manage db_sync, thenopenstack domain list. Only ifheat_user_domainis absent does it create the domain, theheat_domain_adminuser (passwordHEAT_DOMAIN_ADMIN_PASSWORD) and that user'sadminrole on the domain (kolladocker/heat/heat-api/extend_start.sh:5-17). It re-runs on every reconfigure (and on upgrade,tasks/upgrade.yml:6), but once the domain exists it never touches the user. A changedheat_domain_admin_password(etc/kolla/passwords.yml:114) is therefore not pushed to Keystone, whileheat.confgets the new value (templates/heat.conf.j2:11-12). NOT AVAILABLE: whether stack-domain operations then fail authentication. The check looks only at the domain name, so if a run fails partway and leaves the domain without the user or its role, no later run repairs it. The container is removed on exit, after a failure too (module_utils/kolla_docker_worker.py:426-434), so the passwords do not linger indocker inspect. Butkolla_startruns with xtrace: when the bootstrap creates the domain, theopenstack user create … --passwordline goes to stderr withHEAT_DOMAIN_ADMIN_PASSWORDexpanded.kolla_dockerreturns that stderr in the task result (module_utils/kolla_docker_worker.py:421-425), which Ansible prints on failure or with-v(ansible-core 2.14.18lib/ansible/plugins/callback/default.py:50-71,104-105,lib/ansible/plugins/callback/__init__.py:200-202), and the task sets nono_log(kolladocker/base/start.sh:6,docker/heat/heat-api/extend_start.sh:12; Kolla images §2.2).- Keystone: services
heat/orchestration(/v1/%(tenant_id)s) andheat-cfn/cloudformation(/v1), each with 2 endpoints (internal, public); userheat; rolesheat_stack_ownerandheat_stack_user;heat_stack_ownergranted to the cloud admin user in the admin project (defaults/main.yml:173-179,213-240;tasks/register.yml:2-9). This runs on deploy/reconfigure, not on upgrade.
5.7 Dependencies (templates/heat.conf.j2)¶
- Public endpoints reached from inside VMs:
heat_metadata_server_urlandheat_waitcondition_server_urluseheat_cfn_public_base_endpoint(:6-7), i.e.public_protocolandheat_cfn_external_fqdn(defaults/main.yml:176).server_keystone_endpoint_type = public(:23). Instances that use wait conditions or software deployments must be able to reach the external VIP on 8000 and the public keystone endpoint. - All other services:
[clients] endpoint_type = internalURL(:92-94). heat-engine calls nova, neutron, cinder, glance, swift, etc. through their internal catalog endpoints. The real dependency set is whatever the users' templates use. - Keystone:
keystone_authtoken(:31-46), plus[trustee]with the heat service user (:55-60), plus[ec2authtoken](:62-63).stack_domain_admin=heat_domain_admin/heat_domain_admin_password(:11-13). - MariaDB (:25-29); RabbitMQ RPC (:19) and notifications (ceilometer only, otherwise
driver = noop; :65-72,defaults/main.yml:204-208); memcached for authtoken and[cache](:44-52). num_engine_workers = heat_engine_workers=openstack_service_workers(≤5) per engine host (:15-17;defaults/main.yml:191).
5.8 Operational notes¶
- heat-engine restarts with stacks in progress: Docker sends SIGTERM, then SIGKILL if heat-engine is still running
after 60 s (
module_utils/kolla_systemd_worker.py:33;group_vars/all.yml:140,152; §0). NOT AVAILABLE (upstream heat): how that compares with the duration of long resource creates; whether stacks that a killed engine owned stay*_IN_PROGRESSuntil some engine starts again; whether heat-engine'sreset_stack_statusthen marks themFAILED("Engine went down during stack …"); and whether users must then runstack updateor a check/resume. - With serial 0, every heat-engine restarts in the same handler flush on any heat.conf change. A memcached or rabbit
membership change, a password rotation or an
openstack_service_workerschange is enough. Every in-progress stack in the cloud is therefore affected, not just a fraction. - On upgrade the db sync runs while old engines are still active, and all engines restart together after it.
- Heat does not touch compute hosts. Its dependency on instance reachability to the external VIP (above) is the main network coupling.
6. Summary matrix: what reconfigure does per role¶
| role | keystone register | DB create/grant | bootstrap container (schema sync) on reconfigure | extra side effects on reconfigure | restarts (serial 0) |
|---|---|---|---|---|---|
| glance | yes | yes | yes (KOLLA_BOOTSTRAP on glance_api_hosts[0]) |
toggles GLOBAL log_bin_trust_function_creators 1→0 (unless use_preconfigured_databases); the Ussuri optional sanity image does not exist in 2023.1 |
glance-api (and tls-proxy) on glance_api_hosts only (1 host with the file backend) |
| placement | yes | yes | yes (also runs online data migrations) | – | all placement-api |
| cinder | yes | yes | yes | – (the Ussuri optional sanity volume does not exist in 2023.1) | api, scheduler, volume, backup on all storage hosts at once if their config changed |
| horizon | no | only if horizon_backend_database |
only if horizon_backend_database |
env-driven recreate when plugin flags change | all horizon |
| heat | yes (also roles and a grant to admin) | yes | yes (with admin creds in env) | – | api, api-cfn, all engines |
The separate online-data-migration container (KOLLA_OSM) runs only on upgrade, for placement
(roles/placement/tasks/upgrade.yml:16-34) and cinder (roles/cinder/tasks/upgrade.yml:26-44). Glance's
expand/migrate/contract runs only in the rolling path. A reconfigure with a bumped image tag therefore runs cinder's
db sync without its OSM step (kolla docker/cinder/cinder-api/extend_start.sh:6-14), while placement's bootstrap
container runs both db sync and db online_data_migrations (kolla
docker/placement/placement-api/extend_start.sh:5-9). A reconfigure also skips cinder's second "remove RPC version pin"
restart, which only upgrade.yml:24 runs. Whether this leaves cinder in a partially upgraded state is NOT AVAILABLE.
See Kolla images §4.
7. Things that scale with host count¶
- memcached and rabbit host lists in every config (§0). Any membership change rewrites the config of all 5 services and restarts every container of every service.
glance_num_retries = len(groups['glance-api'])(roles/cinder/templates/cinder.conf.j2:20).- HAProxy backend lists: glance uses
glance_api_hosts(roles/glance/defaults/main.yml:84-85); the others usegroups[host_group](roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:96). - Placement and cinder credentials are embedded in nova-compute config on every compute. Password rotation therefore needs a reconfigure of the whole compute fleet (§2.8, §3.7).
run_oncetasks repeat per batch whenANSIBLE_SERIALis set (§0).
8. NOT AVAILABLE list¶
- NOT AVAILABLE: whether an existing glance file-store image stays readable after
enable_swift: yesorglance_backend_ceph: yesis applied by reconfigure. - Answered from code: Glance rolling upgrade with
ANSIBLE_SERIAL=1runs the contract at the end of batch 1, before later batches are upgraded (§1.8 note 4; §0 run_once + serial;roles/glance/tasks/rolling_upgrade.yml:80-102). - NOT AVAILABLE: Glance-api start time with a large NFS
glance_file_datadir_volume(recursive chown,glance-api.json.j2:54-58). - NOT AVAILABLE: whether the glance cinder store works with
cinder_catalog_info = volume:cinder:internalURLand no[cinder]store section. - NOT AVAILABLE: whether cinder-backup works with default
cinder_backup_driver=cephandcinder_backend_ceph=no(kolla-ansible gives it no ceph config, §3.8 oddity 5). - NOT AVAILABLE: whether the volumes owned by one of several RBD cinder-volume nodes (
host) can still be managed while that node is down, and what state an in-flight create-from-image is in after a cinder-volume restart. - Cinder RPC pinning after
kolla-ansible upgrade(is a second restart needed?): answered from code for 2023.1: the role does a second restart of every cinder service after the upgrade flush and a 30 s pause, to remove the RPC version pin (roles/cinder/tasks/upgrade.yml:21-24,tasks/reload.yml:2-10,handlers/main.yml:70-88; §3.8 note 4). Remaining NOT AVAILABLE: whether 30 s is always enough for all services to report their new version. - NOT AVAILABLE: Heat-engine restart with
CREATE_IN_PROGRESSstacks: final state and recovery steps. - NOT AVAILABLE: Horizon restart: session survival (memcached) and start-up duration. When collectstatic/compress run is answered from kolla source (§4.8).
- Answered from kolla source:
bootstrap_heatdoes not update theheat_domain_adminpassword onceheat_user_domainexists (kolladocker/heat/heat-api/extend_start.sh:10-14; §5.6). Remaining NOT AVAILABLE: how heat behaves afterheat_domain_admin_passwordchanges, whenheat.confgets the new value (roles/heat/templates/heat.conf.j2:12) but Keystone keeps the old one. - NOT AVAILABLE: whether the kolla-toolbox
mysql_usertask (fromcommunity.mysql <3, whose source is not among the studied trees) resets the DB user's password on every deploy/reconfigure (§1.6).