Fact sheet B¶
Source: kolla-ansible at tag 2024.1-eom (commit 890396f), the kolla image repository at tag 2024.1-eom (commit
ed7a36a) and ansible-core 2.16.20. Kolla-Ansible paths are relative to ansible/ unless they start with tools/,
etc/ or doc/.
Labels: FACT = read directly from source code (cited): kolla-ansible, the kolla image repository (cited as kolla
docker/...) or Ansible (cited as ansible-core 2.16.20 lib/ansible/...). NOT AVAILABLE = these sources do not
establish it, often because it depends on upstream MariaDB, RabbitMQ, Keystone, HAProxy, Docker or Podman behaviour that
is not in them.
The supported Ansible range is ansible-core 2.15 to 2.16 (roles/prechecks/vars/main.yml:4-5,
tools/kolla-ansible:10-11). Statements about handler, serial and include semantics cite the ansible-core 2.16.20
source; ansible-core 2.15 is NOT AVAILABLE.
What changed since 2023.1, in one paragraph (FACT): the play structure is the same: MariaDB and RabbitMQ are applied
by their own playbooks, which site.yml imports (site.yml:400,468), the role handlers only fill group_by groups and
separate plays restart them in serial: "33%" batches (mariadb.yml:1-92, rabbitmq.yml:1-70). Every play now sets
max_fail_percentage, default 100 (mariadb.yml:8-11, rabbitmq.yml:8-11, site.yml:489-492). The container module
is kolla_container instead of kolla_docker, and the volume paths of mariadb_recovery follow the engine
(roles/mariadb/tasks/recover_cluster.yml:47,106). Outward RabbitMQ is gone, so the rabbitmq role is applied once.
RabbitMQ quorum queues are on by default and the prechecks are stricter (F17); upgrade now
runs the RabbitMQ version check, which refuses to skip a minor version (roles/rabbitmq/tasks/upgrade.yml:2,
roles/rabbitmq/tasks/version-check.yml:44-64). Mariabackup uses the mysql schema and narrower privileges. Keystone
has no admin-port leftovers and upgrade no longer removes the admin endpoint. The changed systemd unit template makes
the first 2024.1 check of each container report drift (F16). No role has a meta/main.yml
dependency on common (sections below).
0. Shared context that applies to all three roles¶
| Topic | Fact | Evidence |
|---|---|---|
| Action dispatch | tasks/main.yml = include_tasks: "{{ kolla_action }}.yml". mariadb first puts every mariadb host into its shard group (mariadb_shard_<id> via add_host, shard 0 by default); rabbitmq adds when: inventory_hostname in groups[role_rabbitmq_groups] |
roles/, group_vars/, roles/, roles/ |
common dependency |
None. The three roles have no meta/ directory. common is its own play, tagged common (site.yml:88-103), so it runs only in an untagged run or with -t common |
ls roles/ (no meta) |
common_run guard |
Still absent: no common_run anywhere in ansible/. The mariadb_recovery and mariadb_backup commands run playbooks that contain only the mariadb role (mariadb_recovery.yml:1-12, mariadb_backup.yml:1-11), so common does not run. The CLI passes kolla_action=deploy for recovery and kolla_action=backup -e mariadb_backup_type=... for backup |
tools/ |
| Serial | kolla_serial is passed only for upgrade and reconfigure, from ANSIBLE_SERIAL (default 0 = all hosts). The comment still says "Serial is not recommended and disabled by default". The MariaDB and RabbitMQ restart plays ignore it: they use a fixed serial: "33%" |
tools/; mariadb.yml:24,; rabbitmq.yml:36 |
max_fail_percentage |
New in 2024.1. Every play of the three playbooks, and the keystone play, sets max_fail_percentage: {{ <svc>_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) }}. With the default 100 it never stops a play early: ansible-core breaks the play only when the failed hosts of the current batch exceed that share of the batch size, and it clears the failed-host list before each batch |
mariadb.yml:8-11,; rabbitmq.yml:8-11,; mariadb_recovery.yml:4-7; mariadb_backup.yml:4-7; site.yml:489-492; ansible-core 2.16.20 lib/, lib/ |
| Config strategy | config_strategy: "COPY_ALWAYS", so recreate_or_restart_container restarts the container when its parameters are unchanged, and recreates it when they differ |
group_vars/; module_utils/ |
| Graceful stop | Effective stop timeout is 60 s (docker_graceful_timeout). kolla_container's graceful_timeout argument has no default; the value of docker_common_options is used, and the module's own fallback of 10 s applies only to a task that passes no common_options (library/; group_vars/). With Docker, a container created with a restart policy other than oneshot gets a systemd unit whose ExecStop is /usr/, and stops and restarts go through that unit (module_utils/; module_utils/). The engine's own stop behaviour (which signal, when it kills) is NOT AVAILABLE from these sources. For MariaDB the image still runs dumb-init without --single-child (kolla docker/). The image's comment says that mysqld_safe ignores SIGTERM and that, without --single-child, TERM is sent to all subprocesses, including mysqld (kolla docker/). NOT AVAILABLE: the dumb-init and mysqld_safe behaviour itself, and so whether mysqld shuts down cleanly within the 60 s |
group_vars/; library/ |
| Bootstrap containers | kolla_container defaults remove_on_exit: True, so detach: False bootstrap/upgrade containers are stopped and removed after they exit. With Docker they use restart_policy: oneshot and get no systemd unit. Containers started with a BOOTSTRAP label are later recreated because compare_labels sees a difference |
library/; module_utils/; module_utils/ |
compare_environment |
Checks only that the desired env keys exist with equal values. Extra env vars on the running container (for example BOOTSTRAP_ARGS) are not detected; labels are what force the recreate |
module_utils/ |
| First 2024.1 check | New in 2024.1. The systemd unit text is part of compare_container, and the template changed, so the first 2024.1 Check <role> containers task of a container whose unit 2023.1 wrote reports it as changed (F16). For MariaDB and RabbitMQ that notification only fills the restart groups, so the restart is batched by the restart plays; for Keystone it restarts the containers through the handlers (3.5) |
module_utils/; module_utils/ |
| Handler order | Ansible runs handlers in the order they are defined in handlers/, not in notify order: at a flush each host walks a copy of the play's handler list and runs those it was notified for. Under the default linear strategy each handler is queued on every notified host before the results are awaited. This matters for the keystone handlers and for the order in which the mariadb group_by handlers fill their groups |
ansible-core 2.16.20 lib/, lib/ |
| Loadbalancer config | The loadbalancer play (site.yml:105-327; hosts loadbalancer + &enable_loadbalancer_True, serial: kolla_serial) runs include_role ... tasks_from: loadbalancer for each service, only when enable_haproxy and kolla_action in ['deploy','reconfigure','upgrade','config'] |
site.yml:105-110, |
Batch semantics of serial: "33%" |
The batch size is int(0.33 × N), at least 1, where N is the number of hosts the restart play targets: 1 host per batch up to 6 hosts, 2 for 7 to 9, 3 for 10 to 12. If every host of a batch fails, the playbook stops and later batches do not run; if only some fail, the next batch runs (with the default max_fail_percentage of 100) |
ansible-core 2.16.20 lib/, lib/ |
Play order in site.yml (relevant subset): loadbalancer :105 -> mariadb (import_playbook: mariadb.yml, :400)
-> memcached :402 -> rabbitmq (import_playbook: rabbitmq.yml, :468) -> etcd :470 -> keystone :483 ->
everything else (glance :544 onward). Keystone therefore still runs after its DB, cache and MQ dependencies, and
before every service that registers with Keystone.
1. MariaDB (ansible/roles/mariadb)¶
1.1 Plays, hosts, tags, serial¶
| Play | Lines | Hosts | Tags | Serial | Notes |
|---|---|---|---|---|---|
Apply role loadbalancer -> include_role mariadb tasks_from: loadbalancer |
site.yml:223-227 |
loadbalancer + &enable_loadbalancer_True (:105-110) |
mariadb |
loadbalancer play has serial: kolla_serial (:110) |
when: enable_mariadb or enable_external_mariadb_load_balancer. Also ensures the monitor user (haproxy, or monitor with ProxySQL) on the first host of each shard, run_once, failed_when: False (roles/) |
Apply role mariadb |
mariadb.yml:3-16 |
mariadb + &enable_mariadb_True |
mariadb |
none | gather_facts: false (facts come from gather-facts.yml, site.yml:2) |
Restart mariadb services |
mariadb.yml:18-34 |
mariadb_restart + &enable_mariadb_True |
mariadb |
"33%" |
runs restart_services.yml on the hosts that handler 5 grouped (1.5) |
Start mariadb services |
mariadb.yml:36-52 |
mariadb_start + &enable_mariadb_True |
mariadb |
"33%" |
same task file, hosts grouped by handler 6 |
Restart bootstrap mariadb service |
mariadb.yml:54-68 |
mariadb_bootstrap_restart + &enable_mariadb_True |
mariadb |
none | the first-bootstrap host only (handler 4) |
Apply mariadb post-configuration |
mariadb.yml:70-92 |
mariadb + &enable_mariadb_True |
mariadb |
none | post-deploy.yml (register.yml, check.yml) for deploy/reconfigure/upgrade; post-upgrade.yml for upgrade |
Recover mariadb |
mariadb_recovery.yml:1-12 |
mariadb (no enable_ intersection) |
mariadb |
none | play var mariadb_recover: true; role when: enable_mariadb. Sets no gather_facts, so the implicit fact gathering applies (ansible-core 2.16.20 lib/). CLI passes kolla_action=deploy (tools/). The playbook has no restart plays |
Backup MariaDB |
mariadb_backup.yml:1-11 |
mariadb |
mariadb |
none | role when: enable_mariabackup. CLI passes kolla_action=backup and mariadb_backup_type=full/ (tools/) |
Every play of the table also sets max_fail_percentage (section 0).
What the absence of kolla_serial means (FACT plus reasoning):
ANSIBLE_SERIAL/kolla_serialis still ignored for MariaDB onreconfigureandupgrade. The role play handles all mariadb hosts in one batch, and the restart plays use their ownserial: "33%".- This is required by the design.
lookup_cluster.ymlusesgroup_byand themariadb_shard_<id>_*groups across the whole cluster in one pass (port liveness, WSREP state, volume existence) to decide bootstrap, fail-fast and which hosts are restarted or started (roles/mariadb/tasks/lookup_cluster.yml:2-80). Under serial these groups would only describe the current batch. - Rolling behaviour comes from the restart plays (
serial: "33%", see 1.5), not from the handlers. --limitstill narrows the plays: Ansible applies the limit when it resolves a play's hosts (ansible-core 2.16.20lib/ansible/executor/playbook_executor.py:281,lib/ansible/inventory/manager.py:404-406), and agroup_bygroup only receives the hosts that ran the task (lib/ansible/plugins/action/group_by.py:44-52,lib/ansible/plugins/strategy/__init__.py:692-694). Group-derived decisions then only see the limited hosts (for examplemariadb_cluster_existsis computed fromgroups[mariadb_shard_group + '_had_volume_True']among play hosts). The "existing but stopped" check uses the full shard lengthgroups[mariadb_shard_group] | length > 1, filled byadd_hostfromgroups['mariadb'](tasks/main.yml:2-7,lookup_cluster.yml:35-42). FACT from these lines:--limitto a single new node on an existing cluster makes that node treat the cluster as new (mariadb_cluster_exists=false,lookup_cluster.yml:10-17), so in theStart mariadb servicesplay it waits for WSREP Synced with only a 60 s budget (restart_services.yml:31-51).
1.2 mariadb_services (ansible/roles/mariadb/defaults/main.yml:2-47)¶
In 2024.1 the dict spans roles/mariadb/defaults/main.yml:2-53, as in 2023.1 (the heading keeps its earlier range so
that links stay stable).
| Key | container_name | group | enabled | Notable |
|---|---|---|---|---|
mariadb |
mariadb |
{{ mariadb_shard_group }} (mariadb_shard_0 by default) |
true |
Volumes: /etc/, named volume mariadb:/ (mariadb_datadir_volume, group_vars/), kolla_logs (:81-86). Environment MYSQL_USERNAME (monitor user), MYSQL_PASSWORD (empty unless ProxySQL), MYSQL_HOST, AVAILABLE_WHEN_DONOR=1 (:11-15). Container healthcheck /usr/ when healthchecks are enabled (:96-107). HAProxy mariadb (internal, enabled unless external LB): mode tcp, client/server timeouts 3600 s, option httpchk when clustercheck is enabled, otherwise no check option (:16-29). custom_member_list = internal_haproxy_members, built from the hosts of the default shard, where only the first host is active and every other host is backup: {% if not loop.first %} backup{% endif %}, check port 4569 (3306 without clustercheck) inter 2000 rise 2 fall 5 (:64; group_vars/). HAProxy mariadb_external_lb uses inventory hostnames as addresses (:30-41,65). With enable_proxysql (default no, group_vars/) ProxySQL is the database load balancer instead (mariadb_loadbalancer, group_vars/); see the loadbalancer page |
mariadb-clustercheck |
mariadb_clustercheck |
{{ mariadb_shard_group }} |
enable_mariadb_clustercheck (defaults to enable_haproxy, :132) |
Same environment as above (:49-53). socat on mariadb_clustercheck_port 4569 running EXEC:clustercheck (templates/). The 2024.1 clustercheck image also ships a socat_wrapper (kolla docker/), but the kolla-ansible template does not call it |
Other defaults: database_max_timeout: 120 (:59); mariadb_service: "{{ mariadb_services['mariadb'] }}" is used by
the recovery and WSREP checks (:112); recovery temp file /tmp/kolla_mariadb_recover_inventory_name_<shard> (:113);
mariadb_backup_host: "{{ groups[mariadb_shard_group][0] }}" (:123); backup user backup (haproxy mode) (:125).
Changed in 2024.1: the backup schema is mysql instead of PERCONA_SCHEMA (:124). Ports: 3306, wsrep 4567, IST
4568, SST 4444, clustercheck 4569 (group_vars/all.yml:85,467-471). enable_mariabackup: "no"
(group_vars/all.yml:804).
1.3 Task flow per kolla_action¶
| Action | Task files (in order) | Key behaviour |
|---|---|---|
| deploy | role play: deploy.yml: config.yml -> check-containers.yml -> bootstrap.yml (= lookup_cluster.yml -> bootstrap_cluster.yml [only if no cluster and host is groups[mariadb_shard_group][0]] -> recover_cluster.yml [only if mariadb_recover]); handlers at the end of the play. Then the restart, start and bootstrap-restart plays, then post-configuration: register.yml -> check.yml |
roles/, bootstrap.yml:1-10, post-deploy.yml:1-4; mariadb.yml:3-86 |
| reconfigure | import_tasks: deploy.yml (identical to deploy) |
tasks/ |
| upgrade | role play: upgrade.yml = deploy.yml; restart plays; post-configuration: post-deploy.yml, then "Run upgrade in MariaDB container" (upgrade_mariadb, KOLLA_UPGRADE, DB_HOST=api_interface_address, detach: False) on every mariadb host, in a play without serial, with no run_once and no image-changed condition. New in the 2024.1 image: mysql_upgrade output is also appended to mariadb-upgrade.log in the log directory (kolla docker/) |
tasks/, tasks/; mariadb.yml:70-92 |
| deploy-containers | check-containers.yml -> lookup_cluster.yml (prerequisite for the handlers; comment # handlers prerequisite). Handlers at the end of the role play, then the restart plays. No post-configuration (when lists deploy/reconfigure/upgrade only) |
tasks/; mariadb.yml:86 |
| pull | service-images-pull (pulls mariadb and mariadb-clustercheck if enabled) |
tasks/pull.yml, roles/ |
| stop | service-stop: stop_container for each enabled, mapped service on all hosts at once, unless the name is in skip_stop_containers |
tasks/stop.yml, roles/ |
| precheck | service-precheck (inventory group exists) plus free-port checks for 3306/4567/4568/4444, only if no mariadb container is listed |
roles/ |
| check | There is no check command in tools/ 2024.1 (tools/). check.yml runs in post-configuration and at the end of recovery: from every host of the default shard, <engine> exec mariadb mysql -h <VIP> -P 3306 -u <shard root user> -p<database_password> -e 'show databases;', 6 retries x 10 s |
roles/; recover_cluster.yml:234 |
| config (genconfig) | config.yml (handlers guarded by kolla_action != "config") |
handlers/ |
| backup | backup.yml: one-shot mariabackup container (detach: False, remove_on_exit: True) on mariadb_backup_host only (and only if mariadb_backup_possible), using the image of the running mariadb container when there is one, mounting mariadb:/ and mariadb_backup:/ and running kolla_set_configs && /usr/ with BACKUP_TYPE |
roles/; defaults/ |
recovery (mariadb_recovery) |
kolla_action=deploy with mariadb_recover=true, so deploy.yml runs and bootstrap.yml includes recover_cluster.yml. The port and WSREP block of lookup_cluster.yml is skipped (when: not mariadb_recover). mariadb_recovery.yml has no restart and no post-configuration plays, so register.yml does not run; recover_cluster.yml ends with check.yml |
lookup_cluster.yml:80, bootstrap.yml:9-10, recover_cluster.yml:234; mariadb_recovery.yml:1-12 |
lookup_cluster.yml (runs on deploy, reconfigure, upgrade and deploy-containers)¶
create_volume mariadb. If the volume already existed (not changed), the host joins groupmariadb_shard_<id>_had_volume_True(:2-13).mariadb_cluster_exists = groups[mariadb_shard_group + '_had_volume_True'] is defined(:15-17).- Unless recovering:
wait_forport 3306 withsearch_regex: MariaDB,timeout: 10,ignore_errors, thengroup_by mariadb_shard_<id>_port_alive_{True,False}(:19-33). - Fail fast with "MariaDB cluster exists but is stopped. Please start it using kolla-ansible mariadb_recovery" if
the shard has more than one node, the cluster exists and no node is alive (
:35-42). - On alive nodes,
SHOW STATUS LIKE "wsrep_local_state_comment", thengroup_by mariadb_shard_<id>_sync_status_<state>(:44-70). - Fail fast with "MariaDB cluster is not synced" unless every alive node is
Synced(:72-78). So a reconfigure refuses to run while any node is Donor or Joining.
First bootstrap (bootstrap_cluster.yml)¶
- On
groups[mariadb_shard_group][0]only, when no host had a volume: runbootstrap_mariadb(KOLLA_BOOTSTRAP,DB_ROOT_PASSWORD,DB_MAX_TIMEOUT),detach: False, notifyBootstrap MariaDB cluster, andset_fact bootstrap_host(roles/mariadb/tasks/bootstrap_cluster.yml:1-27).
Recovery (recover_cluster.yml), in order¶
- Fail if no cluster volume exists ("MariaDB cluster was not found. Is your inventory correct?") (
:2-5). - Delete every
/tmp/kolla_mariadb_recover_inventory_name_*file on the controller (delegate_to: localhost,run_once,with_fileglob) (:7-16). - The following block is skipped if
mariadb_recover_inventory_nameis supplied with-e, which lets the operator force the bootstrap host (:95-96):- Stop the
mariadbcontainer on every host (:19-25). This happens even if some nodes are healthy, so running recovery on a partially-up cluster causes a full DB outage. Changed in 2024.1: the task now passescommon_options(:24), so it uses the configured engine and the 60 s graceful timeout of section 0. - Run
mariadb_wsrep_recoverywithBOOTSTRAP_ARGS=--wsrep-recover,detach: false(:29-43). - Copy
<container_engine_volumes_path>/kolla_logs/_data/mariadb/mariadb.logto/tmp/mariadb_tmp.logon the host, then take the seqno of the lastRecovered positionline in the whole file withawk(:45-54), and delete the copy (:56-62).seqno = stdout_lines[0](:64-67). Changed in 2024.1: the path follows the engine:<docker_runtime_directory or /var/lib/docker>/volumeswith Docker,<docker_runtime_directory or /var/lib/containers>/storage/volumesotherwise (group_vars/all.yml:156-158). NOT AVAILABLE: whether an empty parse makes that host fail and so drops it from the play. - Each host compares its seqno against every other host of its shard, using a bash regex guard. A host with no
larger peer writes its own name to the controller temp file of its shard (
:69-89). On a tie, every tied host writes and the last writer wins (harmless when the seqnos are equal). - Every host reads the chosen name with
lookup('file', ...)(:91-94).
- Stop the
bootstrap_host = mariadb_recover_inventory_name(:98-100). On that host,lineinfile grastate.dat safe_to_bootstrap: 1, editing the volume path on the host directly (:102-112).- On the bootstrap host: start a
mariadbcontainer withBOOTSTRAP_ARGS: --wsrep-new-cluster, labelBOOTSTRAPandrestart_policy: oneshot-> wait for the port ->SET GLOBAL wsrep_provider_options='pc.bootstrap=yes'-> wait forwsrep_evs_stateto reportOPERATIONAL(10 x 6 s) (:114-170). - "Restart slave MariaDB container(s)":
start_containerloops overwith_dict: mariadb_serviceson all other hosts at once, then waits only for the port, not for WSREP Synced (:172-201). - "Restart master MariaDB container(s)":
recreate_or_restart_containeron the bootstrap host (drops the bootstrap args and label), then waits for the port (:203-232). Finallycheck.yml(:234). - Handlers notified earlier by config/check-containers do nothing for
restart mariadb: they need the_port_alive_*groups, which recovery never creates (handlers/main.yml:59-61,68-71), andmariadb_recovery.ymlhas no restart plays anyway. Handler 7 (restart mariadb-clustercheck) still runs on the hosts it was notified on (handlers/main.yml:75-91).
1.4 Config files generated (ansible/roles/mariadb/tasks/config.yml)¶
Destination (/etc/kolla/...) |
Source / merge order | Notifies | Lines |
|---|---|---|---|
mariadb/, mariadb-clustercheck/ dirs |
(directory), only for enabled services | none | :2-13 |
mariabackup/ dir |
only if enable_mariabackup and the host is mariadb_backup_host |
none | :15-25 |
mariabackup/ |
merge_configs: templates/ + /etc/ + /etc/. The template contains backup user/password and host={{ database_address }} (VIP) |
none | :27-40; templates/ |
<svc>/ (both services) |
templates/. mariadb.json.j2 recursively chowns /var/ and /var/lib/mysql on every COPY_ALWAYS start, the bootstrap, upgrade and --wsrep-recover one-shots included (templates/; Kolla images §5.4) |
restart <svc> (restart mariadb / restart mariadb-clustercheck) |
:42-53 |
mariabackup/ |
templates/ |
none | :55-65 |
mariadb/ |
merge_configs: templates/ + /etc/ + /etc/ |
restart mariadb |
:67-83 |
wsrep-notify.sh and the clustercheck xinetd file are still not generated: the role has no such templates in 2024.1
(roles/mariadb/templates/).
Notable galera.cnf.j2 contents:
wsrep_cluster_address=gcomm://lists all hosts of the shard group when there are two or more, and is empty on a single node (:29). Adding or removing a DB node therefore changes galera.cnf on every node of the shard, which triggers a batched restart of the whole shard.innodb_buffer_pool_sizeis 40 % ofansible_facts.memtotal_mb, capped at 8192M (:50-55). The file depends on facts, so a RAM change causes a restart.wsrep_sst_method=mariabackupandwsrep_sst_auth={{ database_user }}:{{ database_password }}(root credentials) (:2,40-41);max_connections=10000(:45). There is nowsrep_notify_cmd: HAProxy's view of a node comes only from the clustercheck HTTP check (or a plain TCP check without clustercheck,defaults/main.yml:28,64).
1.5 Handlers (ansible/roles/mariadb/handlers/main.yml), in definition order¶
Still holds in 2024.1: apart from the module name, the file is the one of 2023.1, at the same lines.
| # | Name | listen | Action / logic | Lines |
|---|---|---|---|---|
| 1 | Starting first MariaDB container | Bootstrap MariaDB cluster |
start_container mariadb with BOOTSTRAP_ARGS: --wsrep-new-cluster, label BOOTSTRAP, restart_policy: oneshot |
:2-20 |
| 2 | Wait for first MariaDB service port liveness | Bootstrap MariaDB cluster |
wait_for 3306 search_regex MariaDB, retries 10, delay 6 |
:23-34 |
| 3 | Wait for first MariaDB service to sync WSREP | Bootstrap MariaDB cluster |
wait until wsrep_local_state_comment == Synced, 10 x 6 s |
:36-49 |
| 4 | Ensure MariaDB is running normally on bootstrap host | Bootstrap MariaDB cluster |
group_by mariadb_bootstrap_restart |
:51-54 |
| 5 | Restart MariaDB on existing cluster members | restart mariadb |
group_by mariadb_restart for hosts in <shard>_port_alive_True, unless kolla_action == "config" |
:56-63 |
| 6 | Start MariaDB on new nodes | restart mariadb |
group_by mariadb_start for hosts in <shard>_port_alive_False that are not bootstrap_host, unless config |
:65-73 |
| 7 | Restart mariadb-clustercheck container | restart mariadb-clustercheck |
recreate_or_restart_container on all notified hosts at once, unless config |
:75-91 |
The monitor user is created by register.yml in post-configuration (tasks/register.yml:19-34) and by the loadbalancer
play (tasks/loadbalancer.yml:18-40).
restart_services.yml (run by the three restart plays of mariadb.yml): recreate_or_restart_container mariadb ->
wait for port 3306 (timeout 60, retries 10, delay 6) -> wait for WSREP Synced (10 x 6 s = about 60 s
budget). The WSREP wait is skipped for new nodes joining an existing cluster ("we don't want to wait for new nodes to
fully sync") (roles/mariadb/tasks/restart_services.yml:1-51).
Rolling semantics (FACT, from the plays and the Ansible source): the role play only fills the groups.
Restart mariadb services then runs restart_services.yml on the hosts of mariadb_restart in batches of
serial: "33%", and Start mariadb services does the same for mariadb_start (mariadb.yml:18-52). The batch size is
int(0.33 × N), at least 1 (section 0): with up to 6 hosts to restart, one node at a time, each waiting for port +
Synced before the next batch starts; with 7 to 9 hosts, 2 at a time; with 10 to 12, 3 at a time. That stays below a
third of the nodes, so a majority stays up while one batch restarts. NOT AVAILABLE: the order of the batches, which
follows the order in which group_by added the hosts to mariadb_restart (ansible-core 2.16.20
lib/ansible/plugins/strategy/__init__.py:692-694; lib/ansible/executor/playbook_executor.py:281), and so in which
batch node 0, the active HAProxy backend, restarts.
- Failure mode: if one node fails its port or 60 s WSREP-sync wait (for example a long IST/SST), only that host is
marked failed. When that is every host of the batch (always so with up to 6 nodes, one host per batch), ansible-core
stops the playbook and the later batches are not restarted; when other hosts of the batch succeed, the next batch
still restarts its nodes (ansible-core 2.16.20
lib/ansible/executor/playbook_executor.py:204-208). No longer holds in 2024.1: "the restart plays set neitherany_errors_fatalnormax_fail_percentage". They setmax_fail_percentage, default 100, which adds no stop; an operator value (mariadb_max_fail_percentageorkolla_max_fail_percentage) lower than the failed share of a batch stops the play at that batch (mariadb.yml:25-28,43-46; ansible-core 2.16.20lib/ansible/plugins/strategy/linear.py:336-350). They still set noany_errors_fatal. NOT AVAILABLE: whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is.
Handler and play order consequences (FACT):
- First deploy: config/check-containers notify
restart mariadbon every host, and the bootstrap host additionally getsBootstrap MariaDB cluster. Order: handlers 1-3 (bootstrap node up with--wsrep-new-clusterand Synced) -> 4 groups the bootstrap node -> 5 groups nothing (no alive nodes) -> 6 groups all other nodes -> 7. Then theStart mariadb servicesplay starts the other nodes in 33 % batches, each waiting for Synced (the cluster did not exist), thenRestart bootstrap mariadb servicerecreates the bootstrap node as a normal one (theBOOTSTRAPlabel differs), then post-configuration creates the users and runscheck.yml. - Scale-out:
Restart mariadb servicesrestarts the existing members first, batch by batch (theirwsrep_cluster_addresschanged), thenStart mariadb servicesstarts the new nodes in batches without waiting for SST to finish.
1.6 Bootstrap, DB and registration; what re-runs on reconfigure¶
bootstrap_mariadbruns only when no host had amariadbvolume and only ongroups[mariadb_shard_group][0](bootstrap.yml:4-7). It does not re-run on reconfigure.register.ymlruns on every deploy/reconfigure/upgrade (post-configuration play), on the first host of the shard, logging in at that host's ownapi_interface_address(not the VIP): it creates the shard root user (rootin HAProxy mode, password re-applied,*.*:ALL,GRANT) and the monitor user (roles/mariadb/tasks/register.yml:1-34). Whenenable_mariabackupis set it also, onmariadb_backup_host, creates the backup user (append_privs, password re-applied) and grants it on the schema (:36-72;defaults/main.yml:137). Changed in 2024.1: there is no "Creating the Mariabackup database" task any more; the backup user getsRELOAD,PROCESS,LOCK TABLES,BINLOG MONITORon*.*plusCREATE,ALTER,INSERTonmysql.mariadb_backup_history(register.yml:49; 2023.1 grantedCREATE TABLESPACEandSUPERand usedPERCONA_SCHEMA), and the second task grantsCREATE,INSERT,SELECTonmysql.*, the schema now beingmysql(register.yml:55-72;defaults/main.yml:124). Withappend_privs, privileges granted by 2023.1 are not removed by these tasks. NOT AVAILABLE: what MariaDB stores inmysql.mariadb_backup_historywhenkolla_mariadb_backup.shpasses--history(kolladocker/mariadb/mariadb-server/backup.sh:10-15,28-34).check.ymlruns on every deploy/reconfigure/upgrade, from every default-shard host through the VIP, as the shard root user withdatabase_passwordon the command line (roles/mariadb/tasks/check.yml:1-16).lookup_cluster.ymlchecks (port, WSREP, fail-fast) run on every deploy/reconfigure/upgrade/deploy-containers.- Backup prerequisite:
my.cnf, the backup user and themariabackupconfig are created only by deploy/reconfigure withenable_mariabackup: yes.kolla-ansible mariadb_backupitself runs onlybackup.yml. A reconfigure is therefore required after enabling backups (config.yml:15-65,register.yml:36-72,backup.yml; the docs atdoc/source/admin/mariadb-backup-and-restore.rst:30-34describe this step).
1.7 Dependencies¶
- Explicit: none in role metadata (no
meta/). The loadbalancer play must have configured themariadbfrontend, which usesloadbalancer-config(tasks/loadbalancer.yml:42-47). - Implicit:
- The VIP (haproxy + keepalived, or ProxySQL) must be up for
check.yml, which connects todatabase_address=kolla_internal_fqdn(group_vars/all.yml:83), so the post-configuration play fails without the load balancer.register.ymlconnects to the node directly. kolla_toolbox(fromcommon) for themysql_usermodules.- Facts:
hostname(HAProxy server names,wsrep_node_name) andmemtotal_mb.
- The VIP (haproxy + keepalived, or ProxySQL) must be up for
- Consumers: the tasks of 29 roles use
kolla_toolbox mysql_db(grep -l "module_name: mysql_db" roles/*/tasks/*.yml); the mariadb role is no longer one of them, since its schema task is gone. The services log in atdatabase_address, the VIP (for exampleroles/keystone/tasks/bootstrap.yml:8). Writes go to a single active backend, the first host of the default shard, because every other server isbackupin HAProxy (defaults/main.yml:64).
1.8 Operationally notable behaviour, risks and oddities¶
- Reconfigure with a galera.cnf change. Every node is notified, and the restart is batched by
Restart mariadb services(serial: "33%": 1 node at a time up to 6 nodes,mariadb.yml:18-34). Each batch waits for the port andSyncedbefore the next. The first host of the default shard is the only active HAProxy backend and is restarted in one of the batches (which one is NOT AVAILABLE, see 1.5). HAProxy checks each server withinter 2000 rise 2 fall 5(defaults/main.yml:64). NOT AVAILABLE: how long HAProxy takes to fail over to a backup node and back, and whether in-flight client connections are dropped at each step. - Clustercheck restart is not rolling. Handler 7 restarts
mariadb_clustercheckon all notified nodes at once, in the role play (handlers/main.yml:75-91). HAProxy health checks for every backend go through port 4569 when clustercheck is enabled (defaults/main.yml:28,64). NOT AVAILABLE: whether HAProxy marks every backend down when that restart outlasts thefall 5xinter 2000check settings, which would mean a short total DB outage through the VIP. On the first 2024.1 run this restart also follows from the unit-template drift (F16), unless the upgrade already recreated the container. - WSREP-sync wait is only about 60 s (
restart_services.yml:31-51, 10 x 6 s at:42-43; handlers:36-49). A restart that needs IST or SST longer than that fails the host, and see the failure-mode note in 1.5. - Fail-fast guards. Reconfigure aborts if the cluster is fully stopped (multi-node) or any alive node is not Synced
(
lookup_cluster.yml:35-42,72-78).kolla-ansible stopstops every node at once (service-stop), so the next deploy/reconfigure fails untilmariadb_recoveryis run. - Recovery takes the whole cluster down. It stops all
mariadbcontainers unconditionally (recover_cluster.yml:19-25), starts all non-bootstrap nodes at once (concurrent SST/IST load on the single bootstrap node,:172-201), and restarts the bootstrap node after the others have only opened their port (:203-232). NOT AVAILABLE: whether restarting the bootstrap/donor node aborts a joiner's SST from it that is still running. - Recovery bug (still present): "Restart slave/master MariaDB container(s)" loop over
with_dict: mariadb_serviceswith noitem.value.enabledor group check (recover_cluster.yml:172-186,203-217). Withenable_mariadb_clustercheck: no, recovery still tries to start amariadb_clustercheckcontainer: with Docker,kolla_containerpulls the image if it is not present, then creates and starts the container (module_utils/kolla_docker_worker.py:365-389). If the pull returns an error, the task fails on that host (:189-224, failures at:210,216). FACT. In deploy and reconfigure the clustercheck container is handled only when enabled (config.yml:42-53,check-containers.yml:13-16). - Recovery assumptions. It relies on the engine's volume directory on the host (
recover_cluster.yml:47,106;group_vars/all.yml:156-158), a fixed/tmp/mariadb_tmp.logon each host (:45-62) and a fixed temp file per shard on the controller (defaults/main.yml:113), and it deletes every shard's temp file first (recover_cluster.yml:7-16), so concurrent recoveries from one controller would collide. Changed in 2024.1: the volume path is no longer hard-coded to/var/lib/docker; it followskolla_container_engineanddocker_runtime_directory. - Upgrade.
mysql_upgrade(upgrade_mariadb) runs on all nodes concurrently and on every upgrade, even when the image is unchanged, in the post-configuration play after the batched restarts (mariadb.yml:70-92,tasks/post-upgrade.yml:2-24). NOT AVAILABLE: whether concurrentmysql_upgradeon Galera nodes conflicts. - Backup. It runs only on the first host of the shard (reading that node's datadir,
defaults/main.yml:123;backup.yml:33-35), butmy.cnfpoints the client atdatabase_address, the VIP (backup.my.cnf.j2:5). NOT AVAILABLE: whether, if HAProxy has failed over away from node 0, the backup tool connects to a different server than the one whose files it copies. Kolla-Ansible does not schedule or rotate backups (no such task inroles/mariadb/tasks/). - Credentials on the command line. Root credentials appear in
<engine> exec ... mysql -uroot -p{{ database_password }}command lines (lookup_cluster.yml:47-51,restart_services.yml:33-38,recover_cluster.yml:149, handlers:38-42) and incheck.yml's-p{{ database_password }}(check.yml:7-10).no_loghides the first ones from Ansible output but not from the host process list;check.ymlhas nono_log(check.yml:5-16). - Scale-out restarts the whole cluster, batch by batch (
wsrep_cluster_address,galera.cnf.j2:29). Every time the loadbalancer play runs,internal_haproxy_membersis also re-rendered in haproxy. - Oddity: hard-coded volume name. The containers mount
mariadb_datadir_volume(defaults/main.yml:85), butlookup_cluster.ymlcreates and tests the named volumemariadb(lookup_cluster.yml:2-8), and the backup container mountsmariadb:/var/lib/mysql(backup.yml:30). NOT AVAILABLE: what bootstrap detection and backup do when an operator setsmariadb_datadir_volumeto another name or a host path.
2. RabbitMQ (ansible/roles/rabbitmq)¶
No longer holds in 2024.1: "used twice". The outward RabbitMQ cluster (the murano agent broker) is gone with its
plays, its HAProxy entries, its prechecks and its definitions: no file under ansible/, etc/ or tools/ mentions
outward (grep -rln outward ansible etc tools finds nothing), and rabbitmq.yml has three plays
(rabbitmq.yml:1-70).
2.1 Plays, hosts, tags, serial¶
| Play | Lines | Hosts | Tags | Serial | Role vars |
|---|---|---|---|---|---|
loadbalancer -> include_role rabbitmq tasks_from: loadbalancer |
site.yml:281-288 |
loadbalancer | rabbitmq |
loadbalancer play serial | vars: role_rabbitmq_cluster_cookie: (null), role_rabbitmq_groups: (null). These are blanked so the lazily templated rabbitmq_services does not fail. Changed in 2024.1: when: enable_rabbitmq only |
Apply role rabbitmq |
rabbitmq.yml:3-28 |
rabbitmq + &enable_rabbitmq_True |
rabbitmq |
none | import_role with vars: role_rabbitmq_cluster_cookie: rabbitmq_cluster_cookie, _cluster_port: 25672, _epmd_port: 4369, _groups: rabbitmq, _management_port: 15672, _monitoring_user/, _password, _port: 5672 (5671 with TLS), _prometheus_port: 15692, _user (rabbitmq.yml:17-28; group_vars/) |
Restart rabbitmq services |
rabbitmq.yml:30-49 |
rabbitmq_restart + &enable_rabbitmq_True |
rabbitmq |
"33%" |
runs restart_services.yml |
Apply rabbitmq post-configuration |
rabbitmq.yml:51-70 |
rabbitmq + &enable_rabbitmq_True |
rabbitmq |
none | post-deploy.yml (rabbitmqctl enable_feature_flag all) for deploy/reconfigure/upgrade |
enable_rabbitmqdefaults to yes when eitherom_rpc_transportorom_notify_transportisrabbit(group_vars/all.yml:732). There is noenable_outward_rabbitmqany more.project_nameisrabbitmq(roles/rabbitmq/vars/main.yml:2), and no play overrides it any more, so the container, the config directory/etc/kolla/rabbitmq, the log dir and the restart grouprabbitmq_restartall use that name (roles/rabbitmq/defaults/main.yml:4,12,16;handlers/main.yml:4).- What the absence of
kolla_serialmeans:kolla_serialis ignored. The role play handles every node in one batch; restarts come from the restart play with a fixedserial: "33%". --tags rabbitmq(or--skip-tags rabbitmq) affects the broker plays and the loadbalancer fragment together.- Commands:
kolla-ansible rabbitmq-upgrade <version>importsrabbitmq.ymlwithkolla_action: upgradeand the imagerabbitmq-<version with dots replaced by dashes>(rabbitmq-upgrade.yml:18-21;tools/kolla-ansible:585-590). The 2024.1 image repository has one such image,rabbitmq-4-1(RabbitMQ 4.1, kolladocker/rabbitmq/rabbitmq-4-1/rabbitmq_packages_rpm.rocky:3), next to the defaultrabbitmqimage (RabbitMQ 3.13, kolladocker/rabbitmq/rabbitmq/rabbitmq_packages_rpm.rocky:1-4).kolla-ansible rabbitmq-reset-staterunsrabbitmqctl stop_app,force_reset,start_appandawait_online_nodeson every rabbitmq host, with noserialand nomax_fail_percentage(rabbitmq-reset-state.yml:1-9;roles/rabbitmq/tasks/reset-state.yml:1-16).
2.2 rabbitmq_services (ansible/roles/rabbitmq/defaults/main.yml:2-39)¶
In 2024.1 the dict spans roles/rabbitmq/defaults/main.yml:2-25 (the heading keeps its earlier range so that links stay
stable).
| Key | container_name | group | enabled | Notable |
|---|---|---|---|---|
rabbitmq |
{{ project_name }} (rabbitmq) |
{{ role_rabbitmq_groups }} |
true |
environment: KOLLA_CONFIG_STRATEGY, RABBITMQ_CLUSTER_COOKIE: role_rabbitmq_cluster_cookie, RABBITMQ_LOG_DIR. bootstrap_environment adds KOLLA_BOOTSTRAP (:8-16). Changed in 2024.1: the data volume is {{ rabbitmq_datadir_volume }}:/ (:52), default rabbitmq (group_vars/). Healthcheck healthcheck_rabbitmq when healthchecks are enabled (:35-46). HAProxy: only rabbitmq_management (http, 15672, host_group: rabbitmq) (:20-25); the outward management and external entries are gone. AMQP 5672 is not load-balanced: clients get every broker in transport_url |
Other defaults: rabbitmq_pid_file: /var/lib/rabbitmq/mnesia/rabbitmq.pid (:59),
rabbitmq_server_additional_erl_args: "+S 2:2 +sbwt none ..." (:60),
rabbitmq_cluster_partition_handling: "pause_minority" (:64), HA policy settings (:65-85),
rabbitmq_version_suffix: "" (:102).
Erlang cookie: comes from the passwords.yml key rabbitmq_cluster_cookie (etc/kolla/passwords.yml:206). It is
passed to the broker container as the env RABBITMQ_CLUSTER_COOKIE (defaults/main.yml:11,15). The common role also
writes it to {{ node_config_directory }}/kolla-toolbox/rabbitmq-erlang.cookie when enable_rabbitmq is set, and
kolla_toolbox copies it to /var/lib/rabbitmq/.erlang.cookie at every COPY_ALWAYS start; a change notifies
Restart kolla-toolbox container (roles/common/tasks/config.yml:272-282,
roles/common/templates/kolla-toolbox.json.j2:4-9). That happens only when the common play runs (an untagged run or
-t common, site.yml:88-103). The kolla image writes /var/lib/rabbitmq/.erlang.cookie only when
KOLLA_BOOTSTRAP is set, then exits; a normal start only handles the log dir (kolla
docker/rabbitmq/rabbitmq/extend_start.sh:7-26; see
Kolla images §4). That
happens only in the rabbitmq_bootstrap container, which runs only when the volume was just created
(tasks/bootstrap.yml:2-8,26). A normal start does not touch the cookie, and the file persists in the data volume
(defaults/main.yml:52). Because the cookie is in the env, compare_container (env compare) still picks up a cookie
change and causes a batched recreate, but the recreated brokers keep their old .erlang.cookie file. NOT AVAILABLE
(it depends on whether rabbitmq-server itself reads RABBITMQ_CLUSTER_COOKIE, which is not in these sources): whether
a partition occurs during the roll, or whether the change is instead silently not applied to existing brokers while
kolla_toolbox (used by the drain step and by service-rabbitmq) and any broker bootstrapped later on a fresh volume
get the new cookie and cannot talk to the existing nodes.
2.3 Task flow per kolla_action¶
| Action | Flow | Evidence |
|---|---|---|
| deploy | version-check.yml (run_once, delegated to the first rabbitmq host, tag rabbitmq-version-check; when a rabbitmq container is listed it fails if the new image's RabbitMQ is more than one minor version ahead of the running one, except 3.13 to 4.0 or 4.1, and on a downgrade) -> remove-ha-all-policy.yml (only when om_enable_rabbitmq_high_availability is false, the default) -> config.yml -> check-containers.yml -> bootstrap.yml; handlers at the end of the role play; then the restart play and post-configuration (enable_feature_flag all) |
roles/, version-check.yml:1-82 (:44-64 minor-version gate, :66-73 downgrade, :75-79 scope), remove-ha-all-policy.yml:1-29; group_vars/; rabbitmq.yml:30-70 |
| reconfigure | = deploy | tasks/ |
| bootstrap.yml | create_volume {{ project_name }}. Only when that volume was just created on that host: run rabbitmq_bootstrap with bootstrap_environment, detach: False |
roles/ |
| upgrade | Changed in 2024.1: version-check.yml first, then remove-ha-all-policy.yml (as above) -> config.yml -> feature-flags.yml (on the running brokers, before their restart) -> check-containers.yml; then the restart play (drained, batched) and post-configuration. There is no bootstrap step and no cluster-wide stop |
roles/ |
| deploy-containers | check-containers.yml only. Handlers at the end of the role play, then the restart play |
tasks/ |
| pull | service-images-pull |
tasks/pull.yml |
| stop | service-stop: all nodes at once |
tasks/stop.yml |
| precheck | group check, free ports, and getent ahostsv4/ for every rabbit host, failing unless every line resolves to that host's api IP (nodename rabbit@<hostname>); TLS files when TLS is on; with om_enable_rabbitmq_high_availability it asserts that an ha-all policy exists; with om_enable_rabbitmq_quorum_queues (now on by default) it lists the queues and asserts that every queue except reply_ and _fanout ones is a quorum queue, plus new transient-quorum and stream assertions. The prechecks role fails when HA and quorum are equal, so also when both are off (F17) |
roles/ (:61-77,); roles/; group_vars/ |
| check | empty file; tools/ 2024.1 has no check command |
tasks/ |
| config | config.yml. The handler is guarded by kolla_action != "config" |
handlers/ |
2.4 Config files (ansible/roles/rabbitmq/tasks/config.yml)¶
All files go into /etc/kolla/{{ project_name }}/. Except config.json, each file uses with_first_found, which
means a full replacement of the template, not a merge:
/etc/kolla/config/rabbitmq/<inventory_hostname>/<file> -> /etc/kolla/config/rabbitmq/<file> -> the role template.
All of them notify Restart rabbitmq container.
| File | Template highlights | Lines |
|---|---|---|
config.json |
rabbitmq.json.j2: /usr/. Copies the config files and, on every COPY_ALWAYS start, recursively chowns /var/ (the data volume, mnesia included) and /var/, one chown per directory and file (templates/; kolla docker/) |
config.yml:15-26 |
rabbitmq-env.conf |
RABBITMQ_NODENAME=rabbit@<hostname>, RABBITMQ_DIST_PORT=role_rabbitmq_cluster_port, RABBITMQ_PID_FILE, ERL_EPMD_ADDRESS=api ip, ERL_EPMD_PORT=role_rabbitmq_epmd_port |
config.yml:28-44; templates/ |
rabbitmq.conf |
listeners.tcp.1 = api_ip:role_rabbitmq_port (listeners.ssl.1 with TLS); cluster_partition_handling = pause_minority; management listener; management.load_definitions = /etc/; cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config with every host in groups[role_rabbitmq_groups] |
config.yml:46-62; templates/ |
erl_inetrc |
{inet6,true} for IPv6 |
config.yml:64-80 |
advanced.config |
inet_dist_use_interface and inet_dist_listen_min/ |
config.yml:82-98; templates/ |
definitions.json |
vhost /; user role_rabbitmq_user (administrator) plus an optional monitoring user. Policies: none by default. Only with om_enable_rabbitmq_high_availability (default false) is there an ha-all policy, on queues other than amq.*, *_fanout_* and reply_*, with ha-mode: exactly and ha-params = N // 2 + 1, ha-promote-on-shutdown: always, a 10 min message TTL and a 1 h queue expiry. The murano vhost, user and policy are gone |
config.yml:100-116; templates/ (policies :13-19); defaults/; group_vars/ |
enabled_plugins |
rabbitmq_management, plus rabbitmq_prometheus when Prometheus is enabled |
config.yml:118-134; defaults/ |
Oddity, now harmless: the override paths are hard-coded to /etc/kolla/config/rabbitmq/... and do not include
{{ project_name }} (config.yml:37-38,55-56). In 2023.1 a custom file there applied to both brokers; with a single
broker in 2024.1 it applies only to rabbitmq.
definitions.json on the first 2024.1 run (FACT plus reasoning): the template lost its outward branches
(templates/definitions.json.j2:1-20). Ansible renders templates with trim_blocks on and lstrip_blocks off by
default (ansible-core 2.16.20 lib/ansible/plugins/action/template.py:59-60). Rendering the 2023.1 and the 2024.1
template with those settings and the default variables gives files that differ in whitespace (the line breaks around the
vhosts list), so the template task reports a change and notifies Restart rabbitmq container on its first 2024.1 run,
unless an override file replaces the template. This reasoning rests on a rendering of the two templates outside any
host; reviewers should re-check it.
2.5 Handlers (ansible/roles/rabbitmq/handlers/main.yml), in definition order¶
| # | Name | listen | Logic | Lines |
|---|---|---|---|---|
| 1 | Restart rabbitmq container | (name) | group_by {{ project_name }}_restart, unless kolla_action == "config" |
:2-6 |
Still holds in 2024.1: there is no first-node-first rule.
restart_services.yml (run by the restart play): kolla_container_facts rabbitmq -> if that container is listed, put
the node into maintenance mode (community.rabbitmq.rabbitmq_upgrade action: drain, through kolla_toolbox) ->
recreate_or_restart_container (with env) ->
<engine> exec <c> rabbitmqctl wait --timeout 60 /var/lib/rabbitmq/mnesia/rabbitmq.pid. There is no
cluster-membership, partition, mirror-sync or quorum check (roles/rabbitmq/tasks/restart_services.yml:1-39). With
Docker, kolla_container_facts calls client.containers() without all=True (library/kolla_container_facts.py:76),
the call the 2023.1 study read as listing running containers only; new in 2024.1, the Podman path lists all
containers, stopped ones included (:97), so under Podman the drain step also runs for a stopped broker. NOT AVAILABLE:
what the drain does on a stopped node.
Rolling semantics (FACT, from the plays and the Ansible source): Restart rabbitmq services runs
restart_services.yml on the hosts of rabbitmq_restart in serial: "33%" batches (rabbitmq.yml:30-49): one node
at a time with up to 6 notified nodes, 2 with 7 to 9 (section 0). Restart time grows linearly with N. Only notified
hosts are in the group. If every host of a batch fails (for example the --timeout 60 wait), the playbook stops before
the next batch (ansible-core 2.16.20 lib/ansible/executor/playbook_executor.py:204-208). NOT AVAILABLE: the order of
the batches, which follows the order in which group_by added the hosts
(lib/ansible/plugins/strategy/__init__.py:692-694).
No longer holds in 2024.1: the 2023.1 oddities about literal names. restart_services.yml checks for a container
named rabbitmq (:6) and feature-flags.yml runs <engine> exec rabbitmq (feature-flags.yml:3), which is the only
broker now; remove-ha-all-policy.yml uses {{ kolla_container_engine }} exec instead of a literal docker exec
(remove-ha-all-policy.yml:14,20).
2.6 Bootstrap and registration¶
- The cluster/user bootstrap is declarative. Users, vhosts and policies come only from
definitions.json, which is loaded at node boot (rabbitmq.conf.j2:13). There is norabbitmq_usertask in the role, so a changedrabbitmq_passwordtakes effect only through a restart (batched, via the restart play). The role'srabbitmqctlcalls are limited to the version check, theha-allremoval, feature flags, the wait andreset-state(version-check.yml:14,remove-ha-all-policy.yml:14,20,feature-flags.yml:3,restart_services.yml:39,reset-state.yml:1-16); the precheck adds read-onlylist_policiesandlist_queues(precheck.yml:110,134). - The
rabbitmq_bootstrapcontainer runs only on hosts whose volume is newly created (bootstrap.yml:26). It does not re-run on reconfigure. - Per-service users and vhosts are created only by the
service-rabbitmqrole (roles/service-rabbitmq/tasks/main.yml:1-45,update_password: alwaysat:24), which in 2024.1 is still used only bynova-cell(roles/nova-cell/tasks/rabbitmq.yml:5,15). Every other service shares theopenstackuser on vhost/(group_vars/all.yml:241-258).
2.7 Dependencies¶
- Explicit: none in role metadata (no
meta/). - Implicit:
- Hostnames must resolve to the api IP on every node (precheck), because nodenames are
rabbit@<hostname>. - Facts (
hostname). kolla_toolbox(fromcommon) for the drain step andservice-rabbitmq.- HAProxy, only for the management UI.
- Hostnames must resolve to the api IP on every node (precheck), because nodenames are
- Consumers: the templates of 25 roles render
rpc_transport_url(26 roles reference sometransport_url, swift included), which lists every host ingroups['rabbitmq'](group_vars/all.yml:248,258;grep -l rpc_transport_url roles/*/templates). Keystone is one of them (keystone.conf.j2:4,54). The counts fell from 30 and 31 in 2023.1 because roles were removed.
2.8 Operationally notable behaviour, risks and oddities¶
- Reconfigure with a config change drains and restarts the brokers in
serial: "33%"batches, one node at a time with up to 6 nodes. At most one node is down at a time in that case, so a 3-node cluster keeps 2/3 and stays above thepause_minoritythreshold. The restart only runsrabbitmqctl wait --timeout 60and has no mirror-sync or quorum check (restart_services.yml:9-39). Mirroring is off by default (om_enable_rabbitmq_high_availability: false,group_vars/all.yml:266; emptypolicies,definitions.json.j2:17-19), and deploy removes an existingha-allpolicy (remove-ha-all-policy.yml:18-22). Quorum queues are on by default (group_vars/all.yml:268). NOT AVAILABLE: what happens to the queues of a drained node, and whether the next node may be stopped while the quorum queues of the freshly restarted node are still catching up. - Upgrade = full outage of the message bus still does not hold:
upgrade.ymlhas no stop step (roles/rabbitmq/tasks/upgrade.yml:1-12); an image change goes through the same drained, batched restart play as a reconfigure (rabbitmq.yml:30-49). No longer holds in 2024.1: "a downgrade is not refused onupgrade".upgrade.ymlnow importsversion-check.yml(upgrade.yml:2), soupgradeandrabbitmq-upgraderefuse a downgrade and also a jump of more than one minor version, except 3.13 to 4.0 or 4.1, with a message that points tokolla-ansible rabbitmq-upgrade(version-check.yml:44-73). The check runs only when arabbitmqcontainer is listed on the first rabbitmq host, and can be skipped with the tagrabbitmq-version-check(version-check.yml:75-79). NOT AVAILABLE: which RabbitMQ version a cloud runs before its 2024.1 upgrade, and whether clients lose RPC while each node is drained and restarted. - Uncontrolled full stop (
kolla-ansible stop, power loss) followed by deploy: the brokers come back through the batched restart play, one batch after another, each waitingrabbitmqctl wait --timeout 60. With Docker the drain step is skipped for a stopped container (restart_services.yml:17;library/kolla_container_facts.py:76). Which node starts first is NOT AVAILABLE (group_byorder). NOT AVAILABLE (RabbitMQ semantics): whether a node that was not the last to stop waits for its peers' tables longer than 60 s. FACT: if it does, the wait fails, and with one node per batch (up to 6 nodes) the playbook stops before the other nodes are started (section 0). - Scaling the rabbitmq group changes
transport_urlin about 25 services, so it restarts almost every OpenStack service cloud-wide when those plays run. It also changesrabbitmq.conf(classic_config node list,rabbitmq.conf.j2:18-21) on every broker, which causes a batched restart. - Password change.
definitions.jsonchanges cause a batched broker restart in the rabbitmq plays. Consumers get the new password only when their own plays run later insite.yml. NOT AVAILABLE: whether there is an auth-failure window in between. - No longer holds in 2024.1: the
kolla-ansible pullfailure with outward RabbitMQ. The lookup of<project_name>_services(roles/service-images-pull/tasks/main.yml:14) now always findsrabbitmq_services(defaults/main.yml:2). check.ymlis empty.- The precheck runs
getentfor every rabbit host on every rabbit host, so it does N² lookups (precheck.yml:61-77). - Quorum queues by default (F17). A cloud that kept the 2023.1 default gets
rabbit_quorum_queue = truein the[oslo_messaging_rabbit]section of its services, Keystone included (roles/keystone/templates/keystone.conf.j2:77-78), and so a changed config and a restart of those services on the first 2024.1 run that renders configs.prechecksfails while classic queues exist and quorum is on (precheck.yml:131-151). NOT AVAILABLE: what oslo.messaging does with existing classic queues. - New oddity: two volume names. The broker mounts
rabbitmq_datadir_volume(defaults/main.yml:52), butbootstrap.ymlcreates and tests the volume{{ project_name }}(bootstrap.yml:2-8). With the default both arerabbitmq(group_vars/all.yml:913). NOT AVAILABLE: whether the cookie bootstrap runs for the real data volume when an operator sets anotherrabbitmq_datadir_volume.
3. Keystone (ansible/roles/keystone)¶
3.1 Plays, hosts, tags, serial¶
| Play | Lines | Hosts | Tags | Serial |
|---|---|---|---|---|
loadbalancer -> include_role keystone tasks_from: loadbalancer |
site.yml:203-207 |
loadbalancer | keystone |
loadbalancer serial |
Apply role keystone |
site.yml:483-495 |
keystone + &enable_keystone_True |
keystone |
serial: '{{ kolla_serial|default("0") }}' (:488) |
What serial means here:
- With the default (
ANSIBLE_SERIALunset, so 0 = all hosts), every keystone node is handled in one batch (ansible-core 2.16.20lib/ansible/executor/playbook_executor.py:300-302), and handlers restart all keystone containers at once (section 0, "Handler order"). - With
ANSIBLE_SERIAL=n,run_oncetasks run once per batch: bootstrap container (db_sync), key distribution, register. Each batch is a separate run on the inventory restricted to that batch, andrun_oncesends a task only to the first host left (playbook_executor.py:185-190,lib/ansible/plugins/strategy/linear.py:200-209). - Fernet bootstrap is blocked when not all hosts are in the batch (3.6).
- On upgrade,
Init(expand/migrate) runs in the batch containinggroups['keystone'][0]andFinish(contract) runs in the batch containinggroups['keystone'][-1].
3.2 keystone_services (ansible/roles/keystone/defaults/main.yml:2-55)¶
In 2024.1 the dict spans roles/keystone/defaults/main.yml:2-56 (the heading keeps its earlier range so that links stay
stable).
| Key | container_name | group | enabled | Notable |
|---|---|---|---|---|
keystone |
keystone |
keystone |
true |
Apache + mod_wsgi. Volumes include keystone_fernet_tokens:/ and, with om_enable_queue_manager, /dev/shm (:153-160). HAProxy keystone_internal (keystone_internal_port, 5000) and keystone_external (keystone_public_port, 5000 unless the single external frontend is on, external_fqdn: keystone_external_fqdn) (:11-30; group_vars/). No longer holds in 2024.1: the keystone_admin (35357) entry "kept only for upgrade compatibility" is gone, and so is the admin Listen/VirtualHost of wsgi-keystone.conf (templates/ has one Listen, :10). Healthcheck healthcheck_curl on port 5000 (:114-125) |
keystone-ssh |
keystone_ssh |
keystone |
true |
sshd -D on api_interface_address:8023 (templates/, group_vars/). Shares the keystone_fernet_tokens volume (:41) |
keystone-fernet |
keystone_fernet |
keystone |
true |
Runs cron with the rotation crontab. Healthcheck fernet-healthcheck.sh = fernet-node-sync.sh --check && fernet-push.sh --check (:140-151; templates/). Shares the keystone_fernet_tokens volume (:54) |
There is still no keystone_token_provider: keystone.conf sets provider = fernet unconditionally
(templates/keystone.conf.j2:32), and the ssh and fernet containers are always enabled.
Relevant globals (ansible/group_vars/all.yml):
| Variable | Value | Line |
|---|---|---|
fernet_token_expiry |
86400 | :977 |
fernet_token_allow_expired_window |
172800 | :979 |
fernet_key_rotation_interval |
expiry + window = 259200 s (3 days) | :984 |
keystone_internal_url |
keystone_internal_fqdn | kolla_url(internal_protocol, keystone_internal_port), i.e. <internal_protocol>://<internal VIP>:5000. Changed in 2024.1: keystone_admin_url, keystone_admin_port and admin_protocol no longer exist |
:957,960 |
openstack_auth.auth_url |
keystone_internal_url |
:990-996 |
multiple_regions_names |
[openstack_region_name] |
:695-696 |
keystone_default_user_role |
member |
:986 |
3.3 Task flow per kolla_action¶
| Action | Flow | Evidence |
|---|---|---|
| deploy | config.yml -> check-containers.yml -> clone.yml (dev mode only) -> bootstrap.yml -> flush_handlers -> distribute_fernet.yml -> register.yml -> register_identity_providers.yml (federation only) |
roles/ |
| reconfigure | = deploy, so bootstrap, distribution and registration all re-run | roles/ |
| upgrade | config.yml -> check-containers.yml -> set log_bin_trust_function_creators=1 (run_once, on groups['keystone'][0]) -> notify Init keystone database upgrade on [0] -> notify Finish keystone database upgrade on [-1] -> flush_handlers -> set log_bin_trust_function_creators=0 (run_once, every batch). No bootstrap or key distribution. Changed in 2024.1: the final service-ks-register call that removed the admin endpoint is gone, so upgrade registers nothing |
roles/ |
| deploy-containers | check-containers.yml |
tasks/ |
| pull / stop | service-images-pull / service-stop (all 3 containers, all hosts at once) |
tasks/pull.yml, tasks/stop.yml |
| precheck | group check; free ports 5000 and 8023; assert fernet_token_expiry is an int between 0 and 604800 |
roles/ |
| check | No check.yml in the role, no kolla_enable_sanity_* variable and no check command in tools/ 2024.1 |
ls roles/ |
| config | config.yml. Handlers are guarded by kolla_action != "config" |
handlers/ |
bootstrap.yml and bootstrap_service.yml (run on every deploy/reconfigure):
mysql_db keystoneandmysql_user keystonewithkeystone_database_password,append_privs(run_once,delegate_to: groups['keystone'][0], skipped withuse_preconfigured_databases) (roles/keystone/tasks/bootstrap.yml:1-36). This re-asserts the DB user password on each run.kolla_container_facts keystone_fernet->group_by keystone_fernet_<State|bootstrap>(roles/keystone/tasks/bootstrap_service.yml:2-18). With Docker the module callsclient.containers()withoutall=True(library/kolla_container_facts.py:76); with Podman it lists all containers and returns their attributes (:97,109). NOT AVAILABLE: theStatevalue the Podman path returns, and so whether thekeystone_fernet_runninggroup is formed under Podman.- Fail if no host has
keystone_fernetrunning and not every keystone host is inansible_play_batch("Please run without --limit or --serial") (:22-31). bootstrap_keystonecontainer (KOLLA_BOOTSTRAP). In the kolla image,kolla_startsourceskolla_extend_start, which seesKOLLA_BOOTSTRAP, runssudo -H -u keystone keystone-manage db_syncand exits 0 without starting httpd (kolladocker/base/start.sh:18-21,docker/keystone/keystone/extend_start.sh:41-46; see Kolla images §4).EXTRA_KEYSTONE_MANAGE_ARGSis empty by default (kolladocker/keystone/keystone/extend_start.sh:21) and kolla-ansible does not set it. It isrun_oncewithoutdelegate_to(first host of the batch) and runs on every deploy/reconfigure (bootstrap_service.yml:33-50).bootstrap_keystone_fernet:keystone-manage fernet_setup,run_once,delegate_to: groups['keystone'][0], only if nokeystone_fernetcontainer is running on any play host (:52-75).
distribute_fernet.yml (every deploy/reconfigure):
- Wait for port 8023 on every host (
:2-10). <engine> exec -t keystone_fernet /usr/bin/fernet-push.sh,run_once,delegate_to= the first host inkeystone_fernet_running(orgroups['keystone'][0]on first bootstrap) (:12-18).
register.yml (every deploy/reconfigure):
<engine> exec keystone kolla_keystone_bootstrap <admin user> <admin pw> <project> admin <internal_url> <public_url> <region>, once per region (run_once); no admin URL (roles/keystone/tasks/register.yml:2-12).service-ks-registerfor the identity service and its endpoints (internal, public) (:14-19;defaults/main.yml:200-208). Changed in 2024.1: the admin endpoint entry withstate: absentis gone andservice-ks-registerno longer passes astatetoos_keystone_endpoint(roles/service-ks-register/tasks/main.yml:24-46); Keystone no longer passesservice_ks_register_roles, so this call no longer creates theservicerole (the role still accepts that list,roles/service-ks-register/tasks/main.yml:97,defaults/main.yml:29).os_keystone_rolekeystone_default_user_role(member) (:21-32).
3.4 Config files (ansible/roles/keystone/tasks/config.yml)¶
Still holds in 2024.1: config.yml is the 2023.1 file, at the same lines.
Destination (/etc/kolla/...) |
Source / merge order | Notifies | Lines |
|---|---|---|---|
<svc>/ |
keystone.json.j2 / keystone-ssh.json.j2 / keystone-fernet.json.j2 |
Restart <svc> container |
:45-56 |
keystone/, keystone-fernet/ |
merge_configs: templates/ + /etc/ + /etc/ + /etc/ + /etc/ |
Restart keystone container / Restart keystone-fernet container |
:58-77 |
keystone/ |
template. Waits up to 36 x 5 s (3 min) for fernet key 0, then exec httpd/ |
Restart keystone container |
:79-91; templates/ |
keystone/ |
Created only if /etc/ exists on the controller. Every file in it is templated (Jinja) into /etc/. keystone.conf gains [identity] domain_specific_drivers_enabled = true, domain_config_dir = /etc/ |
Restart keystone container. A file added once keystone.conf already has the domain settings is not yet in the container and the domains entry is optional, so kolla_set_configs --check exits 2 and "Check keystone containers" fails on every keystone host before the handler runs (templates/, tasks/, module_utils/; Kolla images §3.3) |
:34-39,93-130; keystone.conf.j2:24-28 |
<keystone, keystone-fernet>/ |
First found in /etc/ (supported_policy_format_list) |
Restart <svc> container |
:15-32,132-145 |
keystone/ |
with_first_found: /etc/ -> /etc/ -> template. processes = keystone_api_workers = openstack_service_workers = min(vCPU, 5) (defaults/, group_vars/) |
Restart keystone container |
:151-167; templates/ |
keystone/ |
Only if /etc/ exists |
Restart keystone container |
:169-193 |
| (controller) cron generator | python fernet_rotate_cron_generator.py -t <interval_s//60> -i <index in groups['keystone']> -n <len>, delegate_to: localhost, per host |
none (changed_when: false) |
:195-210 |
keystone-fernet/ |
templates (id_rsa = {{ keystone_ssh_key.private_key }}) |
Restart keystone-fernet container |
:212-233 |
keystone-ssh/ |
templates (id_rsa.pub becomes authorized_keys) |
Restart keystone-ssh container |
:235-250 |
keystone.conf.j2 highlights:
transport_url = rpc_transport_url(:4).[database] connectionvia the VIP,max_retries = -1(:18-22).[token] provider = fernet,expiration,allow_expired_window(:30-34).[fernet_tokens] max_active_keys = ceil((expiry + window) / rotation_interval) + 2, which is 3 with the defaults (:43-46).[cache] backend = oslo_cache.memcache_pool,memcache_servers= everygroups['memcached']host (:48-51).[oslo_messaging_notifications]usesnotify_transport_urland drivermessagingv2only when ceilometer or barbican topics are enabled, otherwisenoop(:53-60;defaults/main.yml:185-194).- Changed in 2024.1:
[oslo_messaging_rabbit]rendersrabbit_quorum_queue = trueby default, plusrabbit_stream_fanout,rabbit_qos_prefetch_countandrabbit_transient_quorum_queuewhen those options are on, anduse_queue_manager = truewithom_enable_queue_manager, which also adds[oslo_concurrency] lock_path(:62-86,116-119;group_vars/all.yml:264-279).
3.5 Handlers (ansible/roles/keystone/handlers/main.yml). Order is load-bearing (comment :2-3)¶
Still holds in 2024.1: apart from the module name, the file is the one of 2023.1, at the same lines.
| # | Name | Logic | Lines |
|---|---|---|---|
| 1 | Init keystone database upgrade | init_upgrade_database container, KOLLA_UPGRADE, run_once, only when kolla_action == upgrade and the host is groups['keystone'][0] |
:5-27 |
| 2 | Restart keystone-ssh container | recreate_or_restart_container. All notified hosts in the batch in parallel |
:29-43 |
| 3 | Restart keystone-fernet container | same | :45-59 |
| 4 | Restart keystone container | same: no rolling, no wait for port or health | :61-75 |
| 5 | Finish keystone database upgrade | finish_upgrade_database, KOLLA_FINISH_UPGRADE, run_once, kolla_action == upgrade. No host guard: it relies on being notified only on groups['keystone'][-1] (upgrade.yml:28-31) |
:77-98 |
ansible-core 2.16.20 runs a run_once handler once and then clears it for every other host
(lib/ansible/plugins/strategy/linear.py:200-209).
3.6 Fernet key lifecycle (FACT unless marked)¶
- Storage: a per-host volume
keystone_fernet_tokens, shared by thekeystone,keystone_fernetandkeystone_sshcontainers (defaults/main.yml:41,54,160). - Initial setup:
keystone-manage fernet_setupongroups['keystone'][0]only when nokeystone_fernetcontainer is running anywhere in the play (bootstrap_service.yml:52-75). The code comment admits the detection is imprecise: "This does not catch some cases ... the keystone_fernet container is created but not running ... Probably what we care about is the existence of Fernet key 0" (:10-14). NOT AVAILABLE (Keystone upstream behaviour): whetherfernet_setupis a no-op when key0already exists, which would make a re-run on a node with an intact volume harmless. On a rebuiltgroups['keystone'][0](empty volume), with everykeystone_fernetcontainer stopped (for example afterkolla-ansible stop), a deploy runsfernet_setupon that node and then pushes its key set withrsync --deleteto every node (:52-75;distribute_fernet.yml:12-18;templates/fernet-push.sh.j2:17-22). NOT AVAILABLE: whetherfernet_setupthen generates brand-new keys and whether that invalidates all tokens; the code comment atbootstrap_service.yml:20-21names this risk. - Rotation: cron in
keystone_fernetrunsfernet-rotate.sh=keystone-manage fernet_rotate&&fernet-push.sh(templates/fernet-rotate.sh.j2:6-8).fernet-push.shruns, for each other keystone host,rsync -az ... --delete /etc/keystone/fernet-keys/ keystone@<host>:/etc/keystone/fernet-keysover ssh port 8023, underset -o errexit(templates/fernet-push.sh.j2:3,17-22). With--checkit does the same as a dry run, which the container healthcheck uses (:6-15). -
Schedule: the generator gives each host a weekly crontab (min hour * * dow) in round-robin with offset
interval * index(files/fernet_rotate_cron_generator.py:39-94;templates/crontab.j2:5-7). It deliberately under-rotates across the week boundary (:53-54,67). Working throughgenerate()for these inputs, which all take the day branch (offsetinterval * index, stepinterval * hosts, a job only whiletime + interval <= 10080;:55-56,65-73), gives:Interval Hosts Result 4320 min (default 3 days) 3 h0 = Sun 00:00, h1 =Wed 00:00, h2 = no jobs. Gaps 3 d / 4 d4320 min 5 only h0 and h1 have jobs 1440 min 3 h0 Sun/Wed/Sat, h1 Mon/Thu, h2 Tue/Fri Crons run in the container's local time (
/etc/localtimeis mounted,defaults/main.yml:51). -
Distribution on every deploy/reconfigure:
fernet-push.shruns from the first running node (distribute_fernet.yml:12-18), so that node's key set overwrites every other node (--delete).
3.7 Bootstrap, DB and registration: what re-runs on reconfigure¶
Re-runs every time:
mysql_dbandmysql_user keystone- the
bootstrap_keystonecontainer (db_sync) - the fernet-bootstrap check
fernet-push.shkolla_keystone_bootstrapfor each region. The keystone image installs kolladocker/keystone/keystone/keystone_bootstrap.shunder that name (kolladocker/keystone/keystone/Dockerfile.j2:35). Called with seven arguments it runs in its "no ADMIN_URL modern mode" (keystone_bootstrap.sh:19-30) and runskeystone-manage bootstrapwith the admin user, password, project, role, internal/public URLs, service namekeystoneand the region (:46). It counts eight 'already exists'/'exists'/'already has'/'skipping' patterns (domain, project, user, role, role assignment, region, internal and public endpoints) but reports unchanged only when the count is nine (:51-63). NOT AVAILABLE (upstreamkeystone-manage bootstrapis not in these sources): whether any output line matches twice, so that the task can ever report unchanged; whetherkeystone-manage bootstrapre-asserts the admin user, project, role and endpoints and resets the admin password tokeystone_admin_passwordif it differs. The wrapper has no pattern for a password reset (:51-63).service-ks-register(identity service and endpoints)- the
memberrole
All of these are run_once per batch, except the cron generator, which runs on the controller once per host.
3.8 Dependencies¶
- Explicit: none in role metadata (no
meta/). - Implicit, upstream:
- MariaDB via the VIP (
keystone_database_address,defaults/main.yml:70). - memcached:
[cache](keystone.conf.j2:48-51). The memcached play (site.yml:402) runs before keystone (:483). - RabbitMQ:
transport_urlis always rendered, and notifications go out only if ceilometer or barbican is enabled (keystone.conf.j2:4,53-60). - HAProxy VIP:
keystone_internal_urlis used byopenstack_authfor registration (group_vars/all.yml:990-996). - kolla_toolbox (
os_*modules). passwords.ymlkeystone_ssh_key(etc/kolla/passwords.yml:174).
- MariaDB via the VIP (
- Downstream: see 3.10.
3.9 Operationally notable behaviour, risks and oddities¶
- Reconfigure with a config change (default serial 0) restarts
keystone_ssh, thenkeystone_fernet, thenkeystone, each on all keystone hosts simultaneously (handlers:29-75). The result is a brief Keystone-wide API outage. Nothing waits for the API to return beforeregister.ymlruns (which usesrun_onceplusuntil/retries insideservice-ks-register,roles/service-ks-register/tasks/main.yml:19-21). The containers have a healthcheck (defaults/main.yml:114-125), but the handlers do not wait for it. NOT AVAILABLE: whether other services keep validating cached tokens via memcached during the restart. SetANSIBLE_SERIALfor a rolling reconfigure. On the first 2024.1 run the same all-at-once restart follows from the unit-template drift (F16) and, with the default quorum setting, from the changedkeystone.conf(keystone.conf.j2:77-78), unless the upgrade already recreated the containers. - What restarts keystone indirectly.
keystone.confembeds every memcached host and every rabbitmq host (keystone.conf.j2:4,51,54), and the fernet crontab andfernet-push.shembed the keystone host list and index. Adding or removing a memcached, rabbitmq or keystone node therefore restarts keystone, and keystone-fernet as well for keystone node changes. - Fernet push has no partial-failure handling.
set -o errexitstops at the first unreachable peer, so later peers are not updated either (fernet-push.sh.j2:3,17-22). The next scheduled rotation runs on another node (round-robin, 3.6), which may be one of the peers left with the stale set. That node runsfernet_rotateand thenfernet-push.sh, whosersync --deleteoverwrites the key set of each peer it reaches, stopping at the first failure (templates/fernet-rotate.sh.j2:6-8,templates/fernet-push.sh.j2:3,17-22). The reconfigure-time push from "first running node" works the same way. Thekeystone_fernethealthcheck (fernet-push.sh --check) reports such a failure as unhealthy (templates/fernet-healthcheck.sh.j2:6). NOT AVAILABLE (Keystone key handling): whether this removes the key that signed live tokens and so causes mass 401s. - Bug in the cron generator hour calculation (still present). In the day branch,
hour = time % HOUR_SPANis computed where(time % DAY_SPAN) // HOUR_SPANwas intended (fernet_rotate_cron_generator.py:68-70). It is harmless when the interval is a whole number of days (the default). Otherwise it can over-rotate. Read from the code: with interval 3600 min (60 h, i.e.fernet_token_expiry=43200), h1 hastime = 3600, soday = 2andhour = 3600 % 60 = 0, givingday 2 00:00instead ofday 2 12:00after h0'sday 0 00:00, which is a 48 h gap with a 60 h interval. NOT AVAILABLE (Keystone key handling): whether, withmax_active_keys=3, a key can then be deleted before the allow-expired window of tokens it signed has elapsed. - The precheck validates the wrong variable (still present). It asserts
fernet_token_expiry <= 604800(precheck.yml:43-60, and its message talks about "rotate the fernet keys"). The generator, however, is fedfernet_key_rotation_interval(= expiry + 2 d by default,config.yml:196-199) and fails above 10080 min (fernet_rotate_cron_generator.py:59-62). The generator exits 0 with{"failed": true}(:31-36,116-117), so thecommandtask succeeds andset_fact cron_jobsthen errors on a missing key outside check mode (config.yml:207-210). Anyfernet_token_expiryabove 5 days with the default window passes precheck but breaks deploy. - Under-rotation by design. With 3 or more hosts and the 3-day default, only 2 hosts ever rotate. Rotations happen
at a fixed local
00:00on Sunday and Wednesday. - Shared SSH key. Every node gets the same
keystone_ssh_key, andssh_configsetsStrictHostKeyChecking no,UserKnownHostsFile /dev/null(templates/ssh_config.j2:1-4). NOT AVAILABLE: whether anyone holding the private key and reaching port 8023 on the API network can overwrite fernet keys asfernet-push.shdoes (templates/fernet-push.sh.j2:20), and whether that allows token forgery. - Domain config cleanup. Removing a file from
/etc/kolla/config/keystone/domainsis never propagated: no task deletes stale files in/etc/kolla/keystone/domains/. FACT: under the defaultCOPY_ALWAYSthe stale file is copied back into the container at every start, even after a recreate:domainsis a directory entry (templates/keystone.json.j2:25-31), sokolla_set_configsdeletes/etc/keystone/domainsand re-copies the whole host directory (kolladocker/base/set_configs.py:136-158,534-541; see Kolla images §2.3).keystone.confkeepsdomain_specific_drivers_enabled = trueanddomain_config_dir = /etc/keystone/domainswhile/etc/kolla/config/keystone/domainsexists (templates/keystone.conf.j2:24-28). NOT AVAILABLE (Keystone semantics): whether Keystone keeps loading the stale file as a domain config. The task also re-uses the variable nameregister: keystone_domains(config.yml:112,123), which is an oddity. - Upgrade. Without serial, all keystone APIs restart at once between
Init(expand/migrate) andFinish(contract). With serial,log_bin_trust_function_creatorsis reset to 0 at the end of every batch (upgrade.yml:36-50), whileEnableruns only in the batch whose first host isgroups['keystone'][0](:6-21). The contract step in the last batch therefore runs with it disabled. Oddity; the impact is NOT AVAILABLE. No longer holds in 2024.1: "the upgrade also removes the admin endpoint". No 2024.1 task removes it (upgrade.yml:1-50;defaults/main.yml:200-202); NOT AVAILABLE: whether a cloud still has one. - Reconfigure re-asserts identity state. It re-applies the admin credentials and endpoints, and the DB user
password, from
passwords.ymland globals. NOT AVAILABLE (os_user andkeystone-manage bootstrapsemantics): whether out-of-band changes are silently reverted. - New oddity: queue-manager
processname. Withom_enable_queue_manager,keystone.conf.j2writesprocessnameonly whenservice_name == 'keystone-api'(keystone.conf.j2:63-68), but the template is rendered withservice_name=keystoneorkeystone-fernet(config.yml:60,74), so that line is never written. NOT AVAILABLE: what oslo.messaging's queue manager does without it.
3.10 Runtime dependency of other services on Keystone (grep across ansible/roles/*/templates)¶
| Measure | Count | Pattern |
|---|---|---|
Roles with www_authenticate_uri (keystonemiddleware [keystone_authtoken]) |
23 | www_authenticate_uri = {{ keystone_internal_url }} (25 lines); 24 roles have a [keystone_authtoken] section (venus without www_authenticate_uri) |
Roles with auth_url |
30 | auth_url = {{ keystone_internal_url }} (66 lines), auth_url = {{ openstack_auth.auth_url }} (1), auth_address = {{ openstack_auth.auth_url }} (1), auth_url: {{ keystone_public_url }} (2); no /v3 and no admin-endpoint variants |
Roles with memcache_servers / memcached_servers |
26 | (29 memcached_servers = lines + 7 memcache_servers =) |
Roles with memcache_secret_key |
23 | Token cache in memcached with memcache_security_strategy = ENCRYPT (for example roles/) |
Roles that call service-ks-register |
31 (keystone included) | During their deploy/reconfigure they create services, endpoints, users, roles and grants through openstack_auth, i.e. keystone_internal_url on the internal VIP:5000 with openstack_interface: internal, run_once, 5 retries x 10 s (roles/, defaults/; group_vars/) |
Every count is lower than in 2023.1 (28, 37, 32, 29 and 37 roles), mainly because roles were removed in 2024.1 (for
example roles/sahara, roles/murano and roles/senlin no longer exist).
Implications:
- FACT: the
[keystone_authtoken]sections of 24 roles point keystonemiddleware at Keystone, and 23 roles keep its token cache in memcached (memcache_secret_key). The counts above are a recursive grep of the 2024.1 templates. NOT AVAILABLE: whether those API services fail to authenticate requests while Keystone or memcached is unavailable. - Keystone, via the internal endpoint, is a hard deploy-time dependency for the
deploy/reconfigureof 31 roles. - Every service's
memcached_serversandtransport_urlembed the full memcached and rabbitmq host lists, so changes to those groups fan out as restarts across the whole control plane.
Related¶
- verification and errata for all sheets
- service index
- findings F16 and F17
- everything marked NOT AVAILABLE