Fact sheet B¶
Source: kolla-ansible at tag 2023.1-eol (commit 4891a21), the kolla image repository at tag 2023.1-eol and
ansible-core 2.14.18. Kolla-Ansible paths are relative to ansible/ unless they start with tools/ or etc/.
Labels: FACT = read directly from source code (cited): kolla-ansible, the kolla image repository (cited as kolla
docker/...) or Ansible (cited as ansible-core 2.14.18 lib/ansible/...). NOT AVAILABLE = these sources do not
establish it, often because it depends on upstream MariaDB, RabbitMQ, Keystone, HAProxy or Docker behaviour that is not
in them.
The supported Ansible range is ansible-core 2.13 to 2.14 (roles/prechecks/vars/main.yml:4-5,
tools/kolla-ansible:10-11). Statements about handler, serial and include semantics cite the ansible-core 2.14.18
source.
What changed since Ussuri, in one paragraph (FACT): MariaDB and RabbitMQ are no longer applied by a play of
site.yml but by their own playbooks, which site.yml imports (site.yml:385,438). In both, the role's handlers no
longer restart anything: they only put the hosts into group_by groups, and separate plays restart those groups in
serial: "33%" batches (mariadb.yml:1-72, rabbitmq.yml:1-115). RabbitMQ drains each node before its restart, and
upgrade no longer stops the whole cluster. MariaDB has shards and an optional ProxySQL load balancer, the xinetd
clustercheck and wsrep-notify.sh are gone, and mysql_upgrade runs in a post-configuration play. Keystone has no
admin endpoint any more, Fernet is the only token provider, and the sanity check is gone. No role has a meta/main.yml
dependency on common (sections below).
0. Shared context that applies to all three roles¶
| Topic | Fact | Evidence |
|---|---|---|
| Action dispatch | tasks/main.yml = include_tasks: "{{ kolla_action }}.yml". mariadb first puts every mariadb host into its shard group (mariadb_shard_<id> via add_host, shard 0 by default); rabbitmq adds when: inventory_hostname in groups[role_rabbitmq_groups] |
roles/, group_vars/, roles/, roles/ |
common dependency |
None. The three roles have no meta/ directory. common is its own play, tagged common (site.yml:86-97), so it runs only in an untagged run or with -t common |
ls roles/ (no meta) |
common_run guard |
Gone: no common_run anywhere in ansible/. The mariadb_recovery and mariadb_backup commands run playbooks that contain only the mariadb role (mariadb_recovery.yml:1-9, mariadb_backup.yml:1-7), so common does not run. The CLI passes kolla_action=deploy for recovery and kolla_action=backup -e mariadb_backup_type=... for backup |
tools/ |
| Serial | kolla_serial is passed only for upgrade and reconfigure, from ANSIBLE_SERIAL (default 0 = all hosts). The comment says "Serial is not recommended and disabled by default". The MariaDB and RabbitMQ restart plays ignore it: they use a fixed serial: "33%" |
tools/; mariadb.yml:20,; rabbitmq.yml:32, |
| Config strategy | config_strategy: "COPY_ALWAYS", so recreate_or_restart_container restarts the container when its parameters are unchanged, and recreates it when they differ |
group_vars/; module_utils/ |
| Graceful stop | Effective stop timeout is 60 s (docker_graceful_timeout). kolla_docker's graceful_timeout argument has no default in 2023.1, so the value from docker_common_options is used (library/; group_vars/). A container created with a restart policy other than oneshot gets a systemd unit whose ExecStop is docker stop <name> -t <graceful_timeout>, and stops go through that unit (module_utils/; module_utils/). The Ussuri erratum (10 s) in Per-role fact sheets does not apply to 2023.1. Docker's own stop behaviour (which signal, when it kills) is NOT AVAILABLE from these sources. For MariaDB the image runs dumb-init without --single-child (kolla docker/). The image's comment says that mysqld_safe ignores SIGTERM and that, without --single-child, TERM is sent to all subprocesses, including mysqld (kolla Dockerfile.j2:64-68). NOT AVAILABLE: the dumb-init and mysqld_safe behaviour itself, and so whether mysqld shuts down cleanly within the 60 s |
group_vars/; library/ |
| Bootstrap containers | kolla_docker defaults remove_on_exit: True, so detach: False bootstrap/upgrade containers are stopped and removed after they exit. They use restart_policy: oneshot, so they get no systemd unit. Containers started with a BOOTSTRAP label are later recreated because compare_labels sees a difference |
library/; module_utils/; module_utils/ |
compare_environment |
Checks only that the desired env keys exist with equal values. Extra env vars on the running container (for example BOOTSTRAP_ARGS) are not detected; labels are what force the recreate |
module_utils/ |
| Handler order | Ansible runs handlers in the order they are defined in handlers/, not in notify order: at a flush each host walks the play's handler list and runs those it was notified for. Under the default linear strategy each handler is queued on every notified host before the results are awaited. This matters for the keystone handlers and for the order in which the mariadb group_by handlers fill their groups |
ansible-core 2.14.18 lib/, lib/ |
| Loadbalancer config | The loadbalancer play (site.yml:99-341; hosts loadbalancer + &enable_loadbalancer_True, serial: kolla_serial) runs include_role ... tasks_from: loadbalancer for each service, only when enable_haproxy and kolla_action in ['deploy','reconfigure','upgrade','config'] |
site.yml:99-104, |
Batch semantics of serial: "33%" |
The batch size is int(0.33 × N), at least 1, where N is the number of hosts the restart play targets: 1 host per batch up to 6 hosts, 2 for 7 to 9, 3 for 10 to 12. If every host of a batch fails, the playbook stops and later batches do not run; if only some fail, the next batch runs |
ansible-core 2.14.18 lib/, lib/ |
Play order in site.yml (relevant subset): loadbalancer :99 -> mariadb (import_playbook: mariadb.yml, :385)
-> memcached :387 -> rabbitmq and outward rabbitmq (import_playbook: rabbitmq.yml, :438) -> keystone
:450 -> everything else (glance :495 onward). Keystone therefore runs after its DB, cache and MQ dependencies, and
before every service that registers with Keystone.
1. MariaDB (ansible/roles/mariadb)¶
1.1 Plays, hosts, tags, serial¶
| Play | Lines | Hosts | Tags | Serial | Notes |
|---|---|---|---|---|---|
Apply role loadbalancer -> include_role mariadb tasks_from: loadbalancer |
site.yml:213-217 |
loadbalancer + &enable_loadbalancer_True (:99-104) |
mariadb |
loadbalancer play has serial: kolla_serial (:104) |
when: enable_mariadb or enable_external_mariadb_load_balancer. Also ensures the monitor user (haproxy, or monitor with ProxySQL) on the first host of each shard, run_once, failed_when: False (roles/) |
Apply role mariadb |
mariadb.yml:3-12 |
mariadb + &enable_mariadb_True |
mariadb |
none | gather_facts: false (facts come from gather-facts.yml, site.yml:2) |
Restart mariadb services |
mariadb.yml:14-26 |
mariadb_restart + &enable_mariadb_True |
mariadb |
"33%" |
runs restart_services.yml on the hosts that handler 5 grouped (1.5) |
Start mariadb services |
mariadb.yml:28-40 |
mariadb_start + &enable_mariadb_True |
mariadb |
"33%" |
same task file, hosts grouped by handler 6 |
Restart bootstrap mariadb service |
mariadb.yml:42-52 |
mariadb_bootstrap_restart + &enable_mariadb_True |
mariadb |
none | the first-bootstrap host only (handler 4) |
Apply mariadb post-configuration |
mariadb.yml:54-72 |
mariadb + &enable_mariadb_True |
mariadb |
none | post-deploy.yml (register.yml, check.yml) for deploy/reconfigure/upgrade; post-upgrade.yml for upgrade |
Recover mariadb |
mariadb_recovery.yml:1-9 |
mariadb (no enable_ intersection) |
mariadb |
none | play var mariadb_recover: true; role when: enable_mariadb. Uses implicit gather_facts (default on, ansible-core 2.14.18 lib/). CLI passes kolla_action=deploy (tools/). The playbook has no restart plays |
Backup MariaDB |
mariadb_backup.yml:1-7 |
mariadb |
mariadb |
none | role when: enable_mariabackup. CLI passes kolla_action=backup and mariadb_backup_type=full/ (tools/) |
What the absence of kolla_serial means (FACT plus reasoning):
ANSIBLE_SERIAL/kolla_serialis ignored for MariaDB onreconfigureandupgrade. The role play handles all mariadb hosts in one batch, and the restart plays use their ownserial: "33%".- This is required by the design.
lookup_cluster.ymlusesgroup_byand themariadb_shard_<id>_*groups across the whole cluster in one pass (port liveness, WSREP state, volume existence) to decide bootstrap, fail-fast and which hosts are restarted or started (roles/mariadb/tasks/lookup_cluster.yml:2-80). Under serial these groups would only describe the current batch. - Rolling behaviour comes from the restart plays (
serial: "33%", see 1.5), not from the handlers. --limitstill narrows the plays: Ansible applies the limit when it resolves a play's hosts (ansible-core 2.14.18lib/ansible/executor/playbook_executor.py:281,lib/ansible/inventory/manager.py:403-406), and agroup_bygroup only receives the hosts that ran the task (lib/ansible/plugins/action/group_by.py:49,lib/ansible/plugins/strategy/__init__.py:667-669). Group-derived decisions then only see the limited hosts (for examplemariadb_cluster_existsis computed fromgroups[mariadb_shard_group + '_had_volume_True']among play hosts). The "existing but stopped" check uses the full shard lengthgroups[mariadb_shard_group] | length > 1, filled byadd_hostfromgroups['mariadb'](tasks/main.yml:2-7,lookup_cluster.yml:35-42). FACT from these lines:--limitto a single new node on an existing cluster makes that node treat the cluster as new (mariadb_cluster_exists=false,lookup_cluster.yml:10-17), so in theStart mariadb servicesplay it waits for WSREP Synced with only a 60 s budget (restart_services.yml:31-51).
1.2 mariadb_services (ansible/roles/mariadb/defaults/main.yml:2-47)¶
In 2023.1 the dict spans roles/mariadb/defaults/main.yml:2-53 (the heading keeps its earlier range so that links stay
stable).
| Key | container_name | group | enabled | Notable |
|---|---|---|---|---|
mariadb |
mariadb |
{{ mariadb_shard_group }} (mariadb_shard_0 by default) |
true |
Volumes: /etc/, named volume mariadb:/ (mariadb_datadir_volume, group_vars/), kolla_logs (:81-86). Environment MYSQL_USERNAME (monitor user), MYSQL_PASSWORD (empty unless ProxySQL), MYSQL_HOST, AVAILABLE_WHEN_DONOR=1 (:11-15). Container healthcheck /usr/ when healthchecks are enabled (:96-107). HAProxy mariadb (internal, enabled unless external LB): mode tcp, client/server timeouts 3600 s, option httpchk when clustercheck is enabled, otherwise no check option (:16-29). custom_member_list = internal_haproxy_members, built from the hosts of the default shard, where only the first host is active and every other host is backup: {% if not loop.first %} backup{% endif %}, check port 4569 (3306 without clustercheck) inter 2000 rise 2 fall 5 (:64; group_vars/). HAProxy mariadb_external_lb uses inventory hostnames as addresses (:30-41,65). With enable_proxysql (default no, group_vars/) ProxySQL is the database load balancer instead (mariadb_loadbalancer, group_vars/); see the loadbalancer page |
mariadb-clustercheck |
mariadb_clustercheck |
{{ mariadb_shard_group }} |
enable_mariadb_clustercheck (defaults to enable_haproxy, :132) |
Same environment as above (:49-53). socat on mariadb_clustercheck_port 4569 running EXEC:clustercheck (templates/). The xinetd template of Ussuri is gone |
Other defaults: database_max_timeout: 120 (:59); mariadb_service: "{{ mariadb_services['mariadb'] }}" is used by
the recovery and WSREP checks (:112); recovery temp file /tmp/kolla_mariadb_recover_inventory_name_<shard> (:113);
mariadb_backup_host: "{{ groups[mariadb_shard_group][0] }}" (:123); backup user backup (haproxy mode), schema
PERCONA_SCHEMA (:124-125). Ports: 3306, wsrep 4567, IST 4568, SST 4444, clustercheck 4569
(group_vars/all.yml:385-389). enable_mariabackup: "no" (group_vars/all.yml:709).
1.3 Task flow per kolla_action¶
| Action | Task files (in order) | Key behaviour |
|---|---|---|
| deploy | role play: deploy.yml: config.yml -> check-containers.yml -> bootstrap.yml (= lookup_cluster.yml -> bootstrap_cluster.yml [only if no cluster and host is groups[mariadb_shard_group][0]] -> recover_cluster.yml [only if mariadb_recover]); handlers at the end of the play. Then the restart, start and bootstrap-restart plays, then post-configuration: register.yml -> check.yml |
roles/, bootstrap.yml:1-10, post-deploy.yml:1-4; mariadb.yml:3-66 |
| reconfigure | import_tasks: deploy.yml (identical to deploy) |
tasks/ |
| upgrade | role play: upgrade.yml = deploy.yml; restart plays; post-configuration: post-deploy.yml, then "Run upgrade in MariaDB container" (upgrade_mariadb, KOLLA_UPGRADE, DB_HOST=api_interface_address, detach: False) on every mariadb host, in a play without serial, with no run_once and no image-changed condition |
tasks/, tasks/; mariadb.yml:54-72 |
| deploy-containers | check-containers.yml -> lookup_cluster.yml (prerequisite for the handlers; comment # handlers prerequisite). Handlers at the end of the role play, then the restart plays. No post-configuration (when lists deploy/reconfigure/upgrade only) |
tasks/; mariadb.yml:66 |
| pull | service-images-pull (pulls mariadb and mariadb-clustercheck if enabled) |
tasks/pull.yml, roles/ |
| stop | service-stop: stop_container for each enabled, mapped service on all hosts at once, unless the name is in skip_stop_containers |
tasks/stop.yml, roles/ |
| precheck | service-precheck (inventory group exists) plus free-port checks for 3306/4567/4568/4444, only if no mariadb container is running |
roles/ |
| check | There is no check command in tools/ 2023.1. check.yml runs in post-configuration and at the end of recovery: from every host of the default shard, docker exec mariadb mysql -h <VIP> -P 3306 -u <shard root user> -p<database_password> -e 'show databases;', 6 retries x 10 s |
roles/; recover_cluster.yml:233 |
| config (genconfig) | config.yml (handlers guarded by kolla_action != "config") |
handlers/ |
| backup | backup.yml: one-shot mariabackup container (detach: False, remove_on_exit: True) on mariadb_backup_host only (and only if mariadb_backup_possible), using the image of the running mariadb container when there is one, mounting mariadb:/ and mariadb_backup:/ and running kolla_set_configs && /usr/ with BACKUP_TYPE |
roles/; defaults/ |
recovery (mariadb_recovery) |
kolla_action=deploy with mariadb_recover=true, so deploy.yml runs and bootstrap.yml includes recover_cluster.yml. The port and WSREP block of lookup_cluster.yml is skipped (when: not mariadb_recover). mariadb_recovery.yml has no restart and no post-configuration plays, so register.yml does not run; recover_cluster.yml ends with check.yml |
lookup_cluster.yml:80, bootstrap.yml:9-10, recover_cluster.yml:233; mariadb_recovery.yml:1-9 |
lookup_cluster.yml (runs on deploy, reconfigure, upgrade and deploy-containers)¶
create_volume mariadb. If the volume already existed (not changed), the host joins groupmariadb_shard_<id>_had_volume_True(:2-13).mariadb_cluster_exists = groups[mariadb_shard_group + '_had_volume_True'] is defined(:15-17).- Unless recovering:
wait_forport 3306 withsearch_regex: MariaDB,timeout: 10,ignore_errors, thengroup_by mariadb_shard_<id>_port_alive_{True,False}(:19-33). - Fail fast with "MariaDB cluster exists but is stopped. Please start it using kolla-ansible mariadb_recovery" if
the shard has more than one node, the cluster exists and no node is alive (
:35-42). - On alive nodes,
SHOW STATUS LIKE "wsrep_local_state_comment", thengroup_by mariadb_shard_<id>_sync_status_<state>(:44-70). - Fail fast with "MariaDB cluster is not synced" unless every alive node is
Synced(:72-78). So a reconfigure refuses to run while any node is Donor or Joining.
First bootstrap (bootstrap_cluster.yml)¶
- On
groups[mariadb_shard_group][0]only, when no host had a volume: runbootstrap_mariadb(KOLLA_BOOTSTRAP,DB_ROOT_PASSWORD,DB_MAX_TIMEOUT),detach: False, notifyBootstrap MariaDB cluster, andset_fact bootstrap_host(roles/mariadb/tasks/bootstrap_cluster.yml:1-27).
Recovery (recover_cluster.yml), in order¶
- Fail if no cluster volume exists ("MariaDB cluster was not found. Is your inventory correct?") (
:2-5). - Delete every
/tmp/kolla_mariadb_recover_inventory_name_*file on the controller (delegate_to: localhost,run_once,with_fileglob) (:7-16). - The following block is skipped if
mariadb_recover_inventory_nameis supplied with-e, which lets the operator force the bootstrap host (:94-95):- Stop the
mariadbcontainer on every host (:19-24). This happens even if some nodes are healthy, so running recovery on a partially-up cluster causes a full DB outage. - Run
mariadb_wsrep_recoverywithBOOTSTRAP_ARGS=--wsrep-recover,detach: false(:28-42). - Copy
<docker_runtime_directory or /var/lib/docker>/volumes/kolla_logs/_data/mariadb/mariadb.logto/tmp/mariadb_tmp.logon the host, then take the seqno of the lastRecovered positionline in the whole file withawk(:44-53), and delete the copy (:55-61).seqno = stdout_lines[0](:63-66). NOT AVAILABLE: whether an empty parse makes that host fail and so drops it from the play. - Each host compares its seqno against every other host of its shard, using a bash regex guard. A host with no
larger peer writes its own name to the controller temp file of its shard (
:68-88). On a tie, every tied host writes and the last writer wins (harmless when the seqnos are equal). - Every host reads the chosen name with
lookup('file', ...)(:90-93).
- Stop the
bootstrap_host = mariadb_recover_inventory_name(:97-99). On that host,lineinfile grastate.dat safe_to_bootstrap: 1, editing the Docker volume path on the host directly (:101-111).- On the bootstrap host: start a
mariadbcontainer withBOOTSTRAP_ARGS: --wsrep-new-cluster, labelBOOTSTRAPandrestart_policy: oneshot-> wait for the port ->SET GLOBAL wsrep_provider_options='pc.bootstrap=yes'-> wait forwsrep_evs_stateto reportOPERATIONAL(10 x 6 s) (:113-169). - "Restart slave MariaDB container(s)":
start_containerloops overwith_dict: mariadb_serviceson all other hosts at once, then waits only for the port, not for WSREP Synced (:171-200). - "Restart master MariaDB container(s)":
recreate_or_restart_containeron the bootstrap host (drops the bootstrap args and label), then waits for the port (:202-231). Finallycheck.yml(:233). - Handlers notified earlier by config/check-containers do nothing for
restart mariadb: they need the_port_alive_*groups, which recovery never creates (handlers/main.yml:59-61,68-71), andmariadb_recovery.ymlhas no restart plays anyway. Handler 7 (restart mariadb-clustercheck) still runs on the hosts it was notified on (handlers/main.yml:75-91).
1.4 Config files generated (ansible/roles/mariadb/tasks/config.yml)¶
Destination (/etc/kolla/...) |
Source / merge order | Notifies | Lines |
|---|---|---|---|
mariadb/, mariadb-clustercheck/ dirs |
(directory), only for enabled services | none | :2-13 |
mariabackup/ dir |
only if enable_mariabackup and the host is mariadb_backup_host |
none | :15-25 |
mariabackup/ |
merge_configs: templates/ + /etc/ + /etc/. The template contains backup user/password and host={{ database_address }} (VIP) |
none | :27-40; templates/ |
<svc>/ (both services) |
templates/. mariadb.json.j2 recursively chowns /var/ and /var/lib/mysql on every COPY_ALWAYS start, the bootstrap, upgrade and --wsrep-recover one-shots included (templates/; Kolla images §5.4) |
restart <svc> (restart mariadb / restart mariadb-clustercheck) |
:42-53 |
mariabackup/ |
templates/ |
none | :55-65 |
mariadb/ |
merge_configs: templates/ + /etc/ + /etc/ |
restart mariadb |
:67-83 |
wsrep-notify.sh and the clustercheck xinetd file are no longer generated: the role has no such templates in 2023.1
(roles/mariadb/templates/).
Notable galera.cnf.j2 contents:
wsrep_cluster_address=gcomm://lists all hosts of the shard group when there are two or more, and is empty on a single node (:29). Adding or removing a DB node therefore changes galera.cnf on every node of the shard, which triggers a batched restart of the whole shard.innodb_buffer_pool_sizeis 40 % ofansible_facts.memtotal_mb, capped at 8192M (:50-55). The file depends on facts, so a RAM change causes a restart.wsrep_sst_method=mariabackupandwsrep_sst_auth={{ database_user }}:{{ database_password }}(root credentials) (:2,40-41);max_connections=10000(:45). There is nowsrep_notify_cmdany more: the Ussuri mechanism that renamed thehaproxyuser while a node was not Synced is gone, and HAProxy's view of a node comes only from the clustercheck HTTP check (or a plain TCP check without clustercheck,defaults/main.yml:28,64).
1.5 Handlers (ansible/roles/mariadb/handlers/main.yml), in definition order¶
| # | Name | listen | Action / logic | Lines |
|---|---|---|---|---|
| 1 | Starting first MariaDB container | Bootstrap MariaDB cluster |
start_container mariadb with BOOTSTRAP_ARGS: --wsrep-new-cluster, label BOOTSTRAP, restart_policy: oneshot |
:2-20 |
| 2 | Wait for first MariaDB service port liveness | Bootstrap MariaDB cluster |
wait_for 3306 search_regex MariaDB, retries 10, delay 6 |
:23-34 |
| 3 | Wait for first MariaDB service to sync WSREP | Bootstrap MariaDB cluster |
wait until wsrep_local_state_comment == Synced, 10 x 6 s |
:36-49 |
| 4 | Ensure MariaDB is running normally on bootstrap host | Bootstrap MariaDB cluster |
group_by mariadb_bootstrap_restart |
:51-54 |
| 5 | Restart MariaDB on existing cluster members | restart mariadb |
group_by mariadb_restart for hosts in <shard>_port_alive_True, unless kolla_action == "config" |
:56-63 |
| 6 | Start MariaDB on new nodes | restart mariadb |
group_by mariadb_start for hosts in <shard>_port_alive_False that are not bootstrap_host, unless config |
:65-73 |
| 7 | Restart mariadb-clustercheck container | restart mariadb-clustercheck |
recreate_or_restart_container on all notified hosts at once, unless config |
:75-91 |
The Ussuri handler "Creating haproxy mysql user" is gone: the monitor user is created by register.yml in
post-configuration (tasks/register.yml:19-34) and by the loadbalancer play (tasks/loadbalancer.yml:18-40).
restart_services.yml (run by the three restart plays of mariadb.yml): recreate_or_restart_container mariadb ->
wait for port 3306 (timeout 60, retries 10, delay 6) -> wait for WSREP Synced (10 x 6 s = about 60 s
budget). The WSREP wait is skipped for new nodes joining an existing cluster ("we don't want to wait for new nodes to
fully sync") (roles/mariadb/tasks/restart_services.yml:1-51).
Rolling semantics (FACT, from the plays and the Ansible source): the role play only fills the groups.
Restart mariadb services then runs restart_services.yml on the hosts of mariadb_restart in batches of
serial: "33%", and Start mariadb services does the same for mariadb_start (mariadb.yml:14-40). The batch size is
int(0.33 × N), at least 1 (section 0): with up to 6 hosts to restart, one node at a time, each waiting for port +
Synced before the next batch starts; with 7 to 9 hosts, 2 at a time; with 10 to 12, 3 at a time. That stays below a
third of the nodes, so a majority stays up while one batch restarts. NOT AVAILABLE: the order of the batches, which
follows the order in which group_by added the hosts to mariadb_restart
(lib/ansible/plugins/strategy/__init__.py:667-669; lib/ansible/executor/playbook_executor.py:281), and so in which
batch node 0, the active HAProxy backend, restarts.
- Failure mode: if one node fails its port or 60 s WSREP-sync wait (for example a long IST/SST), only that host is
marked failed. When that is every host of the batch (always so with up to 6 nodes, one host per batch), ansible-core
stops the playbook and the later batches are not restarted; when other hosts of the batch succeed, the next batch
still restarts its nodes (ansible-core 2.14.18
lib/ansible/executor/playbook_executor.py:204-208,218-219). The restart plays set neitherany_errors_fatalnormax_fail_percentage(mariadb.yml:14-40). NOT AVAILABLE: whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is.
Handler and play order consequences (FACT):
- First deploy: config/check-containers notify
restart mariadbon every host, and the bootstrap host additionally getsBootstrap MariaDB cluster. Order: handlers 1-3 (bootstrap node up with--wsrep-new-clusterand Synced) -> 4 groups the bootstrap node -> 5 groups nothing (no alive nodes) -> 6 groups all other nodes -> 7. Then theStart mariadb servicesplay starts the other nodes in 33 % batches, each waiting for Synced (the cluster did not exist), thenRestart bootstrap mariadb servicerecreates the bootstrap node as a normal one (theBOOTSTRAPlabel differs), then post-configuration creates the users and runscheck.yml. - Scale-out:
Restart mariadb servicesrestarts the existing members first, batch by batch (theirwsrep_cluster_addresschanged), thenStart mariadb servicesstarts the new nodes in batches without waiting for SST to finish.
1.6 Bootstrap, DB and registration; what re-runs on reconfigure¶
bootstrap_mariadbruns only when no host had amariadbvolume and only ongroups[mariadb_shard_group][0](bootstrap.yml:4-7). It does not re-run on reconfigure.register.ymlruns on every deploy/reconfigure/upgrade (post-configuration play), on the first host of the shard, logging in at that host's ownapi_interface_address(not the VIP): it creates the shard root user (rootin HAProxy mode, password re-applied,*.*:ALL,GRANT) and the monitor user. Whenenable_mariabackupis set it also, onmariadb_backup_host, createsPERCONA_SCHEMA, creates the backup user (append_privs, password re-applied) and grants on the schema (roles/mariadb/tasks/register.yml:1-87;defaults/main.yml:137). The Ussuriwait_for_loadbalancerstep is gone.check.ymlruns on every deploy/reconfigure/upgrade, from every default-shard host through the VIP, as the shard root user withdatabase_passwordon the command line (roles/mariadb/tasks/check.yml:1-16).lookup_cluster.ymlchecks (port, WSREP, fail-fast) run on every deploy/reconfigure/upgrade/deploy-containers.- Backup prerequisite:
my.cnf, the backup user and themariabackupconfig are created only by deploy/reconfigure withenable_mariabackup: yes.kolla-ansible mariadb_backupitself runs onlybackup.yml. A reconfigure is therefore required after enabling backups (config.yml:15-65,register.yml:36-87,backup.yml; the docs atdoc/source/admin/mariadb-backup-and-restore.rstdescribe this step).
1.7 Dependencies¶
- Explicit: none in role metadata (no
meta/). The loadbalancer play must have configured themariadbfrontend, which usesloadbalancer-config(tasks/loadbalancer.yml:42-47). - Implicit:
- The VIP (haproxy + keepalived, or ProxySQL) must be up for
check.yml, which connects todatabase_address=kolla_internal_fqdn(group_vars/all.yml:83), so the post-configuration play fails without the load balancer.register.ymlconnects to the node directly. kolla_toolbox(fromcommon) for themysql_user/mysql_dbmodules.- Facts:
hostname(HAProxy server names,wsrep_node_name) andmemtotal_mb.
- The VIP (haproxy + keepalived, or ProxySQL) must be up for
- Consumers: the tasks of 36 roles use
kolla_toolbox mysql_db, the mariadb role's ownPERCONA_SCHEMAtask included (grep -l "module_name: mysql_db" roles/*/tasks/*.yml); the services log in atdatabase_address, the VIP (for exampleroles/keystone/tasks/bootstrap.yml:8). Writes go to a single active backend, the first host of the default shard, because every other server isbackupin HAProxy (defaults/main.yml:64).
1.8 Operationally notable behaviour, risks and oddities¶
- Reconfigure with a galera.cnf change. Every node is notified, and the restart is batched by
Restart mariadb services(serial: "33%": 1 node at a time up to 6 nodes,mariadb.yml:14-26). Each batch waits for the port andSyncedbefore the next. The first host of the default shard is the only active HAProxy backend and is restarted in one of the batches (which one is NOT AVAILABLE, see 1.5). HAProxy checks each server withinter 2000 rise 2 fall 5(defaults/main.yml:64). NOT AVAILABLE: how long HAProxy takes to fail over to a backup node and back, and whether in-flight client connections are dropped at each step. - Clustercheck restart is not rolling. Handler 7 restarts
mariadb_clustercheckon all notified nodes at once, in the role play (handlers/main.yml:75-91). HAProxy health checks for every backend go through port 4569 when clustercheck is enabled (defaults/main.yml:28,64). NOT AVAILABLE: whether HAProxy marks every backend down when that restart outlasts thefall 5xinter 2000check settings, which would mean a short total DB outage through the VIP. - WSREP-sync wait is only about 60 s (
restart_services.yml:31-51, 10 x 6 s at:42-43; handlers:36-49). A restart that needs IST or SST longer than that fails the host, and see the failure-mode note in 1.5. - Fail-fast guards. Reconfigure aborts if the cluster is fully stopped (multi-node) or any alive node is not Synced
(
lookup_cluster.yml:35-42,72-78).kolla-ansible stopstops every node at once (service-stop), so the next deploy/reconfigure fails untilmariadb_recoveryis run. - Recovery takes the whole cluster down. It stops all
mariadbcontainers unconditionally (recover_cluster.yml:19-24), starts all non-bootstrap nodes at once (concurrent SST/IST load on the single bootstrap node,:171-200), and restarts the bootstrap node after the others have only opened their port (:202-231). NOT AVAILABLE: whether restarting the bootstrap/donor node aborts a joiner's SST from it that is still running. - Recovery bug: "Restart slave/master MariaDB container(s)" loop over
with_dict: mariadb_serviceswith noitem.value.enabledor group check (recover_cluster.yml:171-185,202-216). Withenable_mariadb_clustercheck: no, recovery still tries to start amariadb_clustercheckcontainer:kolla_dockerpulls the image if it is not present, then creates and starts the container (module_utils/kolla_docker_worker.py:386-407). If the pull returns an error,kolla_dockerfails the task on that host (:217-230). FACT. In deploy and reconfigure the clustercheck container is handled only when enabled (config.yml:42-53,check-containers.yml:13-18). - Recovery assumptions. It relies on
/var/lib/dockervolume paths on the host (recover_cluster.yml:46,105), a fixed/tmp/mariadb_tmp.logon each host (:44-61) and a fixed temp file per shard on the controller (defaults/main.yml:113), and it deletes every shard's temp file first (recover_cluster.yml:7-16), so concurrent recoveries from one controller would collide. - Upgrade.
mysql_upgrade(upgrade_mariadb) runs on all nodes concurrently and on every upgrade, even when the image is unchanged, in the post-configuration play after the batched restarts (mariadb.yml:54-72,tasks/post-upgrade.yml:2-24). NOT AVAILABLE: whether concurrentmysql_upgradeon Galera nodes conflicts. - Backup. It runs only on the first host of the shard (reading that node's datadir,
defaults/main.yml:123;backup.yml:33-34), butmy.cnfpoints the client atdatabase_address, the VIP (backup.my.cnf.j2:5). NOT AVAILABLE: whether, if HAProxy has failed over away from node 0, the backup tool connects to a different server than the one whose files it copies. Kolla does not schedule or rotate backups (docs). - Credentials on the command line. Root credentials appear in
docker exec ... mysql -uroot -p{{ database_password }}command lines (lookup_cluster.yml:47-51,restart_services.yml:33-38,recover_cluster.yml:148, handlers:38-42) and incheck.yml's-p{{ database_password }}(check.yml:7-10).no_loghides the first ones from Ansible output but not from the host process list;check.ymlhas nono_log(check.yml:5-16). - Scale-out restarts the whole cluster, batch by batch (
wsrep_cluster_address,galera.cnf.j2:29). Every time the loadbalancer play runs,internal_haproxy_membersis also re-rendered in haproxy.
2. RabbitMQ (ansible/roles/rabbitmq), used twice¶
2.1 Plays, hosts, tags, serial¶
| Play | Lines | Hosts | Tags | Serial | Role vars |
|---|---|---|---|---|---|
loadbalancer -> include_role rabbitmq tasks_from: loadbalancer |
site.yml:276-283 |
loadbalancer | rabbitmq |
loadbalancer play serial | vars: role_rabbitmq_cluster_cookie: (null), role_rabbitmq_groups: (null). These are blanked so the lazily templated rabbitmq_services does not fail. when: enable_rabbitmq or enable_outward_rabbitmq |
Apply role rabbitmq |
rabbitmq.yml:3-24 |
rabbitmq + &enable_rabbitmq_True |
rabbitmq |
none | import_role with vars: role_rabbitmq_cluster_cookie: rabbitmq_cluster_cookie, _cluster_port: 25672, _epmd_port: 4369, _groups: rabbitmq, _management_port: 15672, _monitoring_user/, _password, _port: 5672, _prometheus_port: 15692, _user (group_vars/) |
Restart rabbitmq services |
rabbitmq.yml:26-41 |
rabbitmq_restart + &enable_rabbitmq_True |
rabbitmq |
"33%" |
runs restart_services.yml |
Apply rabbitmq post-configuration |
rabbitmq.yml:43-58 |
rabbitmq |
rabbitmq |
none | post-deploy.yml (rabbitmqctl enable_feature_flag all) for deploy/reconfigure/upgrade |
Apply role rabbitmq (outward) |
rabbitmq.yml:60-80 |
outward-rabbitmq + &enable_outward_rabbitmq_True |
rabbitmq (same tag) |
none | project_name: outward_rabbitmq, outward_rabbitmq_cluster_cookie, ports 5674/15674/25674/4371/15694 (group_vars/), _groups: outward-rabbitmq, no monitoring user |
Restart rabbitmq (outward) services |
rabbitmq.yml:82-98 |
outward_rabbitmq_restart + &enable_outward_rabbitmq_True |
rabbitmq |
"33%" |
project_name: outward_rabbitmq |
Apply rabbitmq (outward) post-configuration |
rabbitmq.yml:100-115 |
outward-rabbitmq |
rabbitmq |
none | includes the whole role (no tasks_from), for deploy/reconfigure (RabbitMQ) |
enable_rabbitmqdefaults to yes when eitherom_rpc_transportorom_notify_transportisrabbit(group_vars/all.yml:631).enable_outward_rabbitmq=enable_murano(:632).- Play-level vars override role vars. The
vars:of animport_rolereach the role's tasks as task vars, which Ansible merges after the role'svars/main.yml(project_name: "rabbitmq") (ansible-core 2.14.18lib/ansible/vars/manager.py:397-399). For the outward cluster,project_name=outward_rabbitmqtherefore drives the container name, the volume name, the config dir/etc/kolla/outward_rabbitmq, the log dir, the restart groupoutward_rabbitmq_restartand the murano-specific definitions (roles/rabbitmq/defaults/main.yml:4,12,16,71,74;handlers/main.yml:4;templates/definitions.json.j2:3,9,15). - What the absence of
kolla_serialmeans:kolla_serialis ignored. The role play handles every node in one batch; restarts come from the restart plays with a fixedserial: "33%". --tags rabbitmq(or--skip-tags rabbitmq) affects both clusters and the loadbalancer fragment together.- New commands in 2023.1:
kolla-ansible rabbitmq-upgrade <version>importsrabbitmq.ymlwithkolla_action: upgradeand the imagerabbitmq-<version>(rabbitmq-upgrade.yml:18-21;tools/kolla-ansible:568-572);kolla-ansible rabbitmq-reset-staterunsrabbitmqctl stop_app,force_reset,start_appandawait_online_nodeson every rabbitmq host, with noserial(rabbitmq-reset-state.yml:1-9;roles/rabbitmq/tasks/reset-state.yml:1-16).
2.2 rabbitmq_services (ansible/roles/rabbitmq/defaults/main.yml:2-39)¶
In 2023.1 the dict spans roles/rabbitmq/defaults/main.yml:2-40 (the heading keeps its earlier range so that links stay
stable).
| Key | container_name | group | enabled | Notable |
|---|---|---|---|---|
rabbitmq |
{{ project_name }} (rabbitmq or outward_rabbitmq) |
{{ role_rabbitmq_groups }} |
true |
environment: KOLLA_CONFIG_STRATEGY, RABBITMQ_CLUSTER_COOKIE: role_rabbitmq_cluster_cookie, RABBITMQ_LOG_DIR. bootstrap_environment adds KOLLA_BOOTSTRAP. Volume {{ project_name }}:/. Healthcheck healthcheck_rabbitmq when healthchecks are enabled (:57-68). HAProxy: rabbitmq_management (http, 15672, host_group: rabbitmq), rabbitmq_outward_management (15674), and rabbitmq_outward_external (tcp, external frontend on 5674 for murano agents, 1 h timeouts). AMQP 5672 is not load-balanced: clients get every broker in transport_url |
Other defaults: rabbitmq_pid_file: /var/lib/rabbitmq/mnesia/rabbitmq.pid (:81),
rabbitmq_server_additional_erl_args: "+S 2:2 +sbwt none ..." (:82),
rabbitmq_cluster_partition_handling: "pause_minority" (:86), HA policy settings (:91-107). rabbitmq_hostname,
unused in Ussuri, no longer exists (no match in ansible/).
Erlang cookie: comes from passwords.yml keys rabbitmq_cluster_cookie / outward_rabbitmq_cluster_cookie
(etc/kolla/passwords.yml:220,222). It is passed to the broker container as the env RABBITMQ_CLUSTER_COOKIE
(defaults/main.yml:11,15). The common role also writes rabbitmq_cluster_cookie (main cluster only) to
{{ node_config_directory }}/kolla-toolbox/rabbitmq-erlang.cookie, which kolla_toolbox copies to
/var/lib/rabbitmq/.erlang.cookie at every COPY_ALWAYS start; a change notifies Restart kolla-toolbox container
(roles/common/tasks/config.yml:211-221, roles/common/templates/kolla-toolbox.json.j2:4-9). That happens only when
the common play runs (an untagged run or -t common, site.yml:86-97). The kolla image writes
/var/lib/rabbitmq/.erlang.cookie only when KOLLA_BOOTSTRAP is set, then exits; a normal start only handles the
log dir (kolla docker/rabbitmq/rabbitmq/extend_start.sh:7-26; see
Kolla images §4). That
happens only in the {{ project_name }}_bootstrap container, which runs only when the volume was just created
(tasks/bootstrap.yml:26). A normal start does not touch the cookie, and the file persists in the {{ project_name }}
volume (defaults/main.yml:74). Because the cookie is in the env, compare_container (env compare) still picks up a
cookie change and causes a batched recreate, but the recreated brokers keep their old .erlang.cookie file. NOT
AVAILABLE (it depends on whether rabbitmq-server itself reads RABBITMQ_CLUSTER_COOKIE, which is not in these
sources): whether a partition occurs during the roll, or whether the change is instead silently not applied to existing
brokers while kolla_toolbox (used by the drain step and by service-rabbitmq) and any broker bootstrapped later on a
fresh volume get the new cookie and cannot talk to the existing nodes.
2.3 Task flow per kolla_action¶
| Action | Flow | Evidence |
|---|---|---|
| deploy | version-check.yml (fails on a RabbitMQ downgrade, run_once, tag rabbitmq-version-check) -> remove-ha-all-policy.yml (only when om_enable_rabbitmq_high_availability is false, the default) -> config.yml -> check-containers.yml -> bootstrap.yml; handlers at the end of the role play; then the restart play and post-configuration (enable_feature_flag all) |
roles/, version-check.yml:1-53, remove-ha-all-policy.yml:1-29; group_vars/; rabbitmq.yml:26-58 |
| reconfigure | = deploy | tasks/ |
| bootstrap.yml | create_volume {{ project_name }}. Only when the volume was just created on that host: run {{ project_name }}_bootstrap with bootstrap_environment, detach: False |
roles/ |
| upgrade | remove-ha-all-policy.yml (as above) -> config.yml -> feature-flags.yml (on the running brokers, before their restart) -> check-containers.yml; then the restart play (drained, batched) and post-configuration. There is no bootstrap step and no cluster-wide stop |
roles/ |
| deploy-containers | check-containers.yml only. Handlers at the end of the role play, then the restart play |
tasks/ |
| pull | service-images-pull |
tasks/pull.yml |
| stop | service-stop: all nodes at once |
tasks/stop.yml |
| precheck | group check, free ports for both clusters, and getent ahostsv4/ for every rabbit host, failing unless every line resolves to that host's api IP (nodename rabbit@<hostname>); TLS files when TLS is on; with om_enable_rabbitmq_high_availability it asserts that an ha-all policy exists, with om_enable_rabbitmq_quorum_queues that every queue is a quorum queue |
roles/ (:62-78,) |
| check | empty file; tools/ 2023.1 has no check command |
tasks/ |
| config | config.yml. The handler is guarded by kolla_action != "config" |
handlers/ |
2.4 Config files (ansible/roles/rabbitmq/tasks/config.yml)¶
All files go into /etc/kolla/{{ project_name }}/. Except config.json, each file uses with_first_found, which
means a full replacement of the template, not a merge:
/etc/kolla/config/rabbitmq/<inventory_hostname>/<file> -> /etc/kolla/config/rabbitmq/<file> -> the role template.
All of them notify Restart rabbitmq container.
| File | Template highlights | Lines |
|---|---|---|
config.json |
rabbitmq.json.j2: /usr/. Copies the config files and, on every COPY_ALWAYS start, recursively chowns /var/ (the {{ project_name }} volume, mnesia included) and /var/, one chown per directory and file (templates/; kolla docker/) |
config.yml:15-26 |
rabbitmq-env.conf |
RABBITMQ_NODENAME=rabbit@<hostname>, RABBITMQ_DIST_PORT=role_rabbitmq_cluster_port, RABBITMQ_PID_FILE, ERL_EPMD_ADDRESS=api ip, ERL_EPMD_PORT=role_rabbitmq_epmd_port |
config.yml:28-44; templates/ |
rabbitmq.conf |
listeners.tcp.1 = api_ip:role_rabbitmq_port (listeners.ssl.1 with TLS); cluster_partition_handling = pause_minority; management listener; management.load_definitions = /etc/; cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config with every host in groups[role_rabbitmq_groups] |
config.yml:46-62; templates/ |
erl_inetrc |
{inet6,true} for IPv6 |
config.yml:64-80 |
advanced.config |
inet_dist_use_interface and inet_dist_listen_min/ |
config.yml:82-98; templates/ |
definitions.json |
vhost / (plus the murano agent vhost for outward); user role_rabbitmq_user (administrator) plus an optional monitoring user (plus the murano agent user for outward). Policies: none by default. Only with om_enable_rabbitmq_high_availability (default false) is there an ha-all policy, on queues other than amq.*, *_fanout_* and reply_*, with ha-mode: exactly and ha-params = N // 2 + 1, ha-promote-on-shutdown: always, a 10 min message TTL and a 1 h queue expiry |
config.yml:100-116; templates/ (policies :19-27); defaults/; group_vars/ |
enabled_plugins |
rabbitmq_management, plus rabbitmq_prometheus when Prometheus is enabled |
config.yml:118-134; defaults/ |
Oddity: the override paths are hard-coded to /etc/kolla/config/rabbitmq/... and do not include
{{ project_name }}. A custom rabbitmq.conf placed there replaces the template for both rabbitmq and
outward_rabbitmq, including the listener and management ports. NOT AVAILABLE: whether the two brokers on one host
would then try to bind the same ports (config.yml:37-38,55-56,...).
2.5 Handlers (ansible/roles/rabbitmq/handlers/main.yml), in definition order¶
| # | Name | listen | Logic | Lines |
|---|---|---|---|---|
| 1 | Restart rabbitmq container | (name) | group_by {{ project_name }}_restart, unless kolla_action == "config" |
:2-6 |
The Ussuri pair "Restart first rabbitmq container" / "Restart remaining rabbitmq containers" is gone: there is no first-node-first rule any more.
restart_services.yml (run by the restart plays): kolla_container_facts rabbitmq -> if that container is running,
put the node into maintenance mode (community.rabbitmq.rabbitmq_upgrade action: drain, through kolla_toolbox)
-> recreate_or_restart_container (with env) ->
docker exec <c> rabbitmqctl wait --timeout 60 /var/lib/rabbitmq/mnesia/rabbitmq.pid. There is no cluster-membership,
partition or mirror-sync check (roles/rabbitmq/tasks/restart_services.yml:1-39;
library/kolla_container_facts.py:77 lists running containers only).
Rolling semantics (FACT, from the plays and the Ansible source): Restart rabbitmq services runs
restart_services.yml on the hosts of rabbitmq_restart in serial: "33%" batches (rabbitmq.yml:26-41): one node
at a time with up to 6 notified nodes, 2 with 7 to 9 (section 0). Restart time grows linearly with N. Only notified
hosts are in the group. If every host of a batch fails (for example the --timeout 60 wait), the playbook stops before
the next batch (ansible-core 2.14.18 lib/ansible/executor/playbook_executor.py:204-208). NOT AVAILABLE: the order of
the batches, which follows the order in which group_by added the hosts
(lib/ansible/plugins/strategy/__init__.py:667-669).
Oddity: restart_services.yml checks for a container named rabbitmq (:6), and feature-flags.yml runs
docker exec rabbitmq (feature-flags.yml:3), whatever project_name is; remove-ha-all-policy.yml calls
docker exec literally instead of kolla_container_engine (:14,20). NOT AVAILABLE: which broker the drain step of
the outward restart play talks to, since the rabbitmq_upgrade module runs in kolla_toolbox, whose cookie is the main
cluster's.
2.6 Bootstrap and registration¶
- The cluster/user bootstrap is declarative. Users, vhosts and policies come only from
definitions.json, which is loaded at node boot (rabbitmq.conf.j2:13). There is norabbitmq_usertask in the role, so a changedrabbitmq_passwordtakes effect only through a restart (batched, via the restart play). The role'srabbitmqctlcalls are limited to the version check, theha-allremoval, feature flags, the wait andreset-state(version-check.yml:14,remove-ha-all-policy.yml:14,20,feature-flags.yml:3,restart_services.yml:39,reset-state.yml). - The
{{ project_name }}_bootstrapcontainer runs only on hosts whose volume is newly created (bootstrap.yml:26). It does not re-run on reconfigure. - Per-service users and vhosts are created only by the
service-rabbitmqrole (roles/service-rabbitmq/tasks/main.yml:1-47,update_password: always), which in 2023.1 is used only bynova-cell(roles/nova-cell/tasks/rabbitmq.yml:5,15). Every other service shares theopenstackuser on vhost/(group_vars/all.yml:205-222).
2.7 Dependencies¶
- Explicit: none in role metadata (no
meta/). - Implicit:
- Hostnames must resolve to the api IP on every node (precheck), because nodenames are
rabbit@<hostname>. - Facts (
hostname). kolla_toolbox(fromcommon) for the drain step andservice-rabbitmq.- HAProxy, only for the management UI and the outward external port.
- Hostnames must resolve to the api IP on every node (precheck), because nodenames are
- Consumers: the templates of 30 roles render
rpc_transport_url(31 roles reference sometransport_url), which lists every host ingroups['rabbitmq'](group_vars/all.yml:212,222;grep -l rpc_transport_url roles/*/templates). Keystone is one of them (keystone.conf.j2:4,54).
2.8 Operationally notable behaviour, risks and oddities¶
- Reconfigure with a config change drains and restarts the brokers in
serial: "33%"batches, one node at a time with up to 6 nodes. At most one node is down at a time in that case, so a 3-node cluster keeps 2/3 and stays above thepause_minoritythreshold. The restart only runsrabbitmqctl wait --timeout 60and has no mirror-sync check (restart_services.yml:9-39). Mirroring is off by default (om_enable_rabbitmq_high_availability: false,group_vars/all.yml:229; emptypolicies,definitions.json.j2:25-27), and deploy removes an existingha-allpolicy (remove-ha-all-policy.yml:18-22). NOT AVAILABLE: what happens to the queues of a drained node without mirroring, and, with mirroring enabled, whether the next node may be stopped while mirrors on the freshly restarted node are still unsynchronised. - Upgrade = full outage of the message bus no longer holds in 2023.1.
upgrade.ymlhas no stop step (roles/rabbitmq/tasks/upgrade.yml:1-10); an image change goes through the same drained, batched restart play as a reconfigure (rabbitmq.yml:26-41). A downgrade is refused byversion-check.ymlon deploy/reconfigure (version-check.yml:37-44;deploy.yml:2), not onupgrade(upgrade.ymldoes not import it). NOT AVAILABLE: whether clients lose RPC while each node is drained and restarted. - Uncontrolled full stop (
kolla-ansible stop, power loss) followed by deploy: the brokers come back through the batched restart play, one batch after another, each waitingrabbitmqctl wait --timeout 60. The drain step is skipped for a stopped container (restart_services.yml:17). Which node starts first is NOT AVAILABLE (group_byorder). NOT AVAILABLE (RabbitMQ semantics): whether a node that was not the last to stop waits for its peers' Mnesia tables longer than 60 s. FACT: if it does, the wait fails, and with one node per batch (up to 6 nodes) the playbook stops before the other nodes are started (section 0). - Scaling the rabbitmq group changes
transport_urlin about 30 services, so it restarts almost every OpenStack service cloud-wide when those plays run. It also changesrabbitmq.conf(classic_config node list) on every broker, which causes a batched restart. - Password change.
definitions.jsonchanges cause a batched broker restart in the rabbitmq plays. Consumers get the new password only when their own plays run later insite.yml. NOT AVAILABLE: whether there is an auth-failure window in between. - Bug (FACT):
kolla-ansible pullwith outward rabbitmq enabled. The outward plays setproject_name: outward_rabbitmqand nokolla_role_name(rabbitmq.yml:71), soservice-images-pulllooks uplookup('vars', (kolla_role_name | default(project_name)) + '_services')=outward_rabbitmq_services, and that variable is not defined anywhere (roles/service-images-pull/tasks/main.yml:14;grep outward_rabbitmq_servicesfinds nothing). Thevarslookup raises an undefined-variable error for a name it cannot find when no default is given, so "Pull images" fails on the outward-rabbitmq hosts (ansible-core 2.14.18lib/ansible/plugins/lookup/vars.py:94-104). check.ymlis empty.rabbitmq_hostnameno longer exists in 2023.1.- The precheck runs
getentfor every rabbit host on every rabbit host, so it does N² lookups (precheck.yml:62-69,152-161).
3. Keystone (ansible/roles/keystone)¶
3.1 Plays, hosts, tags, serial¶
| Play | Lines | Hosts | Tags | Serial |
|---|---|---|---|---|
loadbalancer -> include_role keystone tasks_from: loadbalancer |
site.yml:198-202 |
loadbalancer | keystone |
loadbalancer serial |
Apply role keystone |
site.yml:450-458 |
keystone + &enable_keystone_True |
keystone |
serial: '{{ kolla_serial|default("0") }}' (:455) |
What serial means here:
- With the default (
ANSIBLE_SERIALunset, so 0 = all hosts), every keystone node is handled in one batch (ansible-core 2.14.18lib/ansible/executor/playbook_executor.py:300-302), and handlers restart all keystone containers at once (section 0, "Handler order"). - With
ANSIBLE_SERIAL=n,run_oncetasks run once per batch: bootstrap container (db_sync), key distribution, register. Each batch is a separate run on the inventory restricted to that batch, andrun_oncesends a task only to the first host left (playbook_executor.py:184-189,lib/ansible/plugins/strategy/linear.py:217,246-248). - Fernet bootstrap is blocked when not all hosts are in the batch (3.6).
- On upgrade,
Init(expand/migrate) runs in the batch containinggroups['keystone'][0]andFinish(contract) runs in the batch containinggroups['keystone'][-1].
3.2 keystone_services (ansible/roles/keystone/defaults/main.yml:2-55)¶
In 2023.1 the dict spans roles/keystone/defaults/main.yml:2-66 (the heading keeps its earlier range so that links stay
stable).
| Key | container_name | group | enabled | Notable |
|---|---|---|---|---|
keystone |
keystone |
keystone |
true |
Apache + mod_wsgi. Volumes include keystone_fernet_tokens:/ (:163-169). HAProxy keystone_internal (5000), keystone_external (5000, external), and keystone_admin (35357) only when kolla_action == 'upgrade' ("kept only for upgrade compatibility") (:11-40). Healthcheck healthcheck_curl on port 5000 (:124-135) |
keystone-ssh |
keystone_ssh |
keystone |
true |
sshd -D on api_interface_address:8023 (templates/, group_vars/). Shares the keystone_fernet_tokens volume (:51) |
keystone-fernet |
keystone_fernet |
keystone |
true |
Runs cron with the rotation crontab. Healthcheck fernet-healthcheck.sh = fernet-node-sync.sh --check && fernet-push.sh --check (:154-161; templates/). Shares the keystone_fernet_tokens volume (:64) |
There is no keystone_token_provider in 2023.1: keystone.conf sets provider = fernet unconditionally
(templates/keystone.conf.j2:32), and the ssh and fernet containers are always enabled.
Relevant globals (ansible/group_vars/all.yml):
| Variable | Value | Line |
|---|---|---|
fernet_token_expiry |
86400 | :882 |
fernet_token_allow_expired_window |
172800 | :884 |
fernet_key_rotation_interval |
expiry + window = 259200 s (3 days) | :889 |
keystone_admin_url |
keystone_internal_url (internal_protocol://<internal VIP>:5000) |
:864-865 |
openstack_auth.auth_url |
keystone_internal_url |
:895-899 |
multiple_regions_names |
[openstack_region_name] |
:595-596 |
keystone_default_user_role |
member |
:891 |
3.3 Task flow per kolla_action¶
| Action | Flow | Evidence |
|---|---|---|
| deploy | config.yml -> check-containers.yml -> clone.yml (dev mode only) -> bootstrap.yml -> flush_handlers -> distribute_fernet.yml -> register.yml -> register_identity_providers.yml (federation only) |
roles/ |
| reconfigure | = deploy, so bootstrap, distribution and registration all re-run | roles/ |
| upgrade | config.yml -> check-containers.yml -> set log_bin_trust_function_creators=1 (run_once, on groups['keystone'][0]) -> notify Init keystone database upgrade on [0] -> notify Finish keystone database upgrade on [-1] -> flush_handlers -> set log_bin_trust_function_creators=0 (run_once, every batch) -> service-ks-register, which removes the admin endpoint (state: absent). No bootstrap or key distribution |
roles/; defaults/ |
| deploy-containers | check-containers.yml |
tasks/ |
| pull / stop | service-images-pull / service-stop (all 3 containers, all hosts at once) |
tasks/pull.yml, tasks/stop.yml |
| precheck | group check; free ports 5000 and 8023; assert fernet_token_expiry is an int between 0 and 604800 |
roles/ |
| check | No check.yml in the role, no kolla_enable_sanity_* variable and no check command in tools/ 2023.1 |
ls roles/ |
| config | config.yml. Handlers are guarded by kolla_action != "config" |
handlers/ |
bootstrap.yml and bootstrap_service.yml (run on every deploy/reconfigure):
mysql_db keystoneandmysql_user keystonewithkeystone_database_password,append_privs(run_once,delegate_to: groups['keystone'][0], skipped withuse_preconfigured_databases) (roles/keystone/tasks/bootstrap.yml:1-36). This re-asserts the DB user password on each run.kolla_container_facts keystone_fernet(lists running containers only,library/kolla_container_facts.py:77) ->group_by keystone_fernet_<State|bootstrap>(roles/keystone/tasks/bootstrap_service.yml:2-18).- Fail if no host has
keystone_fernetrunning and not every keystone host is inansible_play_batch("Please run without --limit or --serial") (:22-31). bootstrap_keystonecontainer (KOLLA_BOOTSTRAP). In the kolla image,kolla_startsourceskolla_extend_start, which seesKOLLA_BOOTSTRAP, runssudo -H -u keystone keystone-manage db_syncand exits 0 without starting httpd (kolladocker/base/start.sh:18-21,docker/keystone/keystone/extend_start.sh:41-46; see Kolla images §4).EXTRA_KEYSTONE_MANAGE_ARGSis empty by default (kolladocker/keystone/keystone/extend_start.sh:21) and kolla-ansible does not set it. It isrun_oncewithoutdelegate_to(first host of the batch) and runs on every deploy/reconfigure (bootstrap_service.yml:33-50).bootstrap_keystone_fernet:keystone-manage fernet_setup,run_once,delegate_to: groups['keystone'][0], only if nokeystone_fernetcontainer is running on any play host (:52-75).
distribute_fernet.yml (every deploy/reconfigure):
- Wait for port 8023 on every host (
:2-10). docker exec -t keystone_fernet /usr/bin/fernet-push.sh,run_once,delegate_to= the first host inkeystone_fernet_running(orgroups['keystone'][0]on first bootstrap) (:12-18).
register.yml (every deploy/reconfigure):
docker exec keystone kolla_keystone_bootstrap <admin user> <admin pw> <project> admin <internal_url> <public_url> <region>, once per region (run_once); no admin URL any more (roles/keystone/tasks/register.yml:2-12).service-ks-registerfor the identity service and its endpoints (internal, public; adminstate: absent) and theservicerole (:14-20;defaults/main.yml:209-221).os_keystone_rolekeystone_default_user_role(member) (:22-33).
3.4 Config files (ansible/roles/keystone/tasks/config.yml)¶
Destination (/etc/kolla/...) |
Source / merge order | Notifies | Lines |
|---|---|---|---|
<svc>/ |
keystone.json.j2 / keystone-ssh.json.j2 / keystone-fernet.json.j2 |
Restart <svc> container |
:45-56 |
keystone/, keystone-fernet/ |
merge_configs: templates/ + /etc/ + /etc/ + /etc/ + /etc/ |
Restart keystone container / Restart keystone-fernet container |
:58-77 |
keystone/ |
template. Waits up to 36 x 5 s (3 min) for fernet key 0, then exec httpd/ |
Restart keystone container |
:79-91; templates/ |
keystone/ |
Created only if /etc/ exists on the controller. Every file in it is templated (Jinja) into /etc/. keystone.conf gains [identity] domain_specific_drivers_enabled = true, domain_config_dir = /etc/ |
Restart keystone container. A file added once keystone.conf already has the domain settings is not yet in the container and the domains entry is optional, so kolla_set_configs --check exits 2 and "Check keystone containers" fails on every keystone host before the handler runs (templates/, tasks/, module_utils/; Kolla images §3.3) |
:34-39,93-130; keystone.conf.j2:24-28 |
<keystone, keystone-fernet>/ |
First found in /etc/ (supported_policy_format_list) |
Restart <svc> container |
:15-32,132-145 |
keystone/ |
with_first_found: /etc/ -> /etc/ -> template. processes = keystone_api_workers = openstack_service_workers = min(vCPU, 5) (defaults/, group_vars/) |
Restart keystone container |
:151-167; templates/ |
keystone/ |
Only if /etc/ exists |
Restart keystone container |
:169-193 |
| (controller) cron generator | python fernet_rotate_cron_generator.py -t <interval_s//60> -i <index in groups['keystone']> -n <len>, delegate_to: localhost, per host |
none (changed_when: false) |
:195-210 |
keystone-fernet/ |
templates (id_rsa = {{ keystone_ssh_key.private_key }}) |
Restart keystone-fernet container |
:212-233 |
keystone-ssh/ |
templates (id_rsa.pub becomes authorized_keys) |
Restart keystone-ssh container |
:235-250 |
keystone.conf.j2 highlights:
transport_url = rpc_transport_url(:4).[database] connectionvia the VIP,max_retries = -1(:18-22).[token] provider = fernet,expiration,allow_expired_window(:30-34).[fernet_tokens] max_active_keys = ceil((expiry + window) / rotation_interval) + 2, which is 3 with the defaults (:43-46).[cache] backend = oslo_cache.memcache_pool,memcache_servers= everygroups['memcached']host (:48-51).[oslo_messaging_notifications]usesnotify_transport_urland drivermessagingv2only when ceilometer or barbican topics are enabled, otherwisenoop(:53-60;defaults/main.yml:194-203).
3.5 Handlers (ansible/roles/keystone/handlers/main.yml). Order is load-bearing (comment :2-3)¶
| # | Name | Logic | Lines |
|---|---|---|---|
| 1 | Init keystone database upgrade | init_upgrade_database container, KOLLA_UPGRADE, run_once, only when kolla_action == upgrade and the host is groups['keystone'][0] |
:5-27 |
| 2 | Restart keystone-ssh container | recreate_or_restart_container. All notified hosts in the batch in parallel |
:29-43 |
| 3 | Restart keystone-fernet container | same | :45-59 |
| 4 | Restart keystone container | same: no rolling, no wait for port or health | :61-75 |
| 5 | Finish keystone database upgrade | finish_upgrade_database, KOLLA_FINISH_UPGRADE, run_once, kolla_action == upgrade. No host guard: it relies on being notified only on groups['keystone'][-1] (upgrade.yml:28-31) |
:77-98 |
3.6 Fernet key lifecycle (FACT unless marked)¶
- Storage: a per-host Docker volume
keystone_fernet_tokens, shared by thekeystone,keystone_fernetandkeystone_sshcontainers (defaults/main.yml:51,64,169). - Initial setup:
keystone-manage fernet_setupongroups['keystone'][0]only when nokeystone_fernetcontainer is running anywhere in the play (bootstrap_service.yml:52-75). The code comment admits the detection is imprecise: "This does not catch some cases ... the keystone_fernet container is created but not running ... Probably what we care about is the existence of Fernet key 0" (:10-14). NOT AVAILABLE (Keystone upstream behaviour): whetherfernet_setupis a no-op when key0already exists, which would make a re-run on a node with an intact volume harmless. On a rebuiltgroups['keystone'][0](empty volume), with everykeystone_fernetcontainer stopped (for example afterkolla-ansible stop), a deploy runsfernet_setupon that node and then pushes its key set withrsync --deleteto every node (:52-75;distribute_fernet.yml:12-18;templates/fernet-push.sh.j2:17-22). NOT AVAILABLE: whetherfernet_setupthen generates brand-new keys and whether that invalidates all tokens; the code comment atbootstrap_service.yml:20-21names this risk. - Rotation: cron in
keystone_fernetrunsfernet-rotate.sh=keystone-manage fernet_rotate&&fernet-push.sh(templates/fernet-rotate.sh.j2:6-8).fernet-push.shruns, for each other keystone host,rsync -az ... --delete /etc/keystone/fernet-keys/ keystone@<host>:/etc/keystone/fernet-keysover ssh port 8023, underset -o errexit(templates/fernet-push.sh.j2:3,17-22). With--checkit does the same as a dry run, which the container healthcheck uses (:6-15). -
Schedule: the generator gives each host a weekly crontab (min hour * * dow) in round-robin with offset
interval * index(files/fernet_rotate_cron_generator.py:39-94;templates/crontab.j2:5-7). It deliberately under-rotates across the week boundary (:53-54,67). Working throughgenerate()by hand for these inputs, which all take the day branch (offsetinterval * index, stepinterval * hosts, a job only whiletime + interval <= 10080;:55-56,65-73), gives:Interval Hosts Result 4320 min (default 3 days) 3 h0 = Sun 00:00, h1 =Wed 00:00, h2 = no jobs. Gaps 3 d / 4 d4320 min 5 only h0 and h1 have jobs 1440 min 3 h0 Sun/Wed/Sat, h1 Mon/Thu, h2 Tue/Fri Crons run in the container's local time (
/etc/localtimeis mounted,defaults/main.yml:61). -
Distribution on every deploy/reconfigure:
fernet-push.shruns from the first running node (distribute_fernet.yml:12-18), so that node's key set overwrites every other node (--delete).
3.7 Bootstrap, DB and registration: what re-runs on reconfigure¶
Re-runs every time:
mysql_dbandmysql_user keystone- the
bootstrap_keystonecontainer (db_sync) - the fernet-bootstrap check
fernet-push.shkolla_keystone_bootstrapfor each region. The keystone image installs kolladocker/keystone/keystone/keystone_bootstrap.shunder that name (kolladocker/keystone/keystone/Dockerfile.j2:35). Called with seven arguments it runs in its "no ADMIN_URL modern mode" (keystone_bootstrap.sh:19-30) and runskeystone-manage bootstrapwith the admin user, password, project, role, internal/public URLs, service namekeystoneand the region (:46). It counts eight 'already exists'/'exists'/'already has'/'skipping' patterns (domain, project, user, role, role assignment, region, internal and public endpoints) but reports unchanged only when the count is nine (:51-63). NOT AVAILABLE (upstreamkeystone-manage bootstrapis not in these sources): whether any output line matches twice, so that the task can ever report unchanged; whetherkeystone-manage bootstrapre-asserts the admin user, project, role and endpoints and resets the admin password tokeystone_admin_passwordif it differs. The wrapper has no pattern for a password reset (:51-63).service-ks-register(identity service and endpoints)- the
memberrole
All of these are run_once per batch, except the cron generator, which runs on the controller once per host.
3.8 Dependencies¶
- Explicit: none in role metadata (no
meta/). - Implicit, upstream:
- MariaDB via the VIP (
keystone_database_address,defaults/main.yml:80). - memcached:
[cache](keystone.conf.j2:48-51). The memcached play (site.yml:387) runs before keystone (:450). - RabbitMQ:
transport_urlis always rendered, and notifications go out only if ceilometer or barbican is enabled (keystone.conf.j2:4,53-60). - HAProxy VIP:
keystone_internal_urlis used byopenstack_authfor registration (group_vars/all.yml:895-899). - kolla_toolbox (
os_*modules). passwords.ymlkeystone_ssh_key(etc/kolla/passwords.yml:192).
- MariaDB via the VIP (
- Downstream: see 3.10.
3.9 Operationally notable behaviour, risks and oddities¶
- Reconfigure with a config change (default serial 0) restarts
keystone_ssh, thenkeystone_fernet, thenkeystone, each on all keystone hosts simultaneously (handlers:29-75). The result is a brief Keystone-wide API outage. Nothing waits for the API to return beforeregister.ymlruns (which usesrun_onceplusuntil/retries insideservice-ks-register,roles/service-ks-register/tasks/main.yml:21-23). The containers have a healthcheck (defaults/main.yml:124-135), but the handlers do not wait for it. NOT AVAILABLE: whether other services keep validating cached tokens via memcached during the restart. SetANSIBLE_SERIALfor a rolling reconfigure. - What restarts keystone indirectly.
keystone.confembeds every memcached host and every rabbitmq host (keystone.conf.j2:4,51,54), and the fernet crontab andfernet-push.shembed the keystone host list and index. Adding or removing a memcached, rabbitmq or keystone node therefore restarts keystone, and keystone-fernet as well for keystone node changes. - Fernet push has no partial-failure handling.
set -o errexitstops at the first unreachable peer, so later peers are not updated either (fernet-push.sh.j2:3,17-22). The next scheduled rotation runs on another node (round-robin, 3.6), which may be one of the peers left with the stale set. That node runsfernet_rotateand thenfernet-push.sh, whosersync --deleteoverwrites the key set of each peer it reaches, stopping at the first failure (templates/fernet-rotate.sh.j2:6-8,templates/fernet-push.sh.j2:3,17-22). The reconfigure-time push from "first running node" works the same way. Thekeystone_fernethealthcheck (fernet-push.sh --check) reports such a failure as unhealthy (templates/fernet-healthcheck.sh.j2:6). NOT AVAILABLE (Keystone key handling): whether this removes the key that signed live tokens and so causes mass 401s. - Bug in the cron generator hour calculation. In the day branch,
hour = time % HOUR_SPANis computed where(time % DAY_SPAN) // HOUR_SPANwas intended (fernet_rotate_cron_generator.py:68-70). It is harmless when the interval is a whole number of days (the default). Otherwise it can over-rotate. Read from the code: with interval 3600 min (60 h, i.e.fernet_token_expiry=43200), h1 hastime = 3600, soday = 2andhour = 3600 % 60 = 0, givingday 2 00:00instead ofday 2 12:00after h0'sday 0 00:00, which is a 48 h gap with a 60 h interval. NOT AVAILABLE (Keystone key handling): whether, withmax_active_keys=3, a key can then be deleted before the allow-expired window of tokens it signed has elapsed. - The precheck validates the wrong variable. It asserts
fernet_token_expiry <= 604800(precheck.yml:43-60, and its message talks about "rotate the fernet keys"). The generator, however, is fedfernet_key_rotation_interval(= expiry + 2 d by default,config.yml:196-199) and fails above 10080 min (generator:59-62). The generator exits 0 with{"failed": true}(:31-36,116-117), so thecommandtask succeeds andset_fact cron_jobsthen errors on a missing key (config.yml:207-210). Anyfernet_token_expiryabove 5 days with the default window passes precheck but breaks deploy. - Under-rotation by design. With 3 or more hosts and the 3-day default, only 2 hosts ever rotate. Rotations happen
at a fixed local
00:00on Sunday and Wednesday. - Shared SSH key. Every node gets the same
keystone_ssh_key, andssh_configsetsStrictHostKeyChecking no,UserKnownHostsFile /dev/null(templates/ssh_config.j2:1-4). NOT AVAILABLE: whether anyone holding the private key and reaching port 8023 on the API network can overwrite fernet keys asfernet-push.shdoes (templates/fernet-push.sh.j2:20), and whether that allows token forgery. - Domain config cleanup. Removing a file from
/etc/kolla/config/keystone/domainsis never propagated: no task deletes stale files in/etc/kolla/keystone/domains/. FACT: under the defaultCOPY_ALWAYSthe stale file is copied back into the container at every start, even after a recreate:domainsis a directory entry (templates/keystone.json.j2:25-31), sokolla_set_configsdeletes/etc/keystone/domainsand re-copies the whole host directory (kolladocker/base/set_configs.py:136-158,534-541; see Kolla images §2.3).keystone.confkeepsdomain_specific_drivers_enabled = trueanddomain_config_dir = /etc/keystone/domainswhile/etc/kolla/config/keystone/domainsexists (templates/keystone.conf.j2:24-28). NOT AVAILABLE (Keystone semantics): whether Keystone keeps loading the stale file as a domain config. The task also re-uses the variable nameregister: keystone_domains(config.yml:112,123), which is an oddity. - Upgrade. Without serial, all keystone APIs restart at once between
Init(expand/migrate) andFinish(contract). With serial,log_bin_trust_function_creatorsis reset to 0 at the end of every batch (upgrade.yml:36-50), whileEnableruns only in the batch whose first host isgroups['keystone'][0](:6-21). The contract step in the last batch therefore runs with it disabled. Oddity; the impact is NOT AVAILABLE. The upgrade also removes the admin endpoint (:52-59). - Reconfigure re-asserts identity state. It re-applies the admin credentials and endpoints, and the DB user
password, from
passwords.ymland globals. NOT AVAILABLE (os_user andkeystone-manage bootstrapsemantics): whether out-of-band changes are silently reverted.
3.10 Runtime dependency of other services on Keystone (grep across ansible/roles/*/templates)¶
| Measure | Count | Pattern |
|---|---|---|
Roles with www_authenticate_uri (keystonemiddleware [keystone_authtoken]) |
28 | www_authenticate_uri = {{ keystone_internal_url }} (30 lines); 30 roles have a [keystone_authtoken] section (sahara and venus without www_authenticate_uri) |
Roles with auth_url |
37 | auth_url = {{ keystone_internal_url }} (75 lines), {{ openstack_auth.auth_url }} (1); no /v3 and no admin-endpoint variants any more |
Roles with memcache_servers / memcached_servers |
32 | (36 memcached_servers = lines + 7 memcache_servers =) |
Roles with memcache_secret_key |
29 | Token cache in memcached with memcache_security_strategy = ENCRYPT (for example roles/) |
Roles that call service-ks-register |
37 (keystone included) | During their deploy/reconfigure they create services, endpoints, users, roles and grants through openstack_auth, i.e. keystone_internal_url on the internal VIP:5000 with openstack_interface: internal, run_once, 5 retries x 10 s (roles/, defaults/; group_vars/) |
Implications:
- FACT: the
[keystone_authtoken]sections of 30 roles point keystonemiddleware at Keystone, and 29 roles keep its token cache in memcached (memcache_secret_key). The Ussuri erratum on these counts (erratum) concerns the Ussuri tree; the counts above are a recursive grep of the 2023.1 templates. NOT AVAILABLE: whether those API services fail to authenticate requests while Keystone or memcached is unavailable. - Keystone, via the internal endpoint, is a hard deploy-time dependency for the
deploy/reconfigureof 37 roles. - Every service's
memcached_serversandtransport_urlembed the full memcached and rabbitmq host lists, so changes to those groups fan out as restarts across the whole control plane.