MariaDB/Galera¶
This page is the starting point for MariaDB/Galera, the mariadb role: the containers mariadb and
mariadb_clustercheck (the latter only with enable_mariadb_clustercheck, default {{ enable_haproxy }},
roles/mariadb/defaults/main.yml:45,132), on the control nodes of the example inventory (inventory/multinode:89-90),
selected by the tag mariadb. The role is not applied by a play of site.yml itself: site.yml:385 imports
mariadb.yml, whose five plays all carry the tag mariadb: apply the role (no serial), restart the hosts the
handlers put in mariadb_restart, start those in mariadb_start (both serial: "33%"), restart the bootstrap host,
then post-configuration (mariadb.yml:3-72) CODE. The service card below summarises these plays. The lists below
say where the study covers how to operate it, its impact, its dependencies and the evidence, what to check around a run
and what stays open; the service card follows them.
Where to look¶
Operate¶
- Scenario: Change control-plane membership §2
- Command: Database operations, Command quick lookup by tag
Impact and risk¶
- What the tag executes (
mariadb): Tags §3 and Tags §3.1, rowsmariadb - Risks: R9;
mariadb_recovery: R18
Dependencies¶
- Dependencies: Component dependencies §1, Component dependencies §2
Evidence¶
- Fact sheet: fact sheet B §1
- Image side: Kolla images
§4 (row
mariadb), Kolla images §5, Kolla images §5.4 - Also: handlers: Tags §5
Before and after a run¶
- Pre-execution checklist, step 3 (batch size):
ANSIBLE_SERIALcoversreconfigure/upgradeand-e kolla_serial=Ncoversdeploy; mariadb and rabbitmq ignore it. - Pre-execution checklist, step 5 (health gates): before the run, check
Galera:
wsrep_cluster_size,wsrep_local_state_comment=Synced. Repeat after each batch. - Pre-execution checklist, step 8 (rollback material): a MariaDB backup
(
kolla-ansible mariadb_backup). - Tags §3.1, row
mariadb: safe on its own "only on a healthy, fully synced cluster"; prerequisite: every nodeSynced. - R9, precautions for a MariaDB config change or image change: check
wsrep_cluster_sizeandSyncedon all nodes before; take a backup (mariadb_backup); watch between batches. - R18, precautions for
mariadb_recovery: only when the cluster is fully down; identify the node with the highest seqno manually first. - R22, inventory reordering (loadbalancer, mariadb, glance, nova-conductor, keystone
groups): the single active MariaDB HAProxy backend moves, because it is the first host of the default shard
(
roles/mariadb/defaults/main.yml:64). Precautions: treat inventory order as configuration; never reorder casually. - fact sheet B
§1.8,
item 4 (fail-fast guards): reconfigure aborts if the cluster is fully stopped (multi-node) or any alive node is not
Synced.
kolla-ansible stopstops every node at once, so the next deploy/reconfigure fails untilmariadb_recoveryis run.
Known limits and open points¶
- Known bugs and limitations, row
mariadb: the restart batches have noany_errors_fatal; recovery stops healthy nodes; clustercheck started even when disabled. Status in 2023.1 CODE: the restart plays set neitherany_errors_fatalnormax_fail_percentage(mariadb.yml:14-40). A node that fails its port or WSREP wait (roles/mariadb/tasks/restart_services.yml:18-45) marks only its own host failed; when that is every host of the batch (always so with three nodes, one host per batch), ansible-core ends the playbook and the later batches are not restarted; when the batch has other hosts that succeed, the next batch still restarts its nodes (ansible-core 2.14.18lib/ansible/executor/playbook_executor.py:204-208,lib/ansible/utils/helpers.py:25-32; R9). Clustercheck: in deploy and reconfigure, the config and check tasks notifyrestart mariadb-clustercheckonly for enabled items (roles/mariadb/tasks/config.yml:42-53,check-containers.yml:13-18); only recovery still starts it when disabled (recover_cluster.yml:171-185,202-216loop overmariadb_serviceswith noenabledcheck). Whether the failed node is then still joining, and how large the quorum and availability risk is, NOT AVAILABLE. - Q6: "MariaDB rolling restart under load with a failure in wave 1: do later waves proceed?" Status in 2023.1: control flow answered from the source CODE, as in the row above: a failed batch ends the playbook; a batch with a partial failure lets the next batch restart its nodes. NOT AVAILABLE Whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and so how large the quorum and availability risk is.
- What is not
known, row
"MariaDB": the sources do not establish in which order
group_byplaces the hosts inmariadb_restart, hence in which batch node 0 (the active HAProxy backend) restarts (R9); after a failed WSREP wait in a batch that does not stop the run, whether the failed node is then still joining, whether that can leave a single Synced node acting as donor, and how large the quorum and availability risk is; nor whether the DB containers come back as a non-primary cluster after a full stop, and whether a manual start left with--wsrep-new-clusterbootstraps a new cluster. - What is not known:
the sources do not establish dumb-init signal forwarding for
mysqld_safeunder the MariaDB image's own entrypoint, whether xtrace writes one-shot secrets (the MariaDB root password among them) to the container's stderr, or the cost of the recursive chown on large trees (the MariaDB datadir among them). - fact sheet B
§1.8,
"Operationally notable behaviour, risks and oddities", eleven items. Checked against 2023.1 CODE: reconfigure with
a galera.cnf change (now restarted by the
mariadb.ymlrestart plays inserial: "33%"batches, not by the handlers,roles/mariadb/handlers/main.yml:56-73); clustercheck restart is not rolling (the handler runs in the role play, on all notified hosts at once,handlers/main.yml:75-91); WSREP-sync wait is only about 60 s (10 × 6 s,tasks/restart_services.yml:31-45); fail-fast guards (tasks/lookup_cluster.yml:35-42,72-78); recovery takes the whole cluster down (recover_cluster.yml:18-24); recovery bug (clustercheck started when disabled,recover_cluster.yml:171-185,202-216); recovery assumptions (Docker volume path and fixed/tmpfiles,recover_cluster.yml:44-60); upgrade (upgrade_mariadbnow runs in the post-configuration play, which has noserial, so on all nodes at once,mariadb.yml:54-72,tasks/post-upgrade.yml:2-24); backup (client pointed atdatabase_address,templates/backup.my.cnf.j2:5; run on the first shard host,defaults/main.yml:123); credentials on the command line (mysql -uroot -p{{ database_password }},tasks/restart_services.yml:33-38,recover_cluster.yml:148); scale-out restarts the whole cluster, batch by batch (wsrep_cluster_addresslists every shard host,templates/galera.cnf.j2:29). - Kolla images §6, items 10 and 16: recursive chown on data volumes at every start, the MariaDB datadir among them CODE; MariaDB has no automatic Galera recovery.
- R20: a Glance reconfigure or deploy toggles the global MariaDB
log_bin_trust_function_creators1→0 around db sync.
Service card¶
The rows of the card are explained in How to read a service card.
| Item | Value |
|---|---|
| Play | mariadb.yml (import_playbook at site.yml:385; 5 plays) |
| Tags | mariadb |
| Play host patterns | mariadb, mariadb_restart, mariadb_start, mariadb_bootstrap_restart |
| Node types in play (example inventory) | control |
| Node types running its containers | – |
| Containers | mariadb, mariadb_clustercheck |
deploy/reconfigure flow (tasks/) |
config → check-containers → bootstrap |
| Task files | backup.yml, bootstrap.yml, bootstrap_cluster.yml, check-containers.yml, check.yml, config.yml, config_validate.yml, deploy-containers.yml, deploy.yml, loadbalancer.yml, lookup_cluster.yml, main.yml, post-deploy.yml, post-upgrade.yml, precheck.yml, pull.yml, reconfigure.yml, recover_cluster.yml, register.yml, restart_services.yml, stop.yml, upgrade.yml |
| Templates | backup.my.cnf.j2, galera.cnf.j2, hostname.j2, mariabackup.json.j2, mariadb-clustercheck.json.j2, mariadb.json.j2 |
| Handlers (definition order) | Starting first MariaDB container; Wait for first MariaDB service port liveness; Wait for first MariaDB service to sync WSREP; Ensure MariaDB is running normally on bootstrap host; Restart MariaDB on existing cluster members; Start MariaDB on new nodes; Restart mariadb-clustercheck container |
| Key variables | enable_mariadb, mariadb_*, database_address, enable_external_mariadb_load_balancer |
| Role dependencies | none |
Related¶
- Database operations — the
mariadb_backupandmariadb_recoverycommand lines - Change control-plane membership — Galera membership changes restart all DB nodes
- Stop and destroy — after
kolla-ansible stop, MariaDB needsmariadb_recovery - HAProxy + Keepalived — the database is reached through the internal VIP and HAProxy
- Pre-execution checklist — the eight checks before every production run
- Components — the other component pages