Backing up and restoring the MariaDB cluster with kolla-ansible mariadb-backup¶
Context¶
Every OpenStack service of a Kolla-Ansible deployment keeps its state in the MariaDB (Galera) cluster: instances,
volumes, networks, users, quotas. Galera replicates it across controllers, which protects against losing a node, not
against a bad DELETE, a failed schema migration during an upgrade, or losing every controller. For those, only a
backup taken outside the cluster helps.
Kolla-Ansible ships hot backups with Mariabackup: kolla-ansible mariadb-backup takes a full or incremental backup
without stopping the database. It does not schedule backups, does not copy them off the host, and does not restore
them: restoring is manual. This page says what the backup does, where the files land, how to restore a full or an
incremental backup, and what to check, following the Kolla-Ansible 2026.1 guide and code.
Topology¶
flowchart LR
deploy[deploy host: kolla-ansible mariadb-backup] --> c1
subgraph controllers
c1["controller-1: first host of the shard group, runs the mariabackup container"]
c2[controller-2]
c3[controller-3]
end
c1 --> vol[(mariadb_backup volume on controller-1)]
vol -. "copied off the host by the operator" .-> off[(backup storage outside the cluster)]
Network design¶
Not affected. The backup container connects to the MariaDB server of its own host, on api_interface_address and
mariadb_port (the backup.my.cnf template), not through the VIP.
Storage design¶
- Backups land in the Docker volume
mariadb_backupon the backup host only, by default the first host of the shard group (mariadb_backup_host,groups[mariadb_shard_group][0]; with one shard, the first host ofmariadb). On Docker, its files are under/var/lib/docker/volumes/mariadb_backup/_data/. - The volume is on the controller's own disk: plan its size (a full backup is about the size of the data, compressed) and copy every backup elsewhere. A backup that stays on the controller is lost with it.
- With several MariaDB shards, only the default shard (
mariadb_default_database_shard_id) is backed up.
Inventory and configuration layout¶
# globals.yml
enable_mariabackup: "yes"
Then create the backup user and its grants on the running cluster:
⚠️ State-changing. Writes the Mariabackup configuration on the backup host, and creates the backup database user (
RELOAD,PROCESS,LOCK TABLES,BINLOG MONITOR) with rights on themariadb_backup_historytable, which the backup tool itself creates on its first run. If the MariaDB configuration changed since the last run, the MariaDB containers restart one by one. Undo: setenable_mariabackupback to"no"; the user stays until you drop it.
kolla-ansible reconfigure -i <inventory> --tags mariadb
The password of the backup user is mariadb_backup_database_password in passwords.yml; nobody types it, since the
backup container reads it from its own configuration.
Taking backups¶
kolla-ansible mariadb-backup -i <inventory> # full
kolla-ansible mariadb-backup -i <inventory> --incremental # incremental, on top of the last full backup
These are hot backups: the guide says they take no downtime. The run starts a short-lived mariabackup container on
the backup host, with the same image as the running MariaDB server (Kolla-Ansible 2026.1 picks the image of the
running mariadb container on purpose, to avoid version mismatches). The backup is read-only for the platform, but
it adds read load on that controller's disks while it runs.
The guide suggests a daily full backup kept two weeks and hourly incrementals kept one day, scheduled outside Kolla-Ansible (cron or a timer on the deploy host). Then copy the new files off the host, for example:
# on the backup host: list what is there
sudo ls -lh /var/lib/docker/volumes/mariadb_backup/_data/
# copy them to storage outside the cluster (the destination is yours to choose)
sudo rsync -a /var/lib/docker/volumes/mariadb_backup/_data/ <backup-user>@<backup-server>:<backup-path>/
What the volume held on a lab after one full and one incremental backup (each backup in its own directory; the
incremental names the full one it builds on; last_full_file points at the latest full backup):
full-27-10-2026-1793125994/mysqlbackup-27-10-2026-1793125994.qp.xbc.xbs.gz
incr-18-33-53-27-10-2026-since-27-10-2026-1793125994/incremental-18-33-53-27-10-2026-mysqlbackup-27-10-2026-1793125994.qp.xbc.xbs.gz
last_full_file
On that lab (about 100 MB of OpenStack databases once prepared), the full backup took 37 s and the incremental 58 s,
and the compressed full backup was 1.8 MB (ls -lh in the run's backup capture). To list the files themselves, use a root shell (sudo bash -c 'ls -lh
/var/lib/docker/volumes/mariadb_backup/_data/*/'): with sudo ls …/_data/*/ your own shell expands the glob and
cannot read the directory.
Test a restore on a lab from time to time: an untested backup is a hope, not a backup.
Restoring¶
🛑 Destructive. A restore replaces every OpenStack database with the backup's content. Everything written after the backup is lost: instances, volumes, ports and users created since then are unknown to OpenStack, while the hypervisors, Ceph and the network may still hold them. Every API is down from the stop to the end of the recovery. Only do it when the database is lost or corrupted beyond repair, after copying the current data aside (the
mariadbvolume of each node) so that you can go back, and with a decision recorded by whoever owns the platform. Compare with a Galera recovery first if the data is intact but the cluster is stopped:kolla-ansible mariadb-recovery.
The steps follow the Kolla-Ansible guide; run them on the first MariaDB host (the backup host). <mariadb-image> is
the image of the mariadb container: sudo docker inspect mariadb --format '{{.Config.Image}}'.
1. Prepare the backup in a temporary container (it reads the backup and writes a restorable copy under
/backup/restore, inside the mariadb_backup volume; the running database is not touched yet):
sudo docker run --rm -it --volumes-from mariadb --name dbrestore \
--volume mariadb_backup:/backup <mariadb-image> /bin/bash
# inside the container
cd /backup
rm -rf /backup/restore && mkdir -p /backup/restore/full
gunzip <full-backup-file>.qp.xbc.xbs.gz # with its directory: full-<date>-<n>/mysqlbackup-<date>-<n>…
mbstream -x -C /backup/restore/full/ < <full-backup-file>.qp.xbc.xbs
mariabackup --prepare --target-dir /backup/restore/full
For incremental backups, extract each one and apply them in order, oldest first, after the first --prepare: one
--prepare --incremental-dir per incremental (the guide's second example shows one). The guide's example also mounts
--tmpfs /backup/restore; it is left out here on purpose, because step 3 runs in a second container that must still
see /backup/restore.
mkdir -p /backup/restore/inc
gunzip <incremental-backup-file>.qp.xbc.xbs.gz # incr-<time>-<date>-since-<date>-<n>/incremental-…
mbstream -x -C /backup/restore/inc < <incremental-backup-file>.qp.xbc.xbs
mariabackup --prepare --incremental-dir=/backup/restore/inc --target-dir /backup/restore/full
2. Stop MariaDB on every node.
🛑 Destructive. Stops the database cluster: every OpenStack API fails until step 4 ends. Precondition: step 1 succeeded (
mariabackup --prepareended withcompleted OK!).
kolla-ansible stop -i <inventory> --tags mariadb --yes-i-really-really-mean-it
3. Replace the data on the first node.
🛑 Destructive. Deletes the current database files of the first node and copies the backup in their place. Precondition: the current
mariadbvolume of every MariaDB node was copied aside (for examplesudo cp -a /var/lib/docker/volumes/mariadb/_data <safe place>on each), and step 2 stopped every node.
sudo docker run --rm -it --volumes-from mariadb --name dbrestore \
--volume mariadb_backup:/backup <mariadb-image> /bin/bash
# inside the container
rm -rf /var/lib/mysql/*
rm -rf /var/lib/mysql/\.[^\.]*
mariabackup --copy-back --target-dir /backup/restore/full
4. Start the database again.
- One MariaDB node:
sudo systemctl start kolla-mariadb-container.service(in 2026.1 the container is run by this systemd unit;docker startwould bypass it), then readsudo docker logs mariadb. - A Galera cluster: bootstrap it from the restored node, after making the other nodes take a full copy from it.
Why the extra step: after a restore, the other nodes hold data that is newer than the backup, with their own
grastate.dat. mariadb-recovery with mariadb_recover_inventory_name starts the named node with
--wsrep-new-cluster and then starts the others as they are; nothing in the play clears their data
(recover_cluster.yml). A node whose saved position is ahead of the new cluster's normally refuses to join (Galera
reports that the states diverged) instead of taking a full state transfer (SST) by itself. The upstream guide does
not cover this. Emptying the data directory of each other node makes it join as a new node, which always takes a
full SST from the cluster. This follows from how Galera works and from the role's code. The refusal of a node that was
not emptied was not observed on a lab.
The emptying path was run on a lab (3 nodes; full backup plus one incremental; steps 1 to 4 as written, the docker run -it … bash
commands given to bash -c instead): the restored node has no grastate.dat after the copy-back, and the play handled
it; both emptied nodes took a full SST (wsrep_sst_mariabackup) from the restored node, one after the other (the
second logged requested state transfer from '*any*', but it is impossible to select State Transfer donor: No donor
candidates temporarily available until the first had finished); the run took 3 min 9 s and ended with every node
Synced. The database then held what existed at the incremental backup: a network created between the full and the
incremental backups was there, one created after the incremental was gone. A test instance went ACTIVE in 26 s.
First, on each other MariaDB node (never on the restored one), check that its mariadb container is stopped. The
commands of this page assume Docker; with Podman, use podman and the volume path under
/var/lib/containers/storage/volumes/.
sudo docker ps -a --filter name=mariadb --format '{{.Names}}\t{{.Status}}' # read-only: mariadb must show Exited
Go on only if it shows Exited on that host: emptying the data directory of a running node corrupts it.
🛑 Destructive. On every MariaDB node except the restored one: deletes the node's database files, so that it copies everything from the restored node when it joins. Its own newer data is lost unless it was copied aside. Precondition: the
mariadbvolume of this node was copied aside (step 3's precondition, done on every node), every node is stopped (step 2), and you are on the right host (not the restored one). Undo: copy the saved data back into the volume.
# on each other MariaDB node, never on the restored one, after the check above showed Exited
sudo find /var/lib/docker/volumes/mariadb/_data -mindepth 1 -delete
A lighter variant, unverified: moving only grastate.dat aside. When it is missing, MariaDB's start-up recovers
the position from InnoDB (--wsrep-recover), with the same cluster UUID and the node's own newer seqno, so the
node may refuse to join again. Use the full emptying above unless a lab run has shown the lighter variant works on
your release.
🛑 Destructive. Starts a new Galera cluster from the restored node's data; the other nodes, emptied above, join it with a full state transfer. Precondition: steps 1 to 3 succeeded on the node you name, and the step above was done on every other node. If a node still refuses to join, read
docker logs mariadband/var/log/kolla/mariadb/mariadb.logon it before doing anything else: a line saying that the states diverged, orLocal state seqno ... is greater than group seqno, means its data was not cleared. A full state transfer of a large database can outlast the role's waits (recover_cluster.yml: for a joiner's port, 10 retries 6 s apart with up to 60 s per attempt, about 11 minutes; forSynced, 30 retries 10 s apart, about 5 minutes): ifmariadb-recoverytimes out on a joiner, checkdocker logs mariadbthere for the transfer's progress before running it again.
kolla-ansible mariadb-recovery -i <inventory> -e mariadb_recover_inventory_name=<first-mariadb-host>
Key choices and their consequences¶
| Choice | Alternative(s) | Why this one here | Consequence in operation |
|---|---|---|---|
| Kolla-Ansible's Mariabackup | mariadb-dump of each database; storage snapshots |
Hot, consistent, built in, same image as the server | Restore is manual and all-or-nothing; one database alone is easier to restore from a dump |
| Full plus incrementals | Full only | Smaller hourly backups | A restore needs the full backup and every incremental in order |
| Copy off the controller | Keep in the volume | A lost controller takes its volume with it | One more job to schedule and watch |
Restore with mariadb-recovery and a named host |
Bootstrap Galera by hand | The role sets safe_to_bootstrap and starts the others in order |
It must be told the right host; without mariadb_recover_inventory_name it picks the node with the highest sequence number, which is not the restored one |
What went wrong during deployment¶
Nothing went wrong on the lab run (enable, full and incremental backups, restore). Points that follow from the code and the guide:
mariadb-backupdoes nothing unlessenable_mariabackupis set and thereconfigure --tags mariadbran: the play only runs the rolewhen: enable_mariabackup.- The backup is taken on
mariadb_backup_hostonly; looking for files on another controller finds nothing. - The guide's example image name ends in
-rocky-10: use the image of your ownmariadbcontainer instead.
Version notes¶
Kolla-Ansible 2026.1: the backup container uses the running server's image (bug 2058644). The guide still says the
backup reaches the cluster through the VIP; in 2026.1 the backup configuration names the backup host's own
api_interface_address.
Lessons learned¶
- A backup on the controller is not a backup: copy it off, and check the copy.
- Practise the restore on a lab with the same release before you need it; time it.
- After a restore, OpenStack and the infrastructure disagree about everything created after the backup. Plan how to find those resources (instances on hypervisors, volumes in the storage back-end) before you restore.
Validity and limits¶
- Written from the Kolla-Ansible 2026.1 guide (MariaDB database backup and restore, MariaDB guide) and the code of
ansible/mariadb-backup.yml,ansible/mariadb-recovery.ymlandansible/roles/mariadb/(backup.yml,config.yml,register.yml,recover_cluster.yml,templates/backup.my.cnf.j2). Run on a lab (Kolla-Ansible 22.2.0, 3 MariaDB 11.4.13 nodes behind ProxySQL, Docker, about 100 MB of data): enabling, a full and an incremental backup, and the whole restore with the full emptying of the other nodes. Not run: the lightergrastate.datvariant, a single-node restore,rsyncoff the host, several incrementals, a large database (SST and role time-outs). - Docker only; with Podman, replace
dockerand check the volume path. - Not covered: several shards, external databases (
use_preconfigured_databases), point-in-time recovery with binary logs, restoring one database alone.
Related¶
- Kolla-Ansible 2026.1: MariaDB backup and restore, MariaDB guide
- MariaDB: Mariabackup overview
- Rotating the secrets in passwords.yml: take a backup before rotating database passwords.
Provenance: written by agent:ozarland-author, from the Kolla-Ansible 2026.1 documentation and code. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.