kolla-ansible deploy hangs on MariaDB port liveness. MariaDB exits with "Address already in use" because ProxySQL holds port 3306 on a node without HAProxy¶
Symptom¶
A first kolla-ansible deploy on a single node with enable_haproxy: "no" stops progressing on a MariaDB handler,
until it times out:
RUNNING HANDLER [mariadb : Wait for first MariaDB service port liveness] *****
The MariaDB container has exited:
$ sudo docker ps -a --filter name=mariadb --format '{{.Names}}\t{{.Status}}'
mariadb Exited (1) …
and its log says the port is taken:
Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
prechecks passed before the deploy: nothing was listening on port 3306 yet when they ran.
Impact¶
The deploy cannot complete: no MariaDB, so no OpenStack service. Nothing is lost, because the deployment never completed and the MariaDB volume holds no data yet. On a deployment that already ran, the same mismatch stops MariaDB at its next restart (observed on a lab, see Lab observations): every OpenStack service then loses its database.
Quick checks¶
On the node, read-only:
# 1. Who listens on 3306?
sudo ss -ltnp | grep ':3306'
# expected for this problem: proxysql, on the node's address
# 2. Which containers exist? A haproxy container despite enable_haproxy "no" is part of the signature.
sudo docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -E 'mariadb|proxysql|haproxy|keepalived'
# 3. The MariaDB log
sudo tail -n 20 /var/lib/docker/volumes/kolla_logs/_data/mariadb/mariadb.log
# 4. What the configuration asks for
grep -E '^(enable_haproxy|enable_keepalived|enable_proxysql|kolla_internal_vip_address)' /etc/kolla/globals.yml
Diagnosis¶
In Kolla-Ansible 2026.1:
- MariaDB listens on
api_interface_address, portdatabase_port(3306), frombind-addressandportingalera.cnf.j2; - ProxySQL listens on
kolla_internal_vip_address, portdatabase_port, frommysql_variables.interfacesinproxysql.yaml.j2; - ProxySQL is on by default:
enable_proxysql: "{{ enable_mariadb }}"(group_vars/all/proxysql.yml). It was already on by default in 2025.1 (enable_proxysql: "yes"); - the
loadbalancerrole, which deploys HAProxy, keepalived and ProxySQL, is enabled when any of them is (enable_loadbalanceringroup_vars/all/loadbalancer.yml).
With HAProxy turned off on a single node, the usual setting is kolla_internal_vip_address equal to the node's own
address, which is also api_interface_address. ProxySQL and MariaDB then both want the same address and port. The
loadbalancer role runs before MariaDB, so ProxySQL takes the port and MariaDB exits.
On the lab where this was met, the loadbalancer role also left a haproxy container running, although
enable_haproxy was "no".
Possible causes¶
Cause 1: ProxySQL left on, with the VIP equal to the node's address¶
enable_haproxy: "no" was set, enable_proxysql was not, and kolla_internal_vip_address is the node's own
address. This is the case described here.
Cause 2: another process on port 3306¶
A MariaDB or MySQL server installed on the host itself, or a leftover container, holds the port. Quick check 1
names the process: if it is not proxysql, stop or remove that service first. That is outside this page.
Resolution¶
For cause 1¶
Stop the deploy (Ctrl-C, or pkill -f ansible-playbook from another session), then turn ProxySQL off in
/etc/kolla/globals.yml:
enable_haproxy: "no"
enable_proxysql: "no"
kolla_internal_vip_address: "<node-address>"
Then remove the containers the failed deploy left behind:
⚠️ State-changing. Removes the ProxySQL, HAProxy and MariaDB containers, not their volumes. On a fresh deployment that never completed, nothing else depends on them. On a deployment that already ran, this is a database outage for every OpenStack service on this node until the next
deploystarts MariaDB again. Precondition:enable_proxysql: "no"is set as above, and this is the node you think it is. Undo: the nextdeployrecreates the containers.
sudo docker rm -f proxysql haproxy mariadb
On a deployment that already ran (the MariaDB volume holds the databases): keep the mariadb volume and skip
the next block. On the lab, the next deploy found the existing cluster and started MariaDB on its volume
(Establish whether the cluster has already existed, then Restart MariaDB container), and completed with the
data intact.
Only on a fresh deployment that never completed, remove the MariaDB volume, so that the next deploy starts MariaDB on a clean volume:
🛑 Destructive. Deletes the MariaDB data volume. Only on a fresh deployment that never completed, where the volume holds no data yet. On any other deployment this deletes every OpenStack database. Check first that no OpenStack service was ever deployed on this node (
sudo docker ps -alists only what the failed deploy created) and that this is the node you think it is. Cannot be undone.
sudo docker volume rm mariadb
On Kolla-Ansible 2026.1 each container also has a systemd unit, which stays enabled and keeps trying to start the
removed container (kolla-haproxy-container.service showed failed on the lab). Remove the two units, and the
volumes the containers left:
⚠️ State-changing. Disables and removes the
kolla-proxysql-containerandkolla-haproxy-containersystemd units, and deletes theproxysql,proxysql_socketandhaproxy_socketvolumes. Precondition:enable_proxysql: "no"is set and theproxysqlandhaproxycontainers are already removed (above). Undo: adeploywith them enabled recreates the units and the volumes.
sudo systemctl disable --now kolla-proxysql-container.service kolla-haproxy-container.service
sudo rm /etc/systemd/system/kolla-proxysql-container.service /etc/systemd/system/kolla-haproxy-container.service
sudo systemctl daemon-reload
sudo systemctl reset-failed
sudo docker volume rm proxysql proxysql_socket haproxy_socket
Then run the checks and the deploy again:
kolla-ansible prechecks -i <inventory>
kolla-ansible deploy -i <inventory>
The alternative is to keep ProxySQL and use a separate address for kolla_internal_vip_address, with HAProxy and
keepalived on: that is the multinode layout, and needs an unused address on the API network.
Verification¶
sudo ss -ltnp | grep ':3306'
# expected: mariadbd only, on the node's address; no proxysql
sudo docker ps --format '{{.Names}}\t{{.Status}}' | grep -E 'mariadb|proxysql|haproxy'
# expected: mariadb Up (healthy); no proxysql or haproxy container
sudo docker exec -it mariadb mariadb -uroot -p -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"
# the password is database_password in passwords.yml; expected: Synced
The deploy then goes past the MariaDB handlers and ends with failed=0.
Lab observations¶
Met on 2026-10-21, during the first build of the all-in-one lab
(OpenStack 2026.1, Kolla-Ansible 22.2.0, Ubuntu 24.04, Docker, one VM nested in an OpenStack cloud), with
enable_haproxy: "no" and kolla_internal_vip_address set to the node's address:
deploystayed onWait for first MariaDB service port liveness;mariadbwasExited (1)withGot error: 98: Address already in use;ssshowedproxysqlon the node's address and port 3306;- a
haproxycontainer ran althoughenable_haproxywas"no"; - the fix above (
enable_proxysql: "no", the three containers and the empty volume removed,prechecksanddeployagain) let the deploy complete. A second build, following the pattern page withenable_proxysql: "no"from the start, did not meet the error.
The details are in the all-in-one lab pattern's What went wrong during deployment.
Run again on 2026-10-27 on the same kind of lab, already deployed with enable_proxysql: "no", by commenting
out that line and running kolla-ansible deploy:
- the
loadbalancerrole startedhaproxyandproxysql;proxysql.logshowedbind() error for <node-address>:3306: Address already in use(MariaDB held the port), andproxysqlwas still reportedhealthy; - the deploy then stayed on
Wait for MariaDB service to be ready through VIPfor 11 minutes, until it was stopped (pkill -f ansible-playbook); why that task did not finish was not traced; - MariaDB kept running until it restarted. After
systemctl stop kolla-mariadb-container.service, a restart ofkolla-proxysql-container.service(ProxySQL then listened on port 3306) andsystemctl start kolla-mariadb-container.service,mariadbwasExited (1)withGot error: 98: Address already in use, the quick checks showed the signature above, and ProxySQL answered the services withAccess denied for user 'nova'@'<node-address>'; - the fix without
docker volume rm mariadb(enable_proxysql: "no", the three containers removed,prechecks,deploy) completed withfailed=0in 11.5 minutes; the verification above gavemariadbdalone on port 3306 andSynced. The removed containers left their systemd units enabled and failing, and the volumesproxysql,proxysql_socket,haproxy_socket; they were removed as above.
Validity and limits¶
- Observed once, on one all-in-one build of OpenStack 2026.1 with Kolla-Ansible 22.2.0. The configuration analysis (listen addresses, defaults) is read from the Kolla-Ansible 2026.1 source; ProxySQL was already on by default in 2025.1, so a 2025.1 single node with the same settings is expected to behave the same, but that was not run.
- A deployment that already ran (2026-10-27 lab run): ProxySQL turned on again on a running single node fails to
bind, and MariaDB fails at its next restart; the fix without the volume removal worked there. The
docker volume rm mariadbstep itself was not run on 2026-10-27 (the volume held data); it was run on 2026-10-21, on the fresh build that never completed. - Not tested: Podman, multinode layouts with a VIP equal to a host address.
- Why a
haproxycontainer ran withenable_haproxy: "no"was observed, not traced in the source.
Related¶
- All-in-one lab on a single VM: the configuration that avoids this error.
- Single control node versus three controllers: HAProxy, keepalived and ProxySQL, and when they are needed.
- kolla-ansible deploy fails part-way because Ubuntu upgraded packages during the deploy: another MariaDB failure during deploy.
- Kolla-Ansible 2026.1: HAProxy guide, quick start
Provenance: written by agent:ozarland-author, from a lab build by agent:lab-reviewer on 2026.1 and the Kolla-Ansible 2026.1 source. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.