HAProxy + Keepalived¶
This page is the starting point for HAProxy and keepalived. Changed in 2023.1: the Ussuri haproxy role is the
loadbalancer role, applied by the "Apply role loadbalancer" play to the loadbalancer group (the network nodes of
the example inventory, inventory/multinode:86-87) with play tags haproxy, keepalived and loadbalancer
(site.yml:99-108). It runs the containers haproxy, keepalived (enable_keepalived, default: follows
enable_haproxy) and proxysql (enable_proxysql, default no) (roles/loadbalancer/defaults/main.yml:2-24,
group_vars/all.yml:622-624, :742). The same play holds the 44 per-service include_role … tasks_from: loadbalancer
tasks, run only for deploy, reconfigure, upgrade and config (site.yml:110-341); each of them also carries its
service tag. A service's loadbalancer.yml imports loadbalancer-config, which renders the HAProxy frontend through
haproxy-config and, with enable_proxysql, the ProxySQL users and rules through proxysql-config
(roles/loadbalancer-config/tasks/main.yml:2-8). The hacluster role is a separate play (site.yml:620) and is not
covered here. The lists below say where the study
covers how to operate it, its impact, its dependencies and the evidence, what to check around a run and what stays open;
the service card follows them.
Where to look¶
Operate¶
- Scenario: Reconfigure a single service
§2
(
-t haproxy reconfigureon an unchanged cloud) - Command: Reconfigure a single service §1.2, Command quick lookup by tag
Impact and risk¶
- What the tag executes (
haproxy): Tags §3 and Tags §3.1, rowshaproxy; the haproxy play under a service tag: Tags §2.3 - Risks: R8
Dependencies¶
- Dependencies: Component dependencies §1, Component dependencies §2
Evidence¶
- Fact sheet: fact sheet A §1
(the
loadbalancerrole in 2023.1);haproxy-config: fact sheet A §2 - Image side: Kolla images §5 (rows
haproxy,keepalived) - Also: handlers: Tags §5
Before and after a run¶
- Pre-execution checklist, step 5 (health gates): before the run, check HAProxy backends all UP. Repeat after each batch.
- Pre-execution checklist, step 7 (scope): add
--skip-tags haproxywhen the frontends are not meant to change. - Tags §3.1, row
haproxy: safe on its own "yes if nothing reports a change, since the restart handlers then never run CODE"; with a change, prefer a window. Prerequisites: facts of every backend host; certificates if TLS is on. - R8, still applies with a staged restart: the handlers find the host that holds
kolla_internal_vip_address, stop and starthaproxyon every backup host at once and wait forhaproxy_monitor_port, then stop and start it on the VIP holder (roles/loadbalancer/handlers/main.yml:9-34, :54-118, :179-253) CODE. Akeepalivedchange also stops and startshaproxy(andproxysql) on each host, backups first, and stops the master's keepalived last, so the VIP moves (:36-52, :154-177, :210-222, :284-300) CODE. Precautions:ANSIBLE_SERIAL=1for haproxy changes;--skip-tags haproxywhen you don't intend to change frontends. - R22, inventory reordering (loadbalancer, mariadb, glance, nova-conductor, keystone
groups): keepalived priorities change (priority = position in
loadbalancer+ 1,roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15). Precautions: treat inventory order as configuration; never reorder casually. - Reconfigure a single service
§3,
verification of
-t <svc> reconfigure: before, a diff of/etc/kolla/haproxy/services.d/<svc>*.cfgagainst the previous version; after,docker ps --filter name=haproxyshows whether HAProxy restarted.
Known limits and open points¶
- Known bugs and limitations, row
haproxy, read against 2023.1: restart instead of reload still holds (the image'skolla_ensure_haproxy_latest_configis never called, Kolla images §5); staleservices.dfiles are still never removed (haproxy-configonly templates,roles/haproxy-config/tasks/main.yml:2-23; the only removal ismariadb.cfgwhen ProxySQL is enabled,roles/loadbalancer/tasks/config.yml:60); keepalived priority still comes from inventory order; the Ussuri upgrade bug is gone:roles/loadbalancer/tasks/upgrade.yml:2importsdeploy.yml, with no "Stopping all slave keepalived" task. Status: CODE. - Known bugs and limitations, second row
site.yml: the placement loadbalancer include still lackswhen: enable_placement(site.yml:243-246). Status: CODE. - Q5: "HAProxy restart: VIP behaviour". Status: NOT AVAILABLE.
- What is not known, row "HAProxy restart": the sources do not establish whether keepalived drops or moves the VIP, and for how long; whether the node goes to FAULT; whether established connections are cut; whether nova and neutron get DB errors and instance metadata requests fail meanwhile; how HAProxy treats backends that are not up yet on a first deploy. The Ussuri haproxy upgrade bugs no longer exist in 2023.1 (above).
- F3, "
docker_graceful_timeoutanddocker_client_timeoutare ignored": no longer holds as written in 2023.1; containers stop through their systemd unit withdocker_graceful_timeout(60 s) (R24) CODE. Still NOT AVAILABLE: whether services started by a shell wrapper that does notexecthem (e.g.neutron-l3-agent,haproxy:haproxy_run.shends inxargs /usr/sbin/haproxy,roles/loadbalancer/templates/haproxy/haproxy_run.sh.j2:7-9) receive SIGTERM at all. - fact sheet A §1.8, "Operationally
notable behaviour", items (A) to (I), read against 2023.1: any change means a full restart of haproxy, never a
reload, now staged backups first and VIP holder last (R8 above); whether a haproxy restart causes VIP failover or a
VIP outage is NOT AVAILABLE; keepalived priority depends on inventory order; stale frontends are never removed;
-t haproxyno longer runscommon(commonis its own play,site.yml:86-97); the upgrade "Stopping all slave keepalived containers" task is gone, andupgradeis the same task flow asdeploy/reconfigure(roles/loadbalancer/tasks/upgrade.yml:2,reconfigure.yml:2); non-local bind is still set by the role (net.ipv4/ipv6.ip_nonlocal_bind,roles/loadbalancer/tasks/config-host.yml:8-26, withset_sysctl); VRRP router id collisions:keepalived_virtual_router_idstill defaults to51(group_vars/all.yml:189). New in 2023.1:keepalived_traffic_mode: unicast(defaultmulticast,roles/loadbalancer/defaults/main.yml:50,keepalived.conf.j2:17-29) CODE. - fact sheet A §2.4, "Notable" for the
haproxy-configrole: stale files are not removed; the file of a service whoseenabledis false is left as it was; file name; facts are required; precedence. - Kolla images §6, item 15: the VIP follows the HAProxy stats
socket, and the ProxySQL admin socket when ProxySQL is enabled (one
check_alive_<service>.shper enabled service,roles/loadbalancer/tasks/config.yml:74-105) CODE; whether an HAProxy restart puts the node in FAULT, and for how long, is NOT AVAILABLE (keepalived semantics).
Service card¶
The rows of the card are explained in How to read a service card.
| Item | Value |
|---|---|
| Play | site.yml:99 |
| Tags | haproxy, keepalived, loadbalancer; each of the 44 per-service include_role … tasks_from: loadbalancer tasks in this play also carries its service tag |
| Play host patterns | loadbalancer |
| Node types in play (example inventory) | network |
| Node types running its containers | network |
| Containers | haproxy, proxysql, keepalived |
deploy/reconfigure flow (tasks/) |
config-host → config → check-containers |
| Task files | check-containers.yml, check.yml, config-host.yml, config.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, main.yml, precheck.yml, pull.yml, reconfigure.yml, stop.yml, upgrade.yml |
| Templates | haproxy/, haproxy/, haproxy/, keepalived/, keepalived/, keepalived/, keepalived/, proxysql/, proxysql/, proxysql/ |
| Handlers (definition order) | Check IP addresses on the API interface; Group HA nodes by status; Stop backup keepalived container; Stop backup haproxy container; Stop backup proxysql container; Start backup haproxy container; Wait for backup haproxy to start; Start backup proxysql container; Wait for backup proxysql to start; Start backup keepalived container; Stop master haproxy container; Stop master proxysql container; Stop master keepalived container; Start master haproxy container; Wait for master haproxy to start; Start master proxysql container; Wait for master proxysql to start; Start master keepalived container; Wait for haproxy to listen on VIP; Wait for proxysql to listen on VIP |
| Key variables | enable_haproxy, enable_keepalived, kolla_internal_vip_address, kolla_external_vip_address, keepalived_virtual_router_id, kolla_enable_tls_* |
| Role dependencies | none |
Related¶
- Reconfigure a single service — a service tag renders the service's frontend and restarts HAProxy if it changed
- Tags §2.2 — handlers ignore tags, which is why
-t keystonecan restart HAProxy - Component dependencies §1 — every internal, admin and public endpoint and the DB address resolve to the VIPs
- MariaDB/Galera — HAProxy sends DB traffic to one active backend
- Chrony — in Ussuri
chrony-serverwas the haproxy group and the VIP the internal time source; 2023.1 has no chrony role - Pre-execution checklist — the eight checks before every production run
- Components — the other component pages