HAProxy + Keepalived¶
This page is the starting point for HAProxy and keepalived in 2024.1. The loadbalancer role is applied by the "Apply
role loadbalancer" play to the loadbalancer group (the network nodes of the example inventory,
inventory/multinode:84-85), restricted to enable_loadbalancer_True, with play tags haproxy, keepalived and
loadbalancer and, new in 2024.1, a max_fail_percentage (site.yml:105-120). It runs the containers haproxy,
keepalived (enable_keepalived, default: follows enable_haproxy), proxysql (enable_proxysql, default no) and,
new in 2024.1, haproxy_ssh (only with enable_letsencrypt, default no)
(roles/loadbalancer/defaults/main.yml:2-36,
group_vars/all.yml:723-725, :796, :839). The same play holds 39 per-service include_role … tasks_from: loadbalancer
tasks (44 in 2023.1), run only for deploy, reconfigure, upgrade and config (site.yml:121-326); each of them
also carries its service tag. A service's loadbalancer.yml imports loadbalancer-config, which renders the HAProxy
frontend through haproxy-config and, with enable_proxysql, the ProxySQL users and rules through proxysql-config
(roles/loadbalancer-config/tasks/main.yml:2-8). The hacluster role is a separate play (site.yml:709-722) and is
not covered here; nor is the new letsencrypt play (site.yml:328-340), except where it touches HAProxy. The lists
below say where the study covers how to operate it, its impact, its dependencies and the evidence, what to check
around a run and what stays open; the service card follows them.
Note
Changed in 2024.1 CODE
- Includes: the play lost the freezer, murano, sahara, senlin, solum and vitrage includes and gained a
letsencryptone (site.yml:208-212); the rabbitmq include iswhen: enable_rabbitmq | boolonly (:288). The letsencrypt services define nohaproxyentry (roles/letsencrypt/defaults/main.yml:2-16), sohaproxy-configwrites no frontend for them (roles/haproxy-config/tasks/main.yml:17-21); the ACME challenge path is routed by horizon's frontends to anacme_clientbackend (roles/horizon/defaults/main.yml:36-37, :70-73,group_vars/all.yml:951-952). - New container
haproxy_ssh: ansshd -Donapi_interface_address:2985that accepts thehaproxy_ssh_key(roles/loadbalancer/templates/haproxy-ssh/haproxy-ssh.json.j2:2,sshd_config.j2:1-2,id_rsa.pub:1,group_vars/all.yml:408). It shares thehaproxy_socketandletsencrypt_certificatesvolumes with haproxy (roles/loadbalancer/defaults/main.yml:137-143). The lego container pushes a renewed certificate over SSH and runsupdate-haproxy-cert, which loads it through the HAProxy admin socket (set ssl cert, thencommit ssl cert), not through a restart (kolladocker/letsencrypt/letsencrypt-lego/sync-and-update-certificate.sh:83-89,docker/haproxy/haproxy-ssh/update-haproxy-cert.sh:23, :50, :123-125). That is whyhaproxy_socket_level_adminnow followsenable_letsencrypt(roles/loadbalancer/defaults/main.yml:182,templates/haproxy/haproxy_main.cfg.j2:13). - Certificates: the PEM files now live in
/etc/haproxy/certificates/, owned byhaproxy, and are copied only without Let's Encrypt (roles/loadbalancer/templates/haproxy/haproxy.json.j2:21-44,roles/loadbalancer/tasks/config.yml:211-247); with TLS on and no PEM file,haproxy_run.shfirst writes a one-day self-signed certificate (roles/loadbalancer/templates/haproxy/haproxy_run.sh.j2:3-36). The haproxy container always mountsletsencrypt_certificates:/etc/haproxy/certificates(roles/loadbalancer/defaults/main.yml:121), so its definition differs from a 2023.1 one. - Renamed variables:
haproxy_processesandhaproxy_process_cpu_mapare nowhaproxy_threadsandhaproxy_thread_cpu_map, rendered asnbthreadand a fixedcpu-map auto:1/all 0-63(roles/loadbalancer/defaults/main.yml:66-67,templates/haproxy/haproxy_main.cfg.j2:9-12). The old names are read nowhere underansible/: an override that still uses them is silently ignored. - Single external frontend (
haproxy_single_external_frontend, defaultfalse,group_vars/all.yml:931): oneexternal_frontendon the external VIP routes byHostheader through/etc/haproxy/external-frontend-map(roles/loadbalancer/templates/haproxy/haproxy_external_frontend.cfg.j2:3-12, written byroles/loadbalancer/tasks/config.yml:162-175); each service adds its block to the map (roles/haproxy-config/tasks/main.yml:25-53) and skips its own external HTTP frontend (roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:148-152). Every<svc>_public_portnow depends on it (e.g.group_vars/all.yml:335). - HTTP/2 and metrics: TLS binds add
alpn h2,http/1.1(haproxy_enable_http2: "yes",group_vars/all.yml:920-921,roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2:2-5); HAProxy itself serves Prometheus metrics onapi_interface_address:9101(roles/loadbalancer/templates/haproxy/haproxy_main.cfg.j2:45-50,group_vars/all.yml:567, :1213), andupgradefirst stops and removes the oldprometheus_haproxy_exportercontainer and its configuration (roles/loadbalancer/tasks/upgrade.yml:4-19). - Keepalived: the image's
check_alive.shlost its fallback HAProxy socket test (kolladocker/keepalived/check_alive.sh:9-33); the check scripts are still one per enabled service, never forkeepalivedorhaproxy-ssh, and the removal task now runs only withkeepalived_track_script_enabled(roles/loadbalancer/tasks/config.yml:74-111). - Handlers: the same staged dance as 2023.1, with
kolla_containerinstead ofkolla_docker, plus a lastRestart haproxy-ssh container(roles/loadbalancer/handlers/main.yml:331-345).
Where to look¶
Operate¶
- Scenario: Reconfigure a single service
§2
(
-t haproxy reconfigureon an unchanged cloud) - Command: Reconfigure a single service §1.2, Command quick lookup by tag
Impact and risk¶
- What the tag executes (
haproxy): Tags §3 and Tags §3.1, rowshaproxy; the loadbalancer play under a service tag: Tags §2.3 - Risks: R8
Dependencies¶
- Dependencies: Component dependencies §1, Component dependencies §2
Evidence¶
- Fact sheet: fact sheet A §1
(the
loadbalancerrole in 2024.1);haproxy-config: fact sheet A §2 - Image side: Kolla images §5 (rows
haproxy,keepalived) - Also: handlers: Tags §5; deployment order: Deployment execution flow
Before and after a run¶
- Pre-execution checklist, step 5 (health gates): before the run, check HAProxy backends all UP. Repeat after each batch.
- Pre-execution checklist, step 7 (scope): add
--skip-tags haproxywhen the frontends are not meant to change; it skips the whole loadbalancer play, whose three play tags arehaproxy,keepalivedandloadbalancer(site.yml:115-118). - Tags §3.1, row
haproxy: the restart handlers run only when a task reports a change (they are notified, e.g.roles/loadbalancer/tasks/config.yml:141-142,roles/loadbalancer/tasks/check-containers.yml:17-18) CODE; with a change, prefer a window. Prerequisites: facts of every backend host; certificates if TLS is on and Let's Encrypt is off (roles/loadbalancer/tasks/precheck.yml:57-98). - First 2024.1 run on a 2023.1 cloud: expect the restart handlers. The haproxy container definition gained a volume
(
roles/loadbalancer/defaults/main.yml:121), andcompare_containeralso reports a unit file written by 2023.1 (F16);check-containers.ymlthen notifiesRestart <name> containerfor each container (roles/loadbalancer/tasks/check-containers.yml:2-18), which starts the staged sequence below for haproxy and keepalived CODE. -
R8, the restart order, still staged in 2024.1. Handlers run in their definition order, and each
listens toRestart haproxy container,Restart proxysql containerorRestart keepalived containerCODE:- on every host of the batch, read the API interface addresses and group the hosts by whether they hold
kolla_internal_vip_address(roles/loadbalancer/handlers/main.yml:9-34); - on the backup hosts: stop keepalived (keepalived change only,
:36-49), stop haproxy (haproxy or keepalived change,:54-68), stop proxysql (any of the three,:70-88), start haproxy and wait forhaproxy_monitor_port(haproxy or keepalived change,:90-118), start proxysql when enabled and wait for its admin port (any of the three,:120-152), start keepalived (keepalived change only,:154-175); - on the VIP holder: stop haproxy (haproxy or keepalived change,
:179-193), stop proxysql (proxysql or keepalived change,:195-208), stop keepalived (keepalived change only,:210-222), then start them again in the same order, waiting for each port (:224-305); - after a keepalived start, wait for haproxy and, when enabled, proxysql to answer on the internal VIP
(
:307-329); - new in 2024.1 and last:
Restart haproxy-ssh container, on every host at once, with no backup/VIP split (:331-345), notified by the haproxy-ssh files, itsconfig.jsonor a container change (roles/loadbalancer/tasks/config.yml:113-124, :289-304).
Every stop and start handler is skipped for
kolla_action == "config", and the wait handlers are notified only by the start handlers. An haproxy-only change therefore stops HAProxy on the VIP holder while its keepalived keeps running; whether the VIP moves meanwhile, and for how long clients fail, is NOT AVAILABLE (keepalived semantics). Precautions: see R8;--skip-tags haproxywhen you don't intend to change frontends. - on every host of the batch, read the API interface addresses and group the hosts by whether they hold
-
R22, inventory reordering (loadbalancer, mariadb, glance, nova-conductor, keystone groups): keepalived priorities change (priority = position in
loadbalancer+ 1,roles/loadbalancer/templates/keepalived/keepalived.conf.j2:15, stillstate BACKUPandnopreempt,:11-12). Precautions: treat inventory order as configuration; never reorder casually. - Reconfigure a single service
§3,
verification of
-t <svc> reconfigure: before, a diff of/etc/kolla/haproxy/services.d/<svc>*.cfg(and, with the single external frontend, of/etc/kolla/haproxy/external-frontend-map) against the previous version; after,docker ps --filter name=haproxyshows whether HAProxy restarted. The service's frontend file and its map block both notifyRestart haproxy container(roles/haproxy-config/tasks/main.yml:2-23, :25-53) CODE. - Upgrade from 2023.1:
upgradestops and removesprometheus_haproxy_exporterbefore the deploy tasks (roles/loadbalancer/tasks/upgrade.yml:4-21);deployandreconfiguredo not (roles/loadbalancer/tasks/deploy.yml:2-6,reconfigure.yml:2). The old exporter listened on the same port 9101 that HAProxy's ownlisten metricsnow binds (group_vars/all.yml:567,roles/loadbalancer/templates/haproxy/haproxy_main.cfg.j2:45-50) CODE; what happens if adeployruns while the old container still holds the port is NOT AVAILABLE. - Overrides: rename
haproxy_processes/haproxy_process_cpu_maptohaproxy_threads/haproxy_thread_cpu_mapinglobals.ymlbefore the run (roles/loadbalancer/defaults/main.yml:66-67) CODE.
Known limits and open points¶
- Known bugs and limitations, row
haproxy, read against 2024.1: restart instead of reload still holds (the image'skolla_ensure_haproxy_latest_config, kolladocker/haproxy/haproxy/Dockerfile.j2:17, is never called fromansible/, Kolla images §5); only Let's Encrypt certificates are reloaded without a restart (above). Staleservices.dfiles are still never removed (haproxy-configonly templates,roles/haproxy-config/tasks/main.yml:2-23; the only removal ismariadb.cfgwhen ProxySQL is enabled,roles/loadbalancer/tasks/config.yml:60-72); keepalived priority still comes from inventory order;upgradeis still thedeploytask flow, now preceded by the exporter removal (roles/loadbalancer/tasks/upgrade.yml:21). Status: CODE. - New stale files in 2024.1: turning
haproxy_single_external_frontendoff leavesservices.d/external-frontend.cfgin place (it is only templated when the option is on,roles/loadbalancer/tasks/config.yml:162-175), andhaproxy_run.shloads every file ofservices.d(roles/loadbalancer/templates/haproxy/haproxy_run.sh.j2:38-40); a disabled service's block also stays inexternal-frontend-map(blockinfileruns only for enabled services,roles/haproxy-config/tasks/main.yml:47-51). Status: CODE. - Single external frontend, two points to check before enabling it CODE: the map is copied into the container
only when external TLS is on and Let's Encrypt is off (
roles/loadbalancer/templates/haproxy/haproxy.json.j2:21-28), while the frontend always reads/etc/haproxy/external-frontend-map(roles/loadbalancer/templates/haproxy/haproxy_external_frontend.cfg.j2:11); and the frontend bindshaproxy_external_single_frontend_public_port(role default"443",roles/loadbalancer/defaults/main.yml:192, used athaproxy_external_frontend.cfg.j2:10) while the endpoints usehaproxy_single_external_frontend_public_port(443with external TLS, else80,group_vars/all.yml:932). How HAProxy behaves without the map file, and whether the two ports disagree in a working setup without TLS, is NOT AVAILABLE. - Known bugs and limitations, second row
site.yml: the placement loadbalancer include still lackswhen: enable_placement(site.yml:248-251). Status: CODE. - Q5: "HAProxy restart: VIP behaviour". Status: NOT AVAILABLE.
- What is not known, row "HAProxy restart": the sources do not establish whether keepalived drops or moves the VIP, and for how long; whether the node goes to FAULT; whether established connections are cut; whether nova and neutron get DB errors and instance metadata requests fail meanwhile; how HAProxy treats backends that are not up yet on a first deploy.
- F3, "
docker_graceful_timeoutanddocker_client_timeoutare ignored": no longer holds in 2024.1; containers stop through their systemd unit withdocker_graceful_timeout(R24) CODE. Still NOT AVAILABLE: whether services started by a shell wrapper that does notexecthem (e.g.neutron-l3-agent,haproxy:haproxy_run.shends inxargs /usr/sbin/haproxy,roles/loadbalancer/templates/haproxy/haproxy_run.sh.j2:38-40) receive SIGTERM at all. - fact sheet A §1.8, "Operationally
notable behaviour", read against 2024.1: any change means a full restart of haproxy, never a reload, staged backups
first and VIP holder last (R8 above); whether a haproxy restart causes VIP failover or a VIP outage is NOT AVAILABLE; keepalived priority depends on inventory order; stale frontends are never removed;
-t haproxydoes not runcommon(commonis its own play,site.yml:88-103); non-local bind is still set by the role, now through thesysctlrole with the same values and the sameKOLLA_UNSET/KOLLA_SKIPhandling (roles/loadbalancer/tasks/config-host.yml:8-18,roles/sysctl/tasks/main.yml:7-20, withset_sysctl); VRRP router id collisions:keepalived_virtual_router_idstill defaults to51(group_vars/all.yml:224);keepalived_traffic_mode: unicastis still available (defaultmulticast,roles/loadbalancer/defaults/main.yml:62,keepalived.conf.j2:17-29) CODE. - fact sheet A §2.4, "Notable" for the
haproxy-configrole: stale files are not removed; the file of a service whoseenabledis false is left as it was; file name; facts are required; precedence. New in 2024.1: the map block per service (above). - Kolla images §6, item 15: the VIP follows the HAProxy stats
socket, and the ProxySQL admin socket when ProxySQL is enabled (one
check_alive_<service>.shper enabled service,roles/loadbalancer/tasks/config.yml:93-111;vrrp_scriptwithinterval 2,fall 2,rise 10,keepalived.conf.j2:1-8) CODE. Changed in 2024.1: with no script under/checks,check_alive.shno longer tests the HAProxy socket and reaches its end without a failing check (kolladocker/keepalived/check_alive.sh:9-33). Withkeepalived_track_script_enabled: falsethere is novrrp_scriptat all (roles/loadbalancer/defaults/main.yml:187,keepalived.conf.j2:1-8, :45-49). Whether an HAProxy restart puts the node in FAULT, and for how long, is NOT AVAILABLE (keepalived semantics). - ProxySQL, still off by default CODE:
proxysql-configis unchanged;loadbalancer-configskips the HAProxy frontend ofmariadbwhen ProxySQL is on (roles/loadbalancer-config/defaults/main.yml:2); the ProxySQL backends now usemariadb_port(roles/loadbalancer/templates/proxysql/proxysql.yaml.j2:33), which defaults todatabase_port(group_vars/all.yml:467). An haproxy change also stops and starts proxysql on the backup hosts, not on the VIP holder (roles/loadbalancer/handlers/main.yml:83-87, :140-145, :195-208).
Service card¶
The rows of the card are explained in How to read a service card.
| Item | Value |
|---|---|
| Play | site.yml:105 |
| Tags | haproxy, keepalived, loadbalancer; each of the 39 per-service include_role … tasks_from: loadbalancer tasks in this play also carries its service tag |
| Play host patterns | loadbalancer |
| Node types in play (example inventory) | network |
| Node types running its containers | network |
| Containers | haproxy, proxysql, keepalived, haproxy_ssh |
deploy/reconfigure flow (tasks/) |
config-host → config → check-containers |
| Task files | check-containers.yml, check.yml, config-host.yml, config.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, main.yml, precheck.yml, pull.yml, reconfigure.yml, stop.yml, upgrade.yml |
| Templates | haproxy-ssh/, haproxy-ssh/, haproxy/, haproxy/, haproxy/, haproxy/, keepalived/, keepalived/, keepalived/, keepalived/, proxysql/, proxysql/, proxysql/ |
| Handlers (definition order) | Check IP addresses on the API interface; Group HA nodes by status; Stop backup keepalived container; Stop backup haproxy container; Stop backup proxysql container; Start backup haproxy container; Wait for backup haproxy to start; Start backup proxysql container; Wait for backup proxysql to start; Start backup keepalived container; Stop master haproxy container; Stop master proxysql container; Stop master keepalived container; Start master haproxy container; Wait for master haproxy to start; Start master proxysql container; Wait for master proxysql to start; Start master keepalived container; Wait for haproxy to listen on VIP; Wait for proxysql to listen on VIP; Restart haproxy-ssh container |
| Key variables | enable_haproxy, enable_keepalived, kolla_internal_vip_address, kolla_external_vip_address, keepalived_virtual_router_id, kolla_enable_tls_* |
| Role dependencies | none |
Related¶
- Reconfigure a single service — a service tag renders the service's frontend and restarts HAProxy if it changed
- Tags §2.2 — handlers ignore tags, which is why
-t keystonecan restart HAProxy - Component dependencies §1 — every internal and public endpoint and the DB address resolve to the VIPs
- MariaDB/Galera — HAProxy sends DB traffic to one active backend
- Deployment execution flow — where the loadbalancer and letsencrypt plays run, and why the loadbalancer role flushes its handlers only at the end of its play
- Technical findings — F16, the first 2024.1 check of a container whose unit 2023.1 wrote
- Pre-execution checklist — the checks before every production run
- Components — the other component pages