Fact sheet A¶
Scope: roles haproxy (incl. keepalived and the per-service loadbalancer.yml fan-out), haproxy-config,
memcached, chrony, ovn, ovs-dpdk, service-cert-copy, and the certificates.yml playbook/role.
All paths are relative to the repo root ($SRC). Tag ussuri-eol, HEAD 8a94fe9.
Supported Ansible: 2.8–2.9 (tools/kolla-ansible:58-59).
How the evidence was gathered:
- Code reading. Every claim cites
path:line. - Ansible semantics. Tag inheritance, handler order and serial batches cite the Ansible 2.9.27 source as ansible
2.9.27
lib/ansible/<path>:N. Ansible's own parse ofsite.ymlis in list-tags-site.txt. - Labels. A claim with a
path:linecitation is a FACT read from the sources.NOT AVAILABLEmarks what the sources do not establish, for example timings and runtime outcomes.
Common mechanics that apply to every role below (all verified in code):
- All containers use host networking.
kolla_dockeralways uses host networking:build_host_configsets'network_mode': 'host'(ansible/library/kolla_docker.py:717-719). "net=host" is therefore never a role flag. - A config-only change means a full stop+start.
config_strategy: "COPY_ALWAYS"(ansible/group_vars/all.yml:43) is passed asKOLLA_CONFIG_STRATEGYindocker_common_options(ansible/group_vars/all.yml:157-167). With COPY_ALWAYS and no parameter drift,recreate_or_restart_containercallsrestart_container(ansible/library/kolla_docker.py:781-801), which doesdc.stop(name, timeout=graceful_timeout)thendc.start(kolla_docker.py:892-904).docker_graceful_timeout: 60(ansible/group_vars/all.yml:154) arrives only throughcommon_options, which the module defaultgraceful_timeout: 10overrides, so the effective timeout is 10 s (kolla_docker.py:991,1037-1047; stop-timeout erratum in the Per-role fact sheets). There is no "reload" path anywhere in scope. The haproxy image ships a reload helper,kolla_ensure_haproxy_latest_config(kolladocker/haproxy/Dockerfile.j2:20-21,docker/haproxy/ensure_latest_config.sh:5-14), but kolla-ansible ussuri never calls it (grep finds no caller). The helper would not reload haproxy here anyway: it signalshaproxy-systemd-wrapper(:11), whichhaproxy_run.shdoes not start (ansible/roles/haproxy/templates/haproxy_run.sh.j2:9). kolla_serialis only set for two actions. The CLI passes it only forupgradeandreconfigure(tools/kolla-ansible:430,439), fromANSIBLE_SERIAL, which defaults to0= all hosts at once (tools/kolla-ansible:236).- Handlers fire in definition order across all hosts of the batch in lockstep (linear strategy).
- With the default serial 0, a handler restarts that container on every host of the play at the same time.
- Ansible walks the play's handler list in order. It queues each notified handler on every notified host, then waits
for all results before the next handler (ansible 2.9.27
lib/ansible/plugins/strategy/__init__.py:979-987,:1019-1035). A notify sent by a handler is recorded while its results are processed (:617-632,:784;lib/ansible/playbook/handler.py:47-51), so a handler defined later and notified that way runs in the same flush.
- Handlers are not tag-filtered in Ansible 2.9.
- Tags filter only the play's compiled task blocks and dynamically included blocks (ansible 2.9.27
lib/ansible/executor/play_iterator.py:186-189,lib/ansible/plugins/strategy/linear.py:371).run_handlershas no tag check (lib/ansible/plugins/strategy/__init__.py:972-987). So under-t <service>the handlerRestart haproxy containerstill runs, although it only carries the inherited play taghaproxy.
- Tags filter only the play's compiled task blocks and dynamically included blocks (ansible 2.9.27
0. Plays in ansible/site.yml (summary table)¶
| Role | Play (lines) | Host pattern | Tags | serial |
|---|---|---|---|---|
| chrony | site.yml:114-124 |
chrony-server, chrony, &enable_chrony_True (116-119) |
role-level chrony (123) |
kolla_serial (120) |
| haproxy (+keepalived) | site.yml:126-392 |
haproxy, &enable_haproxy_True (128-130) |
play-level haproxy (132-133) |
kolla_serial (131) |
| haproxy-config | no play of its own. 49× include_role <svc> tasks_from: loadbalancer in a block, inside the haproxy play (site.yml:138-392) |
haproxy hosts | per include: <svc>; nova/nova-cell: [nova, nova-api] / [nova, nova-cell] (296-305). Plus the inherited play tag haproxy |
inherited |
| memcached | site.yml:481-490 |
memcached, &enable_memcached_True |
role-level [memcache, memcached] (489) |
kolla_serial (486) |
| ovs-dpdk | site.yml:723-732 |
openvswitch, &enable_openvswitch_True_enable_ovs_dpdk_True |
role-level ovs-dpdk (731) |
kolla_serial (728) |
| ovn | site.yml:734-746 |
ovn-controller, ovn-nb-db, ovn-northd, ovn-sb-db, &enable_ovn_True |
role-level ovn (745) |
kolla_serial (742) |
| service-cert-copy | no play. Imported from <role>/ (52 roles, including haproxy) |
caller's | caller's | caller's |
| certificates | separate playbook ansible/ (kolla-ansible certificates, tools/) |
localhost |
none | none |
- Pattern groups. The
enable_*pattern groups are created bygroup_byinsite.yml:21-95:- chrony line 30, haproxy 43, memcached 59, openvswitch/ovs_dpdk 68, ovn 70.
- Play order matters. It runs chrony (114) → haproxy (126) → … mariadb (473) → memcached (481) → …
nova.yml(710) → openvswitch (712) → ovs-dpdk (723) → ovn (734) → … neutron (~761). - Common dependency. haproxy, chrony, ovn and ovs-dpdk carry the meta dependency
- { role: common }(ansible/roles/{haproxy,chrony,ovn,ovs-dpdk}/meta/main.yml). memcached has nometa/at all, so it has no common dependency.- The common role is guarded by
when: not common_runand setscommon_run: True(ansible/roles/common/tasks/main.yml:2-10). - It has
common_run: Falseas a default (ansible/roles/common/defaults/main.yml:4).
- The common role is guarded by
1. haproxy role (haproxy + keepalived)¶
1.1 Play¶
- Play:
site.yml:126-392. Hostshaproxy∩enable_haproxy_True,serial: kolla_serial, play-leveltags: [haproxy]. - Role entry:
- { role: haproxy, when: enable_haproxy | bool }(135-136). -
The loadbalancer block. The play then has a
tasks:block of 49include_rolestatements (139-389) guarded by (390-392):when: - enable_haproxy | bool - kolla_action in ['deploy', 'reconfigure', 'upgrade', 'config'] -
Role entry point.
ansible/roles/haproxy/tasks/main.yml:2-3isinclude_tasks: "{{ kolla_action }}.yml"withwhen: inventory_hostname in groups['haproxy']. - Default inventory.
[haproxy:children] network(ansible/inventory/multinode:91-92).
1.2 haproxy_services (ansible/roles/haproxy/defaults/main.yml:2-18)¶
| key | container | group | enabled | privileged | notable volumes |
|---|---|---|---|---|---|
| haproxy | haproxy |
haproxy | true |
True | /etc/, named volume haproxy_socket:/ (50-54) |
| keepalived | keepalived |
haproxy | enable_keepalived |
True | /lib/, haproxy_socket:/ (55-60) |
- Keepalived default.
enable_keepalived: "{{ enable_haproxy | bool }}"(ansible/group_vars/all.yml:575) andenable_haproxy: "yes"(574). - Shared socket volume. The
haproxy_socketvolume is shared between both containers. The haproxy stats socket isstats socket /var/lib/kolla/haproxy/haproxy.sock(templates/haproxy_main.cfg.j2:15).- Confirmed in the kolla source: keepalived's
/check_alive.shlives in the kolla image, not in this repo, and runsecho "show info" | socat unix-connect:/var/lib/kolla/haproxy/haproxy.sock stdio(kolladocker/keepalived/check_alive.sh:6, baked in by kolladocker/keepalived/Dockerfile.j2:22,25).
- Confirmed in the kolla source: keepalived's
- No pid_mode or ipc_mode is set.
1.3 Task flow per kolla_action¶
| action | task files (in order) | notes |
|---|---|---|
| deploy | deploy.yml → config-host.yml → config.yml → check-containers.yml (deploy.yml:2-6) |
No flush_handlers. Comment deploy.yml:8-11: "haproxy role handlers should not be flushed early. site.yml handles all haproxy things in a dedicated play… See bug 1875228". The per-service loadbalancer includes then run (site.yml block) and the handlers flush once at the end of the play. |
| reconfigure | reconfigure.yml:2 → import_tasks: deploy.yml |
identical to deploy |
| upgrade | config-host.yml → config.yml → "Stopping all slave keepalived containers" → check-containers.yml (upgrade.yml:2-19) |
see 1.8 |
| deploy-containers | check-containers.yml (deploy-containers.yml:2) |
the loadbalancer block is not run (site.yml:392) |
| pull | service-images-pull (pull.yml:2-3): pulls images from haproxy_services (via lookup('vars', project_name + '_services'), ansible/) |
|
| stop | service-stop with haproxy_services (stop.yml:2-6) |
stops both haproxy and keepalived on all haproxy hosts, so no API frontend runs; whether the VIP is then released is NOT AVAILABLE. skip_stop_containers can exclude them (ansible/). |
| precheck | precheck.yml (939 lines, 79 tasks) |
service-precheck group validation (2-6). Container facts (8-14). Whole-batch detection for the VIP checks (32-40, which skip when --limit/serial does not cover all haproxy hosts). Cert existence on localhost (42-74). External VIP iface present/active (76-86). VIPs not pingable (88-101, only when keepalived runs nowhere). Free ports (103-138). VIP in the same subnet as api_interface (141-153). show stat through the socket (155-164). Then one "free port" check per service on the VIP (166-939), skipped when that frontend already exists in haproxy_stat. |
| check | check.yml is empty |
|
| config (genconfig) | config.yml only (no config-host.yml). The loadbalancer block does run ('config' is in the list, site.yml:392). |
All handlers have when: kolla_action != "config" (handlers/), so nothing restarts. The next deploy-containers will detect drift through kolla_set_configs --check and restart. |
Key tasks:
config-host.yml:2-26. These run on deploy, reconfigure and upgrade:- sysctl writes to
kolla_sysctl_conf_path(/etc/sysctl.conf,group_vars/all.yml:557), only whenset_sysctl(554):net.ipv4.ip_nonlocal_bind=1,net.ipv6.ip_nonlocal_bind=1,net.ipv4.tcp_retries2=<haproxy_host_ipv4_tcp_retries2>,net.unix.max_dgram_qlen=128. - The value
KOLLA_UNSET(the default,defaults/main.yml:76-78) meansstate: absent(config-host.yml:4-9). - Then
module-loadwithip_vs(21-26). This is loaded even whenenable_keepalivedis false, because it has nowhen.
- sysctl writes to
check-containers.yml:2-17.compare_containerfor each enabled service, notifying"Restart {{ item.key }} container".
1.4 Config files generated (ansible/roles/haproxy/tasks/config.yml)¶
| dest (host) | source / override order | notifies |
|---|---|---|
/etc/ dirs (2-13), /etc/ (15-27) |
n/a | none |
/etc/ (29-40) |
templates/, templates/ |
Restart haproxy container / Restart keepalived container |
/etc/ (42-58) |
first_found: {{node_custom_config}}/ → {{node_custom_config}}/ → haproxy_main.cfg.j2 |
Restart haproxy container |
/etc/ (60-74) |
with_fileglob: {{node_custom_config}}/ (operator custom frontends) |
Restart haproxy container |
/etc/ (76-92) |
first_found: …/ → …/ → keepalived.conf.j2 |
Restart keepalived container |
/etc/ (94-109) |
copy from kolla_external_fqdn_cert (/etc/, group_vars/), when kolla_enable_tls_external |
Restart haproxy container |
/etc/ (111-126) |
from kolla_internal_fqdn_cert (801), when kolla_enable_tls_internal |
Restart haproxy container |
/etc/ (128-130 → copy-certs.yml → service-cert-copy) |
{{kolla_certificates_dir}}/, only when kolla_copy_ca_into_containers |
both Restart haproxy container and Restart keepalived container (see section 7) |
/etc/ (132-148) |
first_found: …/ → …/ → haproxy_run.sh.j2 |
Restart haproxy container |
/etc/ |
written by haproxy-config (section 2) from each service role's loadbalancer.yml |
Restart haproxy container |
Inside the container:
- haproxy.json.j2. It copies
haproxy_run.sh,haproxy.cfg, the wholeservices.d/directory and the two pem files (marked optional when TLS is off) (templates/haproxy.json.j2:2-36). - haproxy_run.sh.j2:7-9. It runs
haproxy -W -db -p /run/haproxy.pid -f /etc/haproxy/haproxy.cfgplus one-fper file found byfind /etc/haproxy/services.d/ -mindepth 1. That is every file, not only*.cfg.- It is started in master-worker mode (
-W), which could do a SIGUSR2 reload, but Kolla never uses that.
- It is started in master-worker mode (
- haproxy_main.cfg.j2.
listen statsis onapi_interface_address:haproxy_stats_port(1984) (38-45).frontend status(monitor-uri) is onapi_interface_address:61313and onVIP:61313(47-53).haproxy_monitor_port: "61313"(group_vars/all.yml:320).
- keepalived.conf.j2:
vrrp_script check_alive { script "/check_alive.sh" interval 2 fall 2 rise 10 }(1-6).state BACKUP+nopreempt(9-10).priority {{ groups['haproxy'].index(inventory_hostname) + 1 }}(13).- Unicast peers are built from
groups['haproxy'](15-27). The VIP(s) are at 28-33.track_interfacefor the external VIP (34-38).auth_type PASS(39-42).track_script check_alive(43-45). keepalived_virtual_router_id: "51"(group_vars/all.yml:186).keepalived_traffic_mode: "multicast"(defaults/main.yml:38).
1.5 Handlers (ansible/roles/haproxy/handlers/main.yml). Definition order = run order (see common mechanics)¶
Restart haproxy container(2-20).kolla_docker: recreate_or_restart_container.when: kolla_action != "config", host ingroups['haproxy'], enabled.notify: Waiting for haproxy to start.
Restart keepalived container(22-40). Same as above;notify: Waiting for virtual IP to appear.Waiting for haproxy to start(42-45).wait_for host: api_interface_address port: haproxy_monitor_port(default 300 s timeout).- It only proves that the haproxy process listens. It says nothing about the backends.
Waiting for virtual IP to appear(47-50).wait_for host: kolla_internal_vip_address port: haproxy_monitor_port.
Other properties:
- There is no
run_once,delegate_to,listen, rolling or serial logic inside the handlers. - Restarts are parallel across all haproxy hosts unless
ANSIBLE_SERIAL/kolla_serialis set.- With
kolla_serial=1, Ansible runs the whole play, handler flushes included, once per one-host batch: one restart plus wait per host, batch by batch (ansible 2.9.27lib/ansible/executor/playbook_executor.py:161-169,:276-285; flush pointslib/ansible/playbook/play.py:276-281).
- With
- No other role notifies these handlers. The only external notifier is
ansible/roles/haproxy-config/tasks/main.yml:21(grep overansible/roles).
1.6 Bootstrap / DB / Keystone / RabbitMQ registration¶
None.
1.7 Dependencies¶
- VIP-related variables.
kolla_internal_vip_address,kolla_external_vip_address(group_vars/all.yml:50-52),haproxy_enable_external_vip(796),api_interface,kolla_external_vip_interface(223-224). - TLS:
kolla_enable_tls_{internal,external}and the cert paths (797-801). These are produced by thecertificatesplaybook (section 8) or supplied by the operator. - Everything behind the VIP. Every service's
loadbalancer.yml(section 2) builds backend lists fromgroups[<svc group>]andhostvars[host].ansible_facts.hostname, so facts are needed for all backend hosts (handled byansible/gather-facts.yml, including the--limitpath at 29-50).database_address: "{{ kolla_internal_fqdn }}"(group_vars/all.yml:105) andkolla_internal_fqdn= the VIP (51). So all DB traffic goes through haproxy.
- chrony.
chrony-server=haproxyhosts (inventory/multinode:47-48), and chrony serves on the VIP (section 4). - Logging.
syslog_server: "{{ api_interface_address }}"andsyslog_udp_port(fluentd) (defaults/main.yml:34,group_vars/all.yml:504). This needs fluentd fromcommon.
1.8 Operationally notable behaviour¶
- (A) Any change means a full restart of haproxy on all haproxy nodes at once, never a reload.
- Every haproxy-relevant change goes to
Restart haproxy container, which isrestart_container(docker stop with SIGKILL after 10 s at the latest, see the stop-timeout erratum in the Per-role fact sheets, then docker start). That includes a change in any one service'sservices.d/<svc>.cfg, a cert, haproxy.cfg or haproxy_run.sh. - With the default
serial 0, all haproxy hosts restart in the same handler step. Result: every API, MariaDB (through the VIP,group_vars/all.yml:105), RabbitMQ management, etc. is served by a haproxy that is restarting. - NOT AVAILABLE: how long the outage lasts per run, and whether established connections are cut. From the code:
haproxy_run.shdoes notexechaproxy (haproxy_run.sh.j2:7-9), so dumb-init's child is bash, not haproxy (Kolla images §2.6). Whether haproxy itself gets the SIGTERM ofdocker stopis NOT AVAILABLE; a container still running after 10 s is stopped with SIGKILL.
- Every haproxy-relevant change goes to
- (B) Whether a haproxy restart causes VIP failover or a VIP outage is NOT AVAILABLE.
- keepalived tracks
check_alivewithinterval 2 fall 2 rise 10(keepalived.conf.j2:1-6). - NOT AVAILABLE: whether keepalived goes to FAULT and drops the VIP while haproxy is down on the VIP holder, and how
long the VIP is then absent. The backups restart haproxy in the same handler step (serial 0). The "Waiting for
haproxy" handler checks
api_interface_address, not the VIP (handlers/main.yml:42-45). nopreemptis set (line 10). Which node holds the VIP afterwards is NOT AVAILABLE.- The check itself is known: the image's
/check_alive.shonly asks the haproxy stats socket forshow info(kolladocker/keepalived/check_alive.sh:6), so it fails as soon as haproxy stops answering on the socket. The FAULT andrisetiming is upstream keepalived behaviour. NOT AVAILABLE: the timing and keepalived's initial script state.
- keepalived tracks
- (C) keepalived priority depends on inventory order.
priority = index in groups['haproxy'] + 1(keepalived.conf.j2:13).- Inserting, removing or re-ordering a haproxy host changes
keepalived.confon every host after it in the list. In unicast mode it changes on all hosts (peer list, 18-25). Each changed host restarts keepalived (in parallel). Whether that moves or flaps the VIP is NOT AVAILABLE.
- (D) Stale frontends are never removed.
- Nothing in the tree deletes files in
services.d(grep: nostate: absentinroles/haproxy*). - If a service is disabled (
enable_X: no), itsinclude_roleis skipped (when: enable_X). The old/etc/kolla/haproxy/services.d/X.cfgstays and is loaded byhaproxy_run.shon the next restart. - The same is true for removed custom
node_custom_config/haproxy/services.d/*.cfgfiles. - BUG/oddity (code fact).
- Nothing in the tree deletes files in
- (E)
-t haproxyalso runscommonon haproxy hosts.- common is a meta dependency, and its tasks inherit the play tag
haproxy(ansible 2.9.27lib/ansible/playbook/block.py:351-358). Ansible's own parse listscommonamong the haproxy play's task tags (list-tags-site.txt, play #7). - So
kolla-ansible -t haproxy reconfigurealso reconfigures fluentd, cron and kolla-toolbox on the haproxy hosts (guarded only bycommon_run).
- common is a meta dependency, and its tasks inherit the play tag
-
(F) Upgrade "Stopping all slave keepalived containers" (
upgrade.yml:6-17):vars: key: "{{ 'ipv6' if api_address_family == 'ipv6' else 'ipv4_secondaries' }}" addresses: "{{ hostvars[inventory_hostname].ansible_facts[api_interface].get(key, []) | map(attribute='address') | list }}" kolla_docker: { action: "stop_container", name: "keepalived" } when: kolla_internal_vip_address not in addresses notify: [ Restart keepalived container ]- On upgrade, keepalived is stopped on every non-VIP host and recreated in the handler phase. This is the fix for
LP#1928362 (
releasenotes/notes/fix-keepalived-upgrade-a395e39dc946e618.yaml). - The handler order is: Restart haproxy on all hosts, then Restart keepalived on all hosts (including the VIP holder if its image changed), then the waits.
- Likely BUG 1. The task has no
when: enable_keepalived, andstop_containerfails when the container is missing (ignore_missingdefaults to False,kolla_docker.py:871-881,1012).- With
enable_keepalived: no(external LB,releasenotes/notes/allow-disabling-keepalived-…yaml) the VIP is on no haproxy host. The task therefore runs everywhere and should fail with "No such container: keepalived to stop". - The same applies to a haproxy host added without a keepalived container. The end-to-end upgrade outcome is NOT AVAILABLE.
- With
- Likely BUG 2. It indexes
ansible_facts[api_interface]with the raw interface name. Ansible converts-to_in fact names (ansible 2.9.27lib/ansible/module_utils/facts/namespace.py:41), whereas Kolla's ownkolla_addressfilter does the conversion (kolla_ansible/kolla_address.py:88).- So an
api_interfacelikebr-mgmtis not a key ofansible_factsunder that name. - The same pattern appears in
precheck.yml:86(ansible_facts[kolla_external_vip_interface]['active']). - NOT AVAILABLE: whether
upgrade(or that precheck) then fails at run time.
- So an
- On upgrade, keepalived is stopped on every non-VIP host and recreated in the handler phase. This is the fix for
LP#1928362 (
-
(G) Things that run on every deploy, reconfigure and upgrade:
- sysctl + modprobe (config-host).
- About 12 template/copy tasks.
- 2×
compare_container, each of which does adocker exec kolla_set_configs --check. - Plus up to 49 service includes, each compiling its role and the role's
commondependency, which is then deduped and skipped. - The cost scales with the number of haproxy hosts, not with computes.
- The
sysctltask withKOLLA_UNSETremoves any manualnet.ipv4.tcp_retries2line from/etc/sysctl.confon every run (config-host.yml:4-10).
- (H) Non-local bind is needed.
ip_nonlocal_bind=1is set only whenset_sysctlis true (config-host.yml:17-19). Backup nodes bind VIP addresses (haproxy_main.cfg.j2:50, every frontend).- NOT AVAILABLE: whether haproxy starts on non-VIP nodes with
set_sysctl: noand no manual sysctl.
- (I) VRRP router id collisions.
keepalived_virtual_router_iddefaults to51with multicast. Two Kolla clouds on the same L2 segment collide unless it is changed (group_vars/all.yml:186,defaults/main.yml:38).
1.9 What exactly happens in the haproxy play for -t <service> and -t haproxy¶
How the tags combine (the mechanics):
- Play tag. It is inherited by everything statically in the play: the role tasks, the
block, and eachinclude_rolestatement. A role task ends up with tags{haproxy}, and common's twomain.ymltasks with{common, haproxy}(ansible/roles/common/tasks/main.yml:2-3,9).tagsis an extending attribute that collects the role, dependency chain and play values (ansible 2.9.27lib/ansible/playbook/taggable.py:31,lib/ansible/playbook/block.py:329-358). - Include tags. The
include_rolestatement for keystone has tags{keystone, haproxy}. - Dynamic include boundary.
include_roleis dynamic. Ansible 2.9 does not pass the include's own tags to the included tasks: when the parent is notstatically_loaded,Task._get_parent_attributeskips to the grandparent (ansible 2.9.27lib/ansible/playbook/task.py:466-472; the same for blocks,lib/ansible/playbook/block.py:309-314; an include is not statically loaded,lib/ansible/playbook/task_include.py:56). So the tasks insidekeystone/tasks/loadbalancer.ymlwould only carry{haproxy}, and would be filtered out under-t keystone. -
The
tags: alwaysworkaround. Kolla works around this withtags: alwayson theimport_role: haproxy-configin everyloadbalancer.yml, e.g.ansible/roles/keystone/tasks/loadbalancer.yml:2-7:- name: "Configure haproxy for {{ project_name }}" import_role: name: haproxy-config vars: project_services: "{{ keystone_services }}" tags: alwaysThe same applies to
nova-cell(ansible/roles/nova-cell/tasks/loadbalancer.yml:55,91,127,proxy_loadbalancer.yml:19,cell_proxy_loadbalancer.yml:12). -
Handlers ignore tags. They run whenever they are notified (see common mechanics).
kolla-ansible -t keystone reconfigure. From the rules above and the tag test in ansible 2.9.27
lib/ansible/playbook/taggable.py:66-76 (always always matches):
- haproxy role tasks are skipped. They are tagged only
haproxy. That covers config-host, config.yml (haproxy.cfg, keepalived.conf, pems, haproxy_run.sh, custom services.d), check-containers, and thecommondependency. - Exactly one
include_roleruns: keystone's (site.yml:229-233). It is skipped ifenable_keystoneis false. The other 48include_rolestatements are skipped by tag. - haproxy-config renders
/etc/kolla/haproxy/services.d/keystone.cfgonly, on every haproxy host. Only service keys that have ahaproxy:dict and are enabled are rendered. - If and only if that file changed,
Restart haproxy containerruns at the end of the haproxy play. It runs on all haproxy hosts in parallel (serial 0), thenWaiting for haproxy to start.- keepalived is not restarted, because nothing notifies it. Whether the VIP still moves because of (B) is NOT AVAILABLE.
- The restart re-reads all
services.d/*files.- Any drift left in
/etc/kolla/haproxy/by earlier runs (e.g. a previousgenconfig, or stale files) goes live too. - Other services'
.cfgfiles are not re-rendered. If another service's backend list changed (say, a new controller), it is not updated by-t keystone.
- Any drift left in
- Cert rotation for the VIP needs
-t haproxy. The pem copy is a haproxy role task. --skip-tags haproxyskips the whole play, including all per-service includes: each one carries the inherited play tag, and a task with a skipped tag does not run (ansible 2.9.27lib/ansible/playbook/taggable.py:78-87).- Tag aliases:
-t novahits both nova and nova-cell (296-306).-t memcache(alias tag of the memcached play,site.yml:489) does not hit memcached's include, which is tagged onlymemcached(262).
kolla-ansible -t haproxy reconfigure.
commonruns on the haproxy hosts (whencommon_runis false). Then the haproxy role runsreconfigure.yml→deploy.yml: config-host, config.yml (all files in 1.4), check-containers.- All 49
include_rolestatements match, through the inherited play tag. Every enabled service'sservices.d/*.cfgis re-rendered.placement's include has nowhen(site.yml:289-292).placement-api.cfgis always written, and is empty ifenable_placementis false (ansible/roles/placement/defaults/main.yml:7,12,19).
- End-of-play handlers, in this order:
- Restart haproxy, if anything changed.
- Restart keepalived, if
keepalived.conf, keepalivedconfig.json, CA certs or container params changed. - Wait for haproxy.
- Wait for the VIP.
- No service containers (keystone etc.) are touched.
kolla-ansible reconfigure (no tags). Same as -t haproxy, plus all the other plays. It is still a single haproxy
restart per run, at the end of the haproxy play (the fix for LP#1864810/LP#1875228,
releasenotes/notes/bug-1864810-5a5d0f91c0171b19.yaml). The restart happens only if any rendered file changed.
deploy-containers and -t <svc> deploy-containers. Only haproxy check-containers.yml runs, and only under
-t haproxy or no tags. The services.d includes do not run (site.yml:392).
2. haproxy-config role¶
2.1 How it is applied¶
- Only through
import_role(withtags: always) from the 49ansible/roles/*/tasks/loadbalancer.ymlfiles, which areinclude_role'd by the haproxy play (site.yml:139-389).- Every role that has a
loadbalancer.ymlis included; a diff of the two lists is empty. nova-cellloops over cells (ansible/roles/nova-cell/tasks/proxy_loadbalancer.yml:5-19→cell_proxy_loadbalancer.yml:3-12, usingnamespace_haproxy_for_cell).
- Every role that has a
- No defaults-level
_servicesdict of its own. It consumesproject_services.
2.2 Task (ansible/roles/haproxy-config/tasks/main.yml:2-21), the only task¶
- name: "Copying over {{ project_name }} haproxy config"
vars:
service: "{{ item.value }}"
haproxy_templates:
- "{{ node_custom_config }}/haproxy-config/{{ inventory_hostname }}/{{ haproxy_service_template }}"
- "{{ node_custom_config }}/haproxy-config/{{ haproxy_service_template }}"
- "templates/{{ haproxy_service_template }}"
template_file: "{{ query('first_found', haproxy_templates) | first }}"
template:
src: "{{ template_file }}"
dest: "{{ node_config_directory }}/haproxy/services.d/{{ item.key }}.cfg"
when: [ service.enabled | bool, service.haproxy is defined, enable_haproxy | bool ]
with_dict: "{{ project_services }}"
notify: [ Restart haproxy container ]
- Template choice.
haproxy_service_templatedefaults tohaproxy_single_service_listen.cfg.j2(defaults/main.yml:2). The alternative ishaproxy_single_service_split.cfg.j2(frontend/backend split). Operators can override per host or globally under/etc/kolla/config/haproxy-config/. - Extras and health checks.
- Global extras:
haproxy_frontend_http_extra: [option httplog, option forwardfor]andhaproxy_frontend_tcp_extra: [option tcplog](defaults/main.yml:5-11). - Health check:
check inter 2000 rise 2 fall 5, or withcheck-ssl(13-14).
- Global extras:
- Backend list. From the listen template:
{% for host in groups[host_group] %} server {{ hostvars[host].ansible_facts.hostname }} {{ 'api'|kolla_address(host) }}:{{ listen_port }} …(templates/haproxy_single_service_listen.cfg.j2:73-77).host_groupdefaults toservice.group(96).custom_member_listoverrides it (57-60), as in memcached and mariadb.
- Binding and TLS.
- External entries bind
kolla_external_vip_address; internal entries bindkolla_internal_vip_address(37-52). - External entries are dropped when
haproxy_enable_external_vipis false (87). tls_backendaddsssl verify required ca-file <haproxy_backend_cacert>whenkolla_verify_tls_backend(63-69).
- External entries bind
2.3 Handler¶
It has none of its own. It notifies Restart haproxy container, defined in the haproxy role. The handler is resolvable
because the haproxy role is in the same play.
include_role also appends the included role's handlers to the play (ansible 2.9.27
lib/ansible/playbook/role_include.py:110-114).
They are never notified here, so this is harmless.
2.4 Notable¶
- Stale files are not removed (see 1.8 D).
- If a service's
enabledis false, the task is skipped and the file is left as it was. It is not emptied. - File name. The file is named after the
<svc>_serviceskey (e.g.keystone.cfg,placement-api.cfg), not the project name. - Facts are required. It depends on facts for every backend host
(
hostvars[host].ansible_facts.hostname, listen template line 74). - Precedence. Variables come from the service role's defaults, evaluated on the haproxy host. Inventory or
group_vars overrides of service variables therefore apply when set for the haproxy hosts or for
all.- NOT AVAILABLE: whether per-host overrides set only on backend hosts are seen by haproxy-config, outside the places
where the code explicitly uses
hostvars[groups[…][0]](nova-cell).
- NOT AVAILABLE: whether per-host overrides set only on backend hosts are seen by haproxy-config, outside the places
where the code explicitly uses
3. memcached role¶
3.1 Play¶
site.yml:481-490: hosts memcached ∩ enable_memcached_True, serial: kolla_serial, role tags
[memcache, memcached]. No meta dependency on common. The default inventory is [memcached:children] control
(inventory/multinode:158-159).
3.2 memcached_services (ansible/roles/memcached/defaults/main.yml:2-21)¶
| key | container | group | enabled | flags |
|---|---|---|---|---|
| memcached | memcached |
memcached | true |
not privileged; volumes are only config, localtime and timezone (37-41) |
- Haproxy entry. It has a haproxy sub-entry
memcached(tcp, port 11211) withenabled: enable_memcached and enable_haproxy_memcached(12).enable_haproxy_memcached: "no"by default (group_vars/all.yml:584-587). The comment there: clients do client-side hashing.- Its backend list is
custom_member_listbuilt fromhaproxy_members(26): the first host is active, the others arebackup.
- Tunables.
memcached_connection_limit: 5000andmemcached_max_memory: 256(MB) (46-49). - Container command:
memcached -v -l <api_interface_address> -p 11211 -c 5000 -U 0 -m 256(templates/memcached.json.j2:2), with noconfig_files.
3.3 Task flow¶
main.yml:2isinclude_tasks "{{ kolla_action }}.yml"(no host guard).- deploy =
config.yml+check-containers.yml(deploy.yml:2-4). No flush; the handler runs at the end of the memcached play. - reconfigure = deploy.
- upgrade = config + check-containers (
upgrade.yml:2-4). - deploy-containers = check-containers.
- pull = service-images-pull.
- stop = service-stop.
- precheck =
service-precheck+ free port onapi_interface_address:11211when the container is absent (precheck.yml:2-24). - check = empty.
- config = config.yml.
- Loadbalancer part:
tasks/loadbalancer.yml:2-7, through the haproxy play (site.yml:259-263, tagmemcached).
3.4 Config files¶
- Only
/etc/kolla/memcached/config.json(config.yml:13-26), which notifiesRestart memcached container. - No custom-override path (there is no config file).
- The directory task has no group guard (
config.yml:2-11).
3.5 Handlers¶
Restart memcached container(handlers/main.yml:2-14):recreate_or_restart_container,when: kolla_action != "config".- It runs on all memcached hosts in parallel at the end of the play.
3.6 Bootstrap¶
None.
3.7 Dependencies (consumers of memcached)¶
- Every service builds its memcache server list from
groups['memcached'].- 43 role files reference
groups['memcached']; 48 templates setmemcache(d)_servers. - Keystone example:
backend = oslo_cache.memcache_poolandmemcache_servers = {% for host in groups['memcached'] %}…(ansible/roles/keystone/templates/keystone.conf.j2:49-51). - Horizon uses memcached as the session store when
horizon_backend_databaseis false (SESSION_ENGINE = 'django.contrib.sessions.backends.cache',ansible/roles/horizon/templates/local_settings.j2:177-185).horizon_backend_databasedefaults toenable_murano(group_vars/all.yml:982).
- 43 role files reference
3.8 Operationally notable¶
- All memcached instances restart at once. FACT: they restart simultaneously (serial 0); the container mounts no
data volume (
defaults/main.yml:37-40) and its command has no persistence option (templates/memcached.json.j2:2).- NOT AVAILABLE: whether a restart therefore empties every cache at the same time, whether Horizon users are logged out (cache session backend), and whether the cold Keystone and keystonemiddleware token-validation caches give a load spike on Keystone and the DB.
- Triggers are rare: a change of
api_interface_address, port, memory or connection limit, or an image change on upgrade.
- Adding or removing a memcached host is expensive. It changes the
memcache_serversof almost every service config, which means a restart of most API containers on the next full reconfigure. That cost sits outside this role. - Tag aliases differ.
-t memcachereconfigures the container but not its haproxy file. Only-t memcacheddoes both. - memcached has no
commondependency. Unlike most roles, memcached does not pull incommon.
4. chrony role¶
4.1 Play¶
site.yml:114-124: hosts chrony-server ∪ chrony ∩ enable_chrony_True, serial: kolla_serial, role tag chrony,
when: enable_chrony (enable_chrony: "yes", group_vars/all.yml:597).
- Default inventory:
chrony-server= haproxy (inventory/multinode:47-48);chrony= control, network, compute, storage, monitoring (50-55). So it runs on every host, including all computes. - It is the first service play, before haproxy.
4.2 chrony_services (ansible/roles/chrony/defaults/main.yml:2-10)¶
| key | container | group | enabled | flags |
|---|---|---|---|---|
| chrony | chrony |
chrony | True |
privileged: True (it sets the host clock); volumes: config, localtime, timezone, kolla_logs (24-28). There is no persistent /var/lib/chrony volume, so the drift file is lost on recreate. |
chrony_bindaddress: "{{ kolla_internal_vip_address }}" (12).
4.3 Task flow¶
- deploy =
config.yml→check-containers.yml→meta: flush_handlers(deploy.yml:2-7). - reconfigure = deploy. upgrade = deploy (
upgrade.yml:2). - deploy-containers = check-containers.
- pull and stop use the generic roles.
- precheck = service-precheck only (
precheck.yml:2-9; a TODO notes that there is no UDP/123 check). - check = empty. config = config.yml.
4.4 Config files (ansible/roles/chrony/tasks/config.yml)¶
| dest | source | notifies |
|---|---|---|
/etc/ (19-34) |
chrony.json.j2 |
Restart chrony container |
/etc/ (36-53) |
first_found: {{node_custom_config}}/ → {{node_custom_config}}/ → chrony.conf.j2 |
Restart chrony container |
chrony.conf.j2 content:
server <VIP> iburst(3), plusserverlines for eachexternal_ntp_serversentry, on all hosts (5-9).- The default list is
0-3.pool.ntp.org(group_vars/all.yml:96-100).
- The default list is
- On
chrony-serverhosts:allow all,deny VIP,deny api_interface_address,local stratum 10(30-35). Otherwise:port 0,deny all(36-38). bindaddress {{ chrony_bindaddress }}(41).makestep 3 3(22).
4.5 Handlers¶
Restart chrony container (handlers/main.yml:2-16): recreate_or_restart_container,
when: kolla_action != "config".
It is flushed inside the chrony play.
4.6 Bootstrap¶
None.
4.7 Dependencies¶
- The VIP (keepalived, which is configured later in the same run) and the haproxy group (as
chrony-server). ip_nonlocal_bind, which is set later byhaproxy/config-host.yml.- NOT AVAILABLE: whether, on a first deploy, chronyd on a non-VIP chrony-server host can bind the VIP before
ip_nonlocal_bindis set, and whether it then serves before a later restart.
- NOT AVAILABLE: whether, on a first deploy, chronyd on a non-VIP chrony-server host can bind the VIP before
- External NTP reachability.
- On Ubuntu,
baremetalremoves the host chronyd AppArmor profile (ansible/roles/baremetal/tasks/post-install.yml:195-209).
4.8 Operationally notable¶
- Low data-plane risk.
- NOT AVAILABLE: whether a restart steps the clock.
makestep 3 3is set (chrony.conf.j2:22).
- NOT AVAILABLE: whether a restart steps the clock.
- Cost scales with the number of hosts. One template, one config.json and one
compare_containerwith adocker execrun on every compute on each run. - chrony.conf depends on the haproxy group. It changes on a host when that host's membership in
chrony-serverchanges, or when the VIP orexternal_ntp_serverschange. - The VIP is the internal time source. When the VIP fails over to another haproxy node, clients keep the same server
IP but get a different clock source.
- NOT AVAILABLE: whether reachable external NTP servers mitigate this.
5. ovn role¶
5.1 Play¶
site.yml:734-746: hosts ovn-controller ∪ ovn-nb-db ∪ ovn-northd ∪ ovn-sb-db, ∩ enable_ovn_True;
serial: kolla_serial; role tag ovn.
enable_ovn: "{{ enable_neutron | bool and neutron_plugin_agent == 'ovn' }}"(group_vars/all.yml:685).- Default inventory (
inventory/multinode:788-808):ovn-controller=ovn-controller-compute(= compute) +ovn-controller-network(= network).ovn-database= control.ovn-northd,ovn-nb-dbandovn-sb-dbare eachovn-database.
- So the play covers all computes. It runs after the openvswitch/ovs-dpdk plays and before neutron.
5.2 ovn_services (ansible/roles/ovn/defaults/main.yml:2-30)¶
| key | container | group | enabled | flags / volumes |
|---|---|---|---|---|
| ovn-controller | ovn_controller |
ovn-controller | true | /run/, kolla_logs (60-64); not privileged |
| ovn-northd | ovn_northd |
ovn-northd | true | config, localtime, logs (65-68) |
| ovn-nb-db | ovn_nb_db |
ovn-nb-db | true | named volume ovn_nb_db:/ (69-73) |
| ovn-sb-db | ovn_sb_db |
ovn-sb-db | true | named volume ovn_sb_db:/ (74-78) |
ovn_remote_probe_interval: "60000" (90).
5.3 Task flow¶
main.yml:2:include_tasks "{{ kolla_action }}.yml".- deploy =
config.yml→check-containers.yml→bootstrap.yml→meta: flush_handlers(deploy.yml:2-9). - reconfigure = deploy. upgrade = the same four steps (
upgrade.yml:2-9). - deploy-containers = check-containers only (no bootstrap, no explicit flush).
- pull/stop use the generic roles (
stop.ymlusesovn_services). - precheck = free-port checks for 6641/6642 only (
precheck.yml:2-30). There is noservice-precheckgroup validation. - check = empty. config = config.yml.
bootstrap.yml runs on every deploy, reconfigure and upgrade. Each task is a
command: docker exec openvswitch_vswitchd ovs-vsctl …
with no changed_when, so each reports changed on every run:
| task (lines) | hosts | effect |
|---|---|---|
| Create br-int (2-7) | ovn-controller | --may-exist add-br br-int -- set Bridge br-int fail-mode=secure |
| OVN encapsulation (9-16) | ovn-controller | external_ids:ovn-encap-ip=<tunnel_interface_address> ovn-encap-type=geneve ovn-remote=<ovn_sb_connection> |
| Bridge mappings (18-25) | ovn-controller-network, or computes if computes_need_external_bridge (group_vars/) |
ovn-bridge-mappings=physnet1:<br>,… from neutron_bridge_name |
| Enable chassis as gateway (27-30) | ovn-controller-network | external-ids:ovn-cms-options="enable-chassis-as-gw". This overwrites the whole ovn-cms-options value. |
| Disable gateway (32-37) | hosts in openvswitch but not in ovn-controller-network |
ovs-vsctl remove … external-ids ovn-cms-options |
| Remote probe interval (39-45) | ovn-controller | ovn-remote-probe-interval=60000 |
5.4 Config files¶
Only config.json per service (config.yml:15-26), which notifies Restart <key> container. There are no other files
and no custom-override paths. The entire configuration is on the command line:
- ovn-nb-db and ovn-sb-db (
templates/ovn-nb-db.json.j2:2,ovn-sb-db.json.j2:2):ovn-ctl run_nb_ovsdb --db-nb-create-insecure-remote=yes --db-nb-addr=<api ip> --db-nb-cluster-local-addr=<api ip>.- For every host except
groups['ovn-nb-db'][0]:--db-nb-cluster-remote-addr=<api ip of groups['ovn-nb-db'][0]>. - In other words: always a RAFT cluster, with the first inventory host as the seed. Plaintext TCP, no SSL.
- ovn-northd (
ovn-northd.json.j2:2):--ovnnb-db={{ ovn_nb_connection }} --ovnsb-db={{ ovn_sb_connection }}.- Those are lists of all DB hosts (
group_vars/all.yml:417-420).
- Those are lists of all DB hosts (
- ovn-controller (
ovn-controller.json.j2:2):ovn-controller … tcp:127.0.0.1:{{ ovsdb_port }}, i.e. the local OVSDB on 6640 (group_vars/all.yml:427). The SB remote comes fromexternal_ids:ovn-remoteset in bootstrap.
5.5 Handlers (ansible/roles/ovn/handlers/main.yml), in definition order¶
Restart ovn-nb-db container(2-15):recreate_or_restart_container.Restart ovn-sb-db container(17-30).Wait for ovn-nb-db(32-44).wait_for api_interface_address:6641, timeout 60,until success retries 10 delay 6.listen: "Restart ovn-nb-db container".
Wait for ovn-sb-db(46-58). Same, withlisten: "Restart ovn-sb-db container".Restart ovn-northd container(60-73).Restart ovn-controller container(75-88).
All handlers have when: kolla_action != "config". There is no run_once, serial or rolling logic.
- All NB members restart in the same step, then all SB members.
- The waits check only that the local TCP port is open. They do not check RAFT membership or leadership.
5.6 Bootstrap / registration¶
- No Keystone, MariaDB or RabbitMQ registration.
- Cluster creation. It is implicit, through
ovn-ctland the--db-*-cluster-*flags. - Chassis registration (
bootstrap.yml) re-runs on every deploy, reconfigure and upgrade.
5.7 Dependencies¶
- openvswitch role. It needs the container name
openvswitch_vswitchd, hard-coded in every bootstrap command (bootstrap.yml:5,12,22,29,34,42), and the local OVSDB on TCP 6640 (ansible/roles/openvswitch/templates/start-ovsdb-server.j2:11). - neutron.
ml2_conf.ini.j2:39-40andneutron_ovn_metadata_agent.ini.j2:16-17useovn_nb_connectionandovn_sb_connection.- Adding or removing a DB host therefore also needs a neutron reconfigure, and it changes the ovn-northd command
line (restart on all northd hosts) and
ovn-remoteon every chassis.
- Adding or removing a DB host therefore also needs a neutron reconfigure, and it changes the ovn-northd command
line (restart on all northd hosts) and
- Shared variables.
neutron_bridge_name(group_vars/all.yml:940),tunnel_interface_address(252),computes_need_external_bridge(949).
5.8 Operationally notable¶
- (A) The whole DB cluster restarts at once. Any change to
ovn-nb-db/ovn-sb-dbrestarts every member at once under serial 0. That covers config.json, i.e. the IP, the first-host seed or the group order, and also an image change on upgrade.- NOT AVAILABLE: whether RAFT loses quorum, how the neutron-server OVN driver and ovn-northd reconnect, and whether the data plane keeps its installed flows while the SB is unreachable.
- Mitigation:
ANSIBLE_SERIALgives a rolling restart, at the cost of making every play serial.
- (B) Whether the seed host is a single point of failure for re-deploys is NOT AVAILABLE. The DB on
groups[...][0]gets nocluster-remote-addr.- NOT AVAILABLE: what ovn-ctl does if that first host is replaced or its
ovn_nb_db/ovn_sb_dbvolume is lost, in particular whether it creates a new, separate single-node cluster instead of joining the existing one (split brain, with clients load-balancing across two clusters). - There is no code for member removal or replacement (
cluster/kick,cluster/leave), and none for backups (grep: noovsdb-client backupanywhere in roles).
- NOT AVAILABLE: what ovn-ctl does if that first host is replaced or its
- (C) ovn-controller restarts. They happen on image or container-parameter change (upgrade,
deploy-containers) or when a deploy/reconfigure rewrites itsconfig.json, and hit all computes and network nodes in parallel. The OVN manifests have noconfig_files(roles/ovn/templates/*.json.j2), sokolla_set_configs --checkalways exits 0: a command-line change written bygenconfigis applied neither bydeploy-containersnor by a later run, until the container restarts for another reason (Kolla images §3).- NOT AVAILABLE: whether Ussuri-era OVN clears and re-installs OpenFlow flows on restart, and so whether there is a short data-plane interruption per chassis, plus north-south traffic loss when all gateway chassis (network nodes) restart together.
- NOT AVAILABLE: whether there is also an SB DB reconnect storm whose size scales with the number of computes.
- (D) Bootstrap cost scales with computes. It runs 4
docker exec ovs-vsctlper compute (5 withcomputes_need_external_bridge), counted from thewhenconditions ofbootstrap.yml:2-45for a compute in the defaultovn-controller-computeandopenvswitchgroups (inventory/multinode:144-147,792-793), on every deploy, reconfigure and upgrade, always reported as changed.- NOT AVAILABLE: whether re-setting identical
external_idsis a no-op for ovn-controller, and whether a changedovn_sb_connectionmakes every chassis reconnect.
- NOT AVAILABLE: whether re-setting identical
- (E) Gateway options are clobbered.
enable-chassis-as-gwoverwritesovn-cms-optionson network nodes (bootstrap.yml:27-30).- It is removed entirely on non-gateway openvswitch hosts (32-37) on every run.
- Any manual
ovn-cms-optionsvalue, such as availability-zone hints, is overwritten or removed on every reconfigure (the code above). The impact of that loss is NOT AVAILABLE. - Bridge mappings are only ever set, never removed (18-25).
- (F) Security. NB and SB listen with
create-insecure-remote=yeson plain TCP. - (G) OVN + DPDK cannot work in this release (inference from code):
- With
enable_ovs_dpdkthe openvswitch play is skipped (site.yml:716). - ovs-dpdk containers are only enabled when
neutron_plugin_agent in ['openvswitch'](ansible/roles/ovs-dpdk/defaults/main.yml:10,24). - OVN bootstrap execs into
openvswitch_vswitchd.
- With
- (H) Missing inventory groups are not caught. There is no group validation precheck. An old inventory without the
ovn-*groups only surfaces as Jinja errors ongroups['ovn-…'].
6. ovs-dpdk role¶
6.1 Play¶
site.yml:723-732: hosts openvswitch ∩ enable_openvswitch_True_enable_ovs_dpdk_True, serial: kolla_serial,
role tag ovs-dpdk.
enable_ovs_dpdk: "no"(group_vars/all.yml:686).- The openvswitch play is mutually exclusive with it (
site.yml:712-721). - Default
openvswitchgroup = network, compute, manila-share (inventory/multinode:144-147).
6.2 ovsdpdk_services (ansible/roles/ovs-dpdk/defaults/main.yml:6-35)¶
| key | container | mapping | enabled | flags / volumes |
|---|---|---|---|---|
| ovsdpdk-db | ovsdpdk_db |
host_in_groups: compute, manila-share (if enabled), neutron-dhcp-agent, -l3-agent, -metadata-agent (11-18) |
neutron_plugin_agent in ['openvswitch'] |
/run/, named volume ovsdpdk_db:/ (75-81) |
| ovsdpdk-vswitchd | ovsdpdk_vswitchd |
same expression (26-33) | same | privileged: True; /lib/modules:ro, /run/, /dev:/ (82-89) |
- Container commands:
start-ovsdb-server 127.0.0.1andovs-vswitchd unix:/run/openvswitch/db.sock … --mlockall(templates/*.json.j2). neutron_bridge_name: "br_dpdk"(56) is a role default.- Ansible merges role defaults first and group variables after them (ansible 2.9.27
lib/ansible/vars/manager.py:186-190,205-206,290-293), so it is shadowed by playbookgroup_vars/all.yml:940(br-ex). The role default is dead unless set in globals.
- Ansible merges role defaults first and group variables after them (ansible 2.9.27
- Templating bug in
ovs_cidr_mappings(44). For multiple bridges it emits the literal text{neutron_bridge_name.split(',')[0]}(single braces) instead of a Jinja expression.
6.3 Task flow¶
- deploy =
config.yml→check-containers.yml→meta: flush_handlers(deploy.yml:2-7). - reconfigure and upgrade = deploy.
- deploy-containers = check-containers.
- precheck = service-precheck (
precheck.yml:2-6). - check = empty.
- config (genconfig) = config.yml. This includes host-mutating NIC binding (see below).
- pull = BUG.
- The
vars/main.yml:2valueproject_name: "ovs"makesservice-images-pulllook upovs_services(ansible/roles/service-images-pull/tasks/main.yml:14), which does not exist anywhere (grep). - For an undefined name and no
default, thevarslookup raisesNo variable found with this name: ovs_services(ansible 2.9.27lib/ansible/plugins/lookup/vars.py:80-94). - The lookup runs while Ansible builds the
with_dictloop. ThePull imagestask has nowhenof its own, and the role'swhenis true on DPDK hosts (site.yml:730-732), so Ansible re-raises that error and the task fails on DPDK hosts (ansible 2.9.27lib/ansible/executor/task_executor.py:105-109,179-180,566-568). The end-to-end outcome ofkolla-ansible pullis NOT AVAILABLE.
- The
- stop = service-stop with
ovsdpdk_services(works).
config.yml key tasks (all of them run on deploy, reconfigure, upgrade and genconfig):
- Config dirs and config.json (2-25), gated by
enabledandhost_in_groups.- The
config.jsoncopy has nobecome, although the dirs are root:root 0770 (config_owner_user: root,group_vars/all.yml:20-21). - NOT AVAILABLE: whether this fails with a non-root
ansible_user.
- The
- "Copying ovs-dpdkctl tool" (27-31).
copy src: ../tools/ovs-dpdkctl.sh. It has nobecomeand nowhen. - "Install ovs-dpdkctl service and config" (33-44).
command: ovs-dpdkctl.sh install, which always reports changed and has nowhen. Insidetools/ovs-dpdkctl.sh:gen_configruns only if the conf file does not exist (350-353). Later changes to bridge, port or cidr mappings in globals are silently ignored.modprobeof the DPDK driver.- Install and enable the
ovs-dpdkctlsystemd unit (bind_nicson boot). - Rewrite the ifcfg or interfaces file for the bridge every run, and on RedHat start
ovs-dpdk-bridge(ovs-dpdkctl.sh:218-258, 270-318, 350-367).
- "Binds the interface to the target driver" (46-57).
ovs-dpdkctl.sh bind_nicsunbinds and rebinds the NICs listed inport_mappingswhen their current driver differs (ovs-dpdkctl.sh:116-127).- This runs even under
kolla-ansible genconfig.
- This runs even under
6.4 Config files¶
/etc/kolla/ovsdpdk-{db,vswitchd}/config.json, which notify their restart handlers./etc/kolla/ovsdpdk-db/ovs-dpdkctl.shand/etc/kolla/ovsdpdk-db/ovs-dpdkctl.conf(generated once).- Systemd units
/etc/systemd/system/ovs-dpdkctl.serviceandovs-dpdk-bridge.service, plus the network ifcfg or interfaces file. These notify no handler. - There are no custom-override paths.
6.5 Handlers (ansible/roles/ovs-dpdk/handlers/main.yml), in definition order¶
Restart ovsdpdk-db container(2-22). It notifies 6 handlers: wait db, bridges indexed, Restart vswitchd, bridges named, wait tunnel ip, gather facts.Waiting the ovs db service to be ready(24-34).docker exec ovsdpdk_db ovs-vsctl --no-wait show, retries 30 delay 2.Ensuring ovsdpdk bridges are properly setup indexed(36-52).ovs-dpdkctl.sh init, only ifovs_physical_port_policy == 'indexed'.Restart ovsdpdk-vswitchd container(54-68).Ensuring ovsdpdk bridges are properly setup named(70-86).init, when the policy isnamed(the default, line 54).wait for dpdk tunnel ip(88-92).wait_for: host: <dpdk_tunnel_interface_address> timeout: 300 sleep: 5, with no port and no path.wait_forthen just doestime.sleep(timeout)(ansible 2.9.27lib/ansible/modules/utilities/logic/wait_for.py:538-539).- So every DB restart costs a fixed 300 s sleep per batch.
ovs-dpdk gather facts(94-102).setupre-gathers facts.
6.6 Bootstrap¶
OVS DB init (ovs-dpdkctl.sh init: other_config:dpdk-*, bridges, DPDK ports, ovs-dpdkctl.sh:164-216) runs only
from handlers, i.e. when ovsdpdk-db restarts. It is not re-asserted on a plain reconfigure.
6.7 Dependencies¶
- neutron (openvswitch agent with
ovs_datapath: netdev,group_vars/all.yml:715). - Hugepages (
ovs_hugepage_mountpoint). dpdk_tunnel_interface(defaults toneutron_external_interface,group_vars/all.yml:234) and its address (254).neutron_plugin_agent == 'openvswitch'.
6.8 Operationally notable¶
- Any
ovsdpdk-dbrestart also restartsovs-vswitchd. That means an image change or config.json drift.- NOT AVAILABLE: whether this is a data-plane outage for all VMs on that host (vhost-user reconnect).
- It happens on all DPDK computes in parallel, followed by a mandatory 5-minute sleep.
- NIC rebinding and unit installation run on every deploy, reconfigure, upgrade and genconfig. They are idempotent only as long as the driver already matches.
- Sticky configuration.
ovs-dpdkctl.confis never regenerated, so mapping changes require deleting it manually.
7. service-cert-copy role¶
7.1 How it is applied¶
import_role: service-cert-copyfrom<role>/tasks/copy-certs.ymlin 52 roles (aodh … zun, pluscommonandhaproxy).- Usually it is gated with
when: kolla_copy_ca_into_containers | bool or <svc>_enable_tls_backend | bool, e.g.ansible/roles/keystone/tasks/config.yml:41-43. - haproxy gates only on
kolla_copy_ca_into_containers(ansible/roles/haproxy/tasks/config.yml:128-130). - chrony, memcached, ovn and ovs-dpdk do not use it.
7.2 Tasks (ansible/roles/service-cert-copy/tasks/main.yml)¶
| task (lines) | dest | when | notifies |
|---|---|---|---|
| Copy extra CA certs (2-12) | /etc/ from {{kolla_certificates_dir}}/ |
kolla_copy_ca_into_containers |
Restart <item.key> container |
| Backend TLS cert (14-33) | /etc/. first_found: certificates/ → certificates/ → certificates/ → kolla_tls_backend_cert |
the service has an enabled haproxy entry with tls_backend true |
Restart <item.key> container |
| Backend TLS key (35-54) | …/ (mode 0600), with the same lookup order for -key.pem |
same | same |
The loop is with_dict: project_services | select_services_enabled_and_mapped_to_host
(kolla_ansible/filters.py:82-91).
7.3 Notable¶
- For haproxy, the loop includes
keepalived.- A CA bundle change (e.g. a root CA rotation with
kolla_copy_ca_into_containers: yes) therefore triggersRestart keepalived containeron all haproxy hosts in parallel, together with the haproxy restart. - Whether that flaps the VIP is NOT AVAILABLE.
keepalived.json.j2does not reference the CA files. - The kolla image does install them: keepalived starts through
kolla_start, which runskolla_copy_cacertsat every start. That script deletes the oldkolla-customca-*entries and installs every file of/var/lib/kolla/config_files/ca-certificates/(kolladocker/base/start.sh:13,docker/base/copy_cacerts.sh:7-27; Kolla images §2.4).
- A CA bundle change (e.g. a root CA rotation with
- A CA or backend cert rotation restarts every container of every service using it (whole fleet, per-play parallel).
- It is a pure file copy, with no expiry check.
8. certificates.yml playbook / certificates role¶
8.1 Play¶
ansible/certificates.yml:1-8:
import_playbook: gather-facts.ymlwithwhen: kolla_enable_tls_backend(to obtain the API IPs oftls-backendhosts).- Then
hosts: localhost,roles: [certificates]. - No serial, no tags, no
become. It runs fromkolla-ansible certificates(tools/kolla-ansible:454-457). - The docs call it a "private test Certificate Authority" (
doc/source/admin/tls.rst:65-66).
8.2 Flow (ansible/roles/certificates/tasks/main.yml:2-6)¶
generate-root.yml → generate.yml → generate-backend.yml (when kolla_enable_tls_backend).
- Root CA (
generate-root.yml).root.keyRSA 4096 androot.crt, CN=KollaTestCA, 1024 days, each guarded bycreates:(14-33).- The CA is copied to
{{kolla_certificates_dir}}/ca/root.crt(41-45). That is the source thatservice-cert-copydistributes.
- External and internal certs (
generate.yml).- Key, CSR and cert with 365 days, each
creates:-guarded. - They are assembled into
kolla_external_fqdn_certandkolla_internal_fqdn_cert(14-80, 82-137). - The internal cert is a copy of the external one when the VIPs are the same (70-80).
- SAN is a single DNS or IP entry from
kolla_{external,internal}_fqdn(templates/openssl-kolla*.cnf.j2:15-21).
- Key, CSR and cert with 365 days, each
- Backend cert (
generate-backend.yml).- One shared cert (500 days) whose SANs are the API IPs of all
tls-backendhosts (templates/openssl-kolla-backend.cnf.j2:15-18;[tls-backend:children] control,inventory/multinode:42-43). - Copied to
backend-cert.pemandbackend-key.pem(54-64).
- One shared cert (500 days) whose SANs are the API IPs of all
- Global paths:
kolla_certificates_dir: "{{ node_config }}/certificates"(group_vars/all.yml:799, normally/etc/kolla/certificates).
8.3 Notable¶
- No rotation or renewal. Every generating step is
creates:-guarded, so re-running never renews expired certificates. External and internal certs expire after 365 days; the files must be deleted manually. - Adding a
tls-backendhost does not update the backend SANs.backend.csrandbackend.crtexist (generate-backend.yml:32,46), so the cert is not re-issued.- NOT AVAILABLE: whether haproxy then rejects the new backend with
kolla_verify_tls_backend: yes(group_vars/all.yml:804).
- NOT AVAILABLE: whether haproxy then rejects the new backend with
- Oddity in key generation. The backend key task writes
private/backend/backend.keybut itscreates:checkskolla_tls_backend_key(generate-backend.yml:16-21).- If an operator pre-supplies
backend-key.pem, keygen is skipped but the CSR step still needsbackend.key.
- If an operator pre-supplies
- Distribution needs a separate run. Certs only reach containers through
kolla-ansible -t haproxy reconfigure(pem files) or a per-service reconfigure (backend and CA). Each of those means a container restart (sections 1.4 and 7).
9. Cross-cutting summary: restart triggers and blast radius¶
| Component | Restarted when | Hosts at once (default) | Blast radius |
|---|---|---|---|
| haproxy | any services.d/haproxy.cfg/run.sh/pem/CA/config.json change, image change |
all haproxy hosts | all APIs plus MariaDB and other VIP traffic; VIP failover: NOT AVAILABLE |
| keepalived | keepalived.conf (inventory order!), config.json, CA copy, image change, every upgrade (backups stopped) | all haproxy hosts | VIP move or outage: NOT AVAILABLE |
| memcached | config.json (IP, port, memory, limit), image | all memcached hosts | whether all caches are emptied and Horizon users logged out: NOT AVAILABLE |
| chrony | chrony.conf (VIP, NTP servers, chrony-server membership), image | all hosts incl. computes | low |
| ovn-nb/sb-db | config.json (IPs, seed host = groups[...][0]), image |
all DB hosts | NB/SB API down; quorum re-election: NOT AVAILABLE |
| ovn-northd | NB/SB host list, image | all northd hosts | logical-flow processing pause |
| ovn-controller | image, drift | all computes and network nodes | data-plane blip per chassis: NOT AVAILABLE |
| ovsdpdk-db → vswitchd | image, drift (db restart always restarts vswitchd) | all DPDK hosts | VM data-plane outage: NOT AVAILABLE; then a 300 s sleep |
Things that run on every reconfigure, regardless of changes:
- haproxy config-host (sysctl/modprobe) and the template set, plus 49 service includes (
-t haproxyor no tags). - OVN bootstrap
ovs-vsctl(per chassis). - ovs-dpdk
install+bind_nics(per DPDK host, also on genconfig). compare_containerwith adocker exec kolla_set_configs --checkfor every container on every host.
Things whose cost scales with the number of computes: chrony, ovn-controller config + bootstrap, ovs-dpdk. haproxy, memcached, the OVN DB and certificates scale only with control or haproxy hosts.