Compute scaling and performance¶
Which Kolla-Ansible 2024.1 operations grow with the number of compute nodes, why, and how to bound them.
Note
Key answers
- No timings are given. Work is counted as operations read from the source; task-result totals and wall times are NOT AVAILABLE (§2, §4.1).
- Per compute, for the default 2024.1 service set with Cinder LVM,
nova_compute_virt_type: kvm, the Docker engine and a non-Ubuntu host, a no-change-t nova reconfiguremakes 27 remote module invocations (31 on compute001) and 3compare_containercalls; a fulldeployorreconfiguremakes 10compare_containercalls CODE (§2). Changed in 2024.1: one invocation more than 2023.1 (26, 30 on compute001), from the newhostnqnfile of nova_compute (roles/nova-cell/tasks/config.yml:203-216). Thecommonplay still does not run under-t nova(site.yml:88-103). - Adding computes adds waves, not tasks: the critical path of a play is the number of executed task-items on the slowest host × ⌈H/F⌉ waves (§4.1). Operation counts for 10 to 500 computes: §4.2.
--limitbounds the restarts, but fact gathering stays O(inventory) CODE (§6).- Why a small change becomes a large workload: a work multiplier (every run that includes the
computegroup executes the full per-host chain on every compute, even with no change) and a restart multiplier (global inputs, loop-notify, noserial) (§5). - What to do about it: the ranked optimisations (§8).
serialbounds simultaneous restarts per play, but adds time (§7).
No timings are given here. Work is expressed as operation counts read from the source and as formulas whose parameters depend on your own environment. Nothing was run against a host. The evidence labels CODE and NOT AVAILABLE are defined once, in Evidence labels. Here NOT AVAILABLE covers, for example, task-result totals and wall times. Citations that start with
lib/ansible/are read in ansible-core 2.16.20, the version studied for 2024.1 (tools/kolla-ansible:10-11accepts 2.15 to 2.16; 2.15 is not studied).
1. How ansible-core 2.16 spends time in Kolla-Ansible¶
| Factor | Mechanism | Consequence for Kolla |
|---|---|---|
| Linear strategy (default) | The strategy takes the next task of the play and queues it for every host whose next task it is (up to forks at a time), then waits until all queued results are back before it moves to the next task CODE (ansible-core 2.16.20 lib/ (next task in lockstep), :119, :129-198 (queue for each host), :217-219 (wait for the results)) |
The slowest host sets the pace of every task. One slow compute (disk, SSH, container engine) slows the whole fleet |
forks (Ansible default 5; Kolla's doc/ raises it to 20 as an example) |
Max parallel host executions per task: the task queue manager creates min(forks, batch size) worker slots for each play run, and each queued host/task takes a free slot CODE (ansible-core 2.16.20 lib/, :174-178, :320-321; lib/; default 5: lib/) |
A task on H hosts needs ⌈H/F⌉ waves |
Loops (with_dict: <role>_services, with_items) |
Items run sequentially on each host CODE (ansible-core 2.16.20 lib/, :336); each item that passes its when is a separate module execution (separate remote command); skipped items run no module |
config.yml and check-containers.yml cost items × invocation latency per host |
| Remote module invocation | Without pipelining (off by default: ansible-core 2.16.20 lib/), each invocation = SSH exec to create a tmp dir, transfer the AnsiballZ payload, run it, clean up. With become, sudo is added |
The fixed cost per invocation dominates idempotent runs |
| Deploy-host templating | template and merge_configs render on the deploy host for every host and item, then copy compares checksums remotely CODE (ansible/; ansible-core 2.16.20 lib/, :185-192, lib/) |
Deploy-host CPU scales with hosts × templates. nova.conf.j2 and neutron.conf.j2 are large, with loops over groups |
| Controller-side tasks | group_by, set_fact, include_*, meta run in Ansible workers without SSH |
Cheap per host, but not free: the 50-item group_by loop (site.yml:20-74; 56 items in 2023.1) is templated in a deploy-host worker for every host, and group_by runs no module on the host CODE (ansible-core 2.16.20 lib/). Its CPU cost per host is NOT AVAILABLE |
run_once / delegate_to |
Run once (per batch) on one host: the linear strategy queues a run_once task for the first host and stops there CODE (ansible-core 2.16.20 lib/, :208-209) |
Constant cost unless serial multiplies it |
| Handlers | Run at flush points on notified hosts: flush_handlers switches the notified hosts to the handlers state, and the linear strategy inserts the play's handlers in its task list and queues each one like a task, in parallel up to forks CODE (ansible-core 2.16.20 lib/, lib/, :129-198) |
Restart time of the slowest container on the slowest host |
until/retries loops |
e.g. compute registration wait (retries: 20, delay: 10: roles/), Keystone registration retries (5 × 10 s: roles/), wait_for |
retries: n means up to n + 1 attempts and n sleeps of delay per task, plus the time of each attempt CODE (ansible-core 2.16.20 lib/, :768-790); how many attempts a run needs is NOT AVAILABLE |
| Fact gathering | setup on every host at the start, skipped when facts are already cached (gather-facts.yml:19-25). With --limit, the limited hosts gather all other hosts' facts sequentially in a delegated loop, the hosts being divided between the limited hosts (gather-facts.yml:34-62) |
O(total hosts) serial latency per limited run without fact caching |
max_fail_percentage (new on every play in 2024.1) |
Every play now sets max_fail_percentage: <service>_max_fail_percentage | default(kolla_max_fail_percentage) | default(100) (e.g. gather-facts.yml:7-10, site.yml:96-99, nova.yml:141-144). The check breaks the play only when the failed hosts of the current batch exceed that percentage of the batch CODE (ansible-core 2.16.20 lib/; the failed-host list is cleared for each batch, lib/) |
At the default 100 it never stops a play early: one failed or slow compute does not stop the others. Setting a lower value is an operator choice; its effect on a given run is NOT AVAILABLE |
2. Work per compute host (read from the task files)¶
Counts below are read from the task files of the roles that run on a compute CODE, for the default 2024.1 service
set with Cinder LVM (enable_cinder and enable_cinder_backend_lvm set; this adds iscsid on computes:
group_vars/all.yml:750, :753-754, :794, roles/iscsi/defaults/main.yml:3-6, :62-67, inventory/multinode:305-308),
nova_compute_virt_type: kvm (group_vars/all.yml:1127), the Docker engine (kolla_container_engine: "docker",
group_vars/all.yml:92) and a non-Ubuntu host (Ubuntu adds one systemd call:
roles/nova-cell/tasks/config-host.yml:44-53). Task-result totals depend on how skipped tasks, includes and meta
tasks are counted at run time; the sources do not give them NOT AVAILABLE.
- A "task result" is one task × one host, including skipped tasks.
- A "remote invocation" is one module execution on the host, counting each executed loop item. Loop items skipped by
whenrun no module and are not counted (for example, on a compute 5 of the 8 items of eachnova_cell_servicesloop are skipped:roles/nova-cell/defaults/main.yml:2-72). Awith_first_foundtask whose files are all absent (skip: true) has no item and runs no module (roles/nova-cell/tasks/config.yml:24-34, :184-201). - Controller-only work is excluded from the invocation count:
group_by,set_fact,fail,include_*,meta. - Each
template,copyormerge_configsitem counts as one invocation, although the copy action behind them runs two modules on the host when the file is unchanged (remotestat, thenfile) CODE (ansible-core 2.16.20lib/ansible/plugins/action/copy.py:263, :285, :380). SSH round trips are therefore higher than the invocation count. - Where two figures are given, the higher one is compute001, the first host of the nova-cell play (
computeis its first host pattern:nova.yml:125-134). It also runs the delegatedrun_oncesteps: compute discovery on the first conductor, control01 (3 invocations:roles/nova-cell/tasks/wait_discover_computes.yml:8-43,get_cell_settings.yml:2-21,discover_computes.yml:14-21) and onestatofvendordata.jsonon the deploy host (roles/nova-cell/tasks/config.yml:43-48). - The
commonrole is not part of a-t novarun: it has its own play, taggedcommononly (site.yml:88-103), and the nova-cell role has nometadependency on it (the role has nometa/directory). Changed in 2024.1: its invocations per host are 20 instead of 18. They are 3 config directories, 8 templates, the newensure_imageof the fluentd image anddocker_image_infoof it (to read thefluentd_binary/fluentd_userlabels), 2 ownership fixes, 3compare_container, and thekolla_logsvolume and its link (roles/common/tasks/config.yml:2-24, :42-63, :93-102, :140-323,check-containers.yml:2-17,bootstrap.yml:2-16, withenable_fluentdandenable_rabbitmqtrue,group_vars/all.yml:765, :732). With Podman the label read usespodman_image_infoand two more steps run (/etc/tmpfiles.d/kolla.confandsystemd-tmpfiles --create:roles/common/tasks/config.yml:30-40, :73-83). These run only when thecommonplay runs (no tag or-t common). - N is the number of inventory hosts in
groups['all'], localhost included (inventory/multinode:32-33).
| Action | Task results per compute | Executed (non-skipped) | Remote module invocations per compute | compare_container calls per compute (one in-container kolla_set_configs --check exec each when the container exists) |
Source |
|---|---|---|---|---|---|
deploy (first time) |
NOT AVAILABLE | NOT AVAILABLE | NOT AVAILABLE | 10 calls, 0 execs (containers missing) | check-containers.yml of common (3), iscsi (1), openvswitch (2), neutron (1), nova-cell (3); module_utils/, module_utils/ |
deploy (no change) |
NOT AVAILABLE | NOT AVAILABLE | NOT AVAILABLE | 10 | as above |
reconfigure (no tag, no change) |
NOT AVAILABLE | NOT AVAILABLE | NOT AVAILABLE | 10 | as above |
-t nova reconfigure (no change) |
NOT AVAILABLE | NOT AVAILABLE | 27 (31 on compute001) | 3 (nova-cell only) | breakdown below |
-t nova reconfigure --limit compute001 (no change) |
NOT AVAILABLE | NOT AVAILABLE | 31 + delegated fact loop (N − 1) | 3 | breakdown below; gather-facts.yml:34-62 |
-t nova deploy-containers --limit compute001 (no change) |
NOT AVAILABLE | NOT AVAILABLE | 4 (setup + 3 compare_container) + delegated fact loop (N − 1) |
3 | roles/, roles/ |
pull (all) |
NOT AVAILABLE | NOT AVAILABLE | 11 (setup, 10 image pulls) |
0 | roles/ (one pull_image per enabled service mapped to the host) |
-t haproxy reconfigure |
4 (facts + group_by only) | 4 | 1 (setup) |
0 | plays tagged always: gather-facts.yml:5-26, :34-62 (the second play targets only limited hosts), site.yml:7-75; site.yml:105-118 (the only play tagged haproxy, the loadbalancer play, targets the loadbalancer group) |
The 10 per compute: fluentd, kolla_toolbox, cron (roles/common/defaults/main.yml:3-27), iscsid
(roles/iscsi/defaults/main.yml:3-6), openvswitch_db and openvswitch_vswitchd
(roles/openvswitch/defaults/main.yml:3-34), neutron_openvswitch_agent (roles/neutron/defaults/main.yml:26-48;
the other neutron agents are off on a compute by default, e.g. neutron_enable_ovn_agent: "no",
group_vars/all.yml:1119), nova_libvirt, nova_ssh, nova_compute (roles/nova-cell/defaults/main.yml:3-63).
The same list drives the 10 image pulls. Still holds in 2024.1.
Breakdown of the 27 remote invocations of a no-change -t nova run on one compute, read from the task files CODE
(compute001 adds 4 delegated invocations, for 31):
| Kind | Invocations | Source |
|---|---|---|
template items (module-load persistence, udev rule, config.json ×3, libvirt ×2, SASL ×3, ssh files ×4, hostnqn) |
15 | module-load, nova-cell config-host.yml + config.yml (roles/; roles/; roles/; libvirt_enable_sasl: true, roles/) |
file items (config dirs ×3) |
3 | nova-cell (roles/) |
kolla_container (compare_container ×3, remove_volume nova_libvirt_secrets) |
4 | nova-cell (roles/; roles/, imported by deploy.yml:39) |
merge_configs (nova.conf ×1 on a compute) |
1 | nova-cell config.yml (roles/; roles/) |
sysctl ×2, modprobe, setup |
4 | facts, module-load, nova-cell config-host.yml (gather-facts.yml:19-25; roles/; roles/) |
Compared with 2023.1 (26, 30 on compute001): the only new invocation is the hostnqn template, rendered for every
compute from its host name (roles/nova-cell/tasks/config.yml:203-216, templates/hostnqn.j2:1). config-host.yml,
module-load, check-containers.yml and wait_discover_computes.yml are unchanged apart from the module name
(kolla_docker → kolla_container). The nova.conf merge reads one more optional source,
<node_custom_config>/nova/<host>/<service>.conf (roles/nova-cell/tasks/config.yml:80), a deploy-host file lookup
that adds no invocation. Discovery now runs exec -t (roles/nova-cell/tasks/discover_computes.yml:17).
Note
"No change" means a cloud already deployed with 2024.1 code. The first 2024.1 run after an upgrade from 2023.1 is
not a no-change run, for two reasons read in the code CODE: the systemd unit template changed
(module_utils/kolla_systemd_worker.py:20-39) and compare_container compares the whole unit file text
(module_utils/kolla_container_worker.py:58-65, module_utils/kolla_systemd_worker.py:154-167), so a container
whose unit was written by 2023.1 code is reported as changed; and om_enable_rabbitmq_quorum_queues now defaults
to true (group_vars/all.yml:268), which adds rabbit_quorum_queue = true to the rendered nova.conf
(roles/nova-cell/templates/nova.conf.j2:207-216) of a cloud that kept the 2023.1 default. The restarts that
follow are counted in §6 only for the override case.
Per-controller counts are NOT AVAILABLE. Controllers carry more roles than computes, but they are few. Compute-side work is what scales.
2.1 Fleet totals scale linearly¶
Total task results ≈ fixed + Σ_hosts per-host count:
- Full no-change
reconfigure, 3 computes: the fleet and per-compute task-result totals are NOT AVAILABLE. - Each additional compute adds CODE (counts from §2):
- 10
compare_containercalls (10kolla_set_configs --checkexecs once its containers exist) to every fulldeploy/reconfigure; the task results and remote invocations it adds to a full run are NOT AVAILABLE; - 27 remote invocations and 3 checks to every
-t novarun; the task results it adds are NOT AVAILABLE.
- 10
The same scenarios at 10 and 30 computes, with forks 5/20/50, are in §6.
3. Expensive tasks and their scaling factors¶
Scope legend:
- D = once per deployment (
run_once, delegated); - Ctl = once per controller;
- C = once per compute;
- C×S = once per compute and per container/service;
- H = once per host of any kind;
- Cond = only when a condition holds (e.g. drift).
| Task | Role / file | Target hosts | Scope | Scaling factor | Why it can be slow | Optimisation |
|---|---|---|---|---|---|---|
Fact gathering (setup) |
gather-facts.yml:19-25 |
all | H | ⌈N/F⌉ waves | full setup collects hardware, network (many interfaces with OVS/Neutron), mounts; facter/ohai if present |
kolla_ansible_setup_filter: "ansible_[!qt]*" (skips qvo/tap interface facts), kolla_ansible_setup_gather_subset (both default to omit: group_vars/), fact caching (gathering = smart + jsonfile; with cached facts the setup task is skipped: gather-facts.yml:23-25) — documented in doc/ |
Delegated fact gathering under --limit |
gather-facts.yml:34-62 |
limited hosts gather everyone | N sequential per limited host (N/L each) | O(N) serial | delegate_to loop runs items one after another |
Fact cache; avoid --limit with a single host on huge inventories, or pre-warm the cache with kolla-ansible gather-facts without a limit (doc/) |
group_by enabled services (50 items) |
site.yml:20-74 |
all | H (deploy-host CPU) | N × 50 | templating each item | none needed below ~1,000 hosts; it saves time later by eliminating disabled plays (site.yml:4-6) |
common config + check |
roles/, check-containers.yml, bootstrap.yml |
every host of the common play (site.yml:88-103) |
H × 20 invocations (§2) | N | 3 container checks (docker exec each) + 13 templates/files + fluentd ensure_image and label read (new in 2024.1) + 2 bootstrap steps |
--limit batches; the play is tagged common only, so any service tag (e.g. -t nova) skips it, -t common runs it alone, --skip-tags common removes it |
nova-cell config |
roles/ |
computes + conductors | C × 22 invocations (17 in config.yml, 5 in config-host.yml with its module-load include, §2) |
C | merge_configs renders nova.conf.j2 on the deploy host for every compute; per-item template loops |
Deploy-host CPU (more cores, fewer forks than cores × ~2); --limit |
nova-cell check-containers |
check-containers.yml |
computes | C×S (3 containers) | 3C docker exec |
kolla_set_configs --check starts a Python process in each container |
– |
| Compute registration wait | nova-cell/ |
first compute of each batch, delegated to conductor | D per batch | up to 21 attempts with 20 × 10 s of sleep per batch; multiplied by the number of batches when serial is used |
waits until all computes of the batch appear in compute service list |
Keep batches large for this play, or run it once at the end |
discover_hosts --by-service |
discover_computes.yml |
conductor[0] | D per batch | cell size (it scans services) | one nova-manage call per batch | – |
| Keystone registration | service-ks-register |
first host | D per play/batch | per service (up to 6 attempts, 5 × 10 s of sleep: roles/) |
runs ansible inside kolla_toolbox (container exec + Python startup) |
– |
| DB-sync bootstrap containers | <role>/ |
first host | D (per batch with serial) | per service | container create + DB migration check | Skip by using deploy-containers for config-only applies |
| OVS post-config | roles/ |
network + computes | C (3 kolla_toolbox items, Set system-id, hostname and hw-offload: post-config.yml:3-23; 2 in 2023.1, the hostname item is new. The bridge and port tasks, which now use the openvswitch_bridge and openvswitch_port modules instead of the image script, run on network hosts, and on computes only when computes_need_external_bridge is true: DVR, provider networks or OVN distributed FIP, post-config.yml:25-53, group_vars/) |
C | always runs; each item runs ansible inside the kolla_toolbox container (library/) |
– |
| Neutron config-host | roles/ |
network + computes | C | C | module load (:2-13), an IPv6 sysctl -n check on every host (:15-18), sysctl settings through the sysctl role only where the L3 agent runs (:20-34) |
– |
| Neutron config + check | roles/ |
network + computes + controllers | C (per-compute count not established NOT AVAILABLE) | C | merge_configs of neutron.conf and openvswitch_agent.ini per host (roles/) |
– |
| Image pull | service-images-pull |
every host | C×S (10 images per compute with the default services and Cinder LVM, §2) | C × image size | registry bandwidth, disk I/O, layer extraction | Local registry mirror close to the computes; pre-pull before maintenance; --forks to cap registry load |
| Container (re)creation | handlers | notified hosts | Cond, C×S | number of drifted containers | stop (up to docker_graceful_timeout, 60 s by default: group_vars/), rm, create, start; kolla_set_configs on start |
Batch with --limit/serial; avoid unnecessary drift (§5) |
| nova-libvirt restart | nova-cell handler | computes | Cond (any nova-compute drift) | C | retries: 5; readiness check and SASL user re-creation after it (roles/) |
Avoid compute-side drift; batch |
| Fluentd/cron/toolbox restart | common handlers | all hosts of the common play, only when that play runs |
Cond (any service enabled/disabled) | N | – | Batch fleet-wide changes |
/etc/hosts block |
ansible-collection-kolla roles/ (bootstrap-servers) |
all | H, content O(N) | N² templating | loops over every baremetal host |
Run bootstrap-servers only when adding hosts |
| Docker daemon restart | ansible-collection-kolla roles/, notified by roles/ (daemon.json) |
all | Cond | N simultaneous | restarted by default (docker_systemd_reload: false, roles/); whether every container on the host stops is NOT AVAILABLE (R14) |
-e kolla_serial=… with bootstrap-servers (kolla-host.yml:6) |
| SSH connection setup | Ansible ssh connection plugin | all | H (handshake once per host, again after 60 s idle) | invocations × SSH round trips | ansible-core already multiplexes connections without any ansible.cfg (default ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s: ansible-core 2.16.20 lib/). Kolla ships no ansible.cfg for deployments (the only one in the tree is the CI template tests/), so pipelining stays off and each invocation still makes several round trips (tmp dir, transfer, exec, cleanup) |
pipelining = True (requires no requiretty; doc/). On large fleets, a longer persistence such as ssh_args = -C -o ControlMaster=auto -o ControlPersist=30m |
| Certificate distribution | service-cert-copy (when kolla_copy_ca_into_containers, no by default: group_vars/) |
every service config dir | C×S | per service | copy per service dir (roles/) |
Only enable when needed |
4. Performance model for 10 / 50 / 100 / 200 / 500 computes¶
4.1 Model¶
For a play whose tasks all run on the compute group (the nova-cell, openvswitch and neutron plays are dominated by it):
T_play ≈ Σ_tasks [ ceil(H_task / F_eff) × items_task × t_inv ] (remote work, per-task barrier)
+ Σ_tasks [ H_task × t_render_task / P ] (deploy-host templating, P cores)
+ Σ_delegated/run_once steps (constant per batch)
+ T_handlers (if drift) (≈ max over hosts of restart time)
H= hosts in the play (computes + the few others).F_eff= min(forks, batch size, what the deploy host CPU and SSH can sustain); the first two bounds are in the code (ansible-core 2.16.20lib/ansible/executor/task_queue_manager.py:320-321), the third is NOT AVAILABLE.t_inv= latency of one remote module invocation: SSH round trips, AnsiballZ transfer, Python start,sudo. The sources do not give it NOT AVAILABLE; it depends on your environment (the median per-task time on a single host, from-vvvtimestamps or theprofile_taskscallback).t_render= deploy-host CPU time to render a template for one host NOT AVAILABLE.
Because of the per-task barrier, the critical path of a play is (number of executed task-items on the slowest host) × ⌈H/F⌉ waves. Adding computes adds waves, not tasks.
4.2 Operation counts by fleet size¶
Built from the per-compute counts read from the code in §2 CODE. Columns whose per-compute count the sources do not give are marked n/a NOT AVAILABLE. The "Waves per task" columns give ⌈C/F⌉ for compute hosts at F = 20, 50 and 100; the worked examples below use F = 50. The waves multiply the per-host invocation chain, they do not add tasks.
| Computes (C) | Compute task results, full reconfigure (per-compute count n/a) |
Compute remote invocations, full reconfigure (per-compute count n/a) |
kolla_set_configs --check execs on computes (10·C) |
Compute task results, -t nova (per-compute count n/a) |
Invocations, -t nova (27·C) |
Waves per task at F=20 | Waves per task at F=50 | Waves per task at F=100 | Image pulls for a full image update (10·C) | Delegated fact setup calls with --limit <1 host> (N − 1 = C + 7 with 8 non-compute hosts, localhost included) |
|---|---|---|---|---|---|---|---|---|---|---|
| 10 | n/a | n/a | 100 | n/a | 270 | 1 | 1 | 1 | 100 | 17 |
| 50 | n/a | n/a | 500 | n/a | 1,350 | 3 | 1 | 1 | 500 | 57 |
| 100 | n/a | n/a | 1,000 | n/a | 2,700 | 5 | 2 | 1 | 1,000 | 107 |
| 200 | n/a | n/a | 2,000 | n/a | 5,400 | 10 | 4 | 2 | 2,000 | 207 |
| 500 | n/a | n/a | 5,000 | n/a | 13,500 | 25 | 10 | 5 | 5,000 | 507 |
Reading the table:
- No-change
-t nova reconfigure, C=500, F=50. Each compute runs a chain of 27 remote invocations. With 10 waves per task, the compute-side critical path is ≈ 270 × t_inv plus deploy-host templating (500 × nova.conf renders) and the constant delegated steps. Nothing restarts, and yet the run still costs ~13,500 remote invocations and 1,500docker execconfig checks (3 per compute). - The same run after a global change (e.g.
enable_barbican) adds a handler phase: 500 × (nova_ssh + nova_libvirt- nova_compute) = 1,500 container restarts, all in the same flush unless batched. The common containers
(fluentd, cron, kolla_toolbox) are not touched by
-t nova; they restart at the next run of thecommonplay (site.yml:88-103) when their own files change (forenable_barbican, fluentd and cron:roles/common/tasks/config.yml:140-203, :205-270).
- nova_compute) = 1,500 container restarts, all in the same flush unless batched. The common containers
(fluentd, cron, kolla_toolbox) are not touched by
- Image updates multiply data volume. A tag change downloads 10 images per compute: 5,000 image pulls at C=500.
Registry egress ≈ C × Σ(size of the new layers); image sizes are NOT AVAILABLE
(Q8 in Findings). With the default
forks/serial, this traffic is concurrent across up toforkshosts. - Fact gathering with
--limit. Gathering the 507 other hosts serially from one host costs ≈507 × t_setup before any real work. Fact caching removes it.
4.3 What is constant, per batch, or per host¶
| Constant (D) | Per batch (with serial) |
Per controller | Per compute | Per container | Conditional |
|---|---|---|---|---|---|
Keystone registration, DB create/grant, DB-sync containers, cell create/update, map_cell0, haproxy frontend rendering (per loadbalancer host) |
all run_once steps above, compute registration wait (up to 20 × 10 s of sleep), discover_hosts |
API configs, MariaDB/RabbitMQ rolling restarts (per DB/MQ node) | facts, group_by, common (when its play runs), config-host, nova-cell/OVS/neutron config, OVS and nova-cell post-config |
compare_container exec, image pull, restart/recreate |
handlers (drift), image download (new image), update_cell (URL change), Docker restart (daemon.json change) |
5. Why a small change becomes a large workload¶
Two independent multipliers apply. Still holds in 2024.1.
- Work multiplier (always). Every run that includes the
computegroup executes the full per-host chain on every compute, even with no change (§2): config re-rendering and comparison, container config checks, common role (in untagged runs), sysctl and modules, OVS post-config, discovery. Kolla has no "only if changed" short-circuit at play level. - Restart multiplier (when inputs are global). Global inputs are rendered into every compute's files (Component
dependencies §5). Loop-notify widens a single
drifted container to the whole role on that host: when one item of a looped task changes, the task result is
changed and every non-skipped item's handler is notified (ansible-core 2.16.20
lib/ansible/executor/task_executor.py:126-130,lib/ansible/plugins/strategy/__init__.py:648-671; Tags §2.4). Withoutserial, all restarts land in one handler flush.
6. Scaling at 3, 10 and 30 computes, read from the code¶
The scenarios of §2 at 3, 10 and 30 computes, for an inventory with
3 controllers, 2 network nodes, 1 storage node, 1 monitoring node and localhost (the shape of
inventory/multinode:3-33 with more computes). Counts on computes come from
§2 CODE. Cells marked n/a are NOT AVAILABLE: the sources give
neither task-result totals, nor the non-compute part of the fleet counts, nor durations.
| Scenario | Computes | forks | Task results (fleet) | per compute | compare_container calls on computes |
Restarts/recreates | nova-cell play duration | Total duration |
|---|---|---|---|---|---|---|---|---|
deploy (first time) |
3 | 20 | n/a | n/a | 30 (no check exec) | n/a | n/a | n/a |
deploy (first time) |
10 | 20 | n/a | n/a | 100 (no check exec) | n/a | n/a | n/a |
deploy (first time) |
30 | 20 | n/a | n/a | 300 (no check exec) | n/a | n/a | n/a |
-t nova reconfigure (no change) |
3 | 20 | n/a | n/a | 9 | 0 | n/a | n/a |
-t nova reconfigure (no change) |
10 | 20 | n/a | n/a | 30 | 0 | n/a | n/a |
-t nova reconfigure (no change) |
30 | 20 | n/a | n/a | 90 | 0 | n/a | n/a |
-t nova reconfigure (no change), forks 5 |
30 | 5 | n/a | n/a | 90 | 0 | n/a | n/a |
-t nova reconfigure (no change), forks 50 |
30 | 50 | n/a | n/a | 90 | 0 | n/a | n/a |
-t nova reconfigure with a nova-compute-only override |
30 | 20 | n/a | n/a | 90 | 90 (30 × nova_compute + nova_libvirt + nova_ssh) | n/a | n/a |
-t nova reconfigure --limit compute001 (no change) |
30 | 20 | n/a | n/a | 3 | 0 | n/a | n/a |
What the code shows:
- Per-compute work is constant and fleet work is linear.
- Every compute runs the same task files for the same services
(§2), so the per-compute counts do not depend on C and the
compute part of every fleet count grows by the per-compute count. The per-compute task-result counts for a deploy
and a
-t novarun are NOT AVAILABLE. - The number of
compare_containercalls grows by 10 per compute for a deploy and 3 per compute for-t nova. A call runsdocker exec kolla_set_configs --checkonly when the container exists and no other attribute differs; it then also compares the container's systemd unit file (module_utils/kolla_container_worker.py:58-65,module_utils/kolla_docker_worker.py:146-181). On a first-time deploy every container is missing, so no check exec runs; a no-change run does one exec per call CODE. With Podman the same steps go throughmodule_utils/kolla_podman_worker.py; this page counts the Docker path only. - forks changes none of these counts: it only bounds how many hosts run a task at once
(ansible-core 2.16.20
lib/ansible/executor/task_queue_manager.py:320-321) CODE.
- Every compute runs the same task files for the same services
(§2), so the per-compute counts do not depend on C and the
compute part of every fleet count grows by the per-compute count. The per-compute task-result counts for a deploy
and a
- The compute-wide plays grow with C.
- The plays that run on every compute are common, iscsi (with Cinder LVM), nova-cell, openvswitch and neutron
(
site.yml:88-103, :439-452, :611-624, :671-693;nova.yml:125-147;computeis in thecommon,iscsidandopenvswitchgroups:inventory/multinode:49-53, :105-108, :305-308) CODE. Still holds in 2024.1. The share of run time they take as C grows is NOT AVAILABLE.
- The plays that run on every compute are common, iscsi (with Cinder LVM), nova-cell, openvswitch and neutron
(
- forks and the deploy host's CPU.
- Whether, and at which forks value, the deploy host's CPU becomes the bottleneck is NOT AVAILABLE (Q7 in Findings).
- What runs on the deploy host is in the code CODE: one worker process per queued host/task, up to forks
(ansible-core 2.16.20
lib/ansible/executor/task_queue_manager.py:320-321,lib/ansible/plugins/strategy/__init__.py:387-419), module packaging (AnsiballZ) for every invocation (ansible-core 2.16.20lib/ansible/plugins/action/__init__.py:225,:300), templating ofnova.conf.j2(ansible/action_plugins/merge_configs.py:149-165) and the per-hostgroup_byloop (site.yml:20-74). - Whether forks overlap SSH and remote-execution latency in production, and up to which point, is NOT AVAILABLE.
- Size the deploy host (cores) together with forks.
- A small change restarts the fleet in one step CODE. With one nova-compute-only override on 30 computes,
loop-notify (Tags §2.4)
notifies
nova_compute,nova_libvirtandnova_sshon every compute: 90 container restarts. Withoutserialall 30 computes are in one batch (nova.yml:140,serialdefaults to "0" andtools/kolla-ansible:296setsANSIBLE_SERIAL=0; ansible-core 2.16.20lib/ansible/executor/playbook_executor.py:181-190, :286-288, :300-301), so the restarts run at one handler flush (roles/nova-cell/tasks/deploy.yml:16-17), on at mostforkshosts at a time (ansible-core 2.16.20lib/ansible/plugins/strategy/__init__.py:387-419, :952-963). How close together they land in time is NOT AVAILABLE. --limitbounds the restarts, but fact gathering stays O(inventory) CODE.- With
--limit compute001on the 30-compute inventory (38 hosts, localhost included), compute001 runs 38setupcalls in sequence: its own (gather-facts.yml:19-25), plus one delegated call for each of the 37 other inventory hosts (gather-facts.yml:34-62: with one limited host, the slicegroups['all'][0::1]is the whole inventory, minus the limited host itself,:47,:59). The delegated loop runs its items one after another (ansible-core 2.16.20lib/ansible/executor/task_executor.py:288,:336). Its duration is NOT AVAILABLE. - The delegated
discover_hosts/list_cellswork runs on the first conductor, control01 (roles/nova-cell/tasks/wait_discover_computes.yml:42-43,discover_computes.yml:21). - With SSH, multiply the delegated count by the real per-host
setuptime; at 500 computes it is ≈508 sequential fact gatherings before any work starts.
- With
7. Influence of the main variables¶
| Variable | Effect | Guidance |
|---|---|---|
| Ansible forks | Divides the number of waves per task (⌈H/F⌉). Bounded by deploy-host CPU (templating, AnsiballZ build per invocation) and memory (one worker process per queued host/task, up to forks) | Raise until deploy-host CPU saturates; where that happens is NOT AVAILABLE (§6). Size the deploy host for templating |
Serial (kolla_serial) |
Bounds simultaneous restarts per play, but adds time: every batch re-runs run_once steps and the compute registration wait, and each batch waits for the slowest host |
Use it for restart safety on changes with drift, not for no-change runs. Prefer genconfig + deploy-containers --limit batches |
max_fail_percentage (new in 2024.1) |
Set on every play, 100 unless kolla_max_fail_percentage or a per-service variable is set (etc/, commented out); compared with the failures of the current batch only (§1) |
Leave at 100 for no-change runs; a lower value makes a batch stop the play when too many of its hosts fail |
| Task delegation | delegate_to steps serialise on one host (conductor[0], neutron-server[0], rabbitmq[0]) |
That host's health gates the whole run |
| SSH overhead | Each remote invocation pays several SSH round trips (tmp dir, transfer, exec, cleanup). The connection itself is reused: ansible-core defaults to ControlMaster=auto with ControlPersist=60s (ansible-core 2.16.20 lib/), so a new handshake happens only after a host has been idle for 60 s |
pipelining = True (requires no requiretty in sudoers) is the main lever. On large fleets, a longer ControlPersist (keep -C -o ControlMaster=auto in ssh_args) and a short ControlPath |
| Docker image size | Pull time and disk I/O per host | Local registry, pre-pull, avoid unnecessary tag churn |
| Network bandwidth | Concurrent pulls from up to forks hosts |
Cap with --forks during pull; registry caching |
| Disk I/O | Image extraction, container creation, /etc/kolla writes |
SSD for /var/lib/docker |
| Container startup time | Handlers start the container through its systemd unit and wait until the unit is running, polled every 5 s (module_utils/, module_utils/), not for service readiness, except explicit waits (haproxy, OVS DB, MariaDB WSREP, RabbitMQ) |
– |
| Handler execution | Runs at the flush in parallel; docker_graceful_timeout (60 s by default, group_vars/) bounds stop time |
Batch to bound the blast radius |
| Parallel vs sequential | Hosts run in parallel within a task; items and tasks run sequentially per host; plays are sequential | Reduce items (disable unused services), use tags to skip plays, use --limit to skip hosts |
8. Bottlenecks and optimisation opportunities (ranked)¶
- Avoid running compute-wide plays when you do not need them.
- Use
-t nova-apifor API-only changes (no computes: the nova-cell play is not taggednova-api,nova.yml:110-114, :135-139). - Use
--limitfor host-scoped changes. - Use
--skip-tags haproxywhen frontends are not changing (it skips the whole loadbalancer play:site.yml:105-118).
- Use
- Separate config from apply.
genconfigonce, thendeploy-containers --limit <batch>. With-t nova, the apply step on a compute runs onlysetupand thecheck-containers.ymlloop of nova-cell (4 invocations instead of 27 for-t nova reconfigure, §2), and runs no bootstrap or Keystone steps CODE (roles/nova-cell/tasks/deploy-containers.yml:2,roles/nova/tasks/deploy-containers.yml:2,nova.yml:53-55, :89-91). In 2024.1,genconfigalso runs the common play'sensure_imageof fluentd, which pulls that image when it is absent (roles/common/tasks/config.yml:42-51). - Fact caching + setup filter. This removes the O(N) serial delegated gathering under
--limitand speeds the first play. All options are documented indoc/source/user/ansible-tuning.rst. - SSH pipelining cuts the per-invocation constant that multiplies 27–31 invocations per compute in a
-t novarun (the count for a full run is NOT AVAILABLE). A longerControlPersistthan the 60 s default avoids repeated handshakes on large fleets. - Size forks to the deploy host. Templating of large Jinja templates (
nova.conf.j2,neutron.conf.j2) happens on the deploy host, per compute. - Pre-pull images (
kolla-ansible pull) before upgrade windows, from a nearby registry. - Minimise global drift.
- Batch
enable_*changes. - Avoid touching RabbitMQ/memcached membership casually.
- Keep compute-relevant overrides in
nova/nova-compute.conf(or, new in 2024.1,nova/<host>/nova-compute.conffor one host:roles/nova-cell/tasks/config.yml:80) rather than the globalnova.conf. - Do not put server-only ML2 options in
neutron/ml2_conf.ini.
- Batch
- Keep batches large for the nova-cell play when using serial. The compute registration wait and discovery run once per batch.
- Disable services you do not use. The
group_byoptimisation removes their plays entirely (site.yml:4-6). Every enabled service adds tasks to the common role and to every host in its groups.
Related¶
- Execution flow §5 (batching)
- Why a small change creates a large workload (small change, large workload)
- Write config first, apply in controlled batches §1.1 (batched compute changes)
- Open questions (Q7, Q8: timings and image sizes)
- Kolla-Ansible Academy
To remember
Lumo: if you keep only 5 things from this page, keep these.
- With the default linear strategy, a task is queued for every host that has it next, and the next task starts only when all of them have finished: the slowest host sets the pace. CODE
- Adding computes adds waves, not tasks: a task on H hosts needs ⌈H/F⌉ waves, F being
forks(one worker per fork, at most the batch size). - In 2024.1 a no-change
-t nova reconfiguremakes 27 remote invocations per compute (one more than 2023.1: the newhostnqnfile) and 3compare_containercalls. CODE --limitbounds the restarts, but fact gathering stays O(inventory): one limited host gathers the facts of every other host, one after another. CODEserialbounds simultaneous restarts per play, but adds time;max_fail_percentage, now on every play, stops nothing early at its default of 100. CODE
Check yourself
Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.