Compute scaling and performance¶
Which Kolla-Ansible Ussuri operations grow with the number of compute nodes, why, and how to bound them.
Note
Key answers
- No timings are given. Work is counted as operations read from the source; task-result totals and wall times are NOT AVAILABLE (§2, §4.1).
- Per compute, for the default Ussuri service set with Cinder LVM,
nova_compute_virt_type: kvmand a CentOS host, a no-change-t nova reconfiguremakes 46 remote module invocations (53 on compute001) and 6compare_containercalls; a fulldeployorreconfiguremakes 11compare_containercalls CODE (§2). - Adding computes adds waves, not tasks: the critical path of a play is the number of executed task-items on the slowest host × ⌈H/F⌉ waves (§4.1). Operation counts for 10 to 500 computes: §4.2.
--limitbounds the restarts, but fact gathering stays O(inventory) CODE (§6).- Why a small change becomes a large workload: a work multiplier (every run that includes the
computegroup executes the full per-host chain on every compute, even with no change) and a restart multiplier (global inputs, loop-notify, noserial) (§5). - What to do about it: the ranked optimisations (§8).
serialbounds simultaneous restarts per play, but adds time (§7).
No timings are given here. Work is expressed as operation counts read from the source and as formulas whose parameters depend on your own environment. Nothing was run against a host. The evidence labels CODE and NOT AVAILABLE are defined once, in Evidence labels. Here NOT AVAILABLE covers, for example, task-result totals and wall times.
1. How Ansible 2.9 spends time in Kolla-Ansible¶
| Factor | Mechanism | Consequence for Kolla |
|---|---|---|
| Linear strategy (default) | A task runs on all hosts of the play (up to forks at a time), and the next task starts only when every host has finished the current one CODE (ansible 2.9.27 lib/) |
The slowest host sets the pace of every task. One slow compute (disk, SSH, Docker) slows the whole fleet |
| forks (Ansible default 5; Kolla docs suggest 20–100) | Max parallel host executions per task: one worker process per fork, at most the batch size CODE (ansible 2.9.27 lib/) |
A task on H hosts needs ⌈H/F⌉ waves |
Loops (with_dict: <role>_services, with_items) |
Items run sequentially on each host CODE (ansible 2.9.27 lib/); each item that passes its when is a separate module execution (separate remote command); skipped items run no module |
config.yml and check-containers.yml cost items × invocation latency per host |
| Remote module invocation | Without pipelining, each invocation = SSH exec to create a tmp dir, transfer the AnsiballZ payload, run it, clean up. With become, sudo is added |
The fixed cost per invocation dominates idempotent runs |
| Deploy-host templating | template and merge_configs render on the deploy host for every host and item, then copy compares checksums remotely CODE (ansible/; ansible 2.9.27 lib/, lib/) |
Deploy-host CPU scales with hosts × templates. nova.conf.j2 and neutron.conf.j2 are large, with loops over groups |
| Controller-side tasks | group_by, set_fact, include_*, meta run in Ansible workers without SSH |
Cheap per host, but not free: the 69-item group_by loop (site.yml:21-95) is templated in a deploy-host worker for every host, and group_by runs no module on the host CODE (ansible 2.9.27 lib/). Its CPU cost per host is NOT AVAILABLE |
run_once / delegate_to |
Run once (per batch) on one host | Constant cost unless serial multiplies it |
| Handlers | Run at flush points on notified hosts, in parallel up to forks CODE (ansible 2.9.27 lib/) |
Restart time of the slowest container on the slowest host |
until/retries loops |
e.g. compute registration wait (20 × 10 s), Keystone registration retries (5 × 10 s), wait_for |
Add up to retries × delay of sleep per task, plus the time of each attempt CODE (ansible 2.9.27 lib/); how many attempts a run needs is NOT AVAILABLE |
| Fact gathering | setup on every host at the start (gather-facts.yml:5-21). With --limit, the limited hosts gather all other hosts' facts sequentially in a delegated loop (gather-facts.yml:29-50) |
O(total hosts) serial latency per limited run without fact caching |
2. Work per compute host (read from the task files)¶
Counts below are read from the task files of the roles that run on a compute CODE, for the default Ussuri service
set with Cinder LVM (which adds iscsid on computes: group_vars/all.yml:601, :651,
roles/iscsi/defaults/main.yml:67-72), nova_compute_virt_type: kvm (group_vars/all.yml:966) and a CentOS host
(Ubuntu adds one systemd call: roles/nova-cell/tasks/config-host.yml:48-56). Task-result totals depend on how
skipped tasks, includes and meta tasks are counted at run time; the sources do not give them NOT AVAILABLE.
- A "task result" is one task × one host, including skipped tasks.
- A "remote invocation" is one module execution on the host, counting each executed loop item. Loop items skipped by
whenrun no module and are not counted (for example, on a compute at least 5 of the 8 items of eachnova_cell_servicesloop are skipped:roles/nova-cell/defaults/main.yml:2-64). - Controller-only work is excluded from the invocation count:
group_by,set_fact,include_*. - Each
template,copyormerge_configsitem counts as one invocation, although the copy action behind them runs two modules on the host when the file is unchanged (remotestat, thenfile) CODE (ansible 2.9.27lib/ansible/plugins/action/copy.py:266, :288, :383). SSH round trips are therefore higher than the invocation count. - Where two figures are given, the higher one is compute001, the first host of the nova-cell play (
nova.yml:117-120). It also runs the delegatedrun_oncesteps: compute discovery on the first conductor, control01 (3 invocations:roles/nova-cell/tasks/wait_discover_computes.yml:8-42,get_cell_settings.yml:2-20,discover_computes.yml:14-21) and, in-t novaruns, fourfindcalls on the deploy host (roles/common/tasks/config.yml:73-107). - N is the number of inventory hosts in
groups['all'], localhost included (inventory/multinode:32-33).
| Action | Task results per compute | Executed (non-skipped) | Remote module invocations per compute | compare_container calls per compute (one docker exec kolla_set_configs --check each when the container exists) |
Source |
|---|---|---|---|---|---|
deploy (first time) |
NOT AVAILABLE | NOT AVAILABLE | NOT AVAILABLE | 11 calls, 0 execs (containers missing) | check-containers.yml of common (3), chrony, iscsi, openvswitch (2), neutron, nova-cell (3); library/ |
deploy (no change) |
NOT AVAILABLE | NOT AVAILABLE | NOT AVAILABLE | 11 | as above |
reconfigure (no tag, no change) |
NOT AVAILABLE | NOT AVAILABLE | NOT AVAILABLE | 11 | as above |
-t nova reconfigure (no change) |
NOT AVAILABLE | NOT AVAILABLE | 46 (53 on compute001) | 6 (3 common + 3 nova) | breakdown below |
-t nova reconfigure --limit compute001 (no change) |
NOT AVAILABLE | NOT AVAILABLE | 53 + delegated fact loop (N − 1) | 6 | breakdown below; gather-facts.yml:29-50 |
-t nova deploy-containers --limit compute001 (no change) |
NOT AVAILABLE | NOT AVAILABLE | 7 (setup + 6 compare_container) + delegated fact loop (N − 1) |
6 | roles/, roles/ |
pull (all) |
NOT AVAILABLE | NOT AVAILABLE | 13 (setup, CentOS slurp, 11 image pulls) |
0 | site.yml:97-105; roles/ |
-t haproxy reconfigure |
5 (facts + group_by only) | 5 | 1 (setup) |
0 | docs/ (plays tagged always); site.yml:126-136 (the only play tagged haproxy targets the haproxy group) |
Breakdown of the 46 remote invocations of a no-change -t nova run on one compute, read from the task files CODE
(compute001 adds 7 delegated invocations, for 53):
| Kind | Invocations | Source |
|---|---|---|
template items (config.json ×3 common, td-agent, logrotate, module-load persistence, udev rule, config.json nova-cell ×3, libvirt ×2, SASL ×3, ssh files ×4) |
19 | common, module-load, nova-cell config-host.yml + config.yml (roles/; roles/; roles/; roles/) |
file items (config dirs ×3 common and ×3 nova-cell, ownership ×2, kolla_logs link) |
9 | common + nova-cell (roles/; roles/; roles/) |
kolla_docker (compare_container ×6, ensure_image, create_volume) |
8 | check-containers, common (roles/; roles/; roles/; roles/) |
merge_configs (nova.conf ×1 on a compute) |
1 | nova-cell config.yml (roles/; roles/) |
sysctl ×4, modprobe, stat, copy, setup, docker_image_info |
9 | facts, common, module-load, nova-cell config-host.yml (gather-facts.yml:10-15; roles/; roles/; roles/) |
Per-controller counts are NOT AVAILABLE. Controllers carry more roles than computes, but they are few. Compute-side work is what scales.
2.1 Fleet totals scale linearly¶
Total task results ≈ fixed + Σ_hosts per-host count:
- Full no-change
reconfigure, 3 computes: the fleet and per-compute task-result totals are NOT AVAILABLE. - Each additional compute adds CODE (counts from §2):
- 11
compare_containercalls (11kolla_set_configs --checkexecs once its containers exist) to every fulldeploy/reconfigure; the task results and remote invocations it adds to a full run are NOT AVAILABLE; - 46 remote invocations and 6 checks to every
-t novarun; the task results it adds are NOT AVAILABLE.
- 11
The same scenarios at 10 and 30 computes, with forks 5/20/50, are in §6.
3. Expensive tasks and their scaling factors¶
Scope legend:
- D = once per deployment (
run_once, delegated); - Ctl = once per controller;
- C = once per compute;
- C×S = once per compute and per container/service;
- H = once per host of any kind;
- Cond = only when a condition holds (e.g. drift).
| Task | Role / file | Target hosts | Scope | Scaling factor | Why it can be slow | Optimisation |
|---|---|---|---|---|---|---|
Fact gathering (setup) |
gather-facts.yml:5-21 |
all | H | ⌈N/F⌉ waves | full setup collects hardware, network (many interfaces with OVS/Neutron), mounts; facter/ohai if present |
kolla_ansible_setup_filter: "ansible_[!qt]*" (skips qvo/tap interface facts), kolla_ansible_setup_gather_subset, fact caching (gathering = smart + jsonfile) — documented in doc/ |
Delegated fact gathering under --limit |
gather-facts.yml:29-50 |
limited hosts gather everyone | N sequential per limited host (N/L each) | O(N) serial | delegate_to loop runs items one after another |
Fact cache; avoid --limit with a single host on huge inventories, or pre-warm the cache with a full facts run |
group_by enabled services (69 items) |
site.yml:21-95 |
all | H (deploy-host CPU) | N × 69 | templating each item | none needed below ~1,000 hosts; it saves time later by eliminating disabled plays |
common config + check |
roles/, check-containers.yml |
every host, in the first matching play | H × ~18 invocations | N | 3 container checks (docker exec each) + ~12 templates/files |
--limit batches; it cannot be skipped selectively (-t common is a no-op) |
nova-cell config |
roles/ |
computes + conductors | C × ~24 invocations (with config-host.yml and its module-load include) |
C | merge_configs renders nova.conf.j2 on the deploy host for every compute; per-item template loops |
Deploy-host CPU (more cores, fewer forks than cores × ~2); --limit |
nova-cell check-containers |
check-containers.yml |
computes | C×S (3 containers) | 3C docker exec |
kolla_set_configs --check starts a Python process in each container |
– |
| Compute registration wait | nova-cell/ |
first compute of each batch, delegated to conductor | D per batch | up to 20 × 10 s per batch; multiplied by the number of batches when serial is used |
waits until all computes of the batch appear in compute service list |
Keep batches large for this play, or run it once at the end |
discover_hosts --by-service |
discover_computes.yml |
conductor[0] | D per batch | cell size (it scans services) | one nova-manage call per batch | – |
| Keystone registration | service-ks-register |
first host | D per play/batch | per service (×5 retries max) | runs ansible inside kolla_toolbox (container exec + Python startup) |
– |
| DB-sync bootstrap containers | <role>/ |
first host | D (per batch with serial) | per service | container create + DB migration check | Skip by using deploy-containers for config-only applies |
| OVS post-config | roles/ |
network + computes | C (1 docker exec, Set system-id; the bridge check kolla_ensure_openvswitch_configured runs on network hosts, and on computes only when computes_need_external_bridge is true: DVR, provider networks or OVN distributed FIP) |
C | always runs; Set system-id always "changed" |
– |
| Neutron config-host | roles/ |
network + computes | C | C | module load, sysctl | – |
| Neutron config + check | roles/ |
network + computes + controllers | C × ~5 | C | merge_configs of neutron.conf and openvswitch_agent.ini per host |
– |
| Image pull | service-images-pull |
every host | C×S (11 images per compute with the default services and Cinder LVM, §2) | C × image size | registry bandwidth, disk I/O, layer extraction | Local registry mirror close to the computes; pre-pull before maintenance; --forks to cap registry load |
| Container (re)creation | handlers | notified hosts | Cond, C×S | number of drifted containers | stop (10 s max), rm, create, start; kolla_set_configs on start |
Batch with --limit/serial; avoid unnecessary drift (§5) |
| nova-libvirt restart | nova-cell handler | computes | Cond (any nova-compute drift) | C | retries: 5; SASL user re-creation after it |
Avoid compute-side drift; batch |
| Fluentd/cron/toolbox restart | common handlers | all hosts reached | Cond (any service enabled/disabled) | N | – | Batch fleet-wide changes |
/etc/hosts template |
roles/baremetal (bootstrap-servers) |
all | H, content O(N) | N² templating | loops over every baremetal host | Run bootstrap-servers only when adding hosts |
| Docker daemon restart | roles/ |
all | Cond | N simultaneous | restarts every container on the host | -e kolla_serial=… with bootstrap-servers |
| SSH connection setup | Ansible ssh connection plugin | all | H (handshake once per host, again after 60 s idle) | invocations × SSH round trips | Ansible 2.9 already multiplexes connections without any ansible.cfg (default ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s). Kolla ships no ansible.cfg, so pipelining stays off and each invocation still makes several round trips (tmp dir, transfer, exec, cleanup) |
pipelining = True (requires no requiretty). On large fleets, a longer persistence such as ssh_args = -C -o ControlMaster=auto -o ControlPersist=30m |
| Certificate distribution | service-cert-copy (when kolla_copy_ca_into_containers) |
every service config dir | C×S | per service | copy per service dir |
Only enable when needed |
4. Performance model for 10 / 50 / 100 / 200 / 500 computes¶
4.1 Model¶
For a play whose tasks all run on the compute group (the nova-cell, openvswitch and neutron plays are dominated by it):
T_play ≈ Σ_tasks [ ceil(H_task / F_eff) × items_task × t_inv ] (remote work, per-task barrier)
+ Σ_tasks [ H_task × t_render_task / P ] (deploy-host templating, P cores)
+ Σ_delegated/run_once steps (constant per batch)
+ T_handlers (if drift) (≈ max over hosts of restart time)
H= hosts in the play (computes + the few others).F_eff= min(forks, what the deploy host CPU and SSH can sustain).t_inv= latency of one remote module invocation: SSH round trips, AnsiballZ transfer, Python start,sudo. The sources do not give it NOT AVAILABLE; it depends on your environment (the median per-task time on a single host, from-vvvtimestamps or theprofile_taskscallback).t_render= deploy-host CPU time to render a template for one host NOT AVAILABLE.
Because of the per-task barrier, the critical path of a play is (number of executed task-items on the slowest host) × ⌈H/F⌉ waves. Adding computes adds waves, not tasks.
4.2 Operation counts by fleet size¶
Built from the per-compute counts read from the code in §2 CODE. Columns whose per-compute count the sources do not give are marked n/a NOT AVAILABLE. The "Waves per task" columns give ⌈C/F⌉ for compute hosts at F = 20, 50 and 100; the worked examples below use F = 50. The waves multiply the per-host invocation chain, they do not add tasks.
| Computes (C) | Compute task results, full reconfigure (per-compute count n/a) |
Compute remote invocations, full reconfigure (per-compute count n/a) |
kolla_set_configs --check execs on computes (11·C) |
Compute task results, -t nova (per-compute count n/a) |
Invocations, -t nova (46·C) |
Waves per task at F=20 | Waves per task at F=50 | Waves per task at F=100 | Image pulls for a full image update (11·C) | Delegated fact setup calls with --limit <1 host> (N − 1 = C + 7 with 8 non-compute hosts, localhost included) |
|---|---|---|---|---|---|---|---|---|---|---|
| 10 | n/a | n/a | 110 | n/a | 460 | 1 | 1 | 1 | 110 | 17 |
| 50 | n/a | n/a | 550 | n/a | 2,300 | 3 | 1 | 1 | 550 | 57 |
| 100 | n/a | n/a | 1,100 | n/a | 4,600 | 5 | 2 | 1 | 1,100 | 107 |
| 200 | n/a | n/a | 2,200 | n/a | 9,200 | 10 | 4 | 2 | 2,200 | 207 |
| 500 | n/a | n/a | 5,500 | n/a | 23,000 | 25 | 10 | 5 | 5,500 | 507 |
Reading the table:
- No-change
-t nova reconfigure, C=500, F=50. Each compute runs a chain of 46 remote invocations. With 10 waves per task, the compute-side critical path is ≈ 460 × t_inv plus deploy-host templating (500 × nova.conf renders) and the constant delegated steps. Nothing restarts, and yet the run still costs ~23,000 remote invocations and 3,000docker execconfig checks (6 per compute). - The same run after a global change (e.g.
enable_barbican) adds a handler phase: 500 × (nova_ssh + nova_libvirt- nova_compute + 3 common containers) = 3,000 container restarts, all in the same flush unless batched.
- Image updates multiply data volume. A tag change downloads 11 images per compute: 5,500 image pulls at C=500.
Registry egress ≈ C × Σ(size of the new layers); image sizes are NOT AVAILABLE
(Q8 in Findings). With the default
forks/serial, this traffic is concurrent across up toforkshosts. - Fact gathering with
--limit. Gathering the 507 other hosts serially from one host costs ≈507 × t_setup before any real work. Fact caching removes it.
4.3 What is constant, per batch, or per host¶
| Constant (D) | Per batch (with serial) |
Per controller | Per compute | Per container | Conditional |
|---|---|---|---|---|---|
Keystone registration, DB create/grant, DB-sync containers, cell create/update, map_cell0, haproxy frontend rendering (per haproxy host) |
all run_once steps above, compute registration wait (≤200 s), discover_hosts |
API configs, MariaDB/RabbitMQ rolling restarts (per DB/MQ node) | facts, group_by, common, config-host, nova-cell/OVS/neutron config, OVS post-config |
compare_container exec, image pull, restart/recreate |
handlers (drift), image download (new image), update_cell (URL change), Docker restart (daemon.json change) |
5. Why a small change becomes a large workload¶
Two independent multipliers apply:
- Work multiplier (always). Every run that includes the
computegroup executes the full per-host chain on every compute, even with no change (§2): config re-rendering and comparison, container config checks, common role, sysctl and modules, OVS post-config, discovery. Kolla has no "only if changed" short-circuit at play level. - Restart multiplier (when inputs are global). Global inputs are rendered into every compute's files (Component
dependencies §5). Loop-notify widens a single
drifted container to the whole role on that host (Tags
§2.4). Without
serial, all restarts land in one handler flush.
6. Scaling at 3, 10 and 30 computes, read from the code¶
The scenarios of §2 at 3, 10 and 30 computes, for an inventory with 3 controllers, 2 network nodes, 1 storage node, 1 monitoring node and localhost. Counts on computes come from §2 CODE. Cells marked n/a are NOT AVAILABLE: the sources give neither task-result totals, nor the non-compute part of the fleet counts, nor durations.
| Scenario | Computes | forks | Task results (fleet) | per compute | compare_container calls on computes |
Restarts/recreates | nova-cell play duration | Total duration |
|---|---|---|---|---|---|---|---|---|
deploy (first time) |
3 | 20 | n/a | n/a | 33 (no check exec) | n/a | n/a | n/a |
deploy (first time) |
10 | 20 | n/a | n/a | 110 (no check exec) | n/a | n/a | n/a |
deploy (first time) |
30 | 20 | n/a | n/a | 330 (no check exec) | n/a | n/a | n/a |
-t nova reconfigure (no change) |
3 | 20 | n/a | n/a | 18 | 0 | n/a | n/a |
-t nova reconfigure (no change) |
10 | 20 | n/a | n/a | 60 | 0 | n/a | n/a |
-t nova reconfigure (no change) |
30 | 20 | n/a | n/a | 180 | 0 | n/a | n/a |
-t nova reconfigure (no change), forks 5 |
30 | 5 | n/a | n/a | 180 | 0 | n/a | n/a |
-t nova reconfigure (no change), forks 50 |
30 | 50 | n/a | n/a | 180 | 0 | n/a | n/a |
-t nova reconfigure with a nova-compute-only override |
30 | 20 | n/a | n/a | 180 | 90 (30 × nova_compute + nova_libvirt + nova_ssh) | n/a | n/a |
-t nova reconfigure --limit compute001 (no change) |
30 | 20 | n/a | n/a | 6 | 0 | n/a | n/a |
What the code shows:
- Per-compute work is constant and fleet work is linear.
- Every compute runs the same task files for the same services
(§2), so the per-compute counts do not depend on C and the
compute part of every fleet count grows by the per-compute count. The per-compute task-result counts for a deploy
and a
-t novarun are NOT AVAILABLE. - The number of
compare_containercalls grows by 11 per compute for a deploy and 6 per compute for-t nova. A call runsdocker exec kolla_set_configs --checkonly when the container exists and no other attribute differs (kolla_docker.py:326-332). On a first-time deploy every container is missing, so no check exec runs; a no-change run does one exec per call CODE. - forks changes none of these counts: it only bounds how many hosts run a task at once
(ansible 2.9.27
lib/ansible/executor/task_queue_manager.py:257-258) CODE.
- Every compute runs the same task files for the same services
(§2), so the per-compute counts do not depend on C and the
compute part of every fleet count grows by the per-compute count. The per-compute task-result counts for a deploy
and a
- The compute-wide plays grow with C.
- The plays that run on every compute include chrony (with common), nova-cell, openvswitch and neutron
(
docs/day2/ussuri/reference/listings/list-tasks-nova.txt:127, :226, :268, :280;computeis in the chrony and openvswitch groups:inventory/multinode:50-55, :144-147) CODE. The share of run time they take as C grows is NOT AVAILABLE.
- The plays that run on every compute include chrony (with common), nova-cell, openvswitch and neutron
(
- forks and the deploy host's CPU.
- Whether, and at which forks value, the deploy host's CPU becomes the bottleneck is NOT AVAILABLE (Q7 in Findings).
- What runs on the deploy host is in the code CODE: one worker process per fork (ansible 2.9.27
lib/ansible/executor/task_queue_manager.py:257-258,lib/ansible/plugins/strategy/__init__.py:387-390), module packaging (AnsiballZ) for every invocation (ansible 2.9.27lib/ansible/plugins/action/__init__.py:155, :206), templating ofnova.conf.j2(ansible/action_plugins/merge_configs.py:151-161) and the per-hostgroup_byloop (site.yml:21-95). - Whether forks overlap SSH and remote-execution latency in production, and up to which point, is NOT AVAILABLE.
- Size the deploy host (cores) together with forks.
- A small change restarts the fleet in one step CODE. With one nova-compute-only override on 30 computes,
loop-notify (Tags §2.4)
notifies
nova_compute,nova_libvirtandnova_sshon every compute: 90 container restarts. Withoutserialall 30 computes are in one batch (nova.yml:132,serialdefaults to "0"; ansible 2.9.27lib/ansible/executor/playbook_executor.py:161-169, :263-264, :276-278), so the restarts run at one handler flush (roles/nova-cell/tasks/deploy.yml:21-22), on at mostforkshosts at a time (ansible 2.9.27lib/ansible/plugins/strategy/__init__.py:359-391, :1019-1035). How close together they land in time is NOT AVAILABLE. --limitbounds the restarts, but fact gathering stays O(inventory) CODE.- With
--limit compute001on the 30-compute inventory (38 hosts, localhost included), compute001 runs 38setupcalls in sequence: its own (gather-facts.yml:10-15), plus one delegated call for each of the 37 other inventory hosts (gather-facts.yml:29-50). The delegated loop runs its items one after another (ansible 2.9.27lib/ansible/executor/task_executor.py:317, :367). Its duration is NOT AVAILABLE. - The delegated
discover_hosts/list_cellswork runs on the first conductor, control01 (roles/nova-cell/tasks/wait_discover_computes.yml:41-42,discover_computes.yml:21). - With SSH, multiply the delegated count by the real per-host
setuptime; at 500 computes it is ≈508 sequential fact gatherings before any work starts.
- With
7. Influence of the main variables¶
| Variable | Effect | Guidance |
|---|---|---|
| Ansible forks | Divides the number of waves per task (⌈H/F⌉). Bounded by deploy-host CPU (templating, AnsiballZ build per invocation) and memory (one worker process per fork) | Raise until deploy-host CPU saturates; where that happens is NOT AVAILABLE (§6). Size the deploy host for templating |
Serial (kolla_serial) |
Bounds simultaneous restarts per play, but adds time: every batch re-runs run_once steps and the compute registration wait, and each batch waits for the slowest host |
Use it for restart safety on changes with drift, not for no-change runs. Prefer genconfig + deploy-containers --limit batches |
| Task delegation | delegate_to steps serialise on one host (conductor[0], neutron-server[0], rabbitmq[0]) |
That host's health gates the whole run |
| SSH overhead | Each remote invocation pays several SSH round trips (tmp dir, transfer, exec, cleanup). The connection itself is reused: Ansible 2.9 defaults to ControlMaster=auto with ControlPersist=60s, so a new handshake happens only after a host has been idle for 60 s |
pipelining = True (requires no requiretty in sudoers) is the main lever. On large fleets, a longer ControlPersist (keep -C -o ControlMaster=auto in ssh_args) and a short ControlPath |
| Docker image size | Pull time and disk I/O per host | Local registry, pre-pull, avoid unnecessary tag churn |
| Network bandwidth | Concurrent pulls from up to forks hosts |
Cap with --forks during pull; registry caching |
| Disk I/O | Image extraction, container creation, /etc/kolla writes |
SSD for /var/lib/docker |
| Container startup time | Handlers wait for docker start (not for service readiness), except explicit waits (haproxy, OVS DB, MariaDB WSREP, RabbitMQ) |
– |
| Handler execution | Runs at the flush in parallel; the 10 s SIGKILL bounds stop time | Batch to bound the blast radius |
| Parallel vs sequential | Hosts run in parallel within a task; items and tasks run sequentially per host; plays are sequential | Reduce items (disable unused services), use tags to skip plays, use --limit to skip hosts |
8. Bottlenecks and optimisation opportunities (ranked)¶
- Avoid running compute-wide plays when you do not need them.
- Use
-t nova-apifor API-only changes (no computes). - Use
--limitfor host-scoped changes. - Use
--skip-tags haproxywhen frontends are not changing.
- Use
- Separate config from apply.
genconfigonce, thendeploy-containers --limit <batch>. With-t nova, the apply step on a compute runs onlysetupand thecheck-containers.ymlloops of common and nova-cell (7 invocations instead of 46 for-t nova reconfigure, §2), and runs no bootstrap or Keystone steps CODE (roles/common/tasks/deploy-containers.yml:2,roles/nova-cell/tasks/deploy-containers.yml:2,roles/nova/tasks/deploy-containers.yml:2,nova.yml:49-51, :85-87). - Fact caching + setup filter. This removes the O(N) serial delegated gathering under
--limitand speeds the first play. All options are documented in Ussuriansible-tuning.rst. - SSH pipelining cuts the per-invocation constant that multiplies 46–53 invocations per compute in a
-t novarun (the count for a full run is NOT AVAILABLE). A longerControlPersistthan the 60 s default avoids repeated handshakes on large fleets. - Size forks to the deploy host. Templating of large Jinja templates (
nova.conf.j2,neutron.conf.j2) happens on the deploy host, per compute. - Pre-pull images (
kolla-ansible pull) before upgrade windows, from a nearby registry. - Minimise global drift.
- Batch
enable_*changes. - Avoid touching RabbitMQ/memcached membership casually.
- Keep compute-relevant overrides in
nova/nova-compute.confrather than the globalnova.conf. - Do not put server-only ML2 options in
neutron/ml2_conf.ini.
- Batch
- Keep batches large for the nova-cell play when using serial. The compute registration wait and discovery run once per batch.
- Disable services you do not use. The
group_byoptimisation removes their plays entirely (site.yml:4-6). Every enabled service adds tasks to the common role and to every host in its groups.
Related¶
- Execution flow §5 (batching)
- Why a small change creates a large workload (small change, large workload)
- Write config first, apply in controlled batches §1.1 (batched compute changes)
- Open questions (Q7, Q8: timings and image sizes)
- Kolla-Ansible Academy
To remember
Lumo: if you keep only 5 things from this page, keep these.
- With the default linear strategy, the next task starts only when every host has finished the current one: the slowest host sets the pace. CODE
- Adding computes adds waves, not tasks: a task on H hosts needs ⌈H/F⌉ waves, F being
forks. --limitbounds the restarts, but fact gathering stays O(inventory). CODE- Every run that includes the
computegroup executes the full per-host chain on every compute, even with no change. serialbounds simultaneous restarts per play, but adds time.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.