Write config first, apply in controlled batches¶
Impact Writes config Restarts on drift
How to write the configuration first and apply it batch by batch, with Nova as the example. genconfig, a review of the
diff, then deploy-containers --limit <batch> is the only built-in way to separate "write the config" from "apply it"
(Execution flow §4). The page
also covers a rolling reconfigure with ANSIBLE_SERIAL, and how to make it stop on failures with
max_fail_percentage, new on every play in 2024.1.
The page uses the definition of drift of the Operations hub.
Note
Key answers
- Compute changes in controlled batches:
-t nova-cell genconfig, thendeploy-containers --limitbatch by batch (§1.1). Rolling runs withANSIBLE_SERIAL: §1.2. - Stop a rolling run on failures: lower
kolla_max_fail_percentage(or<service>_max_fail_percentage); the default 100 never stops a play early (§1.3). - What changed since 2023.1: etcd is no longer batched by
ANSIBLE_SERIAL, outward RabbitMQ is gone, and every play carriesmax_fail_percentage. The pattern itself, the drift check and its blind spots still hold.
1. Commands¶
1.1 Compute changes in controlled batches (recommended pattern)¶
# 1. write the config everywhere, restart nothing
kolla-ansible -i multinode -t nova-cell genconfig
# 2. review: diff /etc/kolla/nova-compute/nova.conf on a sample of hosts
# 3. apply batch by batch (only check-containers + handlers run)
kolla-ansible -i multinode -t nova-cell deploy-containers --limit 'compute[0:9]'
kolla-ansible -i multinode -t nova-cell deploy-containers --limit 'compute[10:19]'
genconfigruns the playbooks withkolla_action=config(tools/kolla-ansible:557-560), anddeploy-containerswithkolla_action=deploy-containers(tools/kolla-ansible:499-502). Neither setskolla_serial.-
deploy-containersruns onlycheck-containers.ymland the handlers it notifies CODE (roles/nova-cell/tasks/deploy-containers.yml:2). These restart each container with drift written bygenconfigand, by loop-notify, the other containers of that role on the same host: the nova-cell check is one looped task whosenotifynamesRestart {{ item.key }} container(roles/nova-cell/tasks/check-containers.yml:2-22), and ansible-core 2.16.20 saves the notifications of every item of a looped task once the task as a whole is changed (lib/ansible/plugins/strategy/__init__.py:648-661) CODE (Tags §2.4).Drift is found by
compare_container: the container is missing, its parameters differ,kolla_set_configs --checkreports a change, or the systemd unit text differs (module_utils/kolla_container_worker.py:58-65). Throughkolla_set_configs --checkit sees only drift in the files listed inconfig.json'sconfig_files(existence, content, perm, owner) CODE (kolladocker/base/set_configs.py:165-198,docker/base/set_configs.py:220-242,docker/base/set_configs.py:556-559). It does not see changes toconfig.json's owncommand/permissions(e.g. memcached: emptyconfig_files,roles/memcached/templates/memcached.json.j2:1-4), and no laterdeploy/reconfigurerestarts for those either, because theconfig.jsontemplate task is then unchanged; they take effect at the next restart for another reason. Still holds in 2024.1: kolla'sset_configs.pyis byte-identical to 2023.1 (Kolla images §3).A newly rendered
optionalfile whose destination is not yet in the container (e.g. nova-compute'srelease) counts as drift:--checkexits 1 and the restart copies the new file (kolladocker/base/set_configs.py:165-171,roles/nova-cell/templates/nova-compute.json.j2:52-58). Exit 1 means "changed" for both container engines (Dockermodule_utils/kolla_docker_worker.py:164-172, Podmanmodule_utils/kolla_podman_worker.py:442-449). -
Each batch restarts only its hosts. No bootstrap containers and no Keystone registration run: the two bootstrap plays of
nova.ymlact only fordeployandreconfigure(nova.yml:53-55,nova.yml:89-91), and the role runsdeploy-containers.ymlinstead ofdeploy.yml(roles/nova-cell/tasks/main.yml:2). The service includes of theloadbalancerplay are skipped too (site.yml:324-326). - Caveat:
genconfig -t nova-cellalso renders the conductor and console config on controllers, because the nova-cell play targetsnova-conductor,nova-novncproxy,nova-serialproxyandnova-spicehtml5proxyas well ascompute(nova.yml:125-134). Include them in a batch too, or the drift stays pending there. - New in 2024.1, first run after the code update: nova-cell now writes a
hostnqnfile for nova-compute (roles/nova-cell/tasks/config.yml:203-216) and lists it inconfig_fileswithoutoptional(roles/nova-cell/templates/nova-compute.json.j2:59-63). On a compute whose nova_compute container was created by 2023.1 code, that file is not in the container yet, so the first 2024.1 check reports drift there even if you changed nothing else. - The
compute[0:9]slice syntax is standard Ansible host-pattern slicing.
1.2 Rolling reconfigure with ANSIBLE_SERIAL¶
ANSIBLE_SERIAL=10 kolla-ansible -i multinode -t nova reconfigure
ANSIBLE_SERIALreaches the plays only forreconfigureandupgrade, askolla_serial(tools/kolla-ansible:522,tools/kolla-ansible:531); it defaults to 0, i.e. no batching (tools/kolla-ansible:296). Those-e kolla_serial=…options are added after your own-eoptions (tools/kolla-ansible:366-368), so for these two actions use the environment variable. Plays that setserial: '{{ kolla_serial|default("0") }}'run in batches of 10 hosts: gather-facts, common, theloadbalancerplay, all nova plays and almost every service play (gather-facts.yml:11,gather-facts.yml:40,site.yml:95,site.yml:110,nova.yml:140).- Not batched by it:
- mariadb and rabbitmq, separate playbooks (imported at
site.yml:400andsite.yml:468) whose role plays have noserial(mariadb.yml:3-16,rabbitmq.yml:3-28). Their handlers only add hosts to restart groups, and the restart plays use a fixedserial: "33%"(mariadb.yml:24,mariadb.yml:42,rabbitmq.yml:36). ansible-core rounds the percentage down with a minimum of one host (lib/ansible/utils/helpers.py:30-32): with three nodes, one node per batch. Changed in 2024.1: outward RabbitMQ and its plays are gone;rabbitmq.ymlhas a single restart play. - etcd (changed in 2024.1). In 2023.1 its play used
kolla_serial; in 2024.1 it has noserialkey (site.yml:470-481). Its handlers stage the restarts instead: look up the leader, restart the non-leaders (outsideupgradeandconfig, looping over the four values ofgroups.etcd.index(host) % 4), then the leader, whose handler is defined last (roles/etcd/handlers/main.yml:9-17,roles/etcd/handlers/main.yml:40-55,roles/etcd/handlers/main.yml:57-65). Onupgradea separate handler loops overgroups.etcdfor the non-leaders (roles/etcd/handlers/main.yml:29-36). Each restart waits for the client port and foretcdctl endpoint health(roles/etcd/tasks/restart_services.yml:16-41). NOT AVAILABLE How ansible-core orders, across hosts, the files included by one looped handler, so whether the% 4groups really restart one after another (Execution flow §7). - cyborg, whose play reads
serialinstead ofkolla_serialand so ignoresANSIBLE_SERIAL(site.yml:876). Still holds in 2024.1. - the helper plays (group-by, prechecks), which change nothing (
site.yml:7-75,site.yml:77-86).
- mariadb and rabbitmq, separate playbooks (imported at
- A
run_oncetask runs once per batch of its own play: the linear strategy sends it to the first host of the current batch only (lib/ansible/plugins/strategy/linear.py:181,lib/ansible/plugins/strategy/linear.py:207-209). For-t nova, the API DB sync (nova_api_bootstrap,roles/nova/tasks/bootstrap_service.yml:21),map_cell0(roles/nova/tasks/map_cell0.yml:63-64) and the Keystone registration (roles/service-ks-register/tasks/main.yml:126) arerun_oncein plays that target only controller groups (nova-api, nova-scheduler, nova-super-conductor:nova.yml:29-39,nova.yml:103-115), so they repeat only if those groups are larger than the batch. The cell DB sync (nova_cell_bootstrap) is notrun_once: it is gated on the first nova-conductor and runs once at any serial value (roles/nova-cell/tasks/bootstrap_service.yml:21). Compute discovery runs on the first compute of each nova-cell batch, so it repeats once per batch that contains computes (roles/nova-cell/tasks/deploy.yml:19-37,roles/nova-cell/tasks/wait_discover_computes.yml:87-90). Withkolla_serial=1and the threemultinodecontrollers (ansible/inventory/multinode:3-7,nova-apiandnova-conductorare children ofnova,ansible/inventory/multinode:227-231):nova_api_bootstrap×3,nova_cell_bootstrap×1, discovery once per compute CODE (Tags §2.5). - Batches apply per play: the whole fleet passes the nova-cell play before the next play starts. ansible-core 2.16.20
loops over the plays (
lib/ansible/executor/playbook_executor.py:126) and, inside each play, over its batches, restricting the inventory to one batch per run (lib/ansible/executor/playbook_executor.py:181-190) CODE (Execution flow §5).
1.3 Stop a rolling run on failures (max_fail_percentage)¶
# stop the run as soon as one host of a batch fails
ANSIBLE_SERIAL=10 kolla-ansible -i multinode -t nova reconfigure -e kolla_max_fail_percentage=0
New in 2024.1 CODE:
- Every play of
site.yml,gather-facts.yml,mariadb.ymlandrabbitmq.yml, and the nova plays except the cell bootstrap and the two online-migration plays, setmax_fail_percentagefrom<service>_max_fail_percentage, thenkolla_max_fail_percentage, then 100 (e.g.site.yml:96-99,gather-facts.yml:7-10,mariadb.yml:25-28,nova.yml:141-144). The nova-cell plays readnova_cell_max_fail_percentage, the nova API playsnova_max_fail_percentage(nova.yml:40-43). The cell bootstrap play keepsany_errors_fatal: trueinstead (nova.yml:78-79).kolla_max_fail_percentageis unset by default, andglobals.ymlsays that 100 is equivalent to not setting it (etc/kolla/globals.yml:27-31). - ansible-core 2.16.20 compares the hosts failed in the current batch with the batch size: the play breaks when
failed / batch_sizeis greater than the percentage (lib/ansible/plugins/strategy/linear.py:336-348), and the failed-host list is cleared at the start of each batch (lib/ansible/executor/task_queue_manager.py:332-338). With 100 that never happens. A broken play stops the whole run (lib/ansible/executor/playbook_executor.py:196-198,lib/ansible/executor/playbook_executor.py:219-220). The value may be a string such as"0"or"10%": it is converted to a number (lib/ansible/playbook/base.py:449-454). - Whatever the value, a batch in which every host failed ends the run
(
lib/ansible/executor/playbook_executor.py:204-209). Hosts that failed in an earlier batch or play are marked failed again at the start of each later run of the task queue, so they drop out of the rest of the run (lib/ansible/executor/task_queue_manager.py:332-334). - So, with the default 100 and
ANSIBLE_SERIAL=10, a failed compute drops out and the next batches carry on. With0, the first failed host of a batch stops the run before the next batch; hosts of the current batch that had not failed are marked failed too and stop where they are (lib/ansible/plugins/strategy/linear.py:341-346), so check their state before you resume. - The restart plays of mariadb and rabbitmq read the same variables (
mariadb.yml:25-28,rabbitmq.yml:37-40): with one node per batch, any value below 100 stops the run when that node fails.
2. Separate "write config" from "apply" (recommended pattern)¶
CODE
-
kolla-ansible -t nova genconfigwrites the newnova.confon all hosts. Zero restarts: every container restart handler haswhen: kolla_action != "config"(e.g.roles/nova-cell/handlers/main.yml:89-90,roles/nova-cell/handlers/main.yml:114-115,roles/nova-cell/handlers/main.yml:167-168). The handlers without that condition are waits, checks,group_bysteps and bootstrap steps chained from other handlers, handlers gated onkolla_action == "upgrade"(e.g.roles/etcd/handlers/main.yml:29-36), plus swift's no-opdebughandler (roles/swift/handlers/main.yml:2-4). The etcd handlers that restart outsideupgradecarry the condition too (roles/etcd/handlers/main.yml:40-65), and so do the mariadb and rabbitmq handlers that fill the restart groups (roles/mariadb/handlers/main.yml:56-73,roles/rabbitmq/handlers/main.yml:2-6). Still holds in 2024.1, from a scan of everyroles/*/handlers/main.yml.An untagged
genconfigalso runs the common play, which in 2024.1 makes sure the fluentd image is present, pulling it if absent (roles/common/tasks/config.yml:42-51): no container restarts, but an image pull is possible. -
kolla-ansible -t nova deploy-containers --limit compute001restarts only compute001's nova containers. Drift is detected bykolla_set_configs --checkinsidecompare_container(module_utils/kolla_container_worker.py:20,module_utils/kolla_container_worker.py:58-65). On compute001, with the defaultnova_compute_virt_type: "kvm"(group_vars/all.yml:1127,group_vars/all.yml:827), its tasks make 4 remote module invocations (setupand 3compare_container: nova-libvirt, nova-ssh, nova-compute) plus the delegated fact loop (gather-facts.yml:19-25,gather-facts.yml:49-61;roles/nova-cell/tasks/check-containers.yml:2-22). Still holds in 2024.1: roles have nometadependency and common runs only in its own play, taggedcommon(site.yml:88-103) (Compute scaling §2). No bootstrap, no Keystone registration and no haproxy work run (roles/nova-cell/tasks/deploy-containers.yml:2,nova.yml:53-55,site.yml:324-326). - Repeat per batch. Controllers (API, scheduler, conductor) are applied with their own
--limitbatch.
Caveats:
- HAProxy frontends are not applied by this pattern.
genconfigalso re-renders/etc/kolla/haproxy/services.d/(the loadbalancer block runs forconfig,site.yml:324-326), while every loadbalancer handler is skipped underconfig(roles/loadbalancer/handlers/main.yml:17), anddeploy-containersruns no haproxy work. A frontend change written this way reaches HAProxy only through-t haproxy reconfigureor a run without tags CODE; copyservices.dbeforegenconfigif you want to diff it. - Between steps 1 and 3, any other Kolla run that reaches the nova-cell check on those hosts applies the pending
drift in files listed in
config_files: an untagged run,-t novaor-t nova-cell(the tags of the nova-cell play,nova.yml:135-139). Unrelated tags do not do it through common: common runs only for an untagged run or-t common(site.yml:100-101). - Drift confined to
config.json'scommandorpermissions(e.g. memcached's-m/-c, whoseconfig_filesis empty,roles/memcached/templates/memcached.json.j2:1-4) is never detected:kolla_set_configs --checkcompares onlyconfig_files(kolladocker/base/set_configs.py:556-559, Kolla images §3), and the template task is already unchanged. No later Kolla run restarts the container for it, step 2 included. It takes effect only at the next restart for another reason. - Containers that start again for any other reason (a container engine restart, a reboot) also pick up the new
config, because
COPY_ALWAYScopies the files and rewrites/run_commandat container start (kolladocker/base/set_configs.py:302-316,docker/base/set_configs.py:535-541). Keep the window short.
Related¶
- Execution flow §4 — the action
matrix: what
genconfiganddeploy-containersrun and skip - Execution flow §5 —
batching with
serial, andmax_fail_percentage: a batch runs the complete play before the next batch starts - Execution flow §7 — the staged etcd handlers
- Tags §2.5 — what
run_oncetasks repeat whenserialis set - Kolla images §3 — what
kolla_set_configs --checkcompares and what it never compares - Predict and inspect a run — the
kolla_set_configs --checkcommand, to see what will restart aftergenconfig - Reconfigure Nova — what
-t nova reconfigureruns and restarts - R1 — the risk whose precautions name this pattern and
ANSIBLE_SERIAL
To remember
Lumo: if you keep only 5 things from this page, keep these.
genconfig(zero restarts), a review of the diff, thendeploy-containers --limit <batch>: each batch restarts only its hosts.genconfig -t nova-cellalso renders the conductor and console config on controllers: give them a batch too. CODE- Until all batches are applied, any untagged,
-t novaor-t nova-cellrun, a container restart or a reboot applies the pending drift: keep the window short. CODE - Drift confined to
config.json'scommandorpermissionsis never detected; it takes effect only at the next restart for another reason. CODE ANSIBLE_SERIALreaches onlyreconfigureandupgrade. mariadb and rabbitmq restart with a fixedserial: "33%", etcd has noserialany more and stages its restarts in its handlers, and cyborg ignores it. CODE- Every play now has
max_fail_percentage, default 100, which never stops a play early: setkolla_max_fail_percentage(or<service>_max_fail_percentage) lower to stop a rolling run when a batch has failures. CODE
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 5 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.