Write config first, apply in controlled batches¶
Impact Writes config Restarts on drift
How to write the configuration first and apply it batch by batch, with Nova as the example. genconfig, a review of the
diff, then deploy-containers --limit <batch> is the only built-in way to separate "write the config" from "apply it"
(Execution flow §4). The page
also covers a rolling reconfigure with ANSIBLE_SERIAL.
The page uses the definition of drift of the Operations hub.
Note
Key answers
1. Commands¶
1.1 Compute changes in controlled batches (recommended pattern)¶
# 1. write the config everywhere, restart nothing
kolla-ansible -i multinode -t nova-cell genconfig
# 2. review: diff /etc/kolla/nova-compute/nova.conf on a sample of hosts
# 3. apply batch by batch (only check-containers + handlers run)
kolla-ansible -i multinode -t nova-cell deploy-containers --limit 'compute[0:9]'
kolla-ansible -i multinode -t nova-cell deploy-containers --limit 'compute[10:19]'
-
deploy-containersruns onlycheck-containers.ymland the handlers it notifies. These restart each container with drift written bygenconfigand, by loop-notify, the other containers of that role on the same host CODE (roles/nova-cell/tasks/deploy-containers.yml:2; Tags §2.4).Through
kolla_set_configs --checkit sees only drift in the files listed inconfig.json'sconfig_files(content, owner, perm) CODE (kolladocker/base/set_configs.py:160-191,407-410). It does not see changes toconfig.json's owncommand/permissions(e.g. memcached: emptyconfig_files), and no laterdeploy/reconfigurerestarts for those either, because theconfig.jsontemplate task is then unchanged; they take effect at the next restart for another reason.A newly rendered
optionalfile whose destination is not yet in the container (e.g. nova-compute'srelease) makes--checkexit 2 and the task fail, unless an entry listed before it (herenova.conf, if it changed too) differs: then it exits 1 and the restart copies the new file (Kolla images §3). -
Each batch restarts only its hosts. No bootstrap containers, no Keystone registration and no haproxy play run.
- Caveat:
genconfig -t nova-cellalso renders the conductor and console config on controllers. Include them in a batch too, or the drift stays pending there. - The
compute[0:9]slice syntax is standard Ansible host-pattern slicing.
1.2 Rolling reconfigure with ANSIBLE_SERIAL¶
ANSIBLE_SERIAL=10 kolla-ansible -i multinode -t nova reconfigure
- Plays that set
serial: kolla_serialrun in batches of 10 hosts: gather-facts, the haproxy play, all nova plays and almost every service play. Not batched: mariadb, rabbitmq and outward rabbitmq (noserial; their rolling logic is in the handlers); cyborg, which usesserialinstead ofkolla_serialand so ignoresANSIBLE_SERIAL(bug,site.yml:1028); and the helper plays (group-by, CentOS check, prechecks), which change nothing. - A
run_oncetask runs once per batch of its own play. For-t nova, the API DB sync (nova_api_bootstrap,map_cell0) and the Keystone registration arerun_oncein plays that target only controller groups (nova-api, nova-scheduler, nova-super-conductor), so they repeat only if those groups are larger than the batch. The cell DB sync (nova_cell_bootstrap) is notrun_once: it is gated on the first nova-conductor and runs once at any serial value. Compute discovery runs on the first compute of each nova-cell batch, so it repeats once per batch that contains computes CODE (Tags §2.5 androles/nova-cell/tasks/bootstrap_service.yml:21: withkolla_serial=1and the threemultinodecontrollers,nova_api_bootstrap×3,nova_cell_bootstrap×1, discovery once per compute). - Batches apply per play: the whole fleet passes the nova-cell play before the next play starts CODE (ansible 2.9.27
lib/ansible/executor/playbook_executor.py:161-169).
2. Separate "write config" from "apply" (recommended pattern)¶
CODE
kolla-ansible -t nova genconfigwrites the newnova.confon all hosts. Zero restarts: every container restart handler haswhen: kolla_action != "config"(e.g.roles/nova-cell/handlers/main.yml:84-85, :107-108, :139-140). The only unguarded restart handlers are the deprecated nova-hyperv Windows-service ones, which-t novadoes not select (Pre-execution checklist).kolla-ansible -t nova deploy-containers --limit compute001restarts only compute001's nova containers. Drift is detected bykolla_set_configs --checkinsidecompare_container. On compute001 its tasks make 7 remote module invocations (setupand 6compare_container) plus the delegated fact loop (Compute scaling §2), with no bootstrap, no Keystone registration and no haproxy play (roles/nova-cell/tasks/deploy-containers.yml:2,nova.yml:49-51,site.yml:390-392).- Repeat per batch. Controllers (API, scheduler, conductor) are applied with their own
--limitbatch.
Caveats:
- Between steps 1 and 3, any other Kolla run that reaches those hosts applies the pending drift in files listed in
config_files. That includes an unrelated-t chronyrun that touches common: the nova containers are not checked by it, but the common ones are. - Drift confined to
config.json'scommandorpermissions(e.g. memcached's-m/-c, whoseconfig_filesis empty) is never detected:kolla_set_configs --checkcompares onlyconfig_files(kolladocker/base/set_configs.py:407-410, Kolla images §3), and the template task is already unchanged. No later Kolla run restarts the container for it, step 2 included. It takes effect only at the next restart for another reason. - Hosts restarted by Docker or reboot also pick up the new config, because
COPY_ALWAYScopies the files and rewrites/run_commandat container start (kolladocker/base/set_configs.py:319-324,391-393). Keep the window short.
Related¶
- Execution flow §4 — the action
matrix: what
genconfiganddeploy-containersrun and skip - Execution flow §5 —
batching with
serial: a batch runs the complete play before the next batch starts - Tags §2.5 — what
run_oncetasks repeat whenserialis set - Kolla images §3 — what
kolla_set_configs --checkcompares and what it never compares - Predict and inspect a run — the
kolla_set_configs --checkcommand, to see what will restart aftergenconfig - Reconfigure Nova — what
-t nova reconfigureruns and restarts - R1 — the risk whose precautions name this pattern and
ANSIBLE_SERIAL
To remember
Lumo: if you keep only 5 things from this page, keep these.
genconfig(zero restarts), a review of the diff, thendeploy-containers --limit <batch>: each batch restarts only its hosts. CODE- Until all batches are applied, any other Kolla run that reaches those hosts (even
-t chrony), a Docker restart or a reboot applies the pending drift: keep the window short. CODE genconfig -t nova-cellalso renders the conductor and console config on controllers: include them in a batch too, or the drift stays pending there.- Drift confined to
config.json'scommandorpermissionsis never detected; it takes effect only at the next restart for another reason. CODE - With
ANSIBLE_SERIAL, mariadb, rabbitmq and outward rabbitmq are not batched, and cyborg ignores it.
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.