Write config first, apply in controlled batches¶
Impact Writes config Restarts on drift
How to write the configuration first and apply it batch by batch, with Nova as the example. genconfig, a review of the
diff, then deploy-containers --limit <batch> is the only built-in way to separate "write the config" from "apply it"
(Execution flow §4). The page
also covers a rolling reconfigure with ANSIBLE_SERIAL.
The page uses the definition of drift of the Operations hub.
Note
Key answers
1. Commands¶
1.1 Compute changes in controlled batches (recommended pattern)¶
# 1. write the config everywhere, restart nothing
kolla-ansible -i multinode -t nova-cell genconfig
# 2. review: diff /etc/kolla/nova-compute/nova.conf on a sample of hosts
# 3. apply batch by batch (only check-containers + handlers run)
kolla-ansible -i multinode -t nova-cell deploy-containers --limit 'compute[0:9]'
kolla-ansible -i multinode -t nova-cell deploy-containers --limit 'compute[10:19]'
-
deploy-containersruns onlycheck-containers.ymland the handlers it notifies. These restart each container with drift written bygenconfigand, by loop-notify, the other containers of that role on the same host CODE (roles/nova-cell/tasks/deploy-containers.yml:2; Tags §2.4).Through
kolla_set_configs --checkit sees only drift in the files listed inconfig.json'sconfig_files(content, owner, perm) CODE (kolladocker/base/set_configs.py:165-198, :220-242, :556-559). It does not see changes toconfig.json's owncommand/permissions(e.g. memcached: emptyconfig_files), and no laterdeploy/reconfigurerestarts for those either, because theconfig.jsontemplate task is then unchanged; they take effect at the next restart for another reason.A newly rendered
optionalfile whose destination is not yet in the container (e.g. nova-compute'srelease) counts as drift:--checkexits 1 and the restart copies the new file (kolladocker/base/set_configs.py:165-171,roles/nova-cell/templates/nova-compute.json.j2:52-58). In Ussuri the same case made--checkexit 2 and the task fail; 2023.1 no longer exemptsoptionalfiles from the destination check (Kolla images §3). -
Each batch restarts only its hosts. No bootstrap containers and no Keystone registration run, and the service includes of the
loadbalancerplay are skipped (nova.yml:49-51,site.yml:339-341). - Caveat:
genconfig -t nova-cellalso renders the conductor and console config on controllers. Include them in a batch too, or the drift stays pending there. - The
compute[0:9]slice syntax is standard Ansible host-pattern slicing.
1.2 Rolling reconfigure with ANSIBLE_SERIAL¶
ANSIBLE_SERIAL=10 kolla-ansible -i multinode -t nova reconfigure
ANSIBLE_SERIALreaches the plays only forreconfigureandupgrade, askolla_serial(tools/kolla-ansible:508, :517). Plays that setserial: kolla_serialrun in batches of 10 hosts: gather-facts, common, theloadbalancerplay, all nova plays and almost every service play (gather-facts.yml:7, :32;site.yml;nova.yml). Not batched by it: mariadb, rabbitmq and outward rabbitmq, now separate playbooks whose role plays have noserialand whose restart plays use a fixedserial: "33%"(mariadb.yml:20, :34;rabbitmq.yml:32, :88; imported atsite.yml:385, :438); cyborg, which usesserialinstead ofkolla_serialand so ignoresANSIBLE_SERIAL(bug,site.yml:784); and the helper plays (group-by, prechecks), which change nothing.- A
run_oncetask runs once per batch of its own play. For-t nova, the API DB sync (nova_api_bootstrap,map_cell0) and the Keystone registration arerun_oncein plays that target only controller groups (nova-api, nova-scheduler, nova-super-conductor), so they repeat only if those groups are larger than the batch. The cell DB sync (nova_cell_bootstrap) is notrun_once: it is gated on the first nova-conductor and runs once at any serial value. Compute discovery runs on the first compute of each nova-cell batch, so it repeats once per batch that contains computes CODE (Tags §2.5 andnova.yml:29-39, :56-67,roles/nova-cell/tasks/bootstrap_service.yml:21: withkolla_serial=1and the threemultinodecontrollers,nova_api_bootstrap×3,nova_cell_bootstrap×1, discovery once per compute). - Batches apply per play: the whole fleet passes the nova-cell play before the next play starts CODE (ansible-core
2.14.18
lib/ansible/executor/playbook_executor.py:126, :181-190).
2. Separate "write config" from "apply" (recommended pattern)¶
CODE
kolla-ansible -t nova genconfigwrites the newnova.confon all hosts. Zero restarts: every container restart handler haswhen: kolla_action != "config"(e.g.roles/nova-cell/handlers/main.yml:89-90, :114-115, :167-168). The handlers without that condition are waits, checks and bootstrap steps chained from other handlers, plus swift's no-opdebughandler (roles/swift/handlers/main.yml:2-4). The Ussuri exception, the unguarded nova-hyperv Windows-service handlers, is gone: there is nonova-hypervrole in 2023.1 (Pre-execution checklist).kolla-ansible -t nova deploy-containers --limit compute001restarts only compute001's nova containers. Drift is detected bykolla_set_configs --checkinsidecompare_container. On compute001 its tasks make 4 remote module invocations (setupand 3compare_container: nova-libvirt, nova-ssh, nova-compute) plus the delegated fact loop (gather-facts.yml:15-22, :41-54;roles/nova-cell/tasks/check-containers.yml:2-22). Ussuri made 7, because common ran too through the roles'metadependency; 2023.1 roles have nometaand common runs only in its own play, taggedcommon(site.yml:86-97) (Compute scaling §2), with no bootstrap, no Keystone registration and no haproxy play (roles/nova-cell/tasks/deploy-containers.yml:2,nova.yml:49-51,site.yml:339-341).- Repeat per batch. Controllers (API, scheduler, conductor) are applied with their own
--limitbatch.
Caveats:
- HAProxy frontends are not applied by this pattern.
genconfigalso re-renders/etc/kolla/haproxy/services.d/(the loadbalancer block runs forconfig,site.yml:339-341), while every loadbalancer handler is skipped underconfig(roles/loadbalancer/handlers/main.yml:17), anddeploy-containersruns no haproxy work. A frontend change written this way reaches HAProxy only through-t haproxy reconfigureor a run without tags CODE; copyservices.dbeforegenconfigif you want to diff it. - Between steps 1 and 3, any other Kolla run that reaches the nova-cell check on those hosts applies the pending
drift in files listed in
config_files: an untagged run,-t novaor-t nova-cell. Unrelated tags no longer do it through common: in Ussuri every role pulled common in throughmeta, so even-t chronychecked the common containers; in 2023.1 common runs only for an untagged run or-t common(site.yml:86-97), and the chrony role no longer exists. - Drift confined to
config.json'scommandorpermissions(e.g. memcached's-m/-c, whoseconfig_filesis empty) is never detected:kolla_set_configs --checkcompares onlyconfig_files(kolladocker/base/set_configs.py:556-559, Kolla images §3), and the template task is already unchanged. No later Kolla run restarts the container for it, step 2 included. It takes effect only at the next restart for another reason. - Hosts restarted by Docker or reboot also pick up the new config, because
COPY_ALWAYScopies the files and rewrites/run_commandat container start (kolladocker/base/set_configs.py:302-316, :535-541). Keep the window short.
Related¶
- Execution flow §4 — the action
matrix: what
genconfiganddeploy-containersrun and skip - Execution flow §5 —
batching with
serial: a batch runs the complete play before the next batch starts - Tags §2.5 — what
run_oncetasks repeat whenserialis set - Kolla images §3 — what
kolla_set_configs --checkcompares and what it never compares - Predict and inspect a run — the
kolla_set_configs --checkcommand, to see what will restart aftergenconfig - Reconfigure Nova — what
-t nova reconfigureruns and restarts - R1 — the risk whose precautions name this pattern and
ANSIBLE_SERIAL
To remember
Lumo: if you keep only 5 things from this page, keep these.
genconfig(zero restarts), a review of the diff, thendeploy-containers --limit <batch>: each batch restarts only its hosts. CODE- Until all batches are applied, any untagged,
-t novaor-t nova-cellrun, a Docker restart or a reboot applies the pending drift: keep the window short. CODE genconfig -t nova-cellalso renders the conductor and console config on controllers: include them in a batch too, or the drift stays pending there.- Drift confined to
config.json'scommandorpermissionsis never detected; it takes effect only at the next restart for another reason. CODE ANSIBLE_SERIALreaches onlyreconfigureandupgrade; mariadb and rabbitmq use a fixedserial: "33%", and cyborg ignores it.
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.