Pre-execution checklist¶
Impact Writes config
What to check before running kolla-ansible against a production cloud. The checklist applies to every production run.
Note
Key answers
- Before every production run: the pre-execution checklist. Know the diff, the restart set and the batch size; pull first; health gates and workload gates; scope; rollback material (steps 1 to 8 below).
- Know the diff before you run.
- Run
kolla-ansible genconfigwith the same tags and--limit. This writes config but does not restart any Docker container: every container restart handler haswhen: kolla_action != "config", and the handlers without that guard are no-ops, steps chained to guarded handlers, or bootstrap, check-containers or upgrade steps that genconfig never notifies (tools/kolla-ansible:540-542; e.g.roles/nova-cell/handlers/main.yml:89-90, :114-115, :167-168;roles/swift/handlers/main.yml:2-4). The Ussuri exception, the unguarded handlers of the nova-hyperv role, no longer exists: 2023.1 has nonova-hypervrole and no Hyper-V support inansible/roles/. - Then diff
/etc/kolla/<service>/against the previous version. Keep/etc/kollaunder version control on every host, or snapshot it before the run. - On each container,
docker exec -u root <c> /usr/local/bin/kolla_set_configs --checktells you whether it will be restarted (exit 1 = a listed file differs or is missing from the container, a newoptionalfile included, a required source is missing orconfig.jsonis invalid; exit 2 = unexpected error, such as a missingconfig.json, which fails the check-containers task on that host instead of restarting; see Predict and inspect a run and Kolla images §3).
- Run
- Know the restart set. Remember loop-notify: if one container of a role changes on a host, every container of that role on that host restarts (Tags §2.4).
- Know the batch size.
ANSIBLE_SERIALcoversreconfigure/upgrade;-e kolla_serial=Ncoversdeploy(tools/kolla-ansible:508, :517).- mariadb and rabbitmq ignore it: their role plays have no
serialand their restart plays use a fixedserial: "33%"(mariadb.yml:20, :34;rabbitmq.yml:32, :88). - cyborg uses the wrong variable (
site.yml:784).
- Pull first.
kolla-ansible pull(with the same tags) beforeupgrade/deploy, so that restart windows do not include downloads. -
Health gates. Before the run, check:
- Galera:
wsrep_cluster_size,wsrep_local_state_comment=Synced; - RabbitMQ: cluster status, no partitions;
openstack compute service listandopenstack network agent listall up;- HAProxy backends all UP.
Repeat after each batch.
- Galera:
-
Workload gates. No live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts. This gate is operator practice, not read from the code. Check each in-progress status separately, for example
openstack server list --all-projects --host <host> --status MIGRATING(thenRESIZEandVERIFY_RESIZE) andopenstack volume list --all-projects --status <status>for each in-progress volume status; image uploads need their own check. - Scope.
- Prefer
--limitand the narrowest tag. - Add
--skip-tags haproxywhen the frontends are not meant to change. -t commonnow runs the common role (fluentd, kolla-toolbox, cron) in its own play; in Ussuri it was a no-op. Service tags no longer run common: 2023.1 roles have nometadependency on it (site.yml:86-97,roles/common/tasks/main.yml:2).
- Prefer
- Rollback material.
- Previous
/etc/kolla/config,globals.yml,passwords.ymland inventory under git. - Previous image tags still present locally (do not
prune-imagesbefore you are done). - A MariaDB backup.
kolla-ansible mariadb_backupdoes nothing unlessenable_mariabackupis"yes"and was applied to MariaDB beforehand: the default is"no"(group_vars/all.yml:709) and the play applies the role onlywhen: enable_mariabackup | bool(mariadb_backup.yml:7) CODE. Check that a backup was really produced.
- Previous
Related¶
- Risk register — the 25 risks, each with its precautions
- Predict and inspect a run — the commands of step 1, with the exit codes of the check
- Write config first, apply in controlled batches —
genconfig, thendeploy-containers --limitbatch by batch - Tags §2.4 — loop-notify, the mechanism of step 2
- Execution flow §5 —
batching with
serial, and the plays that are not batched (step 3) - Update container images —
pullbeforeupgrade(step 4) - Database operations — the
mariadb_backupcommand of step 8
To remember
Lumo: if you keep only 5 things from this page, keep these.
- Before every production run: know the diff, the restart set and the batch size; pull first; health gates and workload gates; scope; rollback material.
- Run
kolla-ansible genconfigwith the same tags and--limit: it writes config but does not restart any Docker container. Then diff/etc/kolla/<service>/against the previous version. - Loop-notify: if one container of a role changes on a host, every container of that role on that host restarts.
ANSIBLE_SERIALcoversreconfigure/upgrade; mariadb and rabbitmq ignore it: their restart plays use a fixedserial: "33%".- Check Galera, RabbitMQ, compute services, network agents and HAProxy backends before the run and after each batch, with no migrations, resizes or uploads in flight.
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.