Pre-execution checklist¶
Impact Writes config
What to check before running kolla-ansible against a production cloud, read in Kolla-Ansible 2024.1-eom. The
checklist of §1 applies to every production run;
§2
adds what the first 2024.1 run on a cloud deployed by 2023.1 needs.
Note
Key answers
- Before every production run: the pre-execution checklist. Know the diff, the restart set and the batch size; pull first; health gates and workload gates; scope; rollback material (steps 1 to 8 of §1).
- First 2024.1 run after 2023.1: three code changes restart containers whose operator config did not change: the
systemd unit text, the new
hostnqnfile, and quorum queues on by default. A fourth cause is the image: adeployorreconfiguremoves every service thatkolla-ansible upgradehas not reached to its 2024.1 image. Run untaggedprechecks, settle the queue type, upgrade first, and list the affected containers before the run CODE (§2).
1. The checklist¶
- Know the diff before you run.
- Run
kolla-ansible genconfigwith the same tags and--limit. This writes config but does not restart any container: every container restart handler haswhen: kolla_action != "config", and the handlers without that guard are no-ops, steps chained to guarded handlers, or bootstrap, check-containers or upgrade steps that genconfig never notifies (tools/kolla-ansible:557-560; e.g.roles/nova-cell/handlers/main.yml:89-90, :114-115, :167-168;roles/swift/handlers/main.yml:2-4; Execution flow §4). - New in 2024.1: when the common play runs (no tag, or
-t common),genconfigrunskolla_containerensure_imageon the fluentd image, which pulls it when that tag is not on the host (roles/common/tasks/config.yml:42-51;module_utils/kolla_docker_worker.py:501-503). Pull first (step 4) or expectgenconfigto need the registry. genconfigalso re-renders/etc/kolla/haproxy/services.d/for the tagged services (site.yml:324-326,roles/haproxy-config/tasks/main.yml:2-23), while every loadbalancer handler is skipped underconfig(roles/loadbalancer/handlers/main.yml:16-17). A later-t <svc>reconfigure then finds the file unchanged and does not restart HAProxy: a frontend change written bygenconfigreaches HAProxy only through-t haproxy reconfigureor an untagged run, and that restart is staged, backups first and the VIP holder last (roles/loadbalancer/handlers/main.yml:9-34). Copyservices.dbeforegenconfigto diff it.- Then diff
/etc/kolla/<service>/against the previous version. Keep/etc/kollaunder version control on every host, or snapshot it before the run. - On each container,
docker exec -u root <c> /usr/local/bin/kolla_set_configs --check(Podman:podman exec -u root) tells you whether its config files will make it restart (exit 1 = a listed file differs or is missing from the container, a newoptionalfile included, a required source is missing orconfig.jsonis invalid; exit 2 = unexpected error, such as aconfig.jsonthat cannot be opened, which fails the check task on that host instead of restarting, when the comparison reaches the config check: it is skipped when the container is missing or its parameters already differ,module_utils/kolla_container_worker.py:58-65; with Podman, 137 fails the task too, where Docker treats it as drift). The check does not see a changed image, volume or systemd unit, which restart the container as well (module_utils/kolla_container_worker.py:58-65). See Predict and inspect a run and Kolla images §3.
- Run
- Know the restart set. Remember loop-notify: if one container of a role changes on a host, every container of that role on that host restarts (Tags §2.4).
- Know the batch size.
ANSIBLE_SERIALcoversreconfigure/upgrade(tools/kolla-ansible:522, :531);-e kolla_serial=Ncoversdeploy(the plays readserial: '{{ kolla_serial|default("0") }}', e.g.site.yml:95).- mariadb and rabbitmq ignore it: their role plays have no
serialand their restart plays use a fixedserial: "33%"(mariadb.yml:24, :42;rabbitmq.yml:36). - Changed in 2024.1: the etcd play has no
serialany more (site.yml:470-481); its handlers stage the restarts themselves (Controlled batches §1.2). - cyborg uses the wrong variable (
serial: '{{ serial|default("0") }}',site.yml:876). - New in 2024.1: every play sets
max_fail_percentage, from<service>_max_fail_percentage, thenkolla_max_fail_percentage, then 100 (e.g.site.yml:96-99). With 100 a failed host only drops out and the next batches run, unless every host of a batch fails: ansible-core then stops the playbook (ansible-core 2.16.20lib/ansible/executor/playbook_executor.py:204-208), so withANSIBLE_SERIAL=1, or in the 33% restart plays of a three-node mariadb or rabbitmq, one failed host ends the run. To stop a rolling run at the first bad batch, setkolla_max_fail_percentagelower (Controlled batches §1.3).
- Pull first.
kolla-ansible pull(with the same tags,tools/kolla-ansible:516-518) beforeupgrade/deploy, so that restart windows do not include downloads. -
Health gates. Before the run, check:
- Galera:
wsrep_cluster_size,wsrep_local_state_comment=Synced; - RabbitMQ: cluster status, no partitions;
openstack compute service listandopenstack network agent listall up;- HAProxy backends all UP.
Repeat after each batch.
- Galera:
-
Workload gates. No live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts.
--statustakes one exact status, so run one command per status, for exampleopenstack server list --all-projects --host <host> --status MIGRATING, thenRESIZEandVERIFY_RESIZE;openstack volume list --all-projects --status <status>for each in-progress volume status;openstack image list --status <status>for uploads. Do not writeMIGRATING|RESIZEor*ing: the shell reads them as a pipe and a file pattern. This gate is operator practice, not something the Kolla code checks. - Scope.
- Prefer
--limitand the narrowest tag. - Add
--skip-tags haproxywhen the frontends are not meant to change: every task of the loadbalancer play inherits its play tagshaproxy,keepalivedandloadbalancer(site.yml:115-118; Tags §4). -t commonruns the common role (fluentd, kolla-toolbox, cron) in its own play, as in 2023.1. Service tags do not run common: no 2024.1 role has ametadependency on it (site.yml:88-103,roles/common/tasks/main.yml:2).
- Prefer
- Rollback material.
- Previous
/etc/kolla/config,globals.yml,passwords.ymland inventory under git. - Previous image tags still present locally: do not run
prune-imagesbefore you are done. It runsdocker_prune(orpodman image prune) filtered on thekolla_versionlabel (roles/prune-images/tasks/prune_images.yml:2-16); NOT AVAILABLE which images that removes, dangling only or tagged ones too: it depends on the prune defaults of Docker or Podman. - A MariaDB backup (
kolla-ansible mariadb_backup,tools/kolla-ansible:453-456). It works only withenable_mariabackup: "yes"set and applied to mariadb beforehand: the playbook applies the role only when that flag is true (mariadb_backup.yml:8-11), the default is"no"(group_vars/all.yml:804), and the mariabackup config is rendered only with it (roles/mariadb/tasks/config.yml:17-64). Otherwise the command skips silently: check that a backup file was produced.
- Previous
2. First 2024.1 run on a cloud deployed by 2023.1¶
On the first deploy, reconfigure, deploy-containers or upgrade that the 2024.1 code runs, three changes in the
code restart containers even when the operator changed nothing, and the image tag changes for every service not yet
upgraded. Add these checks to §1 CODE:
- Settle the RabbitMQ queue type first (F17).
om_enable_rabbitmq_quorum_queuesnow defaults totrue(group_vars/all.yml:266-268), and every service config that renders[oslo_messaging_rabbit]then getsrabbit_quorum_queue = true(e.g.roles/nova-cell/templates/nova.conf.j2:207-216): a changed file, so a restart, for those services, computes included.- Run
kolla-ansible precheckswithout-t: theprechecksrole is an untagged play (site.yml:77-86), so a tagged run skips its checks. It fails whenom_enable_rabbitmq_high_availabilityandom_enable_rabbitmq_quorum_queuesare equal, both off included (roles/prechecks/tasks/service_checks.yml:67-72). The rabbitmq precheck, when arabbitmqcontainer exists and quorum is on, asserts that every queue exceptreply_and_fanoutones is a quorum queue (roles/rabbitmq/tasks/precheck.yml:131-152, :185-189); those assertions carry the tagrabbitmq-ha-precheck(:189), and skipping it does not change what the services render. - Both settings that pass
prechecksare migrations. Quorum on needs every existing queue converted to quorum queues. The only way to keep classic queues isom_enable_rabbitmq_high_availability: truewith quorum off and theha-allpolicy in place (roles/rabbitmq/tasks/precheck.yml:107-129); HA on also rendersamqp_durable_queues = truein every service config (e.g.roles/nova-cell/templates/nova.conf.j2:204-206), a changed file, so a restart, for those services too. On a cloud that kept the 2023.1 default (HA off), 2023.1 runs removedha-all(2023.1 kolla-ansible tree: ansible/roles/rabbitmq/tasks/deploy.yml, lines 4-6), so this path is also a manual migration, to mirrored queues, as the HA precheck message says (roles/rabbitmq/tasks/precheck.yml:115-123). NOT AVAILABLE What oslo.messaging does with existing classic queues when the services restart with quorum on.
- List the units that 2023.1 wrote (F16). The check task compares the whole unit
file with the 2024.1 template, which writes
StartLimitInterval=and no longerStartLimitIntervalSec=(module_utils/kolla_systemd_worker.py:20-39, :154-167). On each host,grep -l StartLimitIntervalSec /etc/systemd/system/kolla-*-container.servicelists the containers that the next check will restart (Swiftdeployhas no such check), with the rest of their role on that host, even whenkolla_set_configs --checkexits 0. A container that a 2024.1 run created, recreated or restarted (an upgrade that reached its role does one of these) already has the new unit; after a partial upgrade (upgrade -t <svc>), every other service still has the old one. Plan those restarts in batches with--limit(Controlled batches). - Expect the
hostnqnrestart on computes and cinder-volume hosts. nova-cell and cinder render a newhostnqnfile and notify the restart (roles/nova-cell/tasks/config.yml:203-216,roles/cinder/tasks/config.yml:95-108);config.jsoncopies it, not optional, to/etc/nvme/hostnqn(roles/nova-cell/templates/nova-compute.json.j2:59-63,roles/cinder/templates/cinder-volume.json.j2:23-27). Aftergenconfig, the check innova_computeandcinder_volumeexits 1. On a compute this restartsnova_computeand, by loop-notify, the other nova-cell containers of that host (Nova deep dive §2.4). - Upgrade every service before any
deployorreconfigure.openstack_releasedefaults to2024.1(group_vars/all.yml:679-681) and every image tag follows it (e.g.roles/keystone/defaults/main.yml:96-100). The check task compares the image (module_utils/kolla_container_worker.py:75), and the restart handler recreates a container whose image differs, pulling the image first when it is absent (module_utils/kolla_docker_worker.py:353-361). So adeployorreconfiguremoves every service thatkolla-ansible upgradehas not yet reached (all of them before the upgrade, the services outside the tags afterupgrade -t <svc>) to its 2024.1 image through the deploy path, without the role's upgrade tasks (e.g. keystone's database upgrade handlers,roles/keystone/tasks/upgrade.yml:23-31).kolla_set_configs --checkdoes not show it. Runkolla-ansible upgradefor those services first.
NOT AVAILABLE The effect of these restarts on running VMs and volumes: the sources do not establish it. How to see each cause before the run: Predict and inspect a run §2.
Related¶
- Risk register — the risks, each with its precautions
- Predict and inspect a run — the commands of step 1, with the exit codes of the check
- Write config first, apply in controlled batches —
genconfig, thendeploy-containers --limitbatch by batch, andmax_fail_percentage - Tags §2.4 — loop-notify, the mechanism of step 2
- Execution flow §5 —
batching with
serial, and the plays that are not batched (step 3) - Update container images —
pullbeforeupgrade(step 4) - Database operations — the
mariadb_backupcommand of step 8 - Technical findings F16, F17 — the unit-text drift and the quorum-queue default of §2
- RabbitMQ — the quorum-queue default and the stricter prechecks
To remember
Lumo: if you keep only 5 things from this page, keep these.
- Before every production run: know the diff, the restart set and the batch size; pull first; health gates and workload gates; scope; rollback material.
- Run
kolla-ansible genconfigwith the same tags and--limit: it writes config and restarts no container, but an untagged or-t commonrun can pull the fluentd image. Then diff/etc/kolla/<service>/against the previous version. CODE - Loop-notify: if one container of a role changes on a host, every container of that role on that host restarts. CODE
ANSIBLE_SERIALcoversreconfigure/upgrade; mariadb and rabbitmq restart with a fixedserial: "33%", etcd has noserial, and with the defaultmax_fail_percentageof 100 a failed host only drops out, unless every host of a batch fails: then the playbook stops. CODE- First 2024.1 run after 2023.1: units with
StartLimitIntervalSec=, the newhostnqnfile andrabbit_quorum_queue = trueeach restart containers, and adeployorreconfiguremoves every service thatkolla-ansible upgradehas not reached to its 2024.1 image; run untaggedprechecks, settle the queue type and upgrade first. CODE
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.