Skip to content

Pre-execution checklist

Impact Writes config

What to check before running kolla-ansible against a production cloud. The checklist applies to every production run.

Note

Key answers

  • Before every production run: the pre-execution checklist. Know the diff, the restart set and the batch size; pull first; health gates and workload gates; scope; rollback material (steps 1 to 8 below).
  1. Know the diff before you run.
    • Run kolla-ansible genconfig with the same tags and --limit. This writes config but does not restart any Docker container: every container restart handler has when: kolla_action != "config", and the handlers without that guard are no-ops, steps chained to guarded handlers, or bootstrap, check-containers or upgrade steps that genconfig never notifies (tools/kolla-ansible:540-542; e.g. roles/nova-cell/handlers/main.yml:89-90, :114-115, :167-168; roles/swift/handlers/main.yml:2-4). The Ussuri exception, the unguarded handlers of the nova-hyperv role, no longer exists: 2023.1 has no nova-hyperv role and no Hyper-V support in ansible/roles/.
    • Then diff /etc/kolla/<service>/ against the previous version. Keep /etc/kolla under version control on every host, or snapshot it before the run.
    • On each container, docker exec -u root <c> /usr/local/bin/kolla_set_configs --check tells you whether it will be restarted (exit 1 = a listed file differs or is missing from the container, a new optional file included, a required source is missing or config.json is invalid; exit 2 = unexpected error, such as a missing config.json, which fails the check-containers task on that host instead of restarting; see Predict and inspect a run and Kolla images §3).
  2. Know the restart set. Remember loop-notify: if one container of a role changes on a host, every container of that role on that host restarts (Tags §2.4).
  3. Know the batch size.
    • ANSIBLE_SERIAL covers reconfigure/upgrade; -e kolla_serial=N covers deploy (tools/kolla-ansible:508, :517).
    • mariadb and rabbitmq ignore it: their role plays have no serial and their restart plays use a fixed serial: "33%" (mariadb.yml:20, :34; rabbitmq.yml:32, :88).
    • cyborg uses the wrong variable (site.yml:784).
  4. Pull first. kolla-ansible pull (with the same tags) before upgrade/deploy, so that restart windows do not include downloads.
  5. Health gates. Before the run, check:

    • Galera: wsrep_cluster_size, wsrep_local_state_comment=Synced;
    • RabbitMQ: cluster status, no partitions;
    • openstack compute service list and openstack network agent list all up;
    • HAProxy backends all UP.

    Repeat after each batch.

  6. Workload gates. No live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts. This gate is operator practice, not read from the code. Check each in-progress status separately, for example openstack server list --all-projects --host <host> --status MIGRATING (then RESIZE and VERIFY_RESIZE) and openstack volume list --all-projects --status <status> for each in-progress volume status; image uploads need their own check.

  7. Scope.
    • Prefer --limit and the narrowest tag.
    • Add --skip-tags haproxy when the frontends are not meant to change.
    • -t common now runs the common role (fluentd, kolla-toolbox, cron) in its own play; in Ussuri it was a no-op. Service tags no longer run common: 2023.1 roles have no meta dependency on it (site.yml:86-97, roles/common/tasks/main.yml:2).
  8. Rollback material.
    • Previous /etc/kolla/config, globals.yml, passwords.yml and inventory under git.
    • Previous image tags still present locally (do not prune-images before you are done).
    • A MariaDB backup. kolla-ansible mariadb_backup does nothing unless enable_mariabackup is "yes" and was applied to MariaDB beforehand: the default is "no" (group_vars/all.yml:709) and the play applies the role only when: enable_mariabackup | bool (mariadb_backup.yml:7) CODE. Check that a backup was really produced.

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • Before every production run: know the diff, the restart set and the batch size; pull first; health gates and workload gates; scope; rollback material.
  • Run kolla-ansible genconfig with the same tags and --limit: it writes config but does not restart any Docker container. Then diff /etc/kolla/<service>/ against the previous version.
  • Loop-notify: if one container of a role changes on a host, every container of that role on that host restarts.
  • ANSIBLE_SERIAL covers reconfigure/upgrade; mariadb and rabbitmq ignore it: their restart plays use a fixed serial: "33%".
  • Check Galera, RabbitMQ, compute services, network agents and HAProxy backends before the run and after each batch, with no migrations, resizes or uploads in flight.

What would you do?

Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.

Scenario 1: Live migrations still running on the target hosts

You are about to run a reconfigure on a batch of 10 computes in a cloud with 40 computes. Just before starting, openstack server list --all-projects --status MIGRATING shows two live migrations in flight on hosts of your batch. Your colleague says the run will not take long and suggests starting anyway.

What would you do?

See every choice explained
  • Avoid: Start the run now on the whole batch: the migrations will finish on their own during the run.. This breaks the workload gate of step 6: no live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts.
  • Avoid: Start the run now, but without --limit, so that the whole cloud is done in one pass.. It still targets the hosts with migrations in flight (step 6), and step 7 says the opposite: prefer --limit and the narrowest tag.
  • Best choice: Wait until nothing is in flight on the targeted hosts (check each server status such as MIGRATING and RESIZE with openstack server list --all-projects --host <host> --status <status>, and the volumes the same way), check the health gates, then run on the batch with --limit.. This is steps 5 to 7 of the checklist: no live migrations, resizes, image uploads or volume migrations in flight on the targeted hosts; health gates (Galera, RabbitMQ, compute services and network agents, HAProxy backends) before the run and after each batch; prefer --limit and the narrowest tag.
Scenario 2: Free disk space before an upgrade

You have pulled the new images with kolla-ansible pull before tonight's upgrade, and some hosts are short of disk space. Your colleague suggests running kolla-ansible prune-images now to free space, then starting the upgrade.

What would you do?

See every choice explained
  • Works, but risky: Keep the previous images, but skip the MariaDB backup: the old images are enough to go back.. Keeping the previous image tags is right, but step 8 lists more rollback material: the previous /etc/kolla/config, globals.yml, passwords.yml and inventory under git, and a MariaDB backup (kolla-ansible mariadb_backup, which does nothing unless enable_mariabackup is on).
  • Best choice: Keep the previous image tags until you are done, and gather all the rollback material before the run: config, globals.yml, passwords.yml and inventory under git, plus a MariaDB backup that you checked was really produced (kolla-ansible mariadb_backup needs enable_mariabackup on).. Step 8, rollback material: previous image tags still present locally (do not prune-images before you are done), the previous configuration and inventory under git, and a MariaDB backup.
  • Avoid: Run kolla-ansible prune-images now, then start the upgrade.. Step 8 says it directly: keep the previous image tags present locally and do not prune-images before you are done.

Check yourself

Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. What should you run before upgrade or deploy, so that restart windows do not include downloads?
Show the answer

Answer: kolla-ansible pull, with the same tags. Step 4 of the checklist: pull first.

2. Do mariadb and rabbitmq follow ANSIBLE_SERIAL?
Show the answer

Answer: No, they ignore it. Their role plays have no serial, and their restart plays use a fixed serial: "33%" (step 3).

3. What does -t common do?
Show the answer

Answer: It runs the common role (fluentd, kolla-toolbox, cron) in its own play. In 2023.1 common is its own play, tagged common; in Ussuri the same tag was a no-op (step 7).