Skip to content

Ten things to know before a production run

Ten things to know before running Kolla-Ansible 2024.1 on production. Each item restates a 2024.1 page and links to the section that gives the evidence; the main citations are repeated here.

  1. reconfigure is deploy for 52 of the 54 roles that have a reconfigure.yml: 51 import deploy.yml and skyline includes it (roles/skyline/tasks/reconfigure.yml:2); swift and bifrost are the exceptions. It re-runs Keystone registration, DB creation and the DB-sync bootstrap containers for most services, Neutron's included (roles/neutron/tasks/bootstrap_service.yml:2-21). The only lightweight path is genconfig (tools/kolla-ansible:557-560) followed by deploy-containers (tools/kolla-ansible:499-502) (Repository architecture §4.2, Execution flow §4) CODE.
  2. Config is applied by a full stop/start, never a reload, and the stop timeout is 60 s. Changed in 2024.1: the container module is kolla_container, which drives Docker or Podman according to kolla_container_engine (default docker, group_vars/all.yml:92; library/kolla_container.py:415-420). Containers run under systemd units whose stop is <engine> stop <name> -t <graceful_timeout> (module_utils/kolla_systemd_worker.py:32), with docker_graceful_timeout: 60 (group_vars/all.yml:139); a service still running then is killed (R24, Repository architecture §6) CODE. The only signal Kolla sends to a running container is a SIGHUP to nova_scheduler, which refreshes its cell cache (roles/nova/tasks/refresh_scheduler_cell_cache.yml:10-12). How the Podman path behaves on a real host is NOT AVAILABLE.
  3. One changed container restarts all containers of its role on that host (ansible-core loop-notify, lib/ansible/plugins/strategy/__init__.py:648-661 in 2.16.20). With the default Nova settings, a nova-compute config change restarts nova_libvirt and nova_ssh with nova_compute on every compute (roles/nova-cell/tasks/check-containers.yml:2-22) (Tags §2.4) CODE.
  4. Without batching, handlers restart a service on every host of the play at once. Batching is ANSIBLE_SERIAL for reconfigure/upgrade (tools/kolla-ansible:522, :531) and -e kolla_serial=N for deploy, which ignores ANSIBLE_SERIAL (tools/kolla-ansible:490-493). MariaDB and RabbitMQ roll by themselves: their restarts run in separate plays with serial: "33%" (mariadb.yml:18-52, rabbitmq.yml:30-49; one node at a time with three nodes), and each RabbitMQ node is drained first (roles/rabbitmq/tasks/restart_services.yml:9-17). Changed in 2024.1: the etcd play has no serial (site.yml:470-481), and every play sets max_fail_percentage, 100 by default (site.yml:96-99): a failed host only drops out, unless every host of a batch fails, which stops the playbook (ansible-core 2.16.20 lib/ansible/executor/playbook_executor.py:204-208); with ANSIBLE_SERIAL=1 or in the 33% restart plays of three nodes, one failure ends the run (Execution flow §5) CODE.
  5. A service tag is never just that service. For a service with a frontend it also renders the haproxy frontend config on every loadbalancer host and, if that file changed, restarts HAProxy (roles/haproxy-config/tasks/main.yml:22-23), backups first and the VIP holder last (roles/loadbalancer/handlers/main.yml:54-68, :179-193). For API services it runs Keystone registration and DB sync. It does not run common (Tags §3, Reconfigure a single service §2) CODE. --skip-tags haproxy drops the whole loadbalancer play (Reconfigure a single service §1.2).
  6. -t common does the work, and only it does. common is its own play, tagged common (site.yml:88-103), so fluentd, cron and kolla_toolbox changes need -t common or a run without tags; a service tag leaves them pending (Tags §2.1, Apply common changes) CODE. New in 2024.1: the common config step runs ensure_image on the fluentd image (roles/common/tasks/config.yml:42-51), so -t common and genconfig can pull it when it is absent.
  7. Global settings land in per-host files, and the first 2024.1 run changes some of them everywhere. enable_*, RabbitMQ/memcached membership and passwords end up in compute-side configs, so small central changes cause fleet-wide restarts (Findings F6, Why a small change creates a large workload). On a cloud deployed by 2023.1, three code changes restart containers whose operator config did not change (Predict and inspect a run §2) CODE:

    • quorum queues on by default (F17): om_enable_rabbitmq_quorum_queues: true (group_vars/all.yml:266-268) writes rabbit_quorum_queue = true into 26 templates of 24 roles (e.g. roles/nova-cell/templates/nova.conf.j2:207-216), and prechecks fails while classic queues exist;
    • the systemd unit text (F16): the 2024.1 template differs from 2023.1's (module_utils/kolla_systemd_worker.py:20-39) and the whole unit file is compared (:154-167), so the first 2024.1 check of a container whose unit 2023.1 wrote reports drift;
    • the new hostnqn file of nova_compute and cinder_volume (roles/nova-cell/tasks/config.yml:203-216).

    In addition, every service that kolla-ansible upgrade has not reached still runs a 2023.1 image: openstack_release defaults to 2024.1 (group_vars/all.yml:679-681) and the check task compares the image (module_utils/kolla_container_worker.py:75), so a deploy or reconfigure recreates it on the 2024.1 image without the role's upgrade tasks. Upgrade first, settle the queue type and list the affected containers (Pre-execution checklist §2).

  8. deploy never pulls an image that is already present. Rebuilt images under the same tag need pull (module_utils/kolla_docker_worker.py:365-367; Podman module_utils/kolla_podman_worker.py:675-677). A tag change applied with deploy recreates the entire fleet at once (Update container images, F4) CODE.

  9. --limit does not contain everything. Delegated tasks still run on groups[x][0] controllers (e.g. the Neutron DB creation, roles/neutron/tasks/bootstrap.yml:13-14), and all hosts' facts are gathered serially by the limited hosts (gather-facts.yml:34-62) (Compute scaling, Add a compute node §2) CODE.
  10. --list-tasks cannot predict a run. Use genconfig plus kolla_set_configs --check, or staging (Predict and inspect a run). kolla's set_configs.py is byte-identical to 2023.1's. Three things to know about the check CODE:
    • It compares only the files listed in config.json, not its command or permissions: memcached lists no files (roles/memcached/templates/memcached.json.j2:1-4), so a change to its command line never shows as drift. A plain reconfigure still applies it (the changed config.json notifies the restart handler), but once genconfig has written it, no later run restarts the container for it; it takes effect at the next restart for another reason (Write config first, apply in controlled batches §1.1).
    • A newly rendered optional file whose destination is not yet in the container makes --check exit 1 (kolla docker/base/set_configs.py:165-169), so the container restarts and the file is copied, as in 2023.1 (Kolla images §3).
    • It is not the whole check. compare_container also restarts a container whose image, parameters or unit text differ (module_utils/kolla_container_worker.py:58-65). New in 2024.1, with Podman a container that is not running counts as drift and an exit code other than 0 or 1, 137 included, fails the task (module_utils/kolla_podman_worker.py:428-453) (Kolla images §3.3).
  • Find it fast — typical operator questions and the section that answers them
  • Pre-execution checklist — what to check before a run, and before the first 2024.1 run on a cloud deployed by 2023.1
  • Risk register — the risks R1–R30, with their precautions (R25 does not apply to 2024.1)
  • Technical findings — the seventeen headline findings, each with its evidence (F16 and F17 are new in 2024.1)
  • Operations — the operation pages, each with its commands

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • reconfigure is deploy for 52 of the 54 roles that have a reconfigure.yml; the only lightweight path is genconfig followed by deploy-containers CODE.
  • Config is applied by a full stop/start through the container's systemd unit, never a reload, and the stop timeout is 60 s, with Docker or Podman CODE.
  • One changed container restarts all containers of its role on that host; without batching, handlers restart it on every host of the play at once, and deploy needs -e kolla_serial=N CODE.
  • The first 2024.1 run on a cloud deployed by 2023.1 restarts containers whose operator config did not change: new unit text, the new hostnqn file, quorum queues on by default; and it moves every service that kolla-ansible upgrade has not reached to its 2024.1 image CODE.
  • deploy never pulls an image that is already present: rebuilt images under the same tag need pull CODE.

Check yourself

Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. What is the only lightweight path to apply config?
Show the answer

Answer: genconfig followed by deploy-containers. reconfigure is deploy for 52 of the 54 roles that have a reconfigure.yml (swift and bifrost are the exceptions). The only lightweight path is genconfig followed by deploy-containers CODE.

2. You run kolla-ansible deploy with ANSIBLE_SERIAL=1 set. How do the handlers restart a changed service?
Show the answer

Answer: On every host of the play at once. For deploy the wrapper does not pass ANSIBLE_SERIAL; only reconfigure and upgrade do. Without batching, handlers restart a service on every host of the play at once: pass -e kolla_serial=N to deploy CODE.

3. In 2024.1, what applies fluentd, cron and kolla_toolbox changes?
Show the answer

Answer: -t common or a run without tags. common is its own play, tagged common, so these changes need -t common or a run without tags; a service tag leaves them pending CODE.

4. After the move from 2023.1, a reconfigure with no operator change restarts many containers. Which cause does kolla_set_configs --check not show?
Show the answer

Answer: The changed systemd unit text. The unit text is compared by compare_container, not by kolla_set_configs --check; the two other causes are config files that the check does report CODE.