Skip to content

Ten things to know before a production run

Ten things to know before running Kolla-Ansible 2023.1 on production. Most items link to the section that gives the evidence.

  1. reconfigure is deploy for 57 of the 59 roles that have a reconfigure.yml (swift and bifrost are the exceptions). It re-runs Keystone registration, DB creation and the DB-sync bootstrap containers for most services, Neutron's included in 2023.1. The only lightweight path is genconfig followed by deploy-containers (Execution flow §4).
  2. Config is applied by a full stop/start, never a reload, and the stop timeout is 60 s. Containers run under systemd units whose stop is docker stop -t <docker_graceful_timeout>, 60 s by default; a service still running then is killed (R24, Repository architecture §6). The only signal Kolla sends to a running container is a SIGHUP to nova_scheduler, which refreshes its cell cache.
  3. One changed container restarts all containers of its role on that host (ansible-core loop-notify). With the default Nova settings, a nova-compute config change restarts nova_libvirt and nova_ssh with nova_compute on every compute (Tags §2.4).
  4. Without batching, handlers restart a service on every host of the play at once. Batching is ANSIBLE_SERIAL for reconfigure/upgrade and -e kolla_serial=N for deploy, which ignores ANSIBLE_SERIAL. Only MariaDB and RabbitMQ roll by themselves: their restarts run in separate plays with serial: "33%" (one node at a time with three nodes), and each RabbitMQ node is drained first (Execution flow §5).
  5. A service tag is never just that service. For a service with a frontend it also runs the haproxy frontend config (and can restart HAProxy). For API services it runs Keystone registration and DB sync. In 2023.1 it no longer runs common (Tags §3, Reconfigure a single service §2).
  6. -t common now does the work, and only it does. common is its own play, tagged common, so fluentd, cron and kolla_toolbox changes need -t common or a run without tags; a service tag leaves them pending (Tags §2.1, Apply common changes). The Ussuri workaround -t chrony no longer exists.
  7. Global settings land in per-host files. enable_*, RabbitMQ/memcached membership and passwords end up in compute-side configs, so small central changes cause fleet-wide restarts (Why a small change creates a large workload).
  8. deploy never pulls an image that is already present. Rebuilt images under the same tag need pull. A tag change applied with deploy recreates the entire fleet at once (Update container images).
  9. --limit does not contain everything. Delegated tasks still run on groups[x][0] controllers, and all hosts' facts are gathered serially by the limited hosts (Compute scaling, Add a compute node §2).
  10. --list-tasks cannot predict a run. Use genconfig plus kolla_set_configs --check, or staging (Predict and inspect a run). Two things to know about the check:
    • It compares only the files listed in config.json, not its command or permissions: memcached lists no files, so a change to its command line never shows as drift. A plain reconfigure still applies it (the changed config.json notifies the restart handler), but once genconfig has written it, no later run restarts the container for it; it takes effect at the next restart for another reason (Write config first, apply in controlled batches §1.1).
    • A newly rendered optional file whose destination is not yet in the container makes --check exit 1, so the container restarts and the file is copied. In Ussuri the same case made the check exit 2 and the task fail (Kolla images §3).

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • reconfigure is deploy for 57 of the 59 roles that have a reconfigure.yml; the only lightweight path is genconfig followed by deploy-containers.
  • Config is applied by a full stop/start, never a reload, and the stop timeout is 60 s.
  • One changed container restarts all containers of its role on that host.
  • Without batching, handlers restart a service on every host of the play at once; deploy ignores ANSIBLE_SERIAL and needs -e kolla_serial=N.
  • deploy never pulls an image that is already present: rebuilt images under the same tag need pull.

Check yourself

Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. What is the only lightweight path to apply config?
Show the answer

Answer: genconfig followed by deploy-containers. reconfigure is deploy for 57 of the 59 roles that have a reconfigure.yml. The only lightweight path is genconfig followed by deploy-containers.

2. Without batching, how do handlers restart a service?
Show the answer

Answer: On every host of the play at once. Without batching, handlers restart a service on every host of the play at once. Batching is ANSIBLE_SERIAL for reconfigure/upgrade and -e kolla_serial=N for deploy, which ignores ANSIBLE_SERIAL.

3. In 2023.1, what applies fluentd, cron and kolla_toolbox changes?
Show the answer

Answer: -t common or a run without tags. common is its own play, tagged common, so these changes need -t common or a run without tags; a service tag leaves them pending. The Ussuri workaround -t chrony no longer exists.