Ten things to know before a production run¶
Ten things to know before running Kolla-Ansible 2024.1 on production. Each item restates a 2024.1 page and links to the section that gives the evidence; the main citations are repeated here.
reconfigureisdeployfor 52 of the 54 roles that have areconfigure.yml: 51 importdeploy.ymland skyline includes it (roles/skyline/tasks/reconfigure.yml:2); swift and bifrost are the exceptions. It re-runs Keystone registration, DB creation and the DB-sync bootstrap containers for most services, Neutron's included (roles/neutron/tasks/bootstrap_service.yml:2-21). The only lightweight path isgenconfig(tools/kolla-ansible:557-560) followed bydeploy-containers(tools/kolla-ansible:499-502) (Repository architecture §4.2, Execution flow §4) CODE.- Config is applied by a full stop/start, never a reload, and the stop timeout is 60 s. Changed in 2024.1: the
container module is
kolla_container, which drives Docker or Podman according tokolla_container_engine(defaultdocker,group_vars/all.yml:92;library/kolla_container.py:415-420). Containers run under systemd units whose stop is<engine> stop <name> -t <graceful_timeout>(module_utils/kolla_systemd_worker.py:32), withdocker_graceful_timeout: 60(group_vars/all.yml:139); a service still running then is killed (R24, Repository architecture §6) CODE. The only signal Kolla sends to a running container is aSIGHUPtonova_scheduler, which refreshes its cell cache (roles/nova/tasks/refresh_scheduler_cell_cache.yml:10-12). How the Podman path behaves on a real host is NOT AVAILABLE. - One changed container restarts all containers of its role on that host (ansible-core loop-notify,
lib/ansible/plugins/strategy/__init__.py:648-661in 2.16.20). With the default Nova settings, a nova-compute config change restarts nova_libvirt and nova_ssh withnova_computeon every compute (roles/nova-cell/tasks/check-containers.yml:2-22) (Tags §2.4) CODE. - Without batching, handlers restart a service on every host of the play at once. Batching is
ANSIBLE_SERIALforreconfigure/upgrade(tools/kolla-ansible:522, :531) and-e kolla_serial=Nfordeploy, which ignoresANSIBLE_SERIAL(tools/kolla-ansible:490-493). MariaDB and RabbitMQ roll by themselves: their restarts run in separate plays withserial: "33%"(mariadb.yml:18-52,rabbitmq.yml:30-49; one node at a time with three nodes), and each RabbitMQ node is drained first (roles/rabbitmq/tasks/restart_services.yml:9-17). Changed in 2024.1: the etcd play has noserial(site.yml:470-481), and every play setsmax_fail_percentage, 100 by default (site.yml:96-99): a failed host only drops out, unless every host of a batch fails, which stops the playbook (ansible-core 2.16.20lib/ansible/executor/playbook_executor.py:204-208); withANSIBLE_SERIAL=1or in the 33% restart plays of three nodes, one failure ends the run (Execution flow §5) CODE. - A service tag is never just that service. For a service with a frontend it also renders the haproxy frontend
config on every loadbalancer host and, if that file changed, restarts HAProxy
(
roles/haproxy-config/tasks/main.yml:22-23), backups first and the VIP holder last (roles/loadbalancer/handlers/main.yml:54-68, :179-193). For API services it runs Keystone registration and DB sync. It does not runcommon(Tags §3, Reconfigure a single service §2) CODE.--skip-tags haproxydrops the whole loadbalancer play (Reconfigure a single service §1.2). -t commondoes the work, and only it does.commonis its own play, taggedcommon(site.yml:88-103), so fluentd, cron and kolla_toolbox changes need-t commonor a run without tags; a service tag leaves them pending (Tags §2.1, Applycommonchanges) CODE. New in 2024.1: the common config step runsensure_imageon the fluentd image (roles/common/tasks/config.yml:42-51), so-t commonandgenconfigcan pull it when it is absent.-
Global settings land in per-host files, and the first 2024.1 run changes some of them everywhere.
enable_*, RabbitMQ/memcached membership and passwords end up in compute-side configs, so small central changes cause fleet-wide restarts (Findings F6, Why a small change creates a large workload). On a cloud deployed by 2023.1, three code changes restart containers whose operator config did not change (Predict and inspect a run §2) CODE:- quorum queues on by default (F17):
om_enable_rabbitmq_quorum_queues: true(group_vars/all.yml:266-268) writesrabbit_quorum_queue = trueinto 26 templates of 24 roles (e.g.roles/nova-cell/templates/nova.conf.j2:207-216), andprechecksfails while classic queues exist; - the systemd unit text (F16): the 2024.1 template differs from 2023.1's
(
module_utils/kolla_systemd_worker.py:20-39) and the whole unit file is compared (:154-167), so the first 2024.1 check of a container whose unit 2023.1 wrote reports drift; - the new
hostnqnfile ofnova_computeandcinder_volume(roles/nova-cell/tasks/config.yml:203-216).
In addition, every service that
kolla-ansible upgradehas not reached still runs a 2023.1 image:openstack_releasedefaults to2024.1(group_vars/all.yml:679-681) and the check task compares the image (module_utils/kolla_container_worker.py:75), so adeployorreconfigurerecreates it on the 2024.1 image without the role's upgrade tasks. Upgrade first, settle the queue type and list the affected containers (Pre-execution checklist §2). - quorum queues on by default (F17):
-
deploynever pulls an image that is already present. Rebuilt images under the same tag needpull(module_utils/kolla_docker_worker.py:365-367; Podmanmodule_utils/kolla_podman_worker.py:675-677). A tag change applied withdeployrecreates the entire fleet at once (Update container images, F4) CODE. --limitdoes not contain everything. Delegated tasks still run ongroups[x][0]controllers (e.g. the Neutron DB creation,roles/neutron/tasks/bootstrap.yml:13-14), and all hosts' facts are gathered serially by the limited hosts (gather-facts.yml:34-62) (Compute scaling, Add a compute node §2) CODE.--list-taskscannot predict a run. Usegenconfigpluskolla_set_configs --check, or staging (Predict and inspect a run). kolla'sset_configs.pyis byte-identical to 2023.1's. Three things to know about the check CODE:- It compares only the files listed in
config.json, not itscommandorpermissions: memcached lists no files (roles/memcached/templates/memcached.json.j2:1-4), so a change to its command line never shows as drift. A plainreconfigurestill applies it (the changedconfig.jsonnotifies the restart handler), but oncegenconfighas written it, no later run restarts the container for it; it takes effect at the next restart for another reason (Write config first, apply in controlled batches §1.1). - A newly rendered optional file whose destination is not yet in the container makes
--checkexit 1 (kolladocker/base/set_configs.py:165-169), so the container restarts and the file is copied, as in 2023.1 (Kolla images §3). - It is not the whole check.
compare_containeralso restarts a container whose image, parameters or unit text differ (module_utils/kolla_container_worker.py:58-65). New in 2024.1, with Podman a container that is not running counts as drift and an exit code other than 0 or 1, 137 included, fails the task (module_utils/kolla_podman_worker.py:428-453) (Kolla images §3.3).
- It compares only the files listed in
Related¶
- Find it fast — typical operator questions and the section that answers them
- Pre-execution checklist — what to check before a run, and before the first 2024.1 run on a cloud deployed by 2023.1
- Risk register — the risks R1–R30, with their precautions (R25 does not apply to 2024.1)
- Technical findings — the seventeen headline findings, each with its evidence (F16 and F17 are new in 2024.1)
- Operations — the operation pages, each with its commands
To remember
Lumo: if you keep only 5 things from this page, keep these.
reconfigureisdeployfor 52 of the 54 roles that have areconfigure.yml; the only lightweight path isgenconfigfollowed bydeploy-containersCODE.- Config is applied by a full stop/start through the container's systemd unit, never a reload, and the stop timeout is 60 s, with Docker or Podman CODE.
- One changed container restarts all containers of its role on that host; without batching, handlers restart it on every host of the play at once, and
deployneeds-e kolla_serial=NCODE. - The first 2024.1 run on a cloud deployed by 2023.1 restarts containers whose operator config did not change: new unit text, the new
hostnqnfile, quorum queues on by default; and it moves every service thatkolla-ansible upgradehas not reached to its 2024.1 image CODE. deploynever pulls an image that is already present: rebuilt images under the same tag needpullCODE.
Check yourself
Lumo: 4 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.