Skip to content

Deployment execution flow

How Kolla-Ansible 2023.1 decides what runs when: play order, the steps inside each role, handler flush points, batching, and how deploy, reconfigure, upgrade, deploy-containers, genconfig, pull and stop differ.

Note

Key answers

  • There is no dependency resolution. Order comes from play order (common is now a play of its own, not a role dependency), task order inside a role, handler flush points, serial, and the restart plays that MariaDB and RabbitMQ handlers fill through group_by CODE (§1).
  • What each action does (deploy, reconfigure, upgrade, deploy-containers, genconfig, pull, stop, precheck, validate-config; the wrapper no longer offers check): the action matrix (§4). genconfig, then deploy-containers --limit <batch>, is the only built-in way to separate "write the config" from "apply it" (§4).
  • Batching: a batch runs the complete play, handler restarts included, before the next batch starts, but each play is still traversed batch by batch for the whole fleet before the next play starts. MariaDB and RabbitMQ apply their role to all hosts in one batch, then restart the notified hosts in separate plays with serial: "33%" (§5).
  • reconfigure on a running cloud still runs the DB sync containers of keystone, glance, cinder, placement, nova, neutron, heat and others CODE. In 2023.1 Neutron's DB sync is no longer skipped (§6).
  • What one compute host goes through in a full deploy, in order (§2.4). On a fresh compute, nova_compute is started before openvswitch and neutron_openvswitch_agent exist on that host.
  • Restart order is handler definition order, not notification order CODE, and the handlers of a play run on all notified hosts in parallel (§7).

Deployment order (this page) ≠ runtime communication (see Component dependencies). Arrows in the diagrams below mean "runs before", unless a diagram explicitly says it shows runtime calls.


1. How the order is decided

There is no dependency resolution in Kolla-Ansible 2023.1. Order comes from these mechanisms only CODE:

Mechanism Effect Evidence
Play order in site.yml / nova.yml / mariadb.yml / rabbitmq.yml Services are applied strictly in file order. Ansible's default linear strategy runs play N on all its hosts (per batch) before play N+1 ansible/site.yml, generated list in Generated inventory §1
Role meta dependency on common None in 2023.1. No role has a meta/ directory. common is the Apply role common play, run after the prechecks play and before the loadbalancer play, on the cron, fluentd, kolla-logs and kolla-toolbox hosts, tagged common site.yml:86-97, roles/common/tasks/main.yml:1-2
Task order inside a role Fixed sequence: register → config(-host) → check-containers → bootstrap → flush handlers (§3) roles/*/tasks/deploy.yml
Handler flush points Restarts happen at meta: flush_handlers (end of most deploy.yml) or at the end of the play. The loadbalancer role deliberately defers to the end of its play e.g. roles/nova/tasks/deploy.yml:8-9; roles/loadbalancer/tasks/deploy.yml:8-11
Restart plays filled by handlers MariaDB and RabbitMQ handlers only put hosts into groups (group_by); the following plays of mariadb.yml / rabbitmq.yml restart those groups with serial: "33%" roles/mariadb/handlers/main.yml:51-73, mariadb.yml:14-52; roles/rabbitmq/handlers/main.yml:2-4, rabbitmq.yml:26-41
serial / kolla_serial Splits a play into batches; each batch completes the whole play, handlers included, before the next starts serial: '{{ kolla_serial|default("0") }}' on the gather-facts plays, the common play and every service play, except the plays of mariadb.yml and rabbitmq.yml (no serial, or a fixed "33%" on the restart plays) and cyborg, which reads serial: '{{ serial|default("0") }}' (site.yml:784), so kolla_serial never batches it. The group_by and prechecks plays have no serial key either

Conditions only remove work, never reorder it:

  • when: enable_<svc> on the role;
  • the host-pattern intersection &enable_<svc>_True;
  • per-task when: on kolla_action;
  • placement filters (inventory_hostname in groups[...] / host_in_groups).

2. The five diagrams

2.1 Infrastructure services (deployment order)

CODE (gather-facts.yml, site.yml:1-440, mariadb.yml, rabbitmq.yml)

flowchart TB
  F["gather-facts (all hosts; delegated when --limit)"] --> G["group_by: kolla_action, enable_* flags"]
  G --> P["prechecks (only kolla_action=precheck)"]
  P --> CO["common (own play, tag common: fluentd, kolla_toolbox, cron)"]
  CO --> HA["loadbalancer: haproxy + keepalived (+ proxysql)<br/>+ 44 per-service loadbalancer includes<br/>(handlers flushed at END of this play)"]
  HA --> MON["collectd → influxdb → telegraf → redis"]
  MON --> DB["mariadb.yml: apply role (no serial) → restart / start plays (serial 33%) → bootstrap-host restart → post-config"]
  DB --> MC["memcached"]
  MC --> PR["prometheus → iscsi → multipathd"]
  PR --> MQ["rabbitmq.yml: rabbitmq → restart (serial 33%) → post-config; same for outward rabbitmq"]
  MQ --> QD["etcd"]
  QD --> KS["keystone ⇒ control plane (2.2)"]
  style HA fill:#ffe9b3
  style DB fill:#ffd6d6
  style MQ fill:#ffd6d6

The Ussuri plays chrony, zookeeper, elasticsearch, kibana and qdrouterd no longer exist in site.yml; OpenSearch now comes after Keystone (site.yml:460-468).

2.2 OpenStack control plane (deployment order)

CODE (site.yml:450-968, nova.yml)

flowchart LR
  KS["keystone"] --> OS["opensearch"] --> SW["swift → ceph-rgw"] --> GL["glance"] --> IR["ironic"] --> CI["cinder"] --> PL["placement"] --> NOVA["nova.yml (2.3)"]
  NOVA --> NET["openvswitch / ovs-dpdk → ovn-controller → ovn-db → neutron → kuryr (2.5)"]
  NET --> HC["hacluster"] --> HE["heat"] --> HO["horizon"] --> MISC["murano → solum → magnum → mistral → sahara"]
  MISC --> MA["manila"] --> TEL["gnocchi → ceilometer → aodh"] --> BA["barbican"] --> MORE["cyborg → designate → trove → watcher → grafana → cloudkitty → freezer → senlin → tacker"]
  MORE --> OC["octavia"] --> TAIL["zun → vitrage → blazar → masakari → venus → skyline"]

Notable order facts CODE:

  • Keystone comes after MariaDB, RabbitMQ and memcached, but before any service that registers with it. service-ks-register uses openstack_interface (internal) and authenticates against keystone_internal_url (roles/service-ks-register/defaults/main.yml:5, group_vars/all.yml:609,865,895-896).
  • Placement, Glance and Cinder come before Nova. Neutron comes after Nova (see §2.5).
  • Barbican comes after Nova, Cinder and Glance, even though they can be configured to use it as a key manager. With enable_barbican, Nova and Cinder render a [barbican] section (roles/nova/templates/nova.conf.j2:204-207, roles/nova-cell/templates/nova.conf.j2:246-249, roles/cinder/templates/cinder.conf.j2:238-241), and their plays run before the barbican play (site.yml:519,544,765). On a first deploy they therefore start configured with a [barbican] section before Barbican exists. NOT AVAILABLE Whether this has any effect before encrypted volumes are used.
  • Octavia is near the end. With octavia_auto_configure (default: true when the amphora provider is enabled, group_vars/all.yml:1228), its play creates the amphora flavor, the keypair, the security groups and the management network (roles/octavia/tasks/deploy.yml:4-5, prepare.yml). No task of the role uploads the amphora image, and the certificates come from the separate octavia-certificates command (tools/kolla-ansible:536) (fact sheet D).

2.3 Nova deployment (ansible/nova.yml)

flowchart TB
  subgraph P1["Play 1: Bootstrap nova API databases — hosts nova-api (tags nova, nova-bootstrap, nova-api, nova-api-bootstrap)"]
    A1["mysql_db nova_api + nova_cell0 (run_once, kolla_toolbox)"] --> A2["mysql_user grants (run_once)"] --> A3["config_bootstrap: /etc/kolla/nova-api-bootstrap (run_once)"]
    A3 --> A4["nova_api_bootstrap container: api_db sync + db sync (run_once)"] --> A5["nova_api_map_cell0 + list_cells for cell0 (+ update cell0 if its DB URL differs) (delegate groups['nova-api'][0])"]
    A5 -->|"map_cell0 changed"| A6["nova_api_bootstrap again"]
  end
  subgraph P2["Play 2: Bootstrap nova cell databases — hosts nova-conductor, any_errors_fatal"]
    B1["cell DB + user (on groups[nova_cell_conductor_group][0])"] --> B2["RabbitMQ vhost/user (service-rabbitmq, delegated to the cell RPC group's first host)"] --> B3["nova-cell-bootstrap config"] --> B4["nova_cell_bootstrap container (cell db sync)"] --> B5["list_cells → create_cell or update_cell"]
  end
  subgraph P3["Play 3: Apply role nova — nova-api, nova-scheduler, nova-super-conductor"]
    C1["service-ks-register (run_once)"] --> C2["config.yml"] --> C3["check-containers"] --> C4["flush: super-conductor → scheduler → api"]
  end
  subgraph P4["Play 4: Apply role nova-cell — compute, nova-compute-ironic, nova-conductor, consoles"]
    D1["config-host (br_netfilter, sysctl, udev, qemu-kvm mask)"] --> D2["config.yml (nova.conf, libvirt, ssh, ceph)"] --> D3["check-containers"] --> D4["flush: conductor → consoles → ssh → libvirt → libvirt ready check + SASL user → compute"] --> D5["wait for nova-compute registration (first compute of batch, delegated to conductor[0], 20×10s)"] --> D6["list_cells + discover_hosts --by-service (delegated to conductor[0])"] --> D7["post-config: remove old nova_libvirt_secrets volume"]
  end
  subgraph P5["Play 5: Refresh scheduler cell cache — kill -HUP nova_scheduler (deploy/reconfigure)"]
  end
  subgraph P6["Plays 6-10: reloads + online data migrations — upgrade only"]
  end
  P1 --> P2 --> P3 --> P4 --> P5 --> P6

CODE (nova.yml:29-156; roles/nova/tasks/bootstrap.yml:49-56, map_cell0.yml:2-67; roles/nova-cell/tasks/bootstrap.yml:37-43, rabbitmq.yml:2-24, create_cells.yml:2-55; roles/nova-cell/tasks/deploy.yml:5-39, wait_discover_computes.yml:27-28,42-43,87-90, discover_computes.yml:5,16-21, post-config.yml:3-7; handler order roles/nova/handlers/main.yml:2,20,38, roles/nova-cell/handlers/main.yml:2-151; roles/nova/tasks/refresh_scheduler_cell_cache.yml:10)

Details and Nova runtime dependencies: Nova deep dive.

2.4 Compute node deployment (what one compute host experiences)

Sequence on one compute host during a full deploy, with the default enable flags and the multinode inventory groups (iscsi only with enable_iscsid) CODE (facts and group_by first (site.yml:2,7-77); play order site.yml:86-97,417-426, nova.yml:116-138 (imported at site.yml:544), site.yml:546-555,590-608; task and handler order in each role's tasks/deploy.yml and handlers/main.yml; common in its own play (site.yml:86-97, roles/common/tasks/deploy.yml:2-9); discovery on the first compute of the batch only, delegated to the first conductor, roles/nova-cell/tasks/wait_discover_computes.yml:42-43,87-90 and discover_computes.yml:21):

sequenceDiagram
  autonumber
  participant D as Deploy host
  participant C as compute001
  D->>C: setup (facts), group_by (always)
  Note over D,C: Play "common" (compute is in the common group)
  D->>C: common: config dirs, config.json ×3, td-agent.conf, cron logrotate config, toolbox Erlang cookie, rabbitmq-env.conf, erl_inetrc
  D->>C: common: check containers (fluentd, kolla_toolbox, cron), create kolla_logs volume, link /var/log/kolla
  D->>C: handlers: Restart fluentd → kolla_toolbox (+ toolbox init) → cron
  Note over D,C: Play "iscsi" (only if enable_iscsid)
  D->>C: iscsid config/check → Restart iscsid
  Note over D,C: Play "nova-cell" (after controllers' nova plays have bootstrapped DBs/cell)
  D->>C: br_netfilter, sysctl, udev kvm rule, config dirs, config.json, nova.conf, qemu.conf, libvirtd.conf, SASL, ssh keys
  D->>C: check nova-cell containers
  D->>C: handlers: Restart nova-ssh → nova-libvirt → libvirt ready check → Create libvirt SASL user → nova-compute
  D->>C: (first compute of batch only) wait for nova-compute services, discover_hosts — delegated to conductor[0]
  D->>C: post-config: remove old nova_libvirt_secrets volume
  Note over D,C: Play "openvswitch"
  D->>C: load openvswitch module, config, check → Restart openvswitch_db → wait ovsdb → Restart vswitchd
  D->>C: post-config: set system-id and hw-offload through openvswitch_db (notifies vswitchd restart), bridge ensure only if computes need external bridge
  Note over D,C: Play "neutron"
  D->>C: ip6_tables module, sysctl (only on l3 hosts), config (neutron.conf, openvswitch_agent.ini), check
  D->>C: handler: Restart neutron-openvswitch-agent

The common steps are roles/common/tasks/config.yml, check-containers.yml and bootstrap.yml:2-16. The openvswitch post-config.yml:3-22 runs after the role's flush, so a vswitchd restart it notifies runs at the end of the play. The neutron module and sysctl steps are roles/neutron/tasks/config-host.yml:1-42.

NOT AVAILABLE How many task results, skips and handlers a compute host or a controller produces in a full deploy: that depends on the inventory, the enable flags and the when: conditions met at run time. The first controller also runs the run_once and delegated bootstrap tasks (e.g. roles/nova/tasks/map_cell0.yml:63-64).

Ordering consequence: on a fresh compute, nova_compute is started (and its registration awaited) before openvswitch and neutron_openvswitch_agent exist on that host. NOT AVAILABLE Whether instances can be scheduled there successfully before the later plays finish (Nova deep dive §3 item 6). On a running cloud this ordering matters when you add computes with --limit (see Add a compute node §2).

2.5 Networking components

flowchart TB
  subgraph Order["Deployment order"]
    direction TB
    O1["loadbalancer play: neutron-server frontend"] --> O2["openvswitch play: openvswitch_db → vswitchd (network, compute, manila-share)"]
    O2 --> O2b["post-config: system-id, hw-offload; br-ex + external interface on network nodes"]
    O2b --> O3["ovs-dpdk play (instead of openvswitch when enable_ovs_dpdk)"]
    O3 --> O4["ovn-controller play (if neutron_plugin_agent=ovn): br-int, OVN settings in OVSDB, ovn_controller"]
    O4 --> O4b["ovn-db play: NB/SB DB cluster, northd"]
    O4b --> O5["neutron play: register → config-host → config → check → bootstrap (DB/user and DB-sync container) → flush"]
    O5 --> O6["kuryr (if enabled)"]
  end
  subgraph Within["Inside the neutron play (all hosts in parallel)"]
    N1["neutron-server (controllers)"]
    N2["l3 / dhcp / metadata / ovs agents (network nodes)"]
    N3["ovs agent (+ l3/metadata if DVR) or ovn-metadata agent (computes)"]
  end
  O5 --> Within

CODE (site.yml:546-608; roles/openvswitch/tasks/deploy.yml, post-config.yml:3-37; roles/ovn-controller/tasks/deploy.yml, setup-ovs.yml; roles/ovn-db/tasks/deploy.yml; roles/neutron/tasks/deploy.yml:2-23, bootstrap.yml:38, bootstrap_service.yml:2-21)

In 2023.1 the OVN controller play runs before the OVN database play. The neutron play deploys the server and all agents in a single play and a single handler flush. Agents are not deployed after the server has been verified. NOT AVAILABLE Whether the RPC layer tolerates the agents starting first.


3. Inside a role: the standard deploy sequence

Typical deploy.yml (Nova shown; Neutron adds config-host and runs its bootstrap before the flush):

Step Typical task file Scope Runs on reconfigure?
1. Keystone registration register.yml → service-ks-register run_once (per batch) via kolla_toolbox on the first host of the play yes
2. Host configuration config-host.yml (sysctl, modules, udev) per host yes
3. Config generation config.yml (dirs, config.json, INI merges, policy, certs) per host × per service × per file yes
4. Container check check-containers.yml (compare_container: container parameters, systemd unit file, kolla_set_configs --check) per host × per container yes
5. Bootstrap bootstrap.yml (DB, user, grants) + bootstrap_service.yml (one-shot container: DB sync) run_once / first host of the group yes for most roles, Neutron included: its bootstrap_neutron DB-sync container has no kolla_action or neutron_enable_rolling_upgrade condition in 2023.1 (roles/neutron/tasks/bootstrap.yml:38, bootstrap_service.yml:2-21)
6. Flush handlers meta: flush_handlers notified hosts, in handler definition order yes
7. Post steps e.g. openvswitch post-config.yml, nova-cell discovery and post-config.yml varies yes

CODE (roles/nova/tasks/deploy.yml:2-9; roles/neutron/tasks/deploy.yml:2-23; module_utils/kolla_container_worker.py:59-66)

Order caveats:

  • Some roles bootstrap before containers start, some after. Nova bootstraps in separate plays before the service plays (nova.yml:29-96). Neutron runs its bootstrap after check-containers but before the flush (roles/neutron/tasks/deploy.yml:13-23). Keystone bootstraps before its flush and registers last, after Fernet key distribution (roles/keystone/tasks/deploy.yml:2-21). Glance and Cinder vary (fact sheets B, C).
  • The loadbalancer role does not flush in deploy.yml. The NOTE in roles/loadbalancer/tasks/deploy.yml:8-11 explains that site.yml handles all haproxy changes in that play, so the restart happens once, after all per-service frontends are rendered.

4. Action matrix: what each kolla_action does

Action (CLI) Config written Containers compared Handlers (restarts) Bootstrap / DB sync Keystone registration Images pulled Notes
deploy yes yes yes yes yes only if absent Full flow
reconfigure yes yes yes yes (most roles) yes only if absent Identical to deploy in 57 of the 59 roles that have a reconfigure.yml (swift and bifrost are the exceptions) (Repository architecture §4.2). kolla_serial passed from ANSIBLE_SERIAL (tools/kolla-ansible:517)
upgrade yes yes yes yes: most roles re-run the deploy KOLLA_BOOTSTRAP one-shot with the new image (cinder, heat, barbican, octavia, manila, horizon, placement, legacy glance/neutron). Upgrade-only one-shots: keystone KOLLA_UPGRADE then KOLLA_FINISH_UPGRADE, run as handlers (roles/keystone/tasks/upgrade.yml:23-34); neutron rolling (default) NEUTRON_DB_EXPAND/NEUTRON_DB_CONTRACT; glance rolling (opt-in, glance_enable_rolling_upgrade: "no", group_vars/all.yml:912) GLANCE_DB_EXPAND/MIGRATE/CONTRACT; mariadb upgrade_mariadb (roles/mariadb/tasks/post-upgrade.yml:2-21); KOLLA_OSM online migrations (nova, placement, cinder, rolling ironic). Nova bootstrap_upgrade re-runs its deploy db-sync container, minus DB creation and cell0 mapping (Kolla images §4) yes (some roles) only if absent → run pull first Nova reload plays and the separate KOLLA_OSM online-migration containers are active only here. kolla_serial passed from ANSIBLE_SERIAL (tools/kolla-ansible:508). Placement's bootstrap container also runs placement-manage db online_data_migrations on every deploy/reconfigure (kolla docker/placement/placement-api/extend_start.sh:5-9); so does Ironic's with ironic_enable_rolling_upgrade: no (roles/ironic/tasks/bootstrap_service.yml:16, kolla docker/ironic/ironic-api/extend_start.sh:5-9)
deploy-containers no yes (incl. drift of the files listed in config_files; not command/permissions changes inside config.json, see below) yes no no only if absent tasks/deploy-containers.yml = check-containers.yml (e.g. roles/nova-cell/tasks/deploy-containers.yml:2). Loadbalancer includes skipped (site.yml:339-341)
genconfig (config) yes no all container restarts skipped (when: kolla_action != "config"; the unguarded handlers are waits, checks and bootstrap steps chained from guarded handlers, plus swift's no-op debug handler; the Ussuri exception, the nova-hyperv role, no longer exists, Pre-execution checklist) no no no tools/kolla-ansible:540-542. The Ussuri fluentd label check, which pulled the fluentd image, is gone from roles/common/tasks/config.yml. With octavia_auto_configure, octavia's config tasks query the cloud through kolla_toolbox (roles/octavia/tasks/config.yml:2-3)
pull no no no no no yes, every enabled container mapped to the host, plus common (in its own play) service-images-pull (roles/service-images-pull/tasks/main.yml:2-17)
stop no no no no no no stop_container for every enabled container on the host, honouring skip_stop_containers (roles/service-stop/tasks/main.yml:2-12)
precheck no no no no no no prechecks role + each role's precheck.yml (port checks, group validation)
validate-config (config_validate, new) no no no no no no tools/kolla-ansible:544-546. Runs oslo-config-validator inside each running container that has a validation list, and writes the errors under service_config_validate_output_dir on the deploy host (roles/service-config-validate/tasks/main.yml:2-13, validate.yml:2-49)
check – – – – – – No longer a kolla-ansible command in 2023.1: the wrapper has no check case (tools/kolla-ansible:429-578). The roles still carry a check.yml, reached only if kolla_action=check is passed to the playbooks some other way

The combination genconfig → (review the diff) → deploy-containers --limit <batch> is the only built-in way to separate "write the config" from "apply it", and to apply in controlled batches. It works because compare_container detects drift in the files listed in each config.json's config_files (content, mode, owner) through kolla_set_configs --check CODE (module_utils/kolla_container_worker.py:59-66, module_utils/kolla_docker_worker.py:155-190, kolla docker/base/set_configs.py:165-198,220-242,556-559).

Changes confined to config.json itself (command, permissions; e.g. every memcached setting, roles/memcached/templates/memcached.json.j2:2-3) are NOT detected. deploy-containers therefore does not apply them after genconfig, and a later deploy/reconfigure does not either: the config tasks that notify the restart, the config.json template task included (memcached's only one, roles/memcached/tasks/config.yml:13-26), are then unchanged. Such changes take effect only at the next restart for another reason.

An optional file that newly appears, and whose destination does not exist in the container yet, now counts as drift: --check exits 1 and the restart copies the file. In Ussuri the same case made the check exit 2 and the task fail CODE (kolla docker/base/set_configs.py:36-39,165-171,220-242,576-578, module_utils/kolla_docker_worker.py:178-190). See Write config first, apply in controlled batches §2 and Kolla images §3.


5. Batching (serial) and its interaction with order

gantt
  dateFormat X
  axisFormat %s
  title Play execution with kolla_serial=2 (illustrative, 4 computes)
  section nova-cell play
  batch 1 (compute001-002) config→check→flush→discover :a1, 0, 30s
  batch 2 (compute003-004) config→check→flush→discover :a2, after a1, 30s
  section openvswitch play
  batch 1 :b1, after a2, 15s
  batch 2 :b2, after b1, 15s

The bar lengths are illustrative only; no duration is known from the code.

  • A batch runs the complete play before the next batch starts. That includes handler restarts and the compute discovery. Batches therefore bound the number of simultaneous restarts per play, but each play is still traversed batch by batch for the whole fleet before the next play starts. The neutron agents on compute001 are restarted only after all computes have finished the nova-cell and openvswitch plays.
  • run_once tasks run once per batch (Tags §2.5). Examples: DB sync, Keystone registration, compute discovery (wait_discover_computes.yml runs for all_computes_in_batch[0]).
  • MariaDB and RabbitMQ are not batched by kolla_serial. Their role plays run all hosts in one batch; their handlers only add the hosts to restart groups, and the next plays restart those groups with serial: "33%" (mariadb.yml:14-40, rabbitmq.yml:26-41,82-98). With three nodes that is one node per batch (ansible-core 2.14.18 lib/ansible/utils/helpers.py:32). Each RabbitMQ node is put into maintenance mode before its restart (roles/rabbitmq/tasks/restart_services.yml:9-17) (fact sheet B). cyborg is not batched by kolla_serial either: its play reads serial instead.
  • How kolla_serial is set depends on the action. For reconfigure and upgrade the wrapper sets it from ANSIBLE_SERIAL (tools/kolla-ansible:290,508,517), after your own -e (tools/kolla-ansible:360-362), so use the environment variable. For deploy the wrapper does not set it (tools/kolla-ansible:476-478): pass -e kolla_serial=N yourself.

6. Initial deployment vs reconfiguration: what actually differs

Aspect Initial deploy (empty hosts) reconfigure on a running cloud
Containers All create+start: the container check finds no container, and the restart handler then starts a new one under its systemd unit CODE (module_utils/kolla_container_worker.py:59-66, module_utils/kolla_docker_worker.py:362-371,386-405). Exception: on the bootstrap host (groups[mariadb_shard_group][0]), mariadb runs first as the one-shot bootstrap_mariadb, then a handler starts it with --wsrep-new-cluster and waits for its port and WSREP sync, and the "Restart bootstrap mariadb service" play runs recreate_or_restart_container on it (roles/mariadb/tasks/bootstrap.yml:4-7, bootstrap_cluster.yml:2-27, roles/mariadb/handlers/main.yml:2-54, mariadb.yml:42-52, tasks/restart_services.yml:2-15). NOT AVAILABLE Container counts, which depend on the inventory and the enable flags Restart (config) or recreate (image or params) only if drift is detected; otherwise none CODE (module_utils/kolla_container_worker.py:59-66, module_utils/kolla_docker_worker.py:362-384)
One-shot containers DB bootstrap for every service, MariaDB and RabbitMQ bootstrap, fernet key generation, cell creation DB sync containers still run for keystone, glance, cinder, placement, nova (API and cell DB sync, plus map_cell0, list_cells for cell0 and list_cells ×2), neutron, heat, plus barbican, octavia and others CODE (in these roles reconfigure.yml imports deploy.yml and the bootstrap containers have no kolla_action guard, e.g. roles/keystone/tasks/bootstrap_service.yml:33-50; nova: nova.yml:45-51,81-87, roles/nova/tasks/map_cell0.yml:2-30, list_cells from roles/nova-cell/tasks/create_cells.yml:2 and discover_computes.yml:5; neutron: roles/neutron/tasks/bootstrap_service.yml:2-21). MariaDB/RabbitMQ bootstrap and the keystone Fernet bootstrap (when: groups['keystone_fernet_running'] is not defined, roles/keystone/tasks/bootstrap_service.yml:52-75) are not re-run
Handler order effects Every handler fires Only notified handlers fire, but loop-notify widens them to all containers of the role on that host (Tags §2.4)
Neutron DB sync (bootstrap_neutron) runs runs too in 2023.1 (no condition on kolla_action or neutron_enable_rolling_upgrade, roles/neutron/tasks/bootstrap_service.yml:2-21). The DB/user/grant tasks run as well
Nova cell create_cell update_cell only if the transport URL or DB URL changed (roles/nova-cell/tasks/create_cells.yml:31-55)
Fernet keys generated re-distributed from the first running keystone_fernet node to the others (fact sheet B §3.6, task flow in §3.3)
MariaDB Galera bootstrap on the first host of the shard cluster lookup; if config changed, restart plays with serial: "33%" (mariadb.yml:14-40)
Duration drivers image pulls, first container starts per-task SSH overhead across all hosts, compare_container exec per container (see Compute scaling). NOT AVAILABLE Any duration

7. Dependencies introduced by handlers

Handlers create ordering that is not visible in task files:

  1. Within a role, restart order = handler definition order, not notification order.
    • nova-cell restarts nova_ssh → nova_libvirt (→ libvirt ready check, SASL user) → nova_compute (roles/nova-cell/handlers/main.yml:74-151).
    • openvswitch restarts openvswitch_db → waits → openvswitch_vswitchd (roles/openvswitch/handlers/main.yml:2-29).
    • CODE ansible-core 2.14.18 copies the play's handler list in definition order into each host's state and walks it in that order, running the handlers notified for the host (lib/ansible/executor/play_iterator.py:192-197,430-452).
  2. Handlers of a play run on all notified hosts in parallel (ansible-core 2.14.18: the linear strategy puts all hosts in lockstep on the same handler, lib/ansible/plugins/strategy/linear.py:84-123). With serial: 0, every compute restarts nova_compute (and, through loop-notify, nova_libvirt and nova_ssh) in the same step.
  3. Handler chains:
    • Restart nova-libvirt → Checking libvirt container is ready → Create libvirt SASL user (with libvirt_enable_sasl, roles/nova-cell/handlers/main.yml:96,116).
    • Restart openvswitch-db → Waiting for openvswitch_db.
    • Restart kolla-toolbox → Initializing toolbox (roles/common/handlers/main.yml:34-37).
    • Restart haproxy / proxysql / keepalived container → handlers that listen to these names: check the API interface addresses, group the hosts by whether they hold the VIP, stop and start the backup hosts' containers, then the master's, then wait for HAProxy (and ProxySQL) to listen on the VIP (roles/loadbalancer/handlers/main.yml:9-329).
    • MariaDB and RabbitMQ: restart handlers only group_by the host; the restart itself is a later play (item 5).
  4. Handlers can cross roles. haproxy-config (imported through loadbalancer-config by every service's loadbalancer.yml) notifies the loadbalancer role's Restart haproxy container (roles/haproxy-config/tasks/main.yml:22-23). This is why the loadbalancer play includes the loadbalancer role even for -t <svc>.
  5. The MariaDB restart is a sequence of plays, not of handler waves. The handlers put running members into mariadb_restart and new members into mariadb_start (roles/mariadb/handlers/main.yml:56-73); the "Restart mariadb services" and "Start mariadb services" plays then run restart_services.yml (restart, port wait, WSREP sync wait) with serial: "33%" (mariadb.yml:14-40, roles/mariadb/tasks/restart_services.yml:2-51). The Ussuri index % 4 waves no longer exist (fact sheet B §1.5).

To remember

Lumo: if you keep only 5 things from this page, keep these.

  • There is no dependency resolution: order comes from play order, task order inside a role, handler flush points, serial and the MariaDB and RabbitMQ restart plays. CODE
  • genconfig, then deploy-containers --limit <batch>, is the only built-in way to separate "write the config" from "apply it".
  • A batch runs the complete play, handler restarts included, before the next batch starts, but each play is traversed batch by batch for the whole fleet first.
  • reconfigure on a running cloud still runs the DB sync containers of keystone, glance, cinder, placement, nova, neutron, heat and others. CODE
  • Restart order is handler definition order, not notification order. CODE

Check yourself

Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.

1. In which order do the restart handlers of a play run?
Show the answer

Answer: In the order the handlers are defined. Restart order is handler definition order, not notification order CODE, and the handlers of a play run on all notified hosts in parallel. Read the section.

2. Which built-in sequence separates "write the config" from "apply it"?
Show the answer

Answer: genconfig, then deploy-containers --limit <batch>. It is the only built-in way to separate the two steps; the action matrix says what each action does. Read the section.

3. Does reconfigure on a running cloud run the DB sync containers?
Show the answer

Answer: Yes, of keystone, glance, cinder, placement, nova, neutron, heat and others. reconfigure still runs the DB sync containers CODE. In 2023.1 Neutron's DB sync is no longer skipped. Read the section.