Reconfigure Nova¶
Impact Restarts on drift
What kolla-ansible -t nova reconfigure does on the loadbalancer hosts, on the controllers and on every compute, with
the
two narrower commands -t nova-api reconfigure (only the Nova control plane, no compute impact) and
-t nova-cell reconfigure (only the cell part: conductors, consoles, computes).
The scenario sections are written for the assumed deployment; the page uses the conventions of the command sections, the definition of drift and the notion of code guarantees and environment dependence.
Note
Key answers
-t nova reconfigure(§2 to §6): every compute runs the tasks. Restarts happen only where drift is detected, in practice on no compute or on all of them, andnova_libvirtandnova_sshrestart whenevernova_computerestarts because of config or image drift (§4). Whether running VMs survive is NOT AVAILABLE (§5).- First 2024.1 run after 2023.1: expect drift on every Nova container, computes included: the systemd unit text
changed (F16), quorum queues are on by default (F17)
and
nova_computehas a newhostnqnfile (§3). - Nova without touching computes:
-t nova-api reconfigure(§1.2).
Warning
R1: a nova-compute drift restarts nova_compute, nova_libvirt and nova_ssh on
each compute where it occurs, so a change common to all computes restarts them on every compute — see
§4.
1. Commands¶
1.1 Reconfigure Nova (control plane and computes)¶
kolla-ansible -i multinode -t nova reconfigure
- Scope:
- loadbalancer play (nova and nova-cell frontends;
site.yml:252-265, inside a block that runs fordeploy,reconfigure,upgradeandconfigwhenenable_haproxyis set,site.yml:324-326); - the 10 nova plays (
nova.yml:29-302), the nova-cell play on every compute (nova.yml:125-147); commondoes not run under-t nova: it is its own play, taggedcommononly (site.yml:88-103).
- loadbalancer play (nova and nova-cell frontends;
- One-shot: 6 container runs on
control01(API DB sync,map_cell0,list_cellsfor cell0, cell DB sync,list_cells×2) plus RabbitMQ user/vhost, Keystone registration, SIGHUP ofnova_scheduler,discover_hostsCODE (when cell0 is mapped with an unchanged connection and the cell is unchanged:roles/nova/tasks/bootstrap.yml:54-56,roles/nova/tasks/map_cell0.yml:2-30,roles/nova-cell/tasks/bootstrap.yml:37-43,roles/nova-cell/tasks/create_cells.yml:2-4,roles/nova-cell/tasks/discover_computes.yml:4-18,roles/nova/tasks/deploy.yml:2; Tags §3). - Restarts: depend on the drift (§2 to §6). A
nova-compute drift restarts
nova_compute,nova_libvirtandnova_sshon each compute where it occurs, so a change common to all computes restarts them on every compute CODE (roles/nova-cell/tasks/check-containers.yml:2-22, loop-notify, Tags §2.4). - Failures: new in 2024.1, seven of the ten plays set
max_fail_percentage(default 100, which never stops a play early); the cell DB bootstrap play and the two online-data-migration plays setany_errors_fatal: trueinstead (nova.yml:40-43,nova.yml:79,nova.yml:141-144,nova.yml:269-270,nova.yml:294-295) CODE.
1.2 Reconfigure only the Nova control plane (no compute impact)¶
kolla-ansible -i multinode -t nova-api reconfigure
- Scope: the
nova-api-tagged plays (API DB bootstrap, nova role, scheduler refresh, and the API reload and online-migration plays, which do nothing outsideupgrade) on controllers (nova.yml:29-65,nova.yml:103-122,nova.yml:152-172,nova.yml:180-201,nova.yml:231-277), plus the loadbalancer nova include (site.yml:252-258). - Not touched: conductors and computes (
nova-cellplays,nova.yml:67-100,nova.yml:124-147).
1.3 Reconfigure only the cell part (conductors, consoles, computes)¶
kolla-ansible -i multinode -t nova-cell reconfigure
- Scope: the
nova-cell-tagged plays (cell DB bootstrap, nova-cell role, and the cell reload and online-migration plays, which do nothing outsideupgrade) (nova.yml:67-100,nova.yml:124-147,nova.yml:203-229,nova.yml:283-302), plus the loadbalancer nova-cell include for the console proxies (site.yml:259-265). The nova-cell play reaches every compute, as with-t nova.
2. Execution sequence (kolla-ansible -t nova reconfigure)¶
sequenceDiagram
autonumber
participant HA as loadbalancer hosts
participant C1 as control01 (nova-api[0], conductor[0])
participant CT as all controllers
participant CP as all computes
Note over HA: loadbalancer play — nova + nova-cell includes only
HA->>HA: render services.d/nova*.cfg, restart haproxy only if changed
Note over CT: Play 1 Bootstrap API DBs
C1->>C1: mysql_db/mysql_user (toolbox) · nova_api_bootstrap (db sync) · map_cell0 · list_cells (cell0)
Note over CT: Play 2 Bootstrap cell DBs (any_errors_fatal)
C1->>C1: cell DB/user · RabbitMQ vhost/user · nova_cell_bootstrap (cell db sync) · list_cells → update_cell if URLs changed
Note over CT: Play 3 nova role — register (keystone) · config · check · flush
CT->>CT: restart super-conductor/scheduler/api if drift (all controllers at once)
Note over CT,CP: Play 4 nova-cell — config-host · config · check · flush
CT->>CT: restart conductor/novncproxy if drift (all controllers at once)
CP->>CP: restart nova_ssh → nova_libvirt (→ SASL user) → nova_compute if drift (all computes at once)
CP->>C1: wait for compute registration · list_cells · discover_hosts (delegated)
Note over CT: Play 5 — kill -HUP nova_scheduler (always)
Note over CT,CP: Plays 6-10 — skipped (upgrade only)
Read from the 2024.1 code CODE: the loadbalancer includes render the frontends and notify the HAProxy restart
(roles/nova/tasks/loadbalancer.yml:2-7, roles/haproxy-config/tasks/main.yml:2-23); play 1
(nova.yml:29-65; roles/nova/tasks/bootstrap.yml:2-56, roles/nova/tasks/map_cell0.yml:2-67); play 2
(nova.yml:67-100; roles/nova-cell/tasks/bootstrap.yml:2-43, roles/nova-cell/tasks/create_cells.yml:2-55);
play 3 (nova.yml:103-122; roles/nova/tasks/deploy.yml:1-9); play 4 (nova.yml:125-147;
roles/nova-cell/tasks/deploy.yml:1-39, roles/nova-cell/tasks/wait_discover_computes.yml:7-90,
roles/nova-cell/tasks/discover_computes.yml:4-23); play 5 (nova.yml:152-172;
roles/nova/tasks/refresh_scheduler_cell_cache.yml:4-12); plays 6 to 10 run their tasks only when kolla_action is
upgrade (nova.yml:199-201, :227-229, :251-253, :275-277, :300-302). With the default kolla_serial 0, each
play takes all its hosts in one batch (e.g. nova.yml:140).
3. What changes, per type of change CODE¶
| Change | Files rewritten | Restarted on controllers | Restarted on every compute | Source |
|---|---|---|---|---|
| none | – | none unless a notifying task reports changed | none unless a notifying task reports changed | restart handlers run only when a notifying task reports changed (Tags §2.4), and check-containers reports changed only for a missing container, a differing compared parameter, a kolla_set_configs --check exit 1/137 or Docker client error, or a missing or different systemd unit file kolla-<container>-container.service (module_utils/, module_utils/, module_utils/; Tags §3); NOT AVAILABLE whether every task reports unchanged on an unchanged cloud |
| first 2024.1 run after 2023.1, no operator change (new in 2024.1) | nova.conf of every nova service when quorum queues were off in 2023.1; nova-compute/ and nova-compute/; nova-ssh/ on Debian and Ubuntu images |
every nova container whose systemd unit 2023.1 wrote; every one whose nova.conf changed |
nova_compute, nova_libvirt, nova_ssh | unit text: F16 (module_utils/, :154-167); quorum default: F17 (group_vars/, roles/, roles/); hostnqn: roles/, roles/, config.json loop roles/; sftp line: roles/, roles/. When the run is the upgrade, it renders the same configs and checks the same containers (roles/) |
/etc/ (global override) |
nova.conf of every nova service |
nova_api, nova_scheduler, nova_conductor, nova_novncproxy | nova_compute, nova_libvirt, nova_ssh | both Copying over nova.conf loops merge {{ node_custom_config }}/ (roles/, roles/); Tags §2.4 |
/etc/ |
nova-compute/ only |
– | nova_compute, nova_libvirt, nova_ssh (libvirt and ssh by loop-notify) | roles/, source at :78; Tags §2.4 |
/etc/ (new source in 2024.1) |
nova-compute/ of that host only |
– | nova_compute, nova_libvirt, nova_ssh on that compute only | roles/; the nova role has no such per-service host source (roles/) |
/etc/ |
nova-scheduler/ only |
nova_scheduler + nova_api (loop-notify) | – | roles/; Tags §2.4 |
enable_barbican: yes (Barbican not even deployed) |
nova.conf everywhere (adds [barbican]). td-agent.conf and logrotate.conf change too, but only when the common play runs (no tag or -t common), not under -t nova |
all nova (fluentd and cron only at the next common run) |
all nova incl. libvirt (same for fluentd and cron) | roles/, roles/; roles/, roles/, roles/; site.yml:88-103; Tags §2.4 |
same as the global nova.conf override, with kolla_serial=1 |
same | one controller at a time | one compute at a time; API DB sync and map_cell0 repeated per batch (3×) |
nova.yml:39, :115, :140; roles/, roles/; Tags §2.5 |
4. Are all compute nodes affected? Does nova-compute restart on every compute?¶
- Tasks: yes. The nova-cell play targets the whole
computegroup (nova.yml:125-147). Read from the 2024.1 task files, a no-change run makes 27 remote module invocations per compute and 31 on the first compute (for the default service set with Cinder LVM,kvm, Docker and a non-Ubuntu host); 2023.1 made 26 and 30, and the new invocation is thehostnqntemplate (roles/nova-cell/tasks/config.yml:203-216; Compute scaling §2). Its number of task results is NOT AVAILABLE. - Restarts: only where drift is detected. But most Nova-related changes are global (Component dependencies
§5, Nova deep dive
§2.3), so in practice
either no compute restarts or all computes restart. They restart in the same handler step unless
ANSIBLE_SERIAL/kolla_serialis set or--limitis used (nova.yml:140;tools/kolla-ansible:531). - libvirt goes with it. Whenever nova-compute restarts because of config or image drift, nova-libvirt and nova-ssh
restart too (loop-notify, Tags
§2.4). Kolla's own handler
ordering restarts nova-ssh, then libvirt, before nova-compute CODE (
roles/nova-cell/handlers/main.yml:74-168; handlers run in definition order: the iterator copies the play's handler list in order and, during a flush, walks it and runs each notified one, ansible-core 2.16.20lib/ansible/executor/play_iterator.py:195-200,lib/ansible/executor/play_iterator.py:426-445). Withlibvirt_enable_sasl(defaulttrue,roles/nova-cell/defaults/main.yml:569), the libvirt handler also notifies a readiness check and the SASL user creation, defined before the nova-compute handler (roles/nova-cell/handlers/main.yml:96,roles/nova-cell/handlers/main.yml:116-141).
5. Running VMs, live migration, scheduling¶
| Effect | Guaranteed by code | Depends on environment |
|---|---|---|
nova-compute stop/start (stop with docker_graceful_timeout, 60 s by default, group_vars/) |
yes, when drift | NOT AVAILABLE Whether VMs are unaffected, whether in-flight operations on that host fail or are left mid-task, and whether the service is reported down only if the restart exceeds service_down_time (Nova deep dive §5) |
| libvirtd stop/start | yes, whenever nova-compute drifts | NOT AVAILABLE Whether VMs survive. CODE The container has pid_mode: host, cgroupns_mode: host (now always compared and set by the Docker worker, module_utils/, module_utils/), the host /run bind mount and /sys/fs/cgroup mounted rw (roles/, :374, :377), and the image's start scripts do nothing to QEMU, cgroups or /var/ (kolla docker/, docker/; Kolla images §5). The open questions are therefore libvirt and container-engine behaviour: where libvirtd puts QEMU (for example a host machine.slice scope), the stop (each container is run by a systemd unit kolla-<name>-container.service whose ExecStop is /usr/, 60 s by default; a restart of an unchanged container restarts that unit, a recreate stops it, removes the container and starts a new one, and without a unit file the Docker worker calls the engine's stop with the same timeout; module_utils/, :106-125, module_utils/, :415-431, :439-459, group_vars/, :151; on a container whose entrypoint is dumb-init, kolla docker/; which signals Docker sends, at the stop and after the timeout, is NOT AVAILABLE) and libvirtd re-attaching to running domains when it starts. NOT AVAILABLE Whether live migrations in progress fail, and whether libvirt API calls fail until libvirtd is back (Nova deep dive §5) |
| nova-ssh stop/start | yes, with the above | NOT AVAILABLE Whether resize or cold-migration disk copies over port 8022 (roles/) are interrupted |
| Scheduling | API and scheduler restarts are parallel on all controllers (serial 0, nova.yml:115) |
NOT AVAILABLE Whether API calls fail during the restart window. CODE The scheduler gets SIGHUP on every deploy or reconfigure run that reaches its hosts (cell-cache refresh, nova.yml:152-172, roles/); NOT AVAILABLE whether that is harmless |
| Host mapping | discover_hosts runs every time |
runs nova-manage cell_v2 discover_hosts --by-service to map newly registered computes (roles/, through kolla_container_engine exec -t nova_conductor, -t new in 2024.1); NOT AVAILABLE whether it affects existing instances |
6. Operational risks and verification¶
- Risks: R1, R2, R3, R24 in Risk register; after an upgrade from 2023.1, F16 and F17.
- Before:
kolla-ansible -t nova genconfig(restarts nothing:kolla_actionisconfig,tools/kolla-ansible:557-559, and every Nova restart handler is skipped then,roles/nova/handlers/main.yml:16-17,roles/nova-cell/handlers/main.yml:167-168), then diff/etc/kolla/nova-*/nova.confon one controller and one compute.docker exec -u root nova_compute /usr/local/bin/kolla_set_configs --check; echo $?on a few computes (withkolla_container_engine: podman,podman exec): exit 1 means the container will restart, and so willnova_libvirtandnova_ssh(Kolla images §3). The check does not see the unit-file drift of F16, which the module compares on the host (module_utils/kolla_container_worker.py:58-65).- No active live migrations:
openstack server list --all-projects --status MIGRATING.
- Apply in batches:
deploy-containers --limit(Write config first, apply in controlled batches §2) orANSIBLE_SERIAL; to stop a rolling run when a batch has failures, setnova_cell_max_fail_percentage(cell plays) orkolla_max_fail_percentagelower than the default 100 (nova.yml:141-144; Write config first, apply in controlled batches §1.3). - After:
openstack compute service list(all up);docker ps --filter name=nova_ -a(allUp, recentCreated/Started);virsh list --allinsidenova_libvirton a sample of computes (domains running), with guest pings;nova-manage cell_v2 list_hostscount equals the number of computes.
Related¶
- Risk register — the risks this page names: R1 (any config
change touching nova-compute), R2 (fleet-wide restarts after an
enable_*change), R3 (changing RabbitMQ group membership, password or port) and R24 (container stop timeout:docker_graceful_timeout, 60 s by default) - Technical findings — F16 (the changed systemd unit template) and F17 (quorum queues by default), both new in 2024.1
- Nova — service cards of the roles
novaandnova-cell, what changed in 2024.1, and where to look - Nova deep dive §5 — nova-libvirt: why its restart is special
- Pre-execution checklist — the checks before every production run
- Write config first, apply in controlled batches —
genconfig, thendeploy-containers --limitbatch by batch, rolling runs withANSIBLE_SERIAL, andmax_fail_percentage - Tags §2.4 — loop-notify: one changed item notifies the handlers of all items
- Compute scaling §2 — work per compute host, read from the task files: the source of the per-compute counts of §4