Nova¶
This page is the starting point for Nova, which is split into two roles: nova (API level: nova_api, nova_scheduler
and nova_super_conductor, on the control nodes of the example inventory) and nova-cell (cell level: nova_compute,
nova_libvirt, nova_ssh, nova_conductor, the three console proxies and nova_compute_ironic, on the compute and
control nodes) (nova.yml:1-22, inventory/multinode:18-19, inventory/multinode:99-100,
inventory/multinode:227-249). The tag nova selects both; nova-api does not touch computes and nova-cell touches
every compute (nova.yml:103-147; Tags §3). The lists
below say where the study covers how to operate it, its impact, its dependencies and the evidence, what to check around
a run and what stays open; the two service cards follow them.
What changed in 2024.1¶
- Every Nova play gets
max_fail_percentage, and one more play getsany_errors_fatalCODE. Seven of the ten plays ofnova.ymlnow setmax_fail_percentagefromnova_max_fail_percentage(API-level plays) ornova_cell_max_fail_percentage(cell-level plays), thenkolla_max_fail_percentage, then 100 (e.g.nova.yml:40-43,nova.yml:141-144). The three plays that carryany_errors_fatal: trueinstead are the cell DB bootstrap and the two online-data-migration plays; the API online-migration play is new to that list (nova.yml:79,nova.yml:269-270,nova.yml:294-295). With the default 100 a play is never stopped early by the share of failed hosts of a batch (Controlled batches §1.3). - The default RabbitMQ quorum queues change every Nova
nova.confCODE.om_enable_rabbitmq_quorum_queuesnow defaults totrue(group_vars/all.yml:268), so both templates renderrabbit_quorum_queue = true(roles/nova/templates/nova.conf.j2:156-165,roles/nova-cell/templates/nova.conf.j2:207-216). A cloud that kept the 2023.1 default gets a changednova.conffor every Nova service, computes included, on its first 2024.1 run that renders the Nova configs (F17). Withom_enable_queue_manager(defaultfalse,group_vars/all.yml:269) the templates adduse_queue_manager(roles/nova/templates/nova.conf.j2:142-147,roles/nova-cell/templates/nova.conf.j2:193-198) and the Nova service containers mount/dev/shm(e.g.roles/nova/defaults/main.yml:169,roles/nova-cell/defaults/main.yml:397). - New file for
nova_compute:hostnqnCODE. The nova-cellconfig.ymlrenders/etc/kolla/nova-compute/hostnqnfrom the host name and notifies the nova-compute restart (roles/nova-cell/tasks/config.yml:203-216);config.jsoncopies it to/etc/nvme/hostnqn(roles/nova-cell/templates/nova-compute.json.j2:59-64). On a compute where it is not there yet, the file is new, andkolla_set_configs --checkreports a missing destination as a difference (kolladocker/base/set_configs.py:165-169,docker/base/set_configs.py:240-242). - A per-host, per-service override for the cell role CODE. The nova-cell
Copying over nova.confloop also merges/etc/kolla/config/nova/<host>/<service>.conf, last (roles/nova-cell/tasks/config.yml:69-89, source at:80). The nova role does not have this source (roles/nova/tasks/config.yml:64-82). - Container definitions that may differ from a 2023.1 container CODE. The
nova_libvirtbind of/runis now written/run:/run:sharedwith Docker (2023.1:/run/:/run/:shared), andnova_libvirtandnova_computeadd therun_default_volumes_<engine>list, empty for Docker (roles/nova-cell/defaults/main.yml:11,:62,:374;group_vars/all.yml:165-172). The Docker worker compares the bind strings without normalising them (module_utils/kolla_docker_worker.py:124-144,module_utils/kolla_container_worker.py:400-430); whether Docker reports the 2023.1 bind as/run/or/runis NOT AVAILABLE, so whether this alone makesnova_libvirtdiffer is NOT AVAILABLE. Independently of it, the first 2024.1 check of a container whose systemd unit 2023.1 wrote reports drift (F16). - Smaller changes CODE: console base URLs and the Nova endpoints are built with the
kolla_urlfilter and the new*_public_portvariables, which followhaproxy_single_external_frontend(defaultfalse) (roles/nova-cell/templates/nova.conf.j2:57,roles/nova/defaults/main.yml:204-205,group_vars/all.yml:512,group_vars/all.yml:520,group_vars/all.yml:931-932); host discovery runsexec -t(roles/nova-cell/tasks/discover_computes.yml:17);sshd_configofnova_sshgains ansftpsubsystem line for Debian and Ubuntu images (roles/nova-cell/templates/sshd_config.j2:9-13); the external Ceph files are looked up per host first and named afternova_ceph_cluster(defaultceph) (roles/nova-cell/tasks/external_ceph.yml:1-130,roles/nova-cell/defaults/main.yml:536); the nova role no longer registers an extra Keystone role (roles/nova/tasks/register.yml:1-8). - Still holds in 2024.1 CODE: the handlers of both roles are the same, in the same order, with the module renamed
kolla_container(roles/nova/handlers/main.yml:1-54,roles/nova-cell/handlers/main.yml:1-168); the deploy flows are unchanged (roles/nova/tasks/deploy.yml:1-9,roles/nova-cell/tasks/deploy.yml:1-39); every play still takesserialfromkolla_serial, default0(e.g.nova.yml:140).
Where to look¶
Operate¶
- Scenario (
nova): Reconfigure Nova; batched apply: Write config first, apply in controlled batches §2 - Command (
nova): Reconfigure Nova §1.1, Write config first, apply in controlled batches §1.2, Reconfigure a single service §1.2 - Scenario (
nova-api): Reconfigure Nova §2 (plays 1, 3 and 5) - Command (
nova-api): Reconfigure Nova §1.2 - Scenario (
nova-cell): Reconfigure Nova §2 (plays 2 and 4), Reconfigure Nova §4 - Command (
nova-cell): Reconfigure Nova §1.3, Write config first, apply in controlled batches §1.1; stop a rolling run on failures withnova_cell_max_fail_percentageorkolla_max_fail_percentage: Write config first, apply in controlled batches §1.3 - Scenario (
nova-bootstrap,nova-reload,nova-online-data-migrations(with their-api-/-cell-variants) andnova-refresh-scheduler-cell-cache): the plays themselves: Nova deep dive §4
Impact and risk¶
- What the tag executes (
nova): Tags §3 and Tags §3.1, rowsnova - Risks (
nova): R1, R2, R3, R24 - What the tag executes (
nova-api): Tags §3 and Tags §3.1, rowsnova-api - What the tag executes (
nova-cell): Tags §3 and Tags §3.1, rowsnova-cell - Risks (
nova-cell): R1 - First 2024.1 run after 2023.1: F16 (changed systemd unit text) and F17 (quorum queues on by default), see What changed in 2024.1
- What the tag executes (
nova-bootstrap,nova-reload,nova-online-data-migrations(with their-api-/-cell-variants) andnova-refresh-scheduler-cell-cache): Tags §3, one row each; use as narrower play tags: Tags §4 (item 4)
Dependencies¶
- Dependencies (role
nova): Component dependencies §3 - Dependencies (role
nova-cell): Component dependencies §3, Component dependencies §5
Evidence¶
- Fact sheet (roles
novaandnova-cell): – - Image side (role
nova): Kolla images §4 (rownova-api), Kolla images §4.1, Kolla images §5 - Also (role
nova): Nova deep dive; handlers: Tags §5 - Image side (role
nova-cell): Kolla images §4 (rownova-conductor), Kolla images §5, Kolla images §5.1 - Also (role
nova-cell): Nova deep dive, in particular Nova deep dive §5; handlers: Tags §5
Before and after a run¶
- Pre-execution checklist, step 5 (health gates): before the run, check
openstack compute service listandopenstack network agent listall up. Repeat after each batch. - Pre-execution checklist, step 6 (workload gates): no live migrations,
resizes, image uploads or volume migrations in flight on the targeted hosts; check
openstack server list --all-projects --status MIGRATING|RESIZEandopenstack volume list --status *ing. - Tags §3.1, rows
nova,nova-apiandnova-cell:novais safe on its own "only with batching and a window; prefergenconfig+deploy-containers --limit",nova-cell"asnova",nova-api"yes; does not touch computes". Prerequisites ofnova: MariaDB, RabbitMQ, memcached, Keystone, Placement, Glance, Neutron registered; ofnova-api: asnova; ofnova-cell: the API DB and cell0 mapping (nova-api/nova), RabbitMQ. - Nova deep dive §3, "What must be available before Nova can operate": what Kolla configures is CODE; upstream Nova behaviour is marked NOT AVAILABLE.
- Nova deep dive §2.1, installation (deploy-time) dependencies: what must work while Kolla runs the nova plays, each with its failure mode.
- Nova deep dive §2.2, runtime dependencies: what each Nova component requires at runtime.
- R1, precautions for any config change touching nova-compute:
genconfig+diff, thendeploy-containers --limitin batches, orANSIBLE_SERIAL; drain live migrations first. The code reason is loop-notify: on a compute theCheck nova-cell containersloop coversnova-libvirt,nova-sshandnova-computeand notifies the restart handler of each executed item (roles/nova-cell/tasks/check-containers.yml:2-22,roles/nova-cell/defaults/main.yml:3-21,roles/nova-cell/defaults/main.yml:53-64) CODE. - R2, precautions for enabling or disabling a service that has a logrotate or fluentd
input entry: treat
enable_*changes as fleet-wide changes; roll out with--limitbatches; apply thecommonside deliberately with-t common. A-t novarun does not run thecommonplay, which is taggedcommononly (site.yml:88-103) CODE. - R3, precautions for a change of RabbitMQ group membership, password or port: plan as a
maintenance; batch with
--limit; verify RabbitMQ cluster health first. The cell transport URLs list every host of the RPC and notification groups with user and password (roles/nova-cell/defaults/main.yml:206-207) and are written into every cellnova.conf(roles/nova-cell/templates/nova.conf.j2:37,roles/nova-cell/templates/nova.conf.j2:184) CODE. - R24, container stop timeout: in 2024.1 the stop goes through the container's systemd
unit, whose
ExecStopis/usr/bin/<engine> stop <name> -t <graceful_timeout>(module_utils/kolla_systemd_worker.py:32), withdocker_graceful_timeout60 s (group_vars/all.yml:139) (F3). Precautions: drain long operations before restarts; raisedocker_graceful_timeoutif needed (changing it changes the unit text, which the next check reports as drift, F16). - Reconfigure Nova §6, verification: before,
kolla-ansible -t nova genconfig(restarts nothing: every Nova restart handler is skipped whenkolla_actionisconfig,roles/nova/handlers/main.yml:16-17,roles/nova-cell/handlers/main.yml:167-168), then a diff of/etc/kolla/nova-*/nova.confon one controller and one compute,kolla_set_configs --checkinnova_computeon a few computes (exit 1 means the container will restart, and so willnova_libvirtandnova_ssh), and no active live migrations. After,openstack compute service list(all up), thenova_*containers' state,virsh list --allinsidenova_libvirton a sample of computes with guest pings, and anova-manage cell_v2 list_hostscount equal to the number of computes. - Nova deep dive §7: no task of the Nova upgrade pulls images
(
roles/nova/tasks/upgrade.yml:1-53,roles/nova-cell/tasks/upgrade.yml:1-16); a restart handler pulls a missing image just before it stops the old container (module_utils/kolla_docker_worker.py:353-361) CODE. Runkolla-ansible pullfirst.
Known limits and open points¶
- Q1: "Do running VMs survive a
nova_libvirtrestart or recreate?" Status: NOT AVAILABLE. - Nova deep dive §5, "nova-libvirt: why its restart is
special": what the code shows about the container is CODE:
pid_mode: hostandcgroupns_mode: host(roles/nova-cell/defaults/main.yml:8-9), state on named volumes (roles/nova-cell/defaults/main.yml:379-382), a start path that is passive towards running guests (kolladocker/nova/nova-libvirt/extend_start.sh:1-16), the 60 s stop timeout of its systemd unit, and the restart handler's 5 retries (roles/nova-cell/handlers/main.yml:109-113). What happens to running VMs is NOT AVAILABLE. - Nova deep dive §6, live migration, resize and consoles:
whether a
nova_libvirtornova_sshrestart interrupts operations in progress is NOT AVAILABLE. - What is not known, group "Running VMs and
compute hosts": what the sources do not establish about a
nova_libvirtrestart or recreate, anova_computerestart, anova_sshrestart, the restart of the nova API, scheduler and conductors, upstream Nova behaviour when a dependency is missing, scheduling on a fresh compute, and live migration to a new compute. - What is not known:
the sources do not establish whether the SSH clients accept the new
nova_sshandkeystone_sshhost keys after a recreate without failing, or the cost of the recursive chown on large trees (the NFS instance directory among them). - Q13: "
bootstrap-serversre-run effects on a live compute". Status: NOT AVAILABLE. - Q14, answered from the source CODE: Nova's bootstrap containers run with
KOLLA_UPGRADE(roles/nova/tasks/bootstrap_service.yml:10-12,roles/nova-cell/tasks/bootstrap_service.yml:10-12), which runs the schema syncs and exits (kolladocker/nova/nova-api/extend_start.sh:16-20,docker/nova/nova-conductor/extend_start.sh:9-12); the online data migrations run only in the upgrade-only plays (nova.yml:258-302), and the images'KOLLA_BOOTSTRAPbranches are not reached from the nova roles. - F1, "Loop-notify amplification": a nova-compute-only change restarts
nova_libvirtandnova_ssh; a scheduler-only change restartsnova_apiCODE (roles/nova-cell/tasks/check-containers.yml:2-22,roles/nova/tasks/config.yml:64-82; Tags §2.4). - F3, "
docker_graceful_timeoutanddocker_client_timeoutare ignored": no longer holds in 2024.1 CODE. Both values reach the module throughdocker_common_options(group_vars/all.yml:142-153), and the stop timeout is written into each container's systemd unit (module_utils/kolla_systemd_worker.py:32); fornova_libvirtthis matters only if libvirtd has not exited by then. - F5, "
reconfigure=deploy":-t novastarts 6 one-shot containers oncontrol01when cell0 is mapped with an unchanged connection and the cell is unchanged:nova_api_bootstrap,map_cell0, alist_cellsfor cell0,nova_cell_bootstrap, and alist_cellsin each ofcreate_cells.ymlanddiscover_computes.yml(roles/nova/tasks/bootstrap.yml:54-56,roles/nova/tasks/map_cell0.yml:2-30,roles/nova-cell/tasks/bootstrap.yml:41-43,roles/nova-cell/tasks/create_cells.yml:2-4,roles/nova-cell/tasks/discover_computes.yml:4-5,roles/nova-cell/tasks/get_cell_settings.yml:2-21) CODE. - F6, "Global inputs are rendered into per-host files":
enable_*flags, RabbitMQ/memcached membership, service passwords, the oslo.messaging options and the external FQDN end up in compute-side files (e.g.roles/nova-cell/templates/nova.conf.j2:37,:57,:207-216) CODE. - F8, "
--limitdoes not confine actions": withdeploy --limitset to a compute host,control01runs cell discovery (nova_list_cells,openstack compute service list,nova-manage cell_v2 discover_hosts), delegated to the first conductor (roles/nova-cell/tasks/wait_discover_computes.yml:8-43,roles/nova-cell/tasks/discover_computes.yml:4-23) CODE. - F10, "
kolla_serialmakesrun_oncesteps repeat per batch": API DB sync ×3 with serial 1 and the three controllers of themultinodeinventory (roles/nova/tasks/bootstrap_service.yml:2-21,nova.yml:39); a play's restarts follow host-pattern order, computes before conductors in the nova-cell play (nova.yml:127-134) CODE. - F13, "The Nova deploy order" puts nova-compute before OVS and the Neutron agents on
new computes (
site.yml:609,site.yml:611-624) CODE; NOT AVAILABLE whether instances can be scheduled on such a compute before the later plays finish. - F15, "What the bootstrap containers run differs per image": Nova's bootstrap
containers run with
KOLLA_UPGRADE(schema sync, no online data migrations outsideupgrade) CODE, as in Q14 above. - F16, new in 2024.1: the first 2024.1 check of a Nova container whose unit 2023.1 wrote
reports drift, and on a compute loop-notify then restarts
nova_compute,nova_libvirtandnova_sshtogether (module_utils/kolla_systemd_worker.py:154-167,module_utils/kolla_container_worker.py:58-65,roles/nova-cell/tasks/check-containers.yml:2-22) CODE. - F17, new in 2024.1: quorum queues on by default change every Nova
nova.confCODE; what oslo.messaging does with existing classic queues is NOT AVAILABLE. - Known bugs and limitations, row
masakari, which names nova-libvirt: the Ussuri SASL mismatch does not hold in 2024.1, since withlibvirt_enable_sasl(defaulttrue) the instancemonitor gets anauth.confwith the libvirt SASL credentials (roles/masakari/defaults/main.yml:185,roles/masakari/templates/masakari-instancemonitor.json.j2:9-15). Itsconnection_uriis still hard-codedqemu+tcp(roles/masakari/templates/masakari-monitors.conf.j2:18), while withlibvirt_tlslibvirtd listens on TLS only (roles/nova-cell/templates/libvirtd.conf.j2:1-3). Status: CODE; effect on the instancemonitor NOT AVAILABLE. - Risk items whose VM impact refers to R1: R12, changing
openstack_tag/*_tagand runningdeployinstead ofupgrade; R13, a rebuilt image pushed under the same tag; R14, abootstrap-serversre-run on a live host: a Docker restart (restartedunlessdocker_systemd_reload, defaultfalse; ansible-collection-kollaroles/docker/handlers/main.yml:10-15,roles/docker/defaults/main.yml:65) stops every container,nova_libvirtand OVS included CODE; R15,kolla-ansible stop, stops the Nova cell containers on the targeted hosts throughservice-stop(roles/nova-cell/tasks/stop.yml:2-6) CODE. - Kolla images §6, items 5, 10, 11 and 18: Nova's online data
migrations never run outside
upgradeCODE; onechownper file of/var/lib/novaon computes on everyCOPY_ALWAYSstart (roles/nova-cell/templates/nova-compute.json.j2:90-94; kolladocker/base/set_configs.py:379-386) CODE; nova-libvirt's start path is passive, and VM survival is decided on the stop path CODE (for the start path; NOT AVAILABLE for survival); SSH host keys ofnova_sshchange on every recreate, since/etc/sshis on none of its volumes (roles/nova-cell/defaults/main.yml:384-391) CODE.
Service card¶
The rows of the two cards are explained on the Components page.
Nova (API level) — role nova¶
| Item | Value |
|---|---|
| Play | nova.yml (import_playbook at site.yml:609) |
| Tags | nova, nova-api(-*) |
| Play host patterns | nova-api, nova-scheduler, nova-super-conductor |
| Node types in play (example inventory) | control |
| Node types running its containers | control |
| Containers | nova_api, nova_scheduler, nova_super_conductor |
deploy/reconfigure flow (tasks/) |
register → config → check-containers → flush handlers |
| Task files | bootstrap.yml, bootstrap_service.yml, bootstrap_upgrade.yml, check-containers.yml, check.yml, clone.yml, config.yml, config_bootstrap.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, loadbalancer.yml, main.yml, map_cell0.yml, online_data_migrations.yml, precheck.yml, pull.yml, reconfigure.yml, refresh_scheduler_cell_cache.yml, register.yml, reload_api.yml, reload_super_conductor.yml, rolling_upgrade.yml, stop.yml, upgrade.yml |
| Templates | nova-api-bootstrap.json.j2, nova-api-wsgi.conf.j2, nova-api.json.j2, nova-scheduler.json.j2, nova-super-conductor.json.j2, nova.conf.j2 |
| Handlers (definition order) | Restart nova-super-conductor container; Restart nova-scheduler container; Restart nova-api container |
| Key variables | enable_nova, enable_cells, nova_safety_upgrade, nova_console |
| Role dependencies | none |
Nova (cell level) — role nova-cell¶
| Item | Value |
|---|---|
| Play | nova.yml (import_playbook at site.yml:609) |
| Tags | nova, nova-cell(-*) |
| Play host patterns | compute, nova-compute-ironic, nova-conductor, nova-novncproxy, nova-serialproxy, nova-spicehtml5proxy |
| Node types in play (example inventory) | compute, control |
| Node types running its containers | compute, control |
| Containers | nova_libvirt, nova_ssh, nova_novncproxy, nova_spicehtml5proxy, nova_serialproxy, nova_conductor, nova_compute, nova_compute_ironic |
deploy/reconfigure flow (tasks/) |
clone (if) → config-host → config → check-containers → config-nova-fake (if) → flush handlers → wait_discover_computes (if) → post-config |
| Task files | bootstrap.yml, bootstrap_service.yml, bootstrap_upgrade.yml, cell_proxy_loadbalancer.yml, check-containers.yml, check.yml, clone.yml, config-host.yml, config-libvirt-tls.yml, config-nova-fake.yml, config.yml, config_bootstrap.yml, config_validate.yml, copy-certs.yml, create_cells.yml, deploy-containers.yml, deploy.yml, discover_computes.yml, external_ceph.yml, get_cell_settings.yml, libvirt-cleanup.yml, loadbalancer.yml, main.yml, online_data_migrations.yml, post-config.yml, precheck.yml, proxy_loadbalancer.yml, pull.yml, rabbitmq.yml, reconfigure.yml, reload.yml, rolling_upgrade.yml, stop.yml, upgrade.yml, wait_discover_computes.yml |
| Templates | 99-kolla-kvm.rules.j2, auth.conf.j2, hostnqn.j2, libvirt-secret.j2, libvirtd.conf.j2, nova-cell-bootstrap.json.j2, nova-compute-ironic.json.j2, nova-compute.json.j2, nova-conductor.json.j2, nova-libvirt.json.j2, nova-novncproxy.json.j2, nova-serialproxy.json.j2, nova-spicehtml5proxy.json.j2, nova-ssh.json.j2, nova.conf.d/, nova.conf.j2, qemu.conf.j2, sasl.conf.j2, secret.xml.j2, ssh_config.j2, sshd_config.j2 |
| Handlers (definition order) | Restart nova-conductor container; Restart nova-novncproxy container; Restart nova-spicehtml5proxy container; Restart nova-serialproxy container; Restart nova-ssh container; Restart nova-libvirt container; Checking libvirt container is ready; Create libvirt SASL user; Reload libvirtd; Restart nova-compute container; Restart nova-compute-ironic container; Restart nova-compute-fake containers; Wait for nova-compute services to update service versions |
| Key variables | nova_compute_virt_type, nova_backend_ceph, libvirt_tls, libvirt_enable_sasl, nova_cell_*_group, migration_interface |
| Role dependencies | none |
Related¶
- Nova deep dive — Nova in depth: the two roles and ten plays, the dependencies, what restarts what
and the
nova_libvirtrestart - Reconfigure Nova — what
-t nova reconfigureruns on controllers and on every compute - Write config first, apply in controlled batches —
genconfig, thendeploy-containers --limitbatch by batch - Add a compute node — what
deploy --limittouches on the new compute and elsewhere - RabbitMQ — a change of its group membership changes the transport URL everywhere; quorum queues by default
- Pre-execution checklist — the eight checks before every production run
- Components — the other component pages