Nova¶
This page is the starting point for Nova, which is split into two roles: nova (API level: nova_api, nova_scheduler
and nova_super_conductor, on the control nodes of the example inventory) and nova-cell (cell level: nova_compute,
nova_libvirt, nova_ssh, nova_conductor, the three console proxies and nova_compute_ironic, on the compute and
control nodes). The tag nova selects both; nova-api does not touch computes and nova-cell touches every compute
(Tags §3). The lists below say where the study covers how
to operate it, its impact, its dependencies and the evidence, what to check around a run and what stays open; the two
service cards follow them.
Where to look¶
Operate¶
- Scenario (
nova): Reconfigure Nova; batched apply: Write config first, apply in controlled batches §2 - Command (
nova): Reconfigure Nova §1.1, Write config first, apply in controlled batches §1.2, Reconfigure a single service §1.2 - Scenario (
nova-api): Reconfigure Nova §2 (plays 1, 3 and 5) - Command (
nova-api): Reconfigure Nova §1.2 - Scenario (
nova-cell): Reconfigure Nova §2 (plays 2 and 4), Reconfigure Nova §4 - Command (
nova-cell): Reconfigure Nova §1.3, Write config first, apply in controlled batches §1.1 - Scenario (
nova-bootstrap,nova-reload,nova-online-data-migrations(with their-api-/-cell-variants) andnova-refresh-scheduler-cell-cache): the plays themselves: Nova deep dive §4
Impact and risk¶
- What the tag executes (
nova): Tags §3 and Tags §3.1, rowsnova - Risks (
nova): R1, R2, R3, R24 - What the tag executes (
nova-api): Tags §3 and Tags §3.1, rowsnova-api - What the tag executes (
nova-cell): Tags §3 and Tags §3.1, rowsnova-cell - Risks (
nova-cell): R1 - What the tag executes (
nova-bootstrap,nova-reload,nova-online-data-migrations(with their-api-/-cell-variants) andnova-refresh-scheduler-cell-cache): Tags §3, one row each; use as narrower play tags: Tags §4 (item 4)
Dependencies¶
- Dependencies (role
nova): Component dependencies §3 - Dependencies (role
nova-cell): Component dependencies §3, Component dependencies §5
Evidence¶
- Fact sheet (roles
novaandnova-cell): – - Image side (role
nova): Kolla images §4 (rownova-api), Kolla images §4.1, Kolla images §5 - Also (role
nova): Nova deep dive; handlers: Tags §5 - Image side (role
nova-cell): Kolla images §4 (rownova-conductor), Kolla images §5, Kolla images §5.1 - Also (role
nova-cell): Nova deep dive, in particular Nova deep dive §5; handlers: Tags §5
Before and after a run¶
- Pre-execution checklist, step 5 (health gates): before the run, check
openstack compute service listandopenstack network agent listall up. Repeat after each batch. - Pre-execution checklist, step 6 (workload gates): no live migrations,
resizes, image uploads or volume migrations in flight on the targeted hosts; check
openstack server list --all-projects --status MIGRATING|RESIZEandopenstack volume list --status *ing. - Tags §3.1, rows
nova,nova-apiandnova-cell:novais safe on its own "only with batching and a window; prefergenconfig+deploy-containers --limit",nova-cell"asnova",nova-api"yes; does not touch computes". Prerequisites ofnova: MariaDB, RabbitMQ, memcached, Keystone, Placement, Glance, Neutron registered; ofnova-api: asnova; ofnova-cell: the API DB and cell0 mapping (nova-api/nova), RabbitMQ. - Nova deep dive §3, "What must be available
before Nova can operate", eight items ordered by necessity: MariaDB (through HAProxy and the VIP); RabbitMQ;
Keystone + memcached; Placement; cell mapping and host mapping; Neutron server + an L2 agent on the compute; Glance
for image-backed boots and Cinder for volume-backed boots; libvirt (+ KVM,
br_netfilter, udev rule) on the compute. What Kolla configures is CODE; upstream Nova behaviour is marked NOT AVAILABLE. - Nova deep dive §2.1, installation (deploy-time) dependencies: what must work while Kolla runs the nova plays, each with its failure mode.
- Nova deep dive §2.2, runtime dependencies: what each Nova component requires at runtime.
- R1, precautions for any config change touching nova-compute:
genconfig+diff, thendeploy-containers --limitin batches, orANSIBLE_SERIAL; drain live migrations first; validate libvirt restart behaviour in staging. - R2, precautions for enabling or disabling a service that has a logrotate or fluentd
input entry: treat
enable_*changes as fleet-wide changes; roll out with--limitbatches; apply thecommonside deliberately with-t common(a-t novarun does not run thecommonplay). - R3, precautions for a change of RabbitMQ group membership, password or port: plan as a
maintenance; batch with
--limit; verify RabbitMQ cluster health first. - R24, precautions for the container stop timeout, in 2023.1 the 60 s of
docker_graceful_timeoutwritten into each container's systemd unit: drain long operations before restarts; raisedocker_graceful_timeoutif needed (it reaches the unit on the next restart). - Reconfigure Nova §6, verification: before,
kolla-ansible -t nova genconfig(restarts nothing), then a diff of/etc/kolla/nova-*/nova.confon one controller and one compute,kolla_set_configs --checkinnova_computeon a few computes (exit 1 means the container will restart, and so willnova_libvirtandnova_ssh), and no active live migrations. After,openstack compute service list(all up),docker ps --filter name=nova_ -a,virsh list --allinsidenova_libvirton a sample of computes with guest pings, and anova-manage cell_v2 list_hostscount equal to the number of computes. - Nova deep dive §7: the image pull is not part of
upgrade; runkolla-ansible pullfirst.
Known limits and open points¶
- Q1: "Do running VMs survive a
nova_libvirtrestart or recreate?" Status: NOT AVAILABLE. - Nova deep dive §5, "nova-libvirt: why its restart is
special": what the code shows about the container is CODE (
pid_mode: host, state on named volumes, a start path that is passive towards running guests, the 60 s stop timeout of its systemd unit, the restart handler's retries); what happens to running VMs is NOT AVAILABLE. - Nova deep dive §6, live migration, resize and consoles:
whether a
nova_libvirtornova_sshrestart interrupts operations in progress is NOT AVAILABLE. - What is not known, group "Running VMs and
compute hosts": what the sources do not establish about a
nova_libvirtrestart or recreate, anova_computerestart, anova_sshrestart, the restart of the nova API, scheduler and conductors, upstream Nova behaviour when a dependency is missing, scheduling on a fresh compute, and live migration to a new compute. - What is not known:
the sources do not establish whether the SSH clients accept the new
nova_sshandkeystone_sshhost keys after a recreate without failing, or the cost of the recursive chown on large trees (the NFS instance directory among them). - Q13: "
bootstrap-serversre-run effects on a live compute". Status: NOT AVAILABLE. - Q14, answered from the source CODE: Nova's bootstrap containers run with
KOLLA_UPGRADE, so Nova's online data migrations never run outsideupgrade; the images'KOLLA_BOOTSTRAPbranches are unreachable. - F1, "Loop-notify amplification": a nova-compute-only change restarts
nova_libvirtandnova_ssh; a scheduler-only change restartsnova_apiCODE. - F3, "
docker_graceful_timeoutanddocker_client_timeoutare ignored": no longer holds in 2023.1 CODE. Both values reach the module throughdocker_common_optionsand the stop timeout is written into each container's systemd unit (ExecStop=docker stop <name> -t <graceful_timeout>), so a container still running 60 s after SIGTERM is killed (group_vars/all.yml:140,152,module_utils/kolla_systemd_worker.py:31-36; R24); fornova_libvirtthis matters only if libvirtd has not exited by then. - F5, "
reconfigure=deploy": in 2023.1-t novastarts 6 one-shot containers oncontrol01when cell0 is mapped with an unchanged connection and the cell is unchanged;map_cell0.ymladds a cell0list_cells(Tags §3, rownova) CODE. - F6, "Global inputs are rendered into per-host files":
enable_*flags, RabbitMQ/memcached membership, service passwords and the external FQDN end up in compute-side files CODE. - F8, "
--limitdoes not confine actions": withdeploy --limitset to a compute host,control01runs cell discovery (nova_list_cells,openstack compute service list,nova-manage cell_v2 discover_hosts) CODE. - F10, "
kolla_serialmakesrun_oncesteps repeat per batch": API DB sync ×3 with serial 1 and the three controllers of themultinodeinventory; a play's restarts follow host-pattern order, computes before conductors in the nova-cell play CODE. - F13, "The Nova deploy order" puts nova-compute before OVS and the Neutron agents on new computes CODE; NOT AVAILABLE whether instances can be scheduled on such a compute before the later plays finish.
- F15, "What the bootstrap containers run differs per image": Nova's bootstrap
containers run with
KOLLA_UPGRADE(schema sync, no online data migrations outsideupgrade) CODE. - Known bugs and limitations has no row whose area is Nova; its row
masakarinames nova-libvirt: instancemonitor usesqemu+tcpwithout SASL while nova-libvirt enables SASL. Status: CODE; effect on the instancemonitor NOT AVAILABLE. - Risk items whose VM impact refers to R1: R12, changing
openstack_tag/*_tagand runningdeployinstead ofupgrade("as R1 + R7 + R8 together"); R13, a rebuilt image pushed under the same tag ("as R1"); R14, abootstrap-serversre-run on a live host ("a Docker restart affectsnova_libvirtand OVS"). R15,kolla-ansible stop, stops thenova_libvirtand OVS containers on the targeted hosts. - Kolla images §6, items 5, 10, 11 and 18: Nova's online data
migrations never run outside
upgradeCODE; onechownper file of/var/lib/novaon computes on everyCOPY_ALWAYSstart CODE; nova-libvirt's start path is passive, and VM survival is decided on the stop path CODE (for the start path; NOT AVAILABLE for survival); SSH host keys ofnova_sshandkeystone_sshchange on every recreate CODE.
Service card¶
The rows of the two cards are explained on the Components page.
Nova (API level) — role nova¶
| Item | Value |
|---|---|
| Play | nova.yml (import_playbook at site.yml:544) |
| Tags | nova, nova-api(-*) |
| Play host patterns | nova-api, nova-scheduler, nova-super-conductor |
| Node types in play (example inventory) | control |
| Node types running its containers | control |
| Containers | nova_api, nova_scheduler, nova_super_conductor |
deploy/reconfigure flow (tasks/) |
register → config → check-containers → flush handlers |
| Task files | bootstrap.yml, bootstrap_service.yml, bootstrap_upgrade.yml, check-containers.yml, check.yml, clone.yml, config.yml, config_bootstrap.yml, config_validate.yml, copy-certs.yml, deploy-containers.yml, deploy.yml, loadbalancer.yml, main.yml, map_cell0.yml, online_data_migrations.yml, precheck.yml, pull.yml, reconfigure.yml, refresh_scheduler_cell_cache.yml, register.yml, reload_api.yml, reload_super_conductor.yml, rolling_upgrade.yml, stop.yml, upgrade.yml |
| Templates | nova-api-bootstrap.json.j2, nova-api-wsgi.conf.j2, nova-api.json.j2, nova-scheduler.json.j2, nova-super-conductor.json.j2, nova.conf.j2 |
| Handlers (definition order) | Restart nova-super-conductor container; Restart nova-scheduler container; Restart nova-api container |
| Key variables | enable_nova, enable_cells, nova_safety_upgrade, nova_console |
| Role dependencies | none |
Nova (cell level) — role nova-cell¶
| Item | Value |
|---|---|
| Play | nova.yml (import_playbook at site.yml:544) |
| Tags | nova, nova-cell(-*) |
| Play host patterns | compute, nova-compute-ironic, nova-conductor, nova-novncproxy, nova-serialproxy, nova-spicehtml5proxy |
| Node types in play (example inventory) | compute, control |
| Node types running its containers | compute, control |
| Containers | nova_libvirt, nova_ssh, nova_novncproxy, nova_spicehtml5proxy, nova_serialproxy, nova_conductor, nova_compute, nova_compute_ironic |
deploy/reconfigure flow (tasks/) |
clone (if) → config-host → config → check-containers → config-nova-fake (if) → flush handlers → wait_discover_computes (if) → post-config |
| Task files | bootstrap.yml, bootstrap_service.yml, bootstrap_upgrade.yml, cell_proxy_loadbalancer.yml, check-containers.yml, check.yml, clone.yml, config-host.yml, config-libvirt-tls.yml, config-nova-fake.yml, config.yml, config_bootstrap.yml, config_validate.yml, copy-certs.yml, create_cells.yml, deploy-containers.yml, deploy.yml, discover_computes.yml, external_ceph.yml, get_cell_settings.yml, libvirt-cleanup.yml, loadbalancer.yml, main.yml, online_data_migrations.yml, post-config.yml, precheck.yml, proxy_loadbalancer.yml, pull.yml, rabbitmq.yml, reconfigure.yml, reload.yml, rolling_upgrade.yml, stop.yml, upgrade.yml, wait_discover_computes.yml |
| Templates | 99-kolla-kvm.rules.j2, auth.conf.j2, libvirt-secret.j2, libvirtd.conf.j2, nova-cell-bootstrap.json.j2, nova-compute-ironic.json.j2, nova-compute.json.j2, nova-conductor.json.j2, nova-libvirt.json.j2, nova-novncproxy.json.j2, nova-serialproxy.json.j2, nova-spicehtml5proxy.json.j2, nova-ssh.json.j2, nova.conf.d/, nova.conf.j2, qemu.conf.j2, sasl.conf.j2, secret.xml.j2, ssh_config.j2, sshd_config.j2 |
| Handlers (definition order) | Restart nova-conductor container; Restart nova-novncproxy container; Restart nova-spicehtml5proxy container; Restart nova-serialproxy container; Restart nova-ssh container; Restart nova-libvirt container; Checking libvirt container is ready; Create libvirt SASL user; Reload libvirtd; Restart nova-compute container; Restart nova-compute-ironic container; Restart nova-compute-fake containers; Wait for nova-compute services to update service versions |
| Key variables | nova_compute_virt_type, nova_backend_ceph, libvirt_tls, libvirt_enable_sasl, nova_cell_*_group, migration_interface |
| Role dependencies | none |
Related¶
- Nova deep dive — Nova in depth: the two roles and ten plays, the dependencies, what restarts what
and the
nova_libvirtrestart - Reconfigure Nova — what
-t nova reconfigureruns on controllers and on every compute - Write config first, apply in controlled batches —
genconfig, thendeploy-containers --limitbatch by batch - Add a compute node — what
deploy --limittouches on the new compute and elsewhere - RabbitMQ — a change of its group membership changes the transport URL everywhere
- Pre-execution checklist — the eight checks before every production run
- Components — the other component pages