Nova deep dive¶
Nova is split into two roles and ten plays. It is the component with the most hosts (every compute), the most coupling (Glance, Neutron, Placement, Cinder, Keystone, RabbitMQ, MariaDB) and the highest VM-impact risk.
Note
Key answers
- What restarts what: any change of nova-compute's
nova.conf,config.jsonor image restartsnova_computeand, through loop-notify,nova_libvirtandnova_sshCODE (§2.4). - Do running VMs survive a
nova_libvirtrestart? NOT AVAILABLE. What the code shows about the container is CODE:pid_mode: host, state on named volumes, a start path that is passive towards running guests, the 10 s stop timeout, the restart handler's retries (§5). - Which inputs change
nova.confon every compute (RabbitMQ group membership, several globalenable_*flags, endpoints and service passwords, operator overrides) and which do not reach the nova containers on compute hosts (the memcached membership, MariaDB settings,nova_keystone_password): §2.3. - The ten nova plays: which play runs on which action (plays 6–10 run on upgrade only), what the bootstrap one-shots run, and how compute discovery works, once per batch (§4).
- Live migration, resize and consoles: the containers each one depends on; whether a
nova_libvirtornova_sshrestart interrupts operations in progress is NOT AVAILABLE (§6). - Upgrade: the image pull is not part of
upgrade; runkolla-ansible pullfirst (§7).
1. Components, roles, containers and placement¶
| Component | Container | Role | Inventory group (default) | Placement variable | Notable container flags | Evidence |
|---|---|---|---|---|---|---|
| nova-api (+ metadata API) | nova_api |
nova |
nova-api → nova → control |
group: nova-api |
privileged, /lib/modules |
roles/ |
| nova-scheduler | nova_scheduler |
nova |
nova-scheduler |
group |
– | :42-48 |
| nova-super-conductor | nova_super_conductor |
nova |
nova-super-conductor |
enabled: enable_cells (default no) |
– | :49-55 |
| nova-conductor (cell) | nova_conductor |
nova-cell |
nova-conductor |
group: nova_cell_conductor_group |
– | roles/ |
| nova-novncproxy | nova_novncproxy |
nova-cell |
nova-novncproxy |
enabled: nova_console == 'novnc' |
– | :19-25 |
| nova-spicehtml5proxy | nova_spicehtml5proxy |
nova-cell |
nova-spicehtml5proxy |
nova_console == 'spice' |
– | :26-32 |
| nova-serialproxy | nova_serialproxy |
nova-cell |
nova-serialproxy |
enable_nova_serialconsole_proxy |
– | :33-39 |
| nova-compute | nova_compute |
nova-cell |
compute |
group: nova_cell_compute_group |
privileged, ipc_mode: host, /dev, /run:shared, /lib/modules, iscsi_info (if iscsid), nova_compute volume /var/lib/nova, libvirtd volume |
:47-57 |
| nova-libvirt | nova_libvirt |
nova-cell |
compute |
enabled: nova_compute_virt_type in ['kvm','qemu'] |
privileged, pid_mode: host, /dev, /run:shared, /sys/fs/cgroup (rw), libvirtd:/, nova_libvirt_qemu:/, instance datadir |
:3-11, volumes :237-250 |
| nova-ssh | nova_ssh |
nova-cell |
compute |
enable_nova_ssh (default yes) |
sshd on migration_interface_address:8022 |
:12-18; templates/ |
| nova-compute-ironic | nova_compute_ironic |
nova-cell |
nova-compute-ironic |
enable_ironic and cell match |
– | :58-64 |
Cells: nova_cell_name is empty by default (a single cell named '', the pre-Train layout). For multi-cell, every
nova_cell_*_group variable points at a per-cell group (roles/nova-cell/defaults/main.yml:87-126). enable_cells
then deploys nova_super_conductor at the API level.
2. Dependency classification¶
2.1 Installation (deploy-time) dependencies¶
What must work while Kolla runs the nova plays:
| Dependency | Used by | Evidence | Failure mode |
|---|---|---|---|
MariaDB (through the VIP) + kolla_toolbox on the first nova-api and first nova-conductor host |
mysql_db / mysql_user |
roles/; roles/ |
play aborts; cell play has any_errors_fatal: true (nova.yml:75) |
RabbitMQ (first host of nova_cell_rpc_group_name) |
vhost/user creation via service-rabbitmq, delegated |
roles/ |
cell bootstrap fails, all hosts fail |
| Keystone admin endpoint | service-ks-register (services nova, nova_legacy, user nova) |
roles/; defaults/ |
retries 5 × 10 s, then fail |
| Docker registry (only if the image is absent) | bootstrap and service containers | kolla_docker.start_container pulls if absent |
fail |
Running nova_conductor on groups[nova_cell_conductor_group][0] |
discover_hosts uses docker exec nova_conductor |
roles/ |
discovery fails (any_errors_fatal) |
Running kolla_toolbox on that conductor + Keystone + nova-api |
openstack compute service list to wait for compute registration |
roles/ |
20 retries × 10 s, then fail only the hosts that did not register |
| Facts of all RabbitMQ, memcached and haproxy hosts | transport URLs, memcache lists, haproxy backends | group_vars/; roles/ |
templating error. This is why gather-facts.yml gathers facts for all hosts even with --limit |
2.2 Runtime dependencies (what Nova needs to operate)¶
flowchart LR
subgraph Ctl["Controllers"]
API["nova_api<br/>(+metadata)"] ; SCH["nova_scheduler"] ; CON["nova_conductor"] ; VNC["nova_novncproxy"]
end
subgraph Cmp["Each compute"]
CMP["nova_compute"] -->|"qemu+tcp/tls :16509/16514<br/>(SASL)"| LV["nova_libvirt (libvirtd + qemu)"]
SSH["nova_ssh :8022"]
end
API -->|"SQL nova_api + cell0 (VIP)"| DB[("MariaDB")]
CON -->|"SQL nova (cell) + nova_api (VIP)"| DB
SCH -->|"SQL nova_api"| DB
API & SCH & CON & CMP -->|RPC| MQ[["RabbitMQ (all nodes)"]]
API & SCH & CON -.notifications.-> MQ
API -->|token validation| KS["Keystone (VIP)"]
API --- MC[("memcached")]
SCH -->|allocation candidates| PL["Placement (VIP)"]
CMP -->|resource providers, inventories| PL
API & CMP -->|"images (VIP)"| GL["Glance"]
API & CMP -->|ports, vif plugging| NE["Neutron server"]
NE -->|"network-vif-plugged events"| API
API & CMP -->|volume attach| CI["Cinder"]
CMP -->|vif plug| OVS[("Open vSwitch on the host")]
VNC -->|console tunnel| LV
LV <-->|"live migration (migration_interface)"| LV2["other computes' libvirt"]
SSH <-->|"resize / cold migrate (8022)"| SSH2["other computes' nova_ssh"]
| Nova component | Requires at runtime | Evidence (template) |
|---|---|---|
| nova-api | MariaDB (nova_api and nova_cell0), RabbitMQ, Keystone (authtoken + memcached), Placement, Glance, Neutron, Cinder |
roles/ |
| nova-scheduler | MariaDB (nova_api), RabbitMQ, Placement. Filters: Kolla leaves [filter_scheduler] enabled_filters unset by default, so Nova's upstream Ussuri default applies (AvailabilityZone, Compute, ComputeCapabilities, ImageProperties, ServerGroupAntiAffinity, ServerGroupAffinity) |
nova.conf.j2:32-40 overrides enabled_filters only when enable_blazar (adds BlazarFilter) or enable_nova_fake is set; both default to no. discover_hosts_in_cells_interval = -1 (:166): the scheduler never discovers hosts by itself |
| nova-conductor (cell) | MariaDB (cell DB, and the API DB because nova_cell_conductor_has_api_database: yes), RabbitMQ |
roles/ |
| nova-compute | RabbitMQ (no DB access: the [database] section is excluded for nova-compute*), Keystone, Placement, Glance, Neutron, Cinder, local libvirt, local OVS |
nova-cell/; libvirt connection_uri in nova.conf.d/ |
| nova-libvirt | /dev/kvm (udev rule 99-kolla-kvm.rules), the br_netfilter module, Ceph (if nova_backend: rbd: ceph.conf, secret XML/base64 in nova-libvirt/) |
nova-cell/, external_ceph.yml |
| nova-novncproxy | Reaches libvirt VNC on each compute (server_listen/server_proxyclient_address = api_interface_address) |
nova-cell/ |
2.3 Configuration dependencies (inputs that change Nova's files)¶
These change nova.conf on every compute. The result is a restart of nova_compute (and, through loop-notify,
nova_libvirt and nova_ssh) on every compute during the next run that executes the nova-cell play:
| Input | Template line | Why it is "global" |
|---|---|---|
| RabbitMQ group membership, user, password, port | transport_url (nova-cell/) |
nova_cell_rpc_transport_url / nova_cell_notify_transport_url list every host of nova_cell_rpc_group_name, default om_rpc_group = rabbitmq (roles/). Within Nova, the global rpc_transport_url (group_vars/) is used only by the API-level nova role, so overriding it does not change the compute nova.conf |
enable_ceilometer, enable_searchlight, enable_designate, enable_neutron_infoblox_ipam_agent |
[notifications] / notification driver (:34-40,215-225) |
global flags |
enable_watcher, enable_cinder, enable_osprofiler, enable_barbican |
:37,111,226,234 |
global flags. enable_barbican also changes the common role's td-agent.conf (fluentd input paths, roles/) and cron logrotate.conf (roles/). common is a meta dependency of nova and nova-cell (roles/, roles/), so its container check also restarts cron, fluentd and kolla_toolbox on every nova host (loop-notify, roles/) CODE |
Glance, Neutron, Placement, Keystone endpoints and service passwords (placement_keystone_password, neutron_keystone_password, cinder_keystone_password, metadata_secret) |
[glance], [neutron], [placement], [cinder] (:121,) |
passwords.yml and VIP/FQDN settings |
nova_console, enable_nova_serialconsole_proxy, kolla_external_fqdn, public_protocol |
[vnc]/[spice]/[serial_console] (:48-88) |
console URLs embed the external FQDN |
libvirt_tls, libvirt_enable_sasl, migration_interface, nova_backend, Ceph settings |
nova.conf.d/, libvirtd.conf.j2, qemu.conf.j2 |
change the libvirt and compute containers |
| Operator overrides | /etc/, nova.conf, nova/, nova/ |
merge order in nova-cell/ |
Inputs that do not reach the nova containers on compute hosts: the memcached membership (no memcache settings
in the nova-cell template; it does reach computes through neutron.conf of the Neutron agents and ceilometer.conf),
MariaDB settings (no [database] on computes) and nova_keystone_password. Among the nova templates, the latter
appears only in [keystone_authtoken] of the API-level template (roles/nova/templates/nova.conf.j2:124), rendered for
nova_api, nova_scheduler and nova_super_conductor, so rotating it restarts no nova container on the computes. It
does reach computes through the [nova] section of the Neutron agents' neutron.conf
(roles/neutron/templates/neutron.conf.j2:85).
2.4 Operational dependencies (what restarts what)¶
| Trigger | Restarted on computes | Restarted on controllers | Evidence |
|---|---|---|---|
Any change of nova-compute's nova.conf / config.json / image |
nova_compute + nova_libvirt + nova_ssh (loop-notify), then "Create libvirt SASL user" |
– | CODE roles/, check-containers.yml:2-20, handlers/; loop-notify, Tags §2.4 |
Change of /etc/ or a global flag |
all three on every compute; a global enable_* flag that changes the common role's fluentd/logrotate config also restarts cron, fluentd, kolla_toolbox |
nova_api, nova_scheduler, nova_conductor, nova_novncproxy (+ super conductor if enable_cells); same common restarts for such a flag |
CODE only the nova and nova-cell roles merge /etc/ (roles/, roles/), so a plain override restarts only nova containers; a flag such as enable_barbican also changes common's files (§2.3), and common is a meta dependency (roles/, roles/) |
| Scheduler-only override | – | nova_scheduler + nova_api (loop-notify) |
CODE roles/, check-containers.yml:2-20; loop-notify, Tags §2.4 |
New image for nova-compute (after pull) |
nova_compute recreated, nova_libvirt + nova_ssh restarted |
– | CODE compare_image makes the check report changed (library/); recreate_or_restart_container recreates nova_compute, whose image differs, and restarts the other two under COPY_ALWAYS (kolla_docker.py:781-803, group_vars/); loop-notify, Tags §2.4 |
Any deploy/reconfigure with nova or nova-api tag, even with no change |
– | nova_scheduler receives SIGHUP (cell-cache refresh) |
roles/; play tags and kolla_action condition in nova.yml:140-156 |
| Libvirt TLS certificate change | nova_libvirt + nova_compute (+ nova_ssh by loop-notify) |
– | nova-cell/ |
Ceph keyring / ceph.conf / secret change |
nova_compute + nova_libvirt + nova_ssh, then "Create libvirt SASL user". The files are listed in the config.json of nova-compute or nova-libvirt, so "Check nova-cell containers" finds kolla_set_configs --check drift and loop-notifies all three. Exception: the cinder keyring file is not in nova-compute's config.json, and its key reaches libvirt only through the cinder secret in nova-libvirt/. A key change therefore still restarts all three; an edit that leaves the key unchanged restarts only nova_compute (direct notify) |
– | nova-cell/, check-containers.yml:2-20, templates/, nova-libvirt.json.j2 (same mechanism as the first row) |
3. What must be available before Nova can operate¶
Ordered by necessity. What Kolla configures is CODE (templates cited in §2.2); upstream Nova behaviour is not in these sources and is marked NOT AVAILABLE:
- MariaDB (through HAProxy and the VIP) with the
nova_api,nova_cell0andnova(cell) schemas migrated. NOT AVAILABLE Whether API, scheduler and conductor fail without it, and whether computes keep running VMs and whether their state reporting through the conductor fails. - RabbitMQ with the cell vhost and user. All RPC between API, scheduler, conductor and compute uses it. NOT AVAILABLE Whether computes also report liveness over it, with the service heartbeat going through the conductor.
- Keystone + memcached. The API-side
nova.confhas[keystone_authtoken]with memcached as token cache (roles/nova/templates/nova.conf.j2:116-130); nova-compute's[cinder],[neutron]and[placement]sections carry Keystone credentials (roles/nova-cell/templates/nova.conf.j2:115,131,205). NOT AVAILABLE Whether API requests, and nova-compute's calls to those services, fail without Keystone. - Placement. The scheduler's allocation candidates and nova-compute's resource-provider updates. NOT AVAILABLE Whether scheduling stops and computes log errors without it.
- Cell mapping and host mapping.
map_cell0,create_cellanddiscover_hosts. The scheduler does not discover hosts by itself (discover_hosts_in_cells_interval = -1). NOT AVAILABLE Whether a compute that was not discovered, and so is not in a host mapping, gets no instances. - Neutron server + an L2 agent on the compute (neutron-openvswitch-agent + OVS, or ovn-controller).
NOT AVAILABLE Whether instance boot waits for the
network-vif-pluggedevent: Kolla setsvif_plugging_is_fatalonly for Hyper-V (roles/nova-hyperv/templates/nova_hyperv.conf.j2:15), and Nova's default is not in these sources. - Glance for image-backed boots. Cinder for volume-backed boots.
- libvirt (+ KVM,
br_netfilter, udev rule) on the compute. NOT AVAILABLE Whether nova-compute fails to start its driver without libvirt.
4. The nova plays step by step (ansible/nova.yml)¶
| # | Play (lines) | Hosts | Runs on | Key tasks (evidence) |
|---|---|---|---|---|
| 1 | Bootstrap nova API databases (29-61) |
nova-api |
deploy, reconfigure; upgrade → bootstrap_upgrade |
mysql_db nova_api/cell0 (run_once); grants; nova-api-bootstrap config (run_once); nova_api_bootstrap one-shot container (roles/); map_cell0 delegated to groups['nova-api'][0] (map_cell0.yml:2-26) |
| 2 | Bootstrap nova cell databases (63-96) |
nova-conductor |
deploy, reconfigure; upgrade | cell DB/user on the first conductor; RabbitMQ vhost/user; nova_cell_bootstrap one-shot (cell DB sync); list_cells → create_cell or update_cell (only if the MQ or DB URL changed) |
| 3 | Apply role nova (99-114) |
nova-api, nova-scheduler, nova-super-conductor |
every action | standard role: register, config, check-containers, flush |
| 4 | Apply role nova-cell (117-135) |
compute, nova-compute-ironic, nova-conductor, consoles |
every action | config-host, config, check, flush, wait for compute registration + discover_hosts |
| 5 | Refresh scheduler cell cache (140-156) |
nova-scheduler |
deploy, reconfigure | kill -HUP $(docker inspect … nova_scheduler) |
| 6 | Reload super conductor (164-181) |
nova-super-conductor |
upgrade | recreate_or_restart to drop the RPC pin |
| 7 | Reload cell services (183-205) |
computes, conductors, consoles | upgrade | same |
| 8 | Reload API services (207-225) |
api, scheduler | upgrade | same |
| 9 | API online data migrations (230-247) |
nova-api[0] |
upgrade | nova_api_online_data_migrations one-shot |
| 10 | Cell online data migrations (253-272) |
nova-conductor (first) |
upgrade | any_errors_fatal |
Bootstrap one-shots. nova_api_bootstrap and nova_cell_bootstrap run with KOLLA_UPGRADE set on deploy,
reconfigure and upgrade alike (roles/nova/tasks/bootstrap_service.yml:13,
roles/nova-cell/tasks/bootstrap_service.yml:11). The nova-api image then runs nova-manage api_db sync and
nova-manage db sync, whose [database] is cell0 (kolla docker/nova/nova-api/extend_start.sh:16-20;
roles/nova/templates/nova.conf.j2:97-98). The nova-conductor image runs nova-manage db sync --local_cell (kolla
docker/nova/nova-conductor/extend_start.sh:9-12). Neither runs online data migrations. Only plays 9 and 10 do, on
upgrade, by setting KOLLA_OSM (roles/nova*/tasks/online_data_migrations.yml:11). No nova task sets
KOLLA_BOOTSTRAP, so the images' bootstrap branches, which would add online data migrations, are never used. See Kolla
images §4.
Discovery mechanics (roles/nova-cell/tasks/deploy.yml:24-42, wait_discover_computes.yml,
discover_computes.yml):
- It runs once per batch on the first compute of the batch (
all_computes_in_batch[0]), delegated to the first conductor of the cell. - It waits until
openstack compute service list --service nova-computecontains every compute of the batch, keyed byansible_facts.nodename. It tries 20 times at 10 s intervals (up to about 200 s). - Then it runs
nova-manage cell_v2 discover_hosts --by-service --cell_uuid …once. - A compute that did not register fails only that host, unless
nova_compute_registration_fatal: true.
5. nova-libvirt: why its restart is special¶
Facts CODE:
pid_mode: host, privileged,/sys/fs/cgroupmounted read-write (roles/nova-cell/defaults/main.yml:3-11,237-250). QEMU processes are started by libvirtd inside this container.- Persistent state lives on named volumes. The
libvirtdvolume holds/var/lib/libvirtand thenova_libvirt_qemuvolume holds domain XML in/etc/libvirt/qemu. Instance disks sit innova_instance_datadir_volume(/var/lib/nova). All of them survive a container recreate. One more location matters for a restart and is not a named volume. The host's/runis bind-mounted:shared(defaults/main.yml:242), so whatever libvirtd writes under/runis on the host and survives a restart or recreate. NOT AVAILABLE Whether the image's/var/runis a symlink to/run, which would put the running-domain state in/var/run/libvirton that bind mount, lost only on a host reboot. The Dockerfiles do not show this. The SASL DB, by contrast, stays in the container layer (see below). - The image start path is passive towards running guests.
kolla_startrunskolla_set_configs,kolla_copy_cacertsand the nova-libvirtextend_start.sh, thenexecs/usr/sbin/libvirtd --listen(kolladocker/base/start.sh:8-21;templates/nova-libvirt.json.j2:2).kolla_set_configsdeletes and rewrites only thedestpaths ofnova-libvirt.json.j2, which are all config files (kolladocker/base/set_configs.py:147-148).extend_start.shonly fixes/dev/kvmownership, mounts xenfs if that module is loaded and creates the log file (kolladocker/nova/nova-libvirt/extend_start.sh:6-24). None of these steps touches cgroups, QEMU processes,/var/run/libvirt,/var/lib/libvirtor/etc/libvirt/qemu, andqemu.conf.j2/libvirtd.conf.j2set no cgroup option. Whether guests survive is therefore decided by the stop path, by where libvirt's defaults placed QEMU, and by libvirtd's reconnect on start. Start path in detail: Kolla images §5. - The restart handler retries 5 times. "Restart nova-libvirt container" uses
retries: 5 / until: success. The in-code note says: "retry 5 to remove nova_libvirt container because when guests running, nova_libvirt will raise error even though it is removed" (roles/nova-cell/handlers/main.yml:87-109). The authors therefore expect running guests during libvirt restarts. - After every libvirt restart or recreate,
Create libvirt SASL userruns whenlibvirt_enable_saslis true (the default) (saslpasswd2inside the container,handlers/main.yml:91,111-122). The SASL DB is/etc/libvirt/passwd.db(templates/sasl.conf.j2:2). It is on no volume (under/etc/libvirtonlyqemu/is,defaults/main.yml:249) and no kolla image step creates it. It survives a plain restart (stop and start of the same container) and is lost on every recreate: changed image or parameters, orconfig_strategy: COPY_ONCE(library/kolla_docker.py:793-803,892-904). The in-code note says the SASL user must exist for nova-compute to start (handlers/main.yml:111). - Effective stop timeout is 10 s (Repository architecture
§6), and both signals
go to dumb-init. The container's main process is
dumb-init --single-child -- kolla_start(kolladocker/base/Dockerfile.j2:433-435,455;use-dumb-initdefaults to True, kollakolla/common/config.py:170).kolla_startexecs libvirtd (kolladocker/base/start.sh:21), so libvirtd is dumb-init's direct child.docker stop(timeout 10,library/kolla_docker.py:871-884,892-904) sends SIGTERM to dumb-init and, after 10 s, SIGKILL to dumb-init. NOT AVAILABLE Whether libvirtd itself is SIGKILLed if it is still running then (the cgroup kill below). - Libvirt is restarted before nova-compute. In the handler order
nova-libvirtcomes beforenova-compute, so during the window nova-compute is still running against a libvirtd that is going away. - Kolla ships, but does not install by default, a guest shutdown helper.
tools/kolla-libvirt-guests.service(withvirt-guest-shutdown.target) is for host shutdown. The optional standalone playbooktools/setup-compute-libvirt.yml(hosts:compute) copies both units to/usr/lib/systemd/systemand enables and starts them, but no kolla-ansible action (deploy, reconfigure, upgrade, ...) runs it, andsetup.cfgdata_filesleaves all three files out of a pip install. The units exist on a compute only if the operator runs that playbook by hand from a source checkout.
What happens to running VMs is not established by these sources NOT AVAILABLE:
- NOT AVAILABLE (dumb-init and containerd behaviour are not in these sources) Whether dumb-init forwards SIGTERM to
libvirtd only and exits when libvirtd does, so that the 10 s SIGKILL matters only if libvirtd hangs; whether QEMU
gets either signal directly; and whether, under
pid_mode: host, the runtime kills whatever is still in the container's cgroup when its init exits, which would make VM survival depend on QEMU's cgroup membership at stop time rather than on the 10 s timer. - NOT AVAILABLE (libvirt and systemd behaviour are not in these sources) Where libvirtd puts QEMU processes, and so
whether QEMU survives the stop of the libvirtd container: with libvirt's systemd integration, libvirtd may ask the
host's systemd-machined over D-Bus to put each QEMU process in its own
machine.slicescope on the host cgroup tree, or, without machined, write the cgroups directly. CODE Three settings give the container the host's view: the host/runbind mount (defaults/main.yml:242),pid_mode: host(:8, QEMU PIDs are host PIDs) and/sys/fs/cgroupmounted rw (:244). NOT AVAILABLE Whether, on start, libvirtd re-attaches to running domains from the status XML under/var/run/libvirt/qemu(on the host/runonly if/var/runis that symlink, see above) and the monitor sockets under/var/lib/libvirt/qemu(thelibvirtdvolume,:246) rather than from the persistent definitions in/etc/libvirt/qemu, and whether it stops a domain whose monitor it cannot reconnect to. CODEstdio_handler = "file"(qemu.conf.j2:1), and the container runs onlylibvirtd --listen(templates/nova-libvirt.json.j2:2), so Kolla starts no virtlogd in it. NOT AVAILABLE Whether running QEMU therefore does not depend on a virtlogd. CODE The handler note above andtools/cleanup-containers:5-9both expect QEMU processes to still be running whennova_libvirtis stopped or removed. - NOT AVAILABLE Whether surviving QEMU processes keep the old container's mount namespace, so that after a recreate with a new image they keep running the old QEMU binary until the guest is stopped and started, hard-rebooted or migrated, and whether a reboot inside the guest keeps the same QEMU process.
- NOT AVAILABLE Possible risk windows, that is, whether:
- live migrations in flight fail;
- attach/detach and snapshot operations in progress fail;
virshand nova-compute calls fail until libvirtd is back;- on cgroup v2 hosts, or wherever QEMU is still in the container's cgroup at stop, QEMU processes are killed;
- a libvirtd that does not exit within 10 s of SIGTERM is killed with the container (SIGKILL to dumb-init, then the cgroup kill described above), possibly while it writes domain state.
nova-compute start CODE: with the default config_strategy: COPY_ALWAYS (group_vars/all.yml:43), every start of
nova_compute, a plain restart included, runs kolla_set_configs, and its permissions block chowns /var/lib/nova
to nova:nova recursively: one chown per directory and file of the instance datadir
(roles/nova-cell/templates/nova-compute.json.j2:72-76; kolla docker/base/set_configs.py:378-385,391-393). Under
COPY_ONCE this happens only on the container's first start. A failed chown is only logged (kolla
docker/base/set_configs.py:359-363). NOT AVAILABLE Whether, with enable_shared_var_lib_nova_mnt (cinder NFS or
Quobyte, roles/nova-cell/defaults/main.yml:294,450), the walk also enters the cinder shares still mounted under
/var/lib/nova/mnt, and whether start time grows with the number of files, in particular on NFS-backed trees such as
those shares (Kolla images §6 item 10).
nova-compute restart NOT AVAILABLE (upstream Nova behaviour is not in these sources):
- Whether running VMs are affected.
- Whether the service is marked down only if the restart exceeds
service_down_time(Kolla does not set it). - Whether in-flight operations owned by that compute (boot, resize, migration) fail or are left in a transitional task_state.
- Whether, on start,
init_hostreconciles instance state and power state.
6. Live migration, resize and consoles¶
| Function | Mechanism in Ussuri Kolla | Dependency |
|---|---|---|
| Live migration | libvirt qemu+tcp://<migration_interface_address>/ (or qemu+tls://<migration_hostname> with libvirt_tls), SASL auth by default (libvirt_enable_sasl: true) |
nova_libvirt on both source and destination computes, the migration_interface network, matching SASL credentials, and with TLS, hostnames resolvable on every compute (/etc/hosts from bootstrap-servers, fact sheet E §3) |
| Cold migration / resize | nova_ssh sshd on port 8022, with keys from passwords.yml (nova_ssh_key) |
nova_ssh on both hosts and a shared instance datadir layout |
| VNC console | nova_novncproxy (controllers) → compute server_proxyclient_address |
api_interface reachable from the proxy hosts. Public URL is built from kolla_external_fqdn |
| Scheduling | nova_scheduler + Placement + host mappings |
discovery must have run for new computes |
Impact of a Kolla run on these:
- A run that restarts
nova_libvirt(any nova-compute change, by loop-notify) does so on every affected compute. Withserial: 0, that is every compute at once. NOT AVAILABLE Whether this interrupts every live migration in progress there (§5). - NOT AVAILABLE Whether a
nova_sshrestart interrupts resizes or cold migrations that are copying disks. - Every
deploy/reconfigurewith-t novaor no tag SIGHUPs the schedulers and runsdiscover_hosts(-t nova-apionly SIGHUPs,-t nova-cellonly discovers). NOT AVAILABLE Whether either affects existing instances.
7. Upgrade path specifics¶
| Item | Behaviour | Evidence |
|---|---|---|
nova-status upgrade check |
Runs on nova-api[0]. The play fails if rc ∉ {0,1} |
roles/ |
nova_safety_upgrade: yes |
Stops nova_api, nova_scheduler and nova_super_conductor, and on the cell hosts nova_conductor, nova_novncproxy, nova_spicehtml5proxy and nova_serialproxy, before the upgrade. nova_compute, nova_compute_ironic, nova_libvirt and nova_ssh keep running. Handlers skip restarts of the stopped services during upgrade; the reload plays start them again. Until then the API, the cell conductors and the consoles are all down |
roles/; roles/; handlers when: kolla_action != "upgrade" or not nova_safety_upgrade (nova-cell/) |
| Rolling upgrade | config → check-containers → flush_handlers. Restarts happen per batch only if ANSIBLE_SERIAL is set. The TODO in rolling_upgrade.yml:7-10 admits they are not serialised |
roles/ |
| RPC version pin removal | After upgrade, the reload plays restart conductors → computes → API (recreate_or_restart), after waiting nova_compute_startup_delay (30 s) for computes to report versions |
nova.yml:158-225; nova-cell/ |
| Online data migrations | After all cells are upgraded | nova.yml:227-272 |
| Image pull | Not part of upgrade. start_container pulls only if the image is absent. Run kolla-ansible pull first. Otherwise each host downloads lazily during the handler phase, which lengthens every restart window |
kolla_docker.py:805-808 |
Related¶
- Reconfigure Nova (reconfigure Nova, step by step)
- Tags §2.4 (loop-notify)
- Risk register (R1)
- Kolla images §5.1 (nova-libvirt start path)
- Nova (service cards)
- Conventions and evidence labels (evidence labels)
To remember
Lumo: if you keep only 5 things from this page, keep these.
- Any change of nova-compute's
nova.conf,config.jsonor image restartsnova_computeand, through loop-notify,nova_libvirtandnova_ssh. CODE - Whether running VMs survive a
nova_libvirtrestart is not established by these sources. NOT AVAILABLE - A run that restarts
nova_libvirtdoes so on every affected compute; withserial: 0, that is every compute at once. - Rotating
nova_keystone_passwordrestarts no nova container on the computes. - The image pull is not part of
upgrade: runkolla-ansible pullfirst.
What would you do?
Lumo: a situation you could meet on a real platform, and three ways to react. Pick the one you would choose: I will tell you what this page says about it. Nothing is recorded.
Check yourself
Lumo: 3 quick questions to check the main ideas of this page. Choose an answer to see the explanation and the section that gives the evidence. Your score counts your first choices and stays in this browser only.
Lumo: ready for the next step? Take your time: you can come back to this page whenever you need it.
Lumo: nicely done! One more page on your journey.