Where instance disks live. Local on each compute, in Ceph RBD, or boot from volume¶
Context¶
Every instance has a root disk, and some have ephemeral and swap disks. Nova can keep them in three places, and the choice decides what happens on the worst days: when a compute dies, when it must be drained for maintenance, when its disk fills up.
- Local instance store (Kolla-Ansible's default): the disks are files on each compute.
- Ceph RBD for Nova (
nova_backend_ceph: "yes"): the disks are RBD images in a Ceph pool,vmsby default. - Boot from volume: the root disk is a Cinder volume, on whatever backend Cinder uses. This is per instance (the user or the flavor workflow chooses it), not a platform switch, and it can be combined with either of the other two for the instances that do not boot from a volume.
The Cinder and Glance back-ends page compares the volume and image stores; this page is about the instance disks themselves.
Topology¶
No change in roles: the choice changes where nova-compute and nova_libvirt read and write disks.
flowchart LR
subgraph local["Local instance store"]
c1[compute-1<br/>disks in nova_compute volume]
c2[compute-2<br/>disks in nova_compute volume]
end
subgraph rbd["Nova on Ceph RBD"]
c3[compute-1] --> pool[(Ceph pool vms)]
c4[compute-2] --> pool
end
subgraph bfv["Boot from volume"]
c5[compute-1] --> cinder[(Cinder backend)]
c6[compute-2] --> cinder
end
Network design¶
- Local store: disks do not cross the network in normal operation. A live migration copies them (block migration), over the migration network, with the memory.
- Ceph RBD and boot from volume: every disk read and write crosses the storage network to Ceph or to the Cinder backend. The computes need a fast, redundant path to it; a storage network outage stalls the I/O of every instance on that compute.
Storage design¶
| Local instance store | Nova on Ceph RBD | Boot from volume | |
|---|---|---|---|
| Where the root disk is | A file under /var/ inside the containers. With Kolla-Ansible's containerised libvirt, that is the nova_compute Docker volume (nova_instance_datadir_volume), so on the host under /var/ |
An RBD image in the pool ceph_nova_pool_name (default vms), through [libvirt] images_type = rbd, which Kolla-Ansible sets when nova_backend is rbd |
A Cinder volume, attached by nova-compute through os-brick |
| Disk format | qcow2 overlays on a cached base image by default (Nova's qcow2 backing store), or flat |
RBD images; cloned copy-on-write from the Glance image when Glance is on the same Ceph and the image is raw |
Whatever the Cinder backend uses |
| Capacity | Each compute's own disk; one compute can be full while others have room | One pool for all computes, sized and monitored in Ceph | The Cinder backend, counted in Cinder quotas |
| What a compute failure does to the disks | They stay on the dead host | They stay in Ceph | They stay on the Cinder backend |
What the Kolla-Ansible external Ceph guide asks for, with Ceph:
# globals.yml
nova_backend_ceph: "yes"
ceph_nova_user: "<user>" # defaults to ceph_cinder_user
ceph_nova_pool_name: "vms"
with /etc/kolla/config/nova/ceph.conf and the keyring ceph.client.<user>.keyring beside it, or per host under
/etc/kolla/config/nova/<hostname>/ with ceph_cluster set in the inventory, to give different computes different
Ceph clusters (for example one per availability zone). For copy-on-write clones from Glance, the same guide sets
show_image_direct_url = True in /etc/kolla/config/glance.conf, and warns that this option can be a security risk
when Glance has other backends than Ceph.
Ceph's OpenStack guide recommends raw images in Glance to boot instances from Ceph, ephemeral or volume: a qcow2
image cannot be cloned and is downloaded and converted instead.
Inventory and configuration layout¶
No new inventory group. The choice is global (nova_backend_ceph in globals.yml), or per compute when several Ceph
clusters are used (ceph_cluster per host). The Ceph keyrings are secrets: keep them out of git, as the
configuration layout pattern describes.
Key choices and their consequences¶
| Choice | Alternative(s) | Why one would choose it | Consequence in operation |
|---|---|---|---|
| Local instance store | Ceph RBD, boot from volume | No storage cluster to run; local disk latency; Kolla-Ansible's default | Live migration copies the disks (block migration: more time, more network and storage load, and, per Nova's guide, incompatible with read-only devices such as CD-ROMs and config drives). Evacuation rebuilds the root disk from its image: the data written in the instance is lost (Nova's evacuate guide: shared storage is what preserves it). A full compute disk pauses the instances that write. |
| Nova on Ceph RBD | Local, boot from volume | Disks survive a compute failure; migrations do not copy disks (external Ceph guide); one pool to size instead of one disk per compute | A Ceph cluster to deploy and operate outside Kolla-Ansible. Every instance's I/O depends on the storage network and on Ceph's health. Images should be raw in Glance for copy-on-write. |
| Boot from volume | Local, Ceph RBD | Per instance: the instances that must keep their disk get it, without changing the platform | Root disks count against Cinder quotas and depend on Cinder at build time (see instance fails to boot from volume). The disk survives a compute failure and live migration does not copy it. |
| Local store, but Glance on Ceph | Glance on file |
Faster image downloads to the computes | Nova's KVM guide describes [glance] enable_rbd_download = true on the computes with show_image_direct_url on Glance, to fetch images by RBD export instead of the Glance API. |
What went wrong during deployment¶
Nothing recorded for this page. Two lab records in the runbooks show the local store's consequences:
- Evacuation: on a two-compute lab with local disks, an evacuated instance with a local disk was rebuilt from its image, while a boot-from-volume instance kept its disk (fence before evacuate, lab confirmation of 2026-10-23).
- Full disk: with the instance store on a dedicated 5 GB file system filled by a stray file, an instance that
wrote was paused while Nova still showed it
ACTIVE;dfon the Docker volume path neededsudo(compute disk full).
Version notes¶
- 2026.1:
nova_backendis derived fromnova_backend_ceph(group_vars/all/nova.yml), andimages_type = rbdis written only forrbd(libvirt.conf.j2); read in Kolla-Ansible'sstable/2026.1. - Nova's
rawimages type is deprecated in favour offlat(Nova's KVM guide).
Lessons learned¶
No first-hand lessons on Ceph for this page. What the sources and the lab records stress:
- Decide before production which instances must survive the loss of their compute. With the local store, only those booted from a volume do.
- With the local store, monitor each compute's instance store, not only the cluster's totals: the lab's full-disk case froze instances on one compute while the others were fine.
- With Ceph, Glance images in
raware what makes boots fast; aqcow2image costs a download and a conversion on every new disk.
Validity and limits¶
Written from the Kolla-Ansible 2026.1 external Ceph guide and source (group_vars/all/nova.yml, nova-cell role),
Nova 2026.1 (KVM, live migration and evacuation guides) and Ceph's OpenStack guide, with two lab records from the
runbooks repository, both on the local store. Nova on Ceph RBD was not deployed for this page.
Not covered: LVM as Nova's images type, NFS-shared instance stores, Ceph pool sizing and replication, encryption of
ephemeral disks, images with hw_disk_bus or hw_scsi_model properties, and image caching settings on the computes.
Related¶
- Cinder and Glance back-ends compared
- Runbooks: fence before evacuate, compute disk full, moving instances off a host for maintenance, instance fails to boot from volume
- Kolla-Ansible 2026.1: external Ceph guide
- Nova 2026.1: KVM, configuring migrations, evacuate
- Ceph: block devices and OpenStack
Provenance: written by agent:ozarland-author, from the Kolla-Ansible, Nova and Ceph documentation and Kolla-Ansible source, with two lab records from the runbooks repository. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.