External Ceph for Glance, Cinder, Cinder backup and Nova. Pools, users, the files Kolla-Ansible expects, and the libvirt secrets¶
Context¶
Kolla-Ansible "does not provide support for provisioning and configuring a Ceph cluster directly" (external Ceph
guide): the cluster is deployed with cephadm or another tool, and Kolla-Ansible only configures the OpenStack
clients. Most integration failures come from the hand-off between the two: a pool or a user missing, a capability too
narrow, a keyring under the wrong name or in the wrong directory, a ceph.conf the INI parser cannot read.
Whether to use Ceph, and for which service, is compared elsewhere: Cinder and Glance back-ends compared and where instance disks live. This page is the integration itself, for one Ceph cluster: what to create on the Ceph side, which files go where, what Kolla-Ansible does with them, and how to check each service.
Topology¶
No Kolla-Ansible role on the Ceph hosts. The clients run in the containers that use them: glance_api,
cinder_volume, cinder_backup, nova_compute and nova_libvirt. In Kolla-Ansible 2026.1, cinder-volume and
cinder-backup run on the hosts of the cinder group, which the sample multinode inventory makes the control
group; the storage group is only used by the LVM variants (cinder-volume-lvm, cinder-backup-lvm). The external
Ceph guide still says that with no node in the storage group "Cinder and related services relying on this group"
fail; that sentence predates the move to the cinder group (see
Cinder and Glance back-ends compared). Put the
hosts where cinder-volume and cinder-backup should run in the cinder group, usually the controllers.
Network design¶
Every host running one of those containers must reach the Ceph monitors and every OSD, usually on a storage network or VLAN separate from the API network. Computes carry the instances' disk traffic. The MTU must match end to end. See network interfaces, VLANs and MTU.
Storage design¶
On the Ceph side¶
From Ceph's Block devices and OpenStack guide, run on a Ceph admin host (the pool names are Kolla-Ansible's
defaults). ceph osd pool create without a placement group count relies on the PG autoscaler, on by default in
current Ceph releases; on an older cluster without it, give the count explicitly:
⚠️ State-changing. Creates pools and client users in the Ceph cluster. Nothing in OpenStack changes yet. Undo:
ceph auth del client.<name>; deleting a pool is destructive and needs the cluster's pool deletion setting, so never delete one that has held data without a backup.
for pool in images volumes backups vms; do
ceph osd pool create "$pool"
rbd pool init "$pool"
done
ceph auth get-or-create client.glance mon 'profile rbd' \
osd 'profile rbd pool=images' mgr 'profile rbd pool=images'
ceph auth get-or-create client.cinder mon 'profile rbd' \
osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images' \
mgr 'profile rbd pool=volumes, profile rbd pool=vms'
ceph auth get-or-create client.cinder-backup mon 'profile rbd' \
osd 'profile rbd pool=backups' mgr 'profile rbd pool=backups'
client.cinder covers vms too because Kolla-Ansible's ceph_nova_user defaults to ceph_cinder_user: Nova uses the
Cinder user unless you create a separate client.nova and set ceph_nova_user: "nova". Its read-only access to
images is what lets Nova and Cinder clone images copy-on-write.
Export each keyring and a minimal configuration (read-only):
ceph auth get client.glance -o ceph.client.glance.keyring
ceph auth get client.cinder -o ceph.client.cinder.keyring
ceph auth get client.cinder-backup -o ceph.client.cinder-backup.keyring
ceph config generate-minimal-conf > ceph.conf
The guide warns that generate-minimal-conf writes leading tabs, which "break Kolla Ansible's ini parser":
remove them before copying the file.
Inventory and configuration layout¶
The file names are not free: Kolla-Ansible builds them from the cluster name (ceph_cluster, ceph by default) and
the user, as <cluster>.client.<user>.keyring, and fails or silently misses a file that does not match.
/etc/kolla/
globals.yml
passwords.yml # rbd_secret_uuid, cinder_rbd_secret_uuid (generated by kolla-genpwd)
config/
glance/
ceph.conf
ceph.client.glance.keyring
glance.conf # optional: show_image_direct_url for copy-on-write
cinder/
ceph.conf # merged with the per-service files below
cinder-volume/ceph.client.cinder.keyring
cinder-backup/ceph.client.cinder.keyring # backup reads the volumes pool too
cinder-backup/ceph.client.cinder-backup.keyring
nova/
ceph.conf
ceph.client.cinder.keyring # attach Cinder volumes; also the Nova user by default
# ceph.client.nova.keyring # only with a separate ceph_nova_user
# globals.yml
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes" # instance disks in the vms pool; leave "no" to keep them local
# enable_cinder_backup is on with Cinder; its driver defaults to ceph (cinder_backup_driver)
# defaults, shown for reference:
# ceph_glance_user: "glance" ceph_glance_pool_name: "images"
# ceph_cinder_user: "cinder" ceph_cinder_pool_name: "volumes"
# ceph_cinder_backup_user: "cinder-backup" ceph_cinder_backup_pool_name: "backups"
# ceph_nova_user: "{{ ceph_cinder_user }}" ceph_nova_pool_name: "vms"
The keyrings are secrets: keep them out of git, like passwords.yml
(laying out the configuration). Kolla-Ansible
copies them with Ansible's template module, so they are rendered as Jinja: an ordinary keyring is unaffected.
What Kolla-Ansible does with them (2026.1)¶
- Glance:
ceph.confand the keyring go intoglance_api; withglance_backend_ceph, the first entry ofglance_ceph_backendsbecomes the default store. - Cinder: one back-end section per entry of
cinder_ceph_backends, namedrbd-1by default (cinder_backend_ceph_name), withrbd_user,rbd_poolandrbd_secret_uuid = <cinder_rbd_secret_uuid>. Itsceph.confis merged fromconfig/cinder/<cluster>.confandconfig/cinder/<service>/<cluster>.conf. - Nova: the configuration tasks of
nova-cell/tasks/external_ceph.ymlfail when the keyring for the Nova user (withnova_backend_ceph) or the Cinder user (withcinder_backend_ceph) is not found underconfig/nova/orconfig/nova/<inventory_hostname>/. The key is extracted from it and written as a libvirt secret: UUIDrbd_secret_uuidfor Nova's ephemeral disks,cinder_rbd_secret_uuidfor attached volumes. libvirt uses them to open the RBD images; without the right key, attaches and boots fail on the compute even when Cinder is healthy.
Applying it¶
1. Read-only checks from a host that will be a Ceph client, if it has the Ceph CLI (otherwise from the Ceph admin host): the user sees its pool.
rbd --id cinder --keyring ./ceph.client.cinder.keyring -c ./ceph.conf ls volumes # empty list, no error
kolla-ansible prechecks -i <inventory>
2. Deploy. On a new deployment, the ordinary kolla-ansible deploy. On a running one:
⚠️ State-changing. Reconfigures and restarts
glance_api, the Cinder services andnova_compute/nova_libvirton every compute; writes the libvirt secrets. Instances keep running; API calls to Glance, Cinder and Nova may fail while their containers restart. Switching an existing deployment's back-end does not move existing images, volumes or instance disks: they stay where they were, and new ones go to Ceph. Undo: set the switches back and reconfigure. That is a configuration undo only, safe before any resource has been created on Ceph: afterwards, instances, volumes and images stored in Ceph can no longer be opened by the services until the switches are set again.
kolla-ansible reconfigure -i <inventory> --tags glance,cinder,nova
3. Verify each service (read-only, then one test of each):
openstack volume service list # cinder-volume for rbd-1 and cinder-backup up
sudo docker exec nova_libvirt virsh secret-list # on a compute: ceph-persistent-cinder with cinder_backend_ceph,
# ceph-ephemeral-nova with nova_backend_ceph
⚠️ State-changing. Creates and deletes test resources in a test project.
openstack image create --disk-format raw --container-format bare --file <image.raw> test-ceph-image
openstack volume create --size 1 --image test-ceph-image test-ceph-vol # cloned from images into volumes
openstack volume backup create --name test-ceph-backup test-ceph-vol # written to backups
openstack server create --flavor <flavor> --image test-ceph-image --network <network> test-ceph-vm
openstack server add volume test-ceph-vm test-ceph-vol
# then delete the server, the backup, the volume and the image
On the Ceph side, rbd ls images, rbd ls volumes, rbd ls backups and rbd ls vms (with nova_backend_ceph)
each show the new objects. Raw images let Ceph clone them; with qcow2 images, Glance stores them as they are and
every volume or boot converts them (Ceph's OpenStack guide).
Where to look when it fails¶
| Symptom | First log to read |
|---|---|
| Deploy fails on a missing keyring in the Nova role | the file name under /etc/: <cluster>.client.<user>.keyring, with the Nova user (ceph_nova_user) and the Cinder user |
cinder-volume for rbd-1 down |
/var/: monitor unreachable, permission denied (capabilities), wrong keyring name, or a ceph.conf the parser rejected |
| Backups fail | /var/: the backup service needs both the Cinder and the backup keyrings |
| Image upload fails | /var/ |
| Volume attach or Ceph-backed boot fails on a compute | /var/ and /var/: a missing or wrong libvirt secret, or no route to the OSDs from the compute |
| A container does not start after the change | docker logs <container> (or podman logs) |
Key choices and their consequences¶
| Choice | Alternative(s) | Why this one here | Consequence in operation |
|---|---|---|---|
| Nova shares the Cinder Ceph user (default) | A separate client.nova and ceph_nova_user |
One key less to manage | The Cinder key can read and write instance disks too |
nova_backend_ceph: "yes" |
Local instance disks | Live migration without copying disks; one store | All instance I/O goes over the storage network to Ceph |
| Raw images in Glance | qcow2 | Copy-on-write clones for volumes and instances | Larger uploads |
show_image_direct_url = True for Glance |
Off | Needed for copy-on-write | The guide warns of a security risk when Glance has back-ends other than Ceph |
cinder-volume on the controllers (the cinder group) |
Dedicated hosts | No extra hosts for a client-only service | The cinder group must not be empty, or Cinder has nowhere to run |
What went wrong during deployment¶
Nothing observed yet against a Ceph cluster: written from the Kolla-Ansible external Ceph guide, the 2026.1 roles and Ceph's documentation; only the read-only checks listed in Validity and limits were run on a lab.
Version notes¶
- 2026.1: per-host Ceph clusters for computes (
ceph_clusterper host, files underconfig/nova/<host>/) and several Ceph back-ends for Glance and Cinder (glance_ceph_backends,cinder_ceph_backends) are supported upstream, not covered on this page; see the guide. - RadosGW integration (
enable_ceph_rgw) is a separate topic, not covered here.
Lessons learned¶
No first-hand lessons yet. What the sources make clear:
- Create the pools and users with the capabilities from Ceph's guide, and name the keyrings exactly as Kolla-Ansible expects them.
- Strip the tabs from a generated
ceph.conf. - Test each path separately (image, volume, backup, attach, boot): each uses a different user, file or secret.
Validity and limits¶
Written from Kolla-Ansible 2026.1 (external Ceph guide, ansible/group_vars/all/{ceph,cinder,glance}.yml, the
cinder, glance and nova-cell roles) and Ceph's Block devices and OpenStack and user management guides. Only the
read-only checks that need no Ceph cluster were run on a lab (an all-in-one with Kolla-Ansible 22.2.0, 2026-10-27, see
confirmed_by): the defaults, the keyring checks, the template rendering and the libvirt secret names, and an empty
virsh secret-list. Nothing was run against a Ceph cluster: rbd ls, the prechecks, the reconfigure and the test
resources were not, so no output involving Ceph on this page was observed. Not covered: several Ceph clusters, Manila with CephFS, Gnocchi,
RadosGW, Ceph without cephx, migrating existing data into Ceph, and Ceph's own sizing and operations.
Related¶
- Cinder and Glance back-ends compared
- Where instance disks live
- Kolla-Ansible 2026.1: external Ceph guide
- Ceph: Block devices and OpenStack, user management
Provenance: written by agent:ozarland-author, from the Kolla-Ansible 2026.1 documentation and code, and the Ceph documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your topology, and what happened.