Stale volume attachment. The volume is still in-use for an instance that was deleted or evacuated¶
Symptom¶
One of these, on a volume that is not multi-attach:
- The instance was deleted, but the volume is still
in-useand "Attached to" names the deleted instance. The user can neither delete the volume nor attach it to another instance. - The instance was evacuated to another compute and runs there, but Cinder lists more than one attachment for the volume, or an attachment whose connector names the old host.
$ openstack volume show <volume-id> -c status -c attachments
| status | in-use |
| attachments | [{'server_id': '<server-id>', ...}] |
$ openstack server show <server-id>
No Server found for <server-id>
Impact¶
One volume and its owner: the volume cannot be reused or deleted, and its quota stays consumed. The data is intact. The risk is in the fix: removing an attachment that a running instance still uses, or forcing the status of a volume that a hypervisor still has open, can give the volume two writers and corrupt it.
Quick checks¶
With admin credentials:
# 1. Cinder: the volume and every attachment record for it (3.27 or later)
openstack volume show <volume-id> -c status -c attachments
cinder --os-volume-api-version 3.27 attachment-list --all-tenants 1 --volume-id <volume-id>
openstack --os-volume-api-version 3.27 volume attachment show <attachment-id> # Properties: the connector, with the host
# On an iSCSI backend (LVM) its Properties also hold the CHAP auth_username and auth_password in clear text:
# redact them before you paste the output into a ticket or a chat.
# Not "openstack volume attachment list --volume-id": with python-openstackclient 9.0.0 the filter is ignored
# and every attachment of the cloud is listed. Check the Volume ID column before you delete anything.
# 2. Nova: does the instance exist, where does it run, and which attachment does it use?
openstack server show <server-id> -c status -c OS-EXT-SRV-ATTR:host -c OS-EXT-SRV-ATTR:instance_name
openstack --os-compute-api-version 2.89 server volume list <server-id>
# 3. Was it deleted? The last actions on the instance
openstack server event list <server-id>
Kolla-Ansible: on the host named by the stale attachment's connector, and on the instance's current host:
sudo docker exec nova_libvirt virsh list --all
sudo docker exec nova_libvirt virsh domblklist <instance-name>
Any: the same virsh commands on those hosts.
Diagnosis¶
Classify each attachment record of the volume:
| What you find | Meaning |
|---|---|
| The instance does not exist in Nova, and no domain on its last host uses the volume | Stale: left by the deletion |
The instance exists, and the attachment ID is the one server volume list shows |
In use: keep it |
| The instance exists on host B, and this attachment's connector names host A, with another ID than Nova's | Stale: left by the evacuation |
nova-manage volume_attachment show <server-id> <volume-id> --json also prints the attachment ID that Nova's
database holds for the pair, including the connection information the API does not show.
Kolla-Ansible: sudo docker exec nova_api nova-manage volume_attachment show <server-id> <volume-id> --json on
a controller. For an instance that was deleted it answers No volume Block Device Mapping with id <volume-id>.
Any: run it where nova-manage has access to the Nova database.
For the deletion case, the nova-api and nova-compute logs at the time of the deletion usually show why Nova could
not tell Cinder: Cinder unreachable, or the compute down while the instance was deleted.
What Nova 2026.1 did on a lab, which narrows down when this happens:
-
cinder-apidown during a normal deletion: Nova deleted the instance anyway, in about 15 seconds, and left the attachment in Cinder. This reproduces the symptom.nova-compute.logshows threekeystoneauth.sessionwarningsFailure: Unable to establish connection to http://<cinder-api>:8776/v3/attachments/<attachment-id>, thenIgnoring unknown exception for volume <volume-id>: Connection to cinder host failed: …. Nova had already disconnected the volume on the host (no iSCSI session left); the target stayed exported by the backend until the attachment was deleted. -
cinder-volumedown during a normal deletion: the instance stayeddeleting(7 minutes in the longest run), and the deletion and the detach finished oncecinder-volumewas back. No stale attachment. - The compute down during the deletion:
nova-apideleted the instance from its database (instance's host <host> is down, deleting from database), and Cinder released the volume once it could. No stale attachment either, but the guest kept running on the host with the volume open untilnova-computecame back and destroyed it (Destroying instance with name label '<instance-name>' which is marked as DELETED but still present on host), while Cinder already showed the volumeavailable. The destroyed domain stayed defined, and the host kept an iSCSI session to the volume. Checkvirsh list --alland the storage sessions on the instance's last host before you trust anavailablestatus after such a deletion.
Possible causes¶
Cause 1: the instance was deleted while Cinder or its compute was unreachable¶
Nova deleted the instance but the detach in Cinder failed or was never sent. Tell it by the instance missing from Nova with the attachment still present, and errors towards Cinder in the Nova logs at the time of the deletion.
Cause 2: an evacuation left the source host's attachment behind¶
Evacuation rebuilds the instance on another host and attaches its volumes there. If the rebuild failed part-way or the old attachment was not removed, Cinder keeps a record whose connector names the source host. Tell it by the instance running on another host than the one in the connector, with an attachment ID that Nova does not use.
Resolution¶
Before any step: the host named in the stale attachment's connector must not run a domain that uses the volume
(virsh domblklist). For an evacuation, the source host must be fenced or its domain destroyed; see
fence before you evacuate.
For cause 1: delete the stale attachment¶
⚠️ State-changing. Deleting the attachment record releases the volume in Cinder and removes the export for that attachment on the storage backend. The volume's data is not touched. Precondition: the instance no longer exists in Nova, and no domain on its last host uses the volume. Undo: none needed; the volume can be attached again normally.
openstack --os-volume-api-version 3.27 volume attachment delete <attachment-id>
openstack volume show <volume-id> -c status # expected: available, once no attachment remains
For cause 2: remove the old host's attachment, keep Nova's¶
(Not run on the lab: the evacuation case needs two computes.)
Preferred, documented in the Nova administration guide for 2026.1: a hard reboot makes Nova check the instance's attachments against Cinder and remove the dangling ones from both databases.
⚠️ State-changing. A hard reboot powers the instance off and on: the guest loses what was not written to disk. Agree on a time with the instance's owner.
openstack server reboot --hard <server-id>
If a reboot is not acceptable, delete only the attachment that Nova does not use:
⚠️ State-changing. Same effect as for cause 1, on one record. Precondition: its ID differs from the one in
server volume list(compute API 2.89), and its connector names the old host, which is fenced. Cinder refuses to delete an attachment that the instance still uses, with HTTP 409 (bug 2004555); on deployments without that protection, the check of the ID is all that stands between you and detaching a running instance's disk.
openstack --os-volume-api-version 3.27 volume attachment delete <attachment-id>
When Nova's own attachment is wrong after the evacuation¶
The instance runs, but its attachment carries outdated connection information (for example a connector from the old
host), and nova-manage volume_attachment show confirms it. Nova documents a refresh with nova-manage, available
since Xena:
⚠️ State-changing. The instance must be stopped and unlocked. The refresh disconnects the host's volume connections, deletes the Cinder attachment, creates a new one and updates Nova's database. The instance is down until it is started again.
openstack server stop <server-id>
Kolla-Ansible: not working as written on Kolla-Ansible 22.2.0 (see Lab observations). Each line below failed on a lab, for a different reason:
docker exec nova_compute nova-manage volume_attachment get_connector --jsonfails as the container's default user (FailedToDropPrivileges: privsep helper command exited non-zero); it works withsudo docker exec -u root.docker exec nova_api nova-manage volume_attachment refresh …runs as root and fails with[Errno 13] Permission denied: '/dev/shm/<host>_nova-api_qmanager'(the host'sfs.protected_regular).- Run as the
novauser, the refresh times out waiting for the compute's reply (Kolla-Ansible setsuse_queue_manager = true). It has then already removed the volume's connection and attachment: the volume is leftreservedwith an attachment that has no connection information, and the instance starts without the volume. Repairing that took the last resort below, a hard reboot and a new attach; see the volume stuck in attaching, reserved or detaching runbook. - With
use_queue_manager = falsefor this command only (a second--config-file), the compute's reply arrived. The full refresh was not completed on the lab.
# On the compute that hosts the instance: its connector
sudo docker exec -u root nova_compute nova-manage volume_attachment get_connector --json > connector.json
# On a controller, with that file copied there: give it to the nova_api container and refresh
sudo docker cp connector.json nova_api:/tmp/connector.json
🛑 Destructive on Kolla-Ansible 22.2.0. With
use_queue_manager = true(Kolla-Ansible's setting), therefreshbelow timed out on a lab after it had already removed the volume's connection and its Cinder attachment. The volume was leftreserved, Nova's attachment had no connection information, and the instance started without its disk;openstack server remove volumewas refused. Repair path that worked, with the volume in no domain (virsh domblklist):openstack volume set --state available --detached <volume-id>(the last resort below, with its precondition), thenopenstack server reboot --hard <server-id>, which removed Nova's dangling record, then a normalopenstack server add volume <server-id> <volume-id>. For the compute's reply to arrive, the command neededuse_queue_manager = falsefor this one command, through a second configuration file; the full refresh was still not completed on the lab. Precondition: the instance stopped and unlocked, and a time agreed with its owner, who may lose the volume until the repair.
# Not lab-verified: see the warning above. The second --config-file holds
# [oslo_messaging_rabbit] use_queue_manager = false, copied into the container like connector.json
sudo docker exec -u nova nova_api nova-manage --config-file /etc/nova/nova.conf --config-file /tmp/nm.conf \
volume_attachment refresh <server-id> <volume-id> /tmp/connector.json
Any:
nova-manage volume_attachment get_connector --json > connector.json # on the compute
nova-manage volume_attachment refresh <server-id> <volume-id> connector.json
openstack server start <server-id>
Last resort: the volume stays in-use with no attachment left¶
🛑 Destructive.
openstack volume set --stateand--detachedonly change Cinder's database record, with no check of the real attachment (the client's own help says so). If any hypervisor still has the volume open, a new attachment gives it a second writer and corrupts the data. Precondition: no attachment record left, and on every host named by the volume's past attachments,virsh domblklistshows no domain using it (for RBD volumes,rbd status <pool>/volume-<volume-id>shows no watcher). Undo: set the status back; data written by two writers cannot be recovered without a backup.
openstack volume set --state available --detached <volume-id>
Verification¶
openstack volume show <volume-id> -c status -c attachments
# deleted instance: available, no attachment
# evacuated instance: in-use, one attachment, to the instance on its current host
cinder --os-volume-api-version 3.27 attachment-list --all-tenants 1 --volume-id <volume-id>
openstack --os-compute-api-version 2.89 server volume list <server-id>
For an evacuated instance, virsh domblklist <instance-name> on its current host lists the volume, and the
instance's guest sees its disk.
Lab observations¶
On a lab: OpenStack 2026.1, Kolla-Ansible 22.2.0, all-in-one (one compute), Cinder LVM over iSCSI, python-openstackclient 9.0.0.
- Deletion case, real outage.
cinder-apistopped (systemctl stop kolla-cinder_api-container.service) while the instance was deleted: the instance was gone in about 15 seconds, the volume stayedin-usewith oneattachedrecord,nova-manage volume_attachment showansweredNo volume Block Device Mapping with id <volume-id>., andvolume attachment delete(cause 1) made the volumeavailablein about 2 seconds, with the backend's iSCSI target removed. Outages ofcinder-volumeor of the compute did not leave a stale attachment (see Diagnosis). - Deletion case, synthetic. The symptom was also produced by
creating a Cinder attachment that Nova did not know (
volume attachment create --connectwith the host's connector, thenvolume attachment complete) and deleting the instance:- the volume stayed
in-use, attached to the deleted server; server showansweredNo Server found;cinder attachment-listshowed oneattachedrecord;volume attachment deletemade the volumeavailable, with no attachment and no iSCSI session left.
- the volume stayed
- 409 protection. A
volume attachment deleteof an attachment Nova used was refused:409 … ConflictNovaUsingAttachment. - Hard reboot. With Cinder's attachment gone and Nova still listing the volume, a hard reboot removed Nova's record:
Removing stale volume attachment '<attachment-id>' from instance for volume '<volume-id>'. - Last resort.
volume set --state available --detachedworked as written. Cinder refusedvolume set --state in-useon a volume with no attachment. nova-manage volume_attachment show. Worked as written for an existing instance; itsattachment_idwas the oneserver volume listshowed.get_connectorandrefresh. Failed as written on Kolla-Ansible; see the resolution section.
Not run: the evacuation case (it needs two computes) and RBD.
Validity and limits¶
Written from the upstream documentation and client source for 2026.1: the Nova volume administration guide (hard
reboot clean-up, nova-manage volume_attachment), the Block Storage API reference (attachment statuses, the HTTP 409
protection), and the OpenStack client's volume set and volume attachment commands. Partly observed on an
all-in-one lab (see Lab observations), not on a multinode platform. A run of the evacuation case on a multinode
lab, and of the refresh procedure on a deployment where it completes, are still wanted.
Per release:
nova-manage volume_attachment show,get_connectorandrefreshexist from Nova 24.0.0 (Xena).- Cinder attachment records need volume API 3.27 or later; the attachment ID in
server volume listneeds compute API 2.89 or later (Xena and later). - The HTTP 409 protection on attachment deletion comes from bug 2004555; check that your release and deployment carry it before relying on it.
- The hard-reboot clean-up of dangling attachments is documented in the 2026.1 Nova guide; check the guide of your release before relying on it on an older one.
Not covered: multi-attach volumes, boot volumes of deleted instances (delete_on_termination decides whether they
are kept), host-side clean-up of leftover iSCSI sessions or multipath devices, and direct database edits.
Related¶
- Volume stuck in attaching, reserved or detaching
- Compute host unreachable, instances down. Fence before you evacuate
- Stale records: stuck instances, orphan ports and attachments, phantom allocations
- Nova, Manage volumes and nova-manage volume_attachment
- Block Storage API, attachments
Provenance: written by agent:ozarland-author, from upstream documentation. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.