Skip to content

← Production Guide · Storage

PG-14 · In-use by a VM that is gone?

In-use by a VM that is gone? Operational lesson: A record is stale only when the hypervisor agrees. Check the host before you change the database.

Operational lesson. A record is stale only when the hypervisor agrees. Check the host before you change the database.

The memo as text

Situation. The instance was deleted but its volume stays in-use, or it was evacuated and Cinder keeps an attachment for the old host. The data is intact. The risk is in the fix.

First checks (read-only):

  1. Every attachment of the volume: cinder attachment-list --all-tenants 1 --volume-id <volume>
  2. Does the instance still exist?: openstack server show <server> -c status -c OS-EXT-SRV-ATTR:host
  3. Which attachment Nova uses: openstack server volume list <server>
  4. Does a domain still use the volume?: virsh list --all, then virsh domblklist <domain>, on each host

Think before acting:

  • ⚠️ State-changing. Delete only an attachment Nova does not use. Compare its ID with server volume list before you delete anything.
  • 🛑 Destructive. volume set --state available --detached checks nothing. If a hypervisor still has the volume open, it gets two writers.

Go deeper

This memo is a memory aid. The procedure, with every command, its risk warning and its limits, is in the runbook:

Where this holds. Written from the 2026.1 documentation and partly observed on an all-in-one lab; the evacuation case needs two computes and was not run. Applies to volumes that are not multi-attach. Attachment records need volume API 3.27 or later, the attachment ID in server volume list needs compute API 2.89 or later. With python-openstackclient 9.0.0, "openstack volume attachment list --volume-id" ignored its filter and listed every attachment of the cloud: check the Volume ID column.

LinkedIn kit

The instance was deleted, or evacuated. Its volume is still "in-use", with an attachment that points at something gone. After a deletion it can be neither deleted nor attached anywhere else.

The data is intact. The risk is entirely in the fix.

An attachment record is stale only if the hypervisor agrees:
→ instance gone from Nova, no domain on its last host uses the volume: stale, left by the deletion
→ the attachment ID is the one Nova's server volume list shows: in use, keep it
→ the instance runs on host B, the connector names host A, and the ID is not the one Nova uses: stale, left by the evacuation

Forcing the volume back to "available" checks none of this. If a hypervisor still has it open, the next attach creates a second writer.

Check the host before you touch the database.

Runbook: https://github.com/OzarLand/ozarland-academy/blob/main/docs/runbooks/volumes/stale-volume-attachment-after-instance-deleted-or-evacuated.md
More memos: https://github.com/OzarLand/ozarland-academy/tree/main/docs/production

#OpenStack #Cinder #Storage #DataIntegrity #CloudOperations #OzarLand

Source: volume-in-use-by-a-missing-instance.yml. Found an error, or does your platform behave differently? Open an issue or see CONTRIBUTING.