Instance snapshot never finishes. Task state image_pending_upload or image_uploading, image queued or saving¶
Symptom¶
After openstack server image create --name <name> <server-id>, the instance keeps a snapshot task state for much
longer than the size of its disk explains, and the image does not become active:
$ openstack server show <server-id> -c status -c OS-EXT-STS:task_state
| OS-EXT-STS:task_state | image_uploading | (or image_snapshot_pending, image_snapshot, image_pending_upload)
| status | ACTIVE |
$ openstack image show <image-id> -c status -c size
| status | saving | (or queued)
Other actions on the instance are refused while the task state is set. Or the snapshot fails: the task state clears and the image disappears.
Impact¶
One instance: no other action (reboot, resize, migrate, a new snapshot) until the task state clears. During a cold snapshot, the instance is suspended. On the standard path, the snapshot is written to the compute's disk first, and an upload of a large disk loads the compute's network, the Glance API and its storage.
Quick checks¶
Read-only. With admin credentials:
# 1. The instance's task state, host, and the snapshot action
openstack server show <server-id> -c OS-EXT-STS:task_state -c OS-EXT-STS:power_state -c OS-EXT-SRV-ATTR:host
openstack server event list <server-id> # createImage and its start time (its result stays Success
# even when the upload fails later)
# 2. The image: status, size (shown only once active), when it was created
openstack image show <image-id> -c status -c size -c created_at -c updated_at
# 3. Is the compute up?
openstack compute service list --service nova-compute --host <compute>
On the compute, the steps logged for the server ID, and the space where the snapshot is written:
Kolla-Ansible:
sudo grep <server-id> /var/log/kolla/nova/nova-compute.log | grep -iE 'snapshot|upload' | tail -n 20
sudo docker exec nova_compute df -h /var/lib/nova/instances
sudo docker exec nova_compute ls -la /var/lib/nova/instances/snapshots/
Any: the nova-compute log, and the file system of [libvirt] snapshots_directory (default
$instances_path/snapshots).
Diagnosis¶
How a snapshot runs (Nova 2026.1)¶
- API (
nova/compute/api.py): creates the image record in Glance (statusqueued), sets the task state toimage_snapshot_pending, and sendssnapshot_instanceto the compute. - Compute (
nova/compute/manager.py): sets the task stateimage_snapshot, then waits for a slot: at most[DEFAULT] max_concurrent_snapshots(default 5) snapshots run at once on a compute. In its slot, it logsinstance snapshotting. - libvirt driver (
nova/virt/libvirt/driver.py,snapshot), in this order:- logs
Beginning live snapshot process(the default for a running instance) orBeginning cold snapshot process(instance shut off, LVM, encrypted ephemeral storage, or[workarounds] disable_libvirt_livesnapshot; a running instance is suspended with a managed save for it), and setsimage_pending_upload, thenimage_uploading; - direct snapshot (instance disks and images both on Ceph RBD): a Ceph snapshot and clone in the images pool;
nothing is uploaded through Glance, and the image stays
queueduntil Nova records its location. If Ceph refuses, the driver logsPerforming standard snapshot because direct snapshot failed: <error>, setsimage_pending_uploadagain, and takes the standard path; - standard snapshot: writes the disk to a temporary directory under
snapshots_directoryon the compute, logsSnapshot extracted, beginning image upload, setsimage_uploading, and uploads the file to Glance (image statussaving).
- logs
- End:
Snapshot image upload complete, thenTook <n> seconds to snapshot the instance on the hypervisor.; the task state clears, the image isactive.
On any error, nova-compute deletes the image in Glance (delete_image_on_error) and clears the task state,
logging Successfully reverted task state from <task state> on failure for instance.; the error itself is in the
lines before it (for an upload, Error contacting glance server '<url>' for 'upload', retrying. then done
trying., with tracebacks). On the standard path (local disks, or a direct snapshot that fell back) an error is not
logged as Failed to snapshot image: read the lines before Successfully reverted task state …. On the direct
(Ceph RBD) path, a failure is logged as Failed to snapshot image (ERROR, with a traceback); the lab did not run that
path. A snapshot that failed leaves no image behind; a snapshot that is stuck leaves a queued or saving image.
When nova-compute starts, it clears every snapshot task state left on its instances (_init_instance; the log
line Instance in transitional state <state> at start-up clearing task state is at debug level). The image in
Glance is not cleaned up then.
Where it stops¶
| Last state | Last compute log line | Where to look |
|---|---|---|
image_snapshot_pending |
none for this server | The request did not reach the compute: nova-compute down or busy, message bus |
image_snapshot |
none for this server yet | Waiting for a slot: max_concurrent_snapshots other snapshots run on this compute |
image_pending_upload |
Performing standard snapshot because direct snapshot failed, or Beginning live/ without Ceph |
Writing the snapshot on the compute: disk space under snapshots_directory, disk speed, a large disk |
image_uploading, image queued |
Beginning live/, no Snapshot extracted line (Ceph RBD) |
The direct snapshot is stuck in Ceph: Ceph health, the images pool |
image_uploading, image saving |
Snapshot extracted, beginning image upload |
The upload to Glance: network from the compute, the Glance API, Glance's storage |
| cleared, image gone | Successfully reverted task state from ... on failure for instance., errors before it |
The errors before it name the cause (standard path) |
| cleared, image gone | Failed to snapshot image (ERROR, traceback) |
The direct (Ceph RBD) snapshot failed; the traceback names the cause |
Possible causes¶
Cause 1: a large disk on a slow path¶
Nothing is broken; the copy takes time. While the snapshot is image_pending_upload, the temporary file under
snapshots_directory grows. Once Snapshot extracted, beginning image upload is logged, the file is complete and the
upload has no progress indicator in the API: the image stays saving with no size until it turns active, and
glance-api-uwsgi.log logs the PUT /v2/images/<image-id>/file line only when the request ends. With Kolla-Ansible, HAProxy's client and server timeouts for the Glance API are 6 hours
(haproxy_glance_api_client_timeout, haproxy_glance_api_server_timeout): an upload longer than that is cut.
Cause 2: no space on the compute for the snapshot¶
The whole disk is written under snapshots_directory before the upload, on the same file system as the instances
by default. A full file system makes the extract fail (the snapshot fails, the image is deleted), and can also stop
the instances that write to their disks there (compute disk full).
Cause 3: Glance cannot store the image¶
Glance's storage is full or unreachable: the file store (with Kolla-Ansible's default glance_backend_file, the
glance Docker volume on the first glance-api host), or the Ceph images pool. The upload fails or hangs; the
Glance API log has the error.
Cause 4: nova-compute stopped or restarted during the snapshot¶
The task state was cleared at restart, or stays set if nova-compute is down. The image remains queued or saving
in Glance, empty or partial. The temporary directory under snapshots_directory, with the full extracted disk, is
left behind on the compute.
Resolution¶
For cause 1: wait, and watch progress¶
Repeat quick checks 1 and 2 and the ls of the snapshots directory. If the temporary file grows, or the upload
started (Snapshot extracted) and the task state is still image_uploading with no error in the compute log, let it
finish. For large
instances, plan snapshots where the network and Glance allow, or snapshot their data volumes with Cinder instead.
For causes 2 and 3: free space, then snapshot again¶
(Not run on the lab.)
Free space on the compute (cause 2) or on Glance's storage (cause 3), from the error in the traceback. The failed snapshot cleaned up after itself: run the snapshot again.
For cause 4: clear the task state, delete the stale image¶
If nova-compute is down, start it: it clears the task state at start-up. Then find the image left behind, and check
that it is the one of this snapshot:
openstack image show <image-id> -c status -c size -c owner -c created_at -c properties
# owner: the instance's project; created_at: the time of the failed snapshot;
# properties: instance_uuid matching <server-id>; size not growing between two runs
🛑 Destructive. Deletes the image record and its data in Glance. It belongs to the instance's project, not only to this instance: another
queuedorsavingimage may be another upload still running, by this project or another one. Precondition: no snapshot of this instance is running any more (task state cleared, no upload in progress in the compute log), and the checks above all match this server. Cannot be undone; run the snapshot again afterwards.
openstack image delete <image-id>
Then remove the temporary directory the interrupted snapshot left on the compute:
sudo docker exec nova_compute ls -la /var/lib/nova/instances/snapshots/ # tmp<random>/ with one file each
🛑 Destructive. Deletes the extracted copy of the instance's disk. Precondition: no snapshot is running on this compute (no instance with a snapshot task state on it, no
Snapshot extractedwithoutupload completesince the last start ofnova-compute), and onlytmp*directories older than the last start ofnova_compute(sudo docker inspect -f '{{.State.StartedAt}}' nova_compute); a running snapshot reads its file from there. Cannot be undone.
sudo docker exec -u root nova_compute rm -rf /var/lib/nova/instances/snapshots/<tmp-dir>
With a direct snapshot (Ceph) (not run on the lab: no Ceph), deleting the queued image may not remove what Ceph already created: a snapshot or a
clone can remain in the images pool or on the instance's disk. Look, read-only, with
rbd snap ls <pool>/<server-id>_disk and rbd ls -l <images-pool>, and escalate to whoever runs Ceph rather than
deleting there.
If the task state stays set with nova-compute up and no snapshot in progress on the compute (not run on the lab),
restart nova-compute on that compute: it clears the task state of its instances (above). First check the other instances on that compute
(openstack server list --all-projects --host <compute> --long): a restart interrupts their migrations, resizes and
snapshots too.
⚠️ State-changing. Restarts
nova-computeon that compute; running instances are not affected; operations in progress on its instances fail. Undo: none needed.
Kolla-Ansible: sudo systemctl restart kolla-nova_compute-container.service (about 60 s on the lab: the unit's
ExecStop waits up to Kolla-Ansible's docker_graceful_timeout, 60 s by default)
Any: restart the nova-compute service.
A cold snapshot (not run on the lab) suspends a running instance with a libvirt managed save, while Nova's vm_state stays active.
Interrupted, it most likely leaves the instance SHUTOFF: start it with openstack server start <server-id> once the
task state is clear (openstack server resume is refused unless Nova shows it SUSPENDED).
Verification¶
openstack server show <server-id> -c status -c OS-EXT-STS:task_state # ACTIVE (or as before), no task state
openstack image show <image-id> -c status -c size # active, with a size, for a new snapshot
openstack image list --status queued ; openstack image list --status saving # no leftovers of this instance
To check that a snapshot is usable, boot a test instance from it, then delete the instance.
Lab observations¶
Kolla-Ansible 22.2.0 (2026.1) all-in-one, local instance disks (qcow2), Glance file store, a CirrOS instance, the
upload slowed with tc on the Glance API port, on 2026-10-27:
- Cause 1 (upload at 4 Mbit/s):
image_pending_uploadandqueuedfor about 10 s while the temporary file grew, thenimage_uploadingandsavingwith nosize;Snapshot extractedtoSnapshot image upload complete108 s,Took 117.55 seconds to snapshot the instance on the hypervisor., imageactiveat 53542912 bytes. The Glance file store file stayed at 0 bytes during the upload. - An upload broken by the network (packets dropped):
Error contacting glance server '<url>' for 'upload', retrying.three times,done trying.,Successfully reverted task state from image_uploading on failure for instance.; GlanceFailed to upload image data due to internal error: OSError: unable to receive chunked part; the image was deleted (DELETE204). NoFailed to snapshot imageline (standard path).server event listshowed thecreateImageevent, itscompute_snapshot_instanceresultSuccess. - Cause 4:
systemctl restart kolla-nova_compute-container.service20 s into the upload took 61 s; Glance loggedunable to receive chunked part16 s after the stop;nova-computeloggedStarting compute node5 s after the restart returned and the task state was clear within 11 s after that; the image was leftqueued(nosize), withinstance_uuid,ownerandcreated_atmatching.image deleteworked; a new snapshot wasactiveand an instance booted from it. A 52 MBtmp<random>/directory was left under/var/lib/nova/instances/snapshots/and removed by hand.
Validity and limits¶
- Lab-run for causes 1 and 4 and for an upload failure on the standard snapshot path (see Lab observations). Not
run: cause 2 (compute disk full), cause 3 (Glance store full or unreachable), the cold snapshot, the direct
snapshot on Ceph RBD and its
rbdchecks, themax_concurrent_snapshotswait, the restart ofnova-computewith the task state stuck while it is up. - Written from Nova 2026.1 (
nova/compute/api.pysnapshot,nova/compute/manager.py_snapshot_instance,delete_image_on_error,_init_instance;nova/virt/libvirt/driver.pysnapshot;snapshots_directory,disable_libvirt_livesnapshot) and Kolla-Ansible 2026.1 (Glance role defaults and backends). - Not covered: snapshots of instances booted from volume (they create Cinder volume snapshots, not a disk upload), Glance image import, Glance multi-store, and drivers other than libvirt.
Related¶
- Compute disk full. An instance that writes is frozen
- Stale records
- Nova 2026.1: configuration reference
- Glance 2026.1: image statuses
Provenance: written by agent:ozarland-author, from the Nova and Kolla-Ansible 2026.1 documentation and source. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.