Resize or cold migration not finished. Instance in VERIFY_RESIZE, stuck in RESIZE, or in ERROR after a failed resize¶
Symptom¶
After openstack server resize --flavor <flavor> <server-id>, or a cold migration (openstack server migrate), the
instance is in one of three states:
$ openstack server show <server-id> -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host -c flavor
status |
task_state |
What it means |
|---|---|---|
VERIFY_RESIZE |
None |
The resize worked and waits for a confirm or a revert. It can stay so for days |
RESIZE |
resize_prep, resize_migrating, resize_migrated or resize_finish |
The operation stopped part way: a service died, or a message was lost |
ERROR |
None |
A step failed; the fault says which one |
Impact¶
One instance. In VERIFY_RESIZE, it runs on the new flavor (and host), but the old copy is kept: its disks stay on
the source compute, and Placement keeps resources allocated on both computes until the confirm. Many instances left
so can make the scheduler see computes as fuller than they are. In RESIZE or ERROR, the instance may be shut off,
and its owner cannot act on it.
Quick checks¶
Read-only. With admin credentials:
# 1. The state, host, flavor, and fault if any
openstack server show <server-id> -c status -c OS-EXT-STS:task_state -c OS-EXT-STS:power_state \
-c OS-EXT-SRV-ATTR:host -c flavor -c fault
# 2. The migration record: source and destination computes, status (finished, error, migrating...)
openstack server migration list --server <server-id>
# 3. The instance's actions, and the failed step
openstack server event list <server-id>
openstack server event show <server-id> <request-id>
# 4. Are both computes up?
openstack compute service list --service nova-compute
The nova-compute logs of the source and destination computes, for the server ID, have the error:
Kolla-Ansible: sudo grep <server-id> /var/log/kolla/nova/nova-compute.log
Any: the nova-compute service log on each compute.
Diagnosis¶
How a resize runs (Nova 2026.1, nova/compute/manager.py)¶
- Destination,
prep_resize: claims resources on the destination.task_stateresize_prep. - Source,
resize_instance:task_stateresize_migrating. The libvirt driver powers the guest off, unplugs its ports, disconnects its volumes, renames the instance directory to<uuid>_resize, and copies the local disks to the destination over SSH (or uses the shared instance directory). Then the instance'shostis set to the destination, andtask_statebecomesresize_migrated. - Destination,
finish_resize: switches the instance to the new flavor, resizes and starts the guest, and sets the instance toRESIZED(VERIFY_RESIZEin the API), migration statusfinished. - Confirm (
openstack server resize confirm): deletes the old copy on the source and the source allocation. Revert (openstack server resize revert): deletes the new guest, restarts the old one on the source with the old flavor. Both are allowed only inVERIFY_RESIZE.
[DEFAULT] resize_confirm_window (default 0, disabled) makes the destination compute confirm resizes older than
that many seconds. Kolla-Ansible 2026.1 does not set it, and sets allow_resize_to_same_host = true.
What fails where¶
| Step | Usual errors (fault or compute log) | State after |
|---|---|---|
Scheduling and prep_resize |
No valid host (from the conductor), resources claim failed, anti-affinity | ACTIVE again (or SHUTOFF), migration error |
| API, before any compute | a new root disk smaller than the image's min_disk or the boot (or rebuild) flavor's root disk: HTTP 400 Flavor's disk is smaller than the minimum size specified in image metadata. Flavor disk is <n> bytes, minimum size is <m> bytes. |
Unchanged, no migration record |
| Disk copy | Unable to resize disk down., not able to execute ssh command, Migration is not supported for LVM backed instances |
Rolled back to the previous state: these errors are raised as InstanceFaultRollback |
| Disk copy, other errors | copy interrupted, disk full on the destination | ERROR, host still the source |
finish_resize |
the guest fails to start on the destination (libvirt, volume, network) | ERROR, host the destination, flavor set back to the old one |
A nova-compute that restarts while an instance is in resize_migrating tries to revert the instance on that host,
then clears the task state whatever the result (_init_instance). A restart in resize_prep clears the task state
only. After a restart in resize_migrating, the migration record is not updated: it stays migrating, and the
Placement allocations stay as the move left them (the instance's on the destination, the migration's on the source).
What the compute logs say¶
Messages of nova/compute/manager.py (2026.1), in the nova-compute log of the compute named:
| Message | Where | Meaning |
|---|---|---|
Migrating |
destination | Claim done on the destination, the resize is handed to the source (_prep_resize) |
Setting instance back to <state> after: <error> |
source or destination | A rollback error (InstanceFaultRollback): the instance is back in its previous state, <error> says why |
Setting instance vm_state to ERROR followed by a traceback |
source or destination | Any other error: the instance is in ERROR; the traceback is the cause |
Instance found in migrating state during startup. Resetting task_state |
source | nova-compute restarted during the copy and reverted the instance (_init_instance) |
Failed to revert crashed migration |
source | That revert failed: recover by hand (below) |
Found <n> unconfirmed migrations older than <s> seconds, then Automatically confirming migration <id> for instance <uuid> |
destination | resize_confirm_window at work |
Setting migration <id> to error: In states <vm_state>/ |
destination | The automatic confirm skipped an instance that was not in VERIFY_RESIZE |
Deleting allocations for old flavor on source node <source> after finish_resize failure. You may be able to recover the instance by hard rebooting it. |
destination | finish_resize failed; Nova's own hint is the hard reboot of For causes 3 and 4 |
Confirming resize on source host. / Reverting resize on destination host. |
source / destination | A confirm or revert running |
Possible causes¶
Cause 1: nobody confirmed (VERIFY_RESIZE)¶
Nothing failed. The owner, or the tool that ran the resize, never confirmed it.
Cause 2: a service died or a message was lost (RESIZE with a task state)¶
nova-compute on the source or destination stopped, the message bus failed, or the conductor did not get a reply.
The migration record stays in pre-migrating, migrating or post-migrating.
Cause 3: SSH between computes (ERROR or rollback at the disk copy)¶
With local instance disks, the source copies the disks to the destination over SSH as the nova user. With
Kolla-Ansible, through the nova_ssh containers, which must run on every compute and accept the keys of the others.
Cause 4: the destination cannot run the guest (ERROR after finish)¶
A problem on the destination compute: libvirt or QEMU error, a volume connection failure, a port binding failure, no space for the disks.
Cause 5: a smaller disk (refused or rolled back)¶
Nova does not shrink disks, except a root disk booted from volume. The API compares the new flavor's root disk with
the instance's image_min_disk: the larger of the image's min_disk and the root disk of the flavor it was booted
(or rebuilt) with. A root disk below that is refused at once (HTTP 400, Flavor's disk is smaller than the minimum
size specified in image metadata): nothing is left to recover. Any other shrink passes the API and is rolled back by
the source compute with Unable to resize disk down.: a smaller ephemeral disk, or, after an earlier resize up, a
root disk smaller than the current one but not below the boot flavor's.
Resolution¶
For cause 1: confirm or revert¶
Both choices belong to the instance's owner, and neither can be undone.
🛑 Destructive.
confirmdeletes the instance's previous copy (disks and definition) on the source compute and frees its resources: the old copy is lost for good, and going back to the previous flavor or host takes a new resize or migration. Precondition: the owner has checked the resized instance and agrees. Cannot be undone.
openstack server resize confirm <server-id> # or: openstack server migration confirm <server-id>
🛑 Destructive.
revertdestroys the resized guest and restarts the previous copy on the source, with the previous flavor: everything the guest wrote to its local disks since the resize is lost (data on volumes is kept). Precondition: the owner agrees; if the data written since the resize matters, copy it out from inside the guest first, or confirm instead of reverting. A Nova snapshot is not possible here:openstack server image createis refused (HTTP 409) for an instance inVERIFY_RESIZE. Cannot be undone.
openstack server resize revert <server-id> # or: openstack server migration revert <server-id>
To confirm automatically after a delay, set resize_confirm_window on the computes:
Kolla-Ansible: in /etc/kolla/config/nova/nova-compute.conf, [DEFAULT] resize_confirm_window = <seconds>,
then kolla-ansible reconfigure -i <inventory> --tags nova.
⚠️ State-changing.
reconfigurerestarts thenova_computecontainers; running instances are not affected. From then on, every resize older than the window is confirmed without the owner, and can no longer be reverted.
Any: the same option in nova.conf on every compute, then restart nova-compute.
For cause 2: restart the service, then check again¶
Bring back the service that stopped (quick check 4), or the message bus
(RabbitMQ, shared
dependencies). On Kolla-Ansible
2026.1, each container runs under a systemd unit (kolla-<container>-container.service, Restart=always,
RestartSec 11 s with the Kolla defaults, 59 s on the lab, which sets docker_client_timeout: 600): a container
that stopped or was stopped with docker stop is started again by systemd after that delay, without anyone typing
docker start. If the instance was in resize_migrating on that compute, its restart tries to revert the instance.
Then check quick check 1 again, and the source nova-compute log for Failed to revert crashed migration.
If that revert failed, the instance is back on the source with its disks but stopped (SHUTOFF after the next power
state sync), and its migration record stays migrating. Start it, then check the allocations (Verification): the
instance's allocation is still on the destination and the migration's on the source.
⚠️ State-changing. Starts the guest on the source compute. Before it, check that the disks are on the source, and that
virsh list --allshows the domain on the source and none on the destination. Undo:openstack server stop <server-id>.
openstack server start <server-id>
If the instance stays in RESIZE with the services up, find where its disks are (the host field and the migration
record), and on that compute, whether the guest exists:
Kolla-Ansible: sudo docker exec nova_libvirt virsh list --all --uuid | grep <server-id>
Any: sudo virsh list --all --uuid | grep <server-id>
Then reset the state and hard reboot, as for an instance in ERROR (below).
For causes 3 and 4: fix the compute, then recover the instance¶
Fix what the error names: the nova_ssh containers and keys (cause 3), or the destination's own problem (cause 4, not
reproduced on the lab, with the runbooks of boot from volume,
port binding, disk full).
Then recover the instance where Nova says it is (host). The python-openstackclient help for server set --state
warns: "If the server is in an error state then this is almost never the correct command to run and you should
prefer hard reboot where possible. In particular, if the server is in an error state due to a move operation,
setting the state can result in instances that are no longer usable."
⚠️ State-changing. Recreates the guest on the compute in the instance's
hostfield, from the disks there, with the flavor Nova records. Precondition: the instance's disks are on that compute (local disks), or on shared storage or volumes. If the disks are only on the other compute, stop here: see Validity and limits.
openstack server reboot --hard <server-id>
If the instance is still in a RESIZE task state, the reboot is refused; clear the state first:
⚠️ State-changing. Sets the instance to
ACTIVEwith no task state in the database only; nothing changes on the compute. Undo:openstack server set --state error <server-id>.
# with python-openstackclient 10.5.0 add --auto-approve (see below)
openstack server set --state active <server-id>
openstack server reboot --hard <server-id>
python-openstackclient 10.5.0 asks Do you wish to continue? [y/n] before server set --state and --auto-approve
skips the question; without a terminal it exits without changing the state (seen on two labs, once with exit code
1). Whether earlier clients (9.0.0 on other labs) prompt or accept --auto-approve is not recorded here: check
openstack server set --help.
When the failure came after the disk copy (in finish_resize), the source compute keeps the previous copy as
<instances-path>/<server-id>_resize. Leave it until the instance runs and its owner has checked it; it is the only
copy if the recovery goes wrong. When the copy itself failed, the libvirt driver renamed the directory back and
tried to remove the partial copy on the destination: the instance's disks are on the source. That clean-up is best
effort (_cleanup_remote_migration ignores its own errors): if the copy failed because SSH failed, a partial
<server-id> directory can remain on the destination; remove it once the instance runs on the source.
For cause 5: choose a flavor with a disk at least as large¶
Nothing to recover: the instance is back in its previous state. Resize again to a flavor whose root and ephemeral disks are at least the current sizes.
Verification¶
openstack server show <server-id> -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host -c flavor
# expected: ACTIVE (or SHUTOFF if it was), no task state, the expected flavor and host
openstack server migration list --server <server-id>
# expected: the last migration confirmed, reverted or error, none in progress.
# After a nova-compute restart during the copy (cause 2), the record stays "migrating": see below.
Then check the allocations: after a failed or abandoned resize, the migration may still hold resources on the source
compute (openstack resource provider allocation show <server-id> and ... show <migration-uuid>, the UUID
column of openstack --os-compute-api-version 2.59 server migration list, with the osc-placement plugin). For a
migration stuck in migrating, deleting the instance does not remove the migration's allocation. After a copy
interrupted by a nova-compute restart, a partial <server-id> directory can also remain on the destination (no
valid host with free capacity and stale
records show how to find them).
Validity and limits¶
- Written from Nova 2026.1 (
nova/compute/manager.py,nova/compute/api.py,nova/virt/libvirt/driver.py, the resize guides,resize_confirm_window), python-openstackclient 2026.1 (server resize,server migration,server set --state) and Kolla-Ansible 2026.1 (nova.conf.j2,nova_ssh). - Lab run (2026-10-26, Kolla-Ansible 22.2.0, 3 controllers + 2 computes, ML2/OVS, local qcow2 disks): causes 1, 2,
3 and 5 reproduced. Cause 1: resize to
VERIFY_RESIZEin 34 s, confirm 12 s, revert 19 s,server image createrefused with 409Cannot 'createImage' instance <server-id> while it is in vm_state resized. Cause 5: root shrink refused by the API (400), ephemeral shrink rolled back in 13 s. Cause 3:docker stop nova_sshon the destination, rollback in 16 s withssh: connect to host <dest> port 8022: Connection refused. Cause 2:docker stop nova_computeon the source during the copy left the instance inRESIZE/resize_migrating; systemd restarted the container 60 s later, the revert failed withAttributeError: 'NoneType' object has no attribute 'get'in_finish_revert_migration_vtpm, the task state was reset and the instance ended stopped on the source, migrationmigrating, allocations swapped. Cause 4 not reproduced. - With local disks, an instance in
ERRORafter a failed copy whosehostpoints to a compute without its disks is not covered: moving the disks by hand is outside this page. - Not covered: cross-cell resize, resize of instances with PCI devices, vGPUs or NUMA topology, and VMware or Ironic drivers.
Related¶
- Moving instances off a compute host for planned maintenance: cold migration as a maintenance tool, and its confirm.
- Stale records: resetting states and orphan allocations.
- New instance in ERROR with "Exceeded maximum number of retries"
- Nova 2026.1: resize an instance, resize configuration
Provenance: written by agent:ozarland-author, from the Nova, python-openstackclient and Kolla-Ansible 2026.1 documentation and source. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.