CPU steal time on instances with dedicated (pinned) CPUs. Live migration moves the symptom, not the cause¶
Instances created with hw:cpu_policy=dedicated should not see steal time: each guest vCPU owns a host CPU. When
they do, live-migrating them makes the symptom disappear for a while, which turns live migration into the
response. The symptom comes back, the same workloads get migrated again and again, each migration carrying its own
risk. The cause is on the host, and it travels with the configuration.
Symptom¶
Inside the guest, top or /proc/stat shows non-zero steal (st) on a pinned instance:
%Cpu(s): 71.2 us, 3.1 sy, 0.0 ni, 18.0 id, 0.0 wa, 0.0 hi, 0.4 si, 7.3 st
Latency-sensitive workloads (telco, trading, real-time) report jitter. The instance's flavor has
hw:cpu_policy=dedicated. After a live migration, steal drops to zero, then returns days or weeks later on the new
host.
Impact¶
Performance degradation of exactly the workloads that paid for isolation. Repeated live migrations add risk: each one pauses the instance briefly and, on older releases, could itself produce an invalid pinning on the destination (see Validity).
Quick checks¶
On the compute host running the instance:
# 1. Which host CPUs the guest vCPUs are pinned to, and where the emulator threads run
virsh vcpupin <instance-name> # instance-name is OS-EXT-SRV-ATTR:instance_name, e.g. instance-0000abcd
virsh emulatorpin <instance-name>
virsh dumpxml <instance-name> | sed -n '/<cputune>/,/<\/cputune>/p'
# Kolla-Ansible: virsh runs in the libvirt container, e.g. sudo docker exec nova_libvirt virsh vcpupin <instance-name>
# 2. What else runs on those CPUs
ps -eLo pid,tid,psr,comm | awk '$3==<host-cpu>' # for each pinned host CPU
cat /proc/interrupts | awk '{print $1, $<host-cpu+2>}' | sort -k2 -n | tail
# 3. Nova's view of the dedicated set
grep -E 'cpu_dedicated_set|cpu_shared_set|vcpu_pin_set' /etc/nova/nova.conf # Kolla-Ansible: /etc/kolla/nova-compute/nova.conf
Diagnosis¶
Nova pins the guest's vCPUs, nothing else. Two things routinely share "dedicated" CPUs:
- Host processes. Nova does not keep the kernel's own threads, interrupt handlers, KSM, or daemons off the
CPUs in
cpu_dedicated_set. That isolation must be arranged on the host:isolcpus,nohz_full,rcu_nocbskernel parameters, a tuned profile (cpu-partitioning,realtime-virtual-host), IRQ affinity. Checkcat /sys/devices/system/cpu/isolatedand/proc/cmdline. - The instance's own emulator threads. By default QEMU's emulator (I/O) thread may run on the instance's
pinned CPUs.
hw:emulator_threads_policy=sharemoves it tocpu_shared_set;isolategives it an extra dedicated CPU. Without either, the emulator thread steals from the vCPUs it serves.
Only after those two: overlapping pinning between instances (two domains pinned to the same host CPU, visible by
comparing virsh vcpupin across all domains), and a mismatch between Nova's belief and the host (for example
cpu_dedicated_set changed on a host that already runs pinned instances, or a migration performed by a release
that did not recalculate the mapping).
Why migration "fixes" it: the destination host may happen to have less background activity on the chosen CPUs, or a different interrupt distribution. The configuration gap is identical, so the symptom returns.
Possible causes¶
| Cause | How to tell |
|---|---|
| Host threads and IRQs not isolated from the dedicated set | /sys/ empty; ps -eLo psr shows kernel threads or daemons on pinned CPUs; interrupts counted on them |
| Emulator threads sharing the pinned CPUs | virsh emulatorpin equals the vCPU set; flavor lacks hw:emulator_threads_policy |
| Overlapping pinning between instances | Two domains' vcpupin share a host CPU |
| Nova/host mismatch | cpu_dedicated_set differs from what the domains use; history of migrations on pre-Train releases |
Resolution¶
Stop migrating first. The book's team did exactly that: accepted a bounded amount of steal while investigating, rather than relocating the symptom.
Isolate the host CPUs¶
(Not run on the lab.)
⚠️ State-changing. Kernel parameters and tuned profiles need a host reboot, hence moving or stopping its instances (see Moving instances off a host). Do one host, measure, then roll out.
Configure the isolation the hardware vendor and your OS document for virtualisation hosts (isolcpus=, nohz_full=,
rcu_nocbs= matching cpu_dedicated_set, or a tuned profile), pin IRQs away from the set, and keep housekeeping
CPUs for the host in cpu_shared_set.
Move the emulator threads¶
⚠️ State-changing. An instance keeps a copy of its flavor from when it was created or last resized, so changing a flavor's extra specs does not change running instances, and a rebuild does not either. Create a new flavor with the policy and resize the existing instances to it (a resize reboots the instance). Changing the extra specs of a flavor in use only affects instances created from it afterwards.
openstack flavor create <new-flavor> --vcpus <n> --ram <mb> --disk <gb> \
--property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=share
openstack server resize --flavor <new-flavor> <server> # then confirm the resize once the instance is checked
# and make sure every host has a [compute] cpu_shared_set for them to land on
Fix overlapping pinning or mismatches¶
(Not run on the lab.)
⚠️ State-changing. Moving an instance changes its pinning. Live-migrate or cold-migrate one instance per overlap to a host with free dedicated CPUs, then verify with
virsh vcpupinon the destination.
Verification¶
# Inside the guest, over a representative period
vmstat 5 12 # st column stays at 0
# On the host
virsh vcpupin <instance-name>; virsh emulatorpin <instance-name> # Kolla-Ansible: sudo docker exec nova_libvirt virsh ...
ps -eLo psr,comm | awk '$1==<pinned-cpu>' | sort | uniq -c
# expected: the instance's vCPU threads (CPU n/KVM) plus the per-CPU kernel threads that always exist on every
# CPU (ksoftirqd/N, migration/N, cpuhp/N, kworker/N); no other process or daemon
Verify over days, not minutes: the symptom was intermittent.
Validity and limits¶
Derived from an operator's incident on a platform with dedicated-CPU workloads; release not recorded. Two release
facts matter: live migration of instances with a NUMA topology (including pinned CPUs) only became NUMA-aware in
Train (Nova 20.0); on Stein with the enable_numa_live_migration workaround, and earlier, the instance moved
without recalculating the guest-to-host mapping and could itself create overlapping pinning. hw:emulator_threads_policy=share
with cpu_shared_set dates from Rocky (Nova 18.0); check the release notes of your release. The host-isolation steps depend on the OS and hardware; check your vendor's
guidance for real-time or NFV hosts.
The lab confirmation (OpenStack 2026.1, Kolla-Ansible, 1 controller and 2 computes with nested KVM, 2026-10-23)
ran the quick checks, reproduced the steal (0.8 % idle, 26.7 % with host processes on the pinned CPU) and moved
the emulator threads with hw:emulator_threads_policy=share. Host isolation and moving instances were not run.
Related¶
- Moving instances off a host for planned maintenance.
- Source: OpenStack, the Day After Tomorrow (S. Zaouam), chapter 15, "From the field" and technical note on pinned CPUs.
Provenance: written by agent:ozarland-author, from the book OpenStack, the Day After Tomorrow. Reviewed in its pull request (see the page history). Checked by a human operator: not yet.
Worked for you, or not? Tell us in one line: your release, your deployment method, and what happened.