id: PG-34
category: compute
title: Steal time on pinned CPUs?
situation: >-
  An instance with dedicated CPUs shows steal time. A live migration makes it vanish, then it returns on the
  new host.

diagram:
  type: branches
  head: [On the compute you find, The cause is]
  split: 52
  rows:
    - see: Daemons or interrupts on the pinned CPUs
      means: Host not isolated
      sub: "`/sys/.../cpu/isolated` is empty"
      tone: stop
    - see: "`virsh emulatorpin` equals the vCPU set"
      means: Emulator threads
      sub: "No `hw:emulator_threads_policy` on the flavor"
      tone: warn
    - see: Two domains pinned to the same host CPU
      means: Overlapping pinning
      sub: "Compare `virsh vcpupin` across domains"
      tone: warn
    - see: "Domains pinned outside `cpu_dedicated_set`"
      means: Nova and host disagree
      sub: The set changed under running instances
      tone: warn

checks:
  - label: Where the vCPUs are pinned
    hint: virsh vcpupin <instance-name>
  - label: Where emulator threads run
    hint: virsh emulatorpin <instance-name>
  - label: What else runs on those CPUs
    hint: "ps -eLo pid,tid,psr,comm ; /proc/interrupts"
  - label: Nova's dedicated set
    hint: "grep -E 'cpu_(dedicated|shared)_set' nova.conf"

caution:
  - level: avoid
    text: >-
      Live-migrating again as the response. The cause travels with the configuration.
  - level: state-changing
    text: >-
      Host CPU isolation needs kernel parameters and a reboot. Do one host, measure, then roll out.

lesson: Live migration moves the symptom, not the cause. The cause is on the host.

runbook:
  path: migration/cpu-steal-on-pinned-instances.md
  status: lab-verified
  release: "2026.1"

scope: >-
  Derived from an operator's incident. On a lab (2026.1) the steal was reproduced and emulator threads moved;
  host isolation (it depends on the OS and hardware: check your vendor's guidance) and the overlap fix were not
  run. Per-CPU kernel threads (ksoftirqd, kworker...) exist on every CPU and are expected. With Kolla-Ansible,
  virsh runs in the nova_libvirt container. Live
  migration of pinned instances only became NUMA-aware in Train; before that a migration could itself create
  overlapping pinning. Changing a flavor's extra specs does not change running instances.

linkedin:
  title: "Steal time on pinned CPUs: live migration is not the fix"
  caption: |
    An instance was created with dedicated CPUs. It should never see steal time: each vCPU owns a host CPU. And yet top shows 7% steal, and the latency-sensitive workload reports jitter.

    Someone live-migrates it. Steal drops to zero. Problem solved.

    It comes back days or weeks later on the new host. So it gets migrated again. And again.

    Live migration moves the symptom, not the cause. The cause is on the host, and it travels with the configuration:
    → daemons or interrupts running on the pinned CPUs: the host is not isolated
    → emulator threads sharing the vCPU set
    → two instances pinned to the same host CPU
    → domains pinned outside the set Nova believes is dedicated

    All of them are visible from the compute, read-only, with virsh vcpupin, virsh emulatorpin and a look at what else runs on those CPUs.
  hashtags: [OpenStack, Nova, Performance, NFV, CloudOperations, OzarLand]
