The ESXi CPU Scheduler, Part 1: Ready Time Is the Only Metric That Matters
An advanced series on how ESXi actually schedules CPU. Part 1: why %RDY beats every other counter, what co-scheduling really costs wide VMs, and how to read esxtop like the scheduler sees it.
This series assumes you run production vSphere and are comfortable in esxtop. We’re going under the DRS dashboards to how the scheduler actually makes decisions — because every “the VM is slow but CPU usage is low” mystery ends up here eventually.
The core misunderstanding
CPU usage tells you how much work a VM did. It tells you nothing about how much work the VM wanted to do but couldn’t. That gap — vCPUs runnable but not running because no physical core was available — is ready time, and it is invisible in every guest-level tool. Windows Task Manager inside the VM will happily show 40% CPU while the VM spends a third of its life waiting for a core.
The number to know: %RDY in esxtop is per-vCPU accumulation over the sample window.
As a working rule, sustained %RDY above ~5% per vCPU is user-noticeable latency,
and above 10% is a genuine problem.
ID NAME %USED %RUN %RDY %CSTP
1201 sql-prd-02 388.1 390.4 41.2 9.8 ← 8 vCPU: ~5%/vCPU RDY + CSTP
1188 app-prd-11 61.0 62.3 2.1 0.0 ← healthy
1240 etl-batch-04 712.6 718.0 18.7 0.2 ← busy but scheduling fineCo-scheduling: the tax on wide VMs
The scheduler must maintain the illusion that a VM’s vCPUs advance roughly together —
a guest OS scheduled on drifting clocks makes catastrophically bad decisions. ESXi uses
relaxed co-scheduling: it tracks per-vCPU skew and only intervenes when skew exceeds
a threshold, stopping the leading vCPUs (%CSTP — co-stop) until the laggards catch up.
The practical consequence: an idle vCPU is not free. Every vCPU you grant a VM is another runner the scheduler must keep loosely synchronized. The 8-vCPU SQL VM above with 9.8% co-stop is paying that tax — its four busy vCPUs keep getting halted so the four idle ones don’t fall behind on bookkeeping interrupts.
Reading the host, not the VM
Ready time is a symptom on the VM; the cause lives at the host level. The triage sequence I use:
- Host-wide pressure? Check the PCPU line in esxtop — sustained UTIL above ~80% means simple contention; the fix is DRS doing its job or capacity.
- NUMA imbalance? Press
mfor the memory/NUMA view. A VM whoseNRMEM(remote memory) is high is scheduled across NUMA nodes — part 2 of this series is entirely about this. - Contention with no host pressure? Look for CPU limits (the silent killer — check
%MLMTD), resource pool starvation, or a latency-sensitivity setting monopolizing cores.
# PowerCLI: find every VM in the cluster with a CPU limit — there is never a good reason
Get-Cluster PROD-A | Get-VM |
Get-VMResourceConfiguration |
Where-Object { $_.CpuLimitMhz -ne -1 } |
Select-Object VM, CpuLimitMhzHomework before part 2
Pull a week of Ready (summation) for your ten widest VMs from vCenter performance
charts, convert to percentage (ready-ms ÷ (interval-ms × vCPUs)), and find your worst
offender. In part 2 — NUMA scheduling and why your monster VM should fit inside one
node — you’ll likely discover the same VM at the top of that list too. That is not a
coincidence, and the why is where this series is headed.