Anatomy of a vSAN Resync Storm (and How to Calm One Down)

It started, as these things do, with a routine maintenance window. One host in a six-node vSAN cluster needed a reboot after a firmware update. Thirty minutes later, latency on the entire cluster had tripled, VMs were stuttering, and the resync dashboard showed 14 TB of data in flight.

This post walks through what actually happened, how to read the signals vSAN gives you, and how to keep a resync from becoming a storm in the first place.

What triggers a resync

vSAN resynchronizes components whenever the cluster believes data no longer meets the storage policy. The common triggers:

The one that bit us: the repair delay timer. By default, vSAN waits 60 minutes (ClomRepairDelay) after a host goes absent before rebuilding its components elsewhere. Our firmware update overran the window by eleven minutes. At minute 60, vSAN dutifully began rebuilding roughly 14 TB of components from the “failed” host onto the remaining five — right as the host came back online.

Reading the situation

Before touching anything, get a factual picture. The H5 client’s resync dashboard is fine, but esxcli and RVC give you more resolution:

# What is resyncing, and how much is left
esxcli vsan debug resync summary get

# Per-object detail — look for objects stuck at the same percentage
esxcli vsan debug resync list | head -40

# Is the cluster otherwise healthy?
esxcli vsan health cluster list

Three things to check before you intervene:

1. Is the resync making progress? Sample the bytes-left figure twice, ten minutes apart. A storm that’s progressing is usually best left alone.
2. Is it competing with production I/O? Check VM observed latency vs. resync throughput. vSAN 7 and later use Adaptive Resync to cap resync traffic under contention — verify it’s actually engaging.
3. Is anything failing repeatedly? A flapping NIC or a marginal disk can cause resync work to restart in a loop. That’s not a storm, that’s a hardware problem wearing a storm costume.

Calming it down

In our case the resync was legitimate but poorly timed. The options, roughly in order of preference:

Let Adaptive Resync do its job

From vSAN 6.7 U3 onward, resync traffic is dynamically throttled when VM latency crosses thresholds. If your version is current and latency is still unacceptable, verify the congestion thresholds haven’t been tuned into oblivion by a previous administrator:

# PowerCLI — check for non-default vSAN advanced settings across the cluster
Get-Cluster "PROD-VSAN" | Get-VMHost | ForEach-Object {
  Get-AdvancedSetting -Entity $_ -Name "VSAN.Dom*" |
    Where-Object { $_.Value -ne $_.DefaultValue } |
    Select-Object @{n='Host';e={$_.Entity}}, Name, Value, DefaultValue
}

Manually cap resync throughput (last resort)

On older builds you can throttle resync per host. Use sparingly — a slower resync means a longer window with reduced redundancy:

# Cap resync traffic to ~80 MB/s on this host (value is in Mbps)
esxcli vsan resync throttle set --mbps 640

Preventing the next one

The fixes that actually stuck for us:

• Extend the repair delay before planned maintenance. If a host operation might exceed 60 minutes, raise ClomRepairDelay for the duration — deliberately, and set it back after.
• Pre-check resync state before every maintenance window. Never put a host into maintenance mode while a previous resync is still draining. This compounds fast.
• Use “Ensure accessibility” for short maintenance, not “Full data migration”. Full migration for a 45-minute firmware update moves terabytes for no benefit.
• Automate the pre-flight. Our maintenance runbook now starts with a PowerCLI gate that refuses to proceed if resync bytes are non-zero or cluster health has any red checks.

# Maintenance pre-flight gate
$resync = Get-VsanResyncingComponent -Cluster "PROD-VSAN"
if ($resync.Count -gt 0) {
  throw "Resync in progress ($($resync.Count) components) — aborting maintenance."
}

The takeaway

vSAN resync behavior is deterministic — the storm wasn’t vSAN misbehaving, it was us handing it a rebuild order we didn’t mean to place. Understand the repair delay timer, respect in-flight resyncs, and put the pre-checks in code rather than in a wiki nobody reads at 2 a.m.