Skip to content

VolumePopulator can leave a PVC Pending forever when the PV rebind reconcile is missed #2178

Closed

Description

@JohnStrunk

Summary

The VolumePopulator controller can leave a PVC that uses a ReplicationDestination as its dataSourceRef stuck in Pending indefinitely. It creates the temporary vs-prime-<uid> PVC from the latest snapshot, but if the reconcile that performs the PV-claim rebind is missed, nothing ever retries. The PVC never binds, so pods that use it cannot be scheduled and callers hang until their timeout.

This is currently the largest single source of e2e flakes. It accounts for at least 7 of the 10 e2e job failures between 2026-10-08 and 2026-10-23, and shows up in unrelated tests (test_simple_rsync, test_rsync_tls_priv, test_simple_rsync_longname, test_volumepopulator, test_multi_sync_snapshot_rsync, test_multi_sync_snapshot_rclone, test_rclone_with_customca_configmap).

Observed behaviour

The controller logs the creation of the prime PVC and then nothing more for that PVC, even though the prime subsequently binds:

21:44:03.420  INFO  controller.VolumePopulator  Creating temp populator pvc from snapshot
              {"pvc": {"name":"data-dest-2","namespace":"test-846112-0"},
               "volpop pvc name":"vs-prime-97fb4a28-..."}
              <no further reconcile for data-dest-2>

At the end of the test run the object state is:

  • vs-prime-97fb4a28-... — Bound
  • data-dest-2 — Pending (spec.volumeName empty)
  • compare-pvcs Job — FailedScheduling: 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims

Examples (all main branch, same signature):

Root cause analysis

In internal/controller/volumepopulator_controller.go, reconcilePVC() creates the prime PVC and then calls rebindPVClaim(). Every "not ready yet" path returns an empty result with no requeue:

  • :417-420 — prime PVC has no volume yet
  • :426-434 — PV not found yet
  • :257-263 — "Waiting for pv rebind" (bind controller has not updated the claim yet)
  • :469-471 — after patching the PV claimRef

The controller therefore relies entirely on a later event (normally the prime PVC update) to run again. If that event is missed/dropped — or if a reconcile runs while the prime is still unbound and the later "bound" update is missed — the PVC is stuck forever. The ReplicationDestination watch at :484 normally masks this while the RD is still updating, but an idle RD produces no events.

There is also a related variable-shadowing bug at :240:

pvcPrime, err := GetVolumePopulatorPVCPrime(...)   // outer
...
if !isPVCBoundToVolume(pvc) {
    pvcPrime, primeResult := r.reconcilePVCPrime(...)  // <- shadows outer pvcPrime
    ...
}
if pvcPrime != nil { ... }                         // checks the outer (still nil) value

On the reconcile that creates the prime, the outer pvcPrime is nil, so the wait-for-rebind block (:258-263) is skipped. The code falls through to the "population done" path, emits a spurious Populator finished event, and calls cleanup(..., nil) (:266-272), which prematurely removes the snapshot's "in use" label.

Proposed fix

  1. Fix the shadowing at :240 so the code uses the prime PVC returned by reconcilePVCPrime().
  2. Return a bounded ctrl.Result{RequeueAfter: ...} (e.g. 5-10s) from every path that waits on external state, so a missed watch event self-heals. This is the key fix.
  3. After patching the PV claimRef, requeue as well rather than relying on the bind controller's update event.

Improve diagnosability (include with this fix)

Today the failure surfaces as a 15-minute compare-pvcs Job timeout with a generic message, and the populator log is buried in a large dump. Update test-e2e/roles/compare_pvc_data/tasks/main.yml to verify that both PVCs are Bound before creating the compare Job, and include the PVC names/phases in the failure message. The test would then fail in seconds and point directly at the unbound PVC. This also makes non-populator unbound-PVC failures obvious.

Acceptance criteria

  • A missed reconcile no longer leaves the target PVC Pending; with no further object changes, the controller re-reconciles and completes the rebind within a bounded time.
  • Test coverage: a unit/envtest case where the prime PVC is created out-of-band (simulating the create reconcile) and a subsequent reconcile completes the rebind; assert the requeue behaviour while waiting.
  • The pvcPrime shadowing is fixed and no spurious Populator finished event/cleanup occurs on the creation reconcile.
  • compare_pvc_data fails fast with a clear message when either PVC is not Bound.
  • Affected e2e tests no longer hit the 15-minute compare timeout.

(AI generated)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions