Repository navigation
Description
Summary
The VolumePopulator controller can leave a PVC that uses a ReplicationDestination as its dataSourceRef stuck in Pending indefinitely. It creates the temporary vs-prime-<uid> PVC from the latest snapshot, but if the reconcile that performs the PV-claim rebind is missed, nothing ever retries. The PVC never binds, so pods that use it cannot be scheduled and callers hang until their timeout.
This is currently the largest single source of e2e flakes. It accounts for at least 7 of the 10 e2e job failures between 2026-10-08 and 2026-10-23, and shows up in unrelated tests (test_simple_rsync, test_rsync_tls_priv, test_simple_rsync_longname, test_volumepopulator, test_multi_sync_snapshot_rsync, test_multi_sync_snapshot_rclone, test_rclone_with_customca_configmap).
Observed behaviour
The controller logs the creation of the prime PVC and then nothing more for that PVC, even though the prime subsequently binds:
21:44:03.420 INFO controller.VolumePopulator Creating temp populator pvc from snapshot
{"pvc": {"name":"data-dest-2","namespace":"test-846112-0"},
"volpop pvc name":"vs-prime-97fb4a28-..."}
<no further reconcile for data-dest-2>
At the end of the test run the object state is:
vs-prime-97fb4a28-...—Bounddata-dest-2—Pending(spec.volumeNameempty)compare-pvcsJob —FailedScheduling:0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims
Examples (all main branch, same signature):
- run 39855046386 / job 120093523831 —
test_rclone_with_customca_configmap - run 38417053443 / job 115366606936 —
test_volumepopulator(data-dest3) - run 37961417997 / job 113927302320 —
test_multi_sync_snapshot_rsync - run 38963538142 / job 117186716768 —
test_multi_sync_snapshot_rsync - run 38767759369 / job 116537770447 —
test_rsync_tls_priv - run 38767759369 / job 116537770655 —
test_simple_rsync_longname - run 37808644530 / job 113421502416 —
test_simple_rsync
Root cause analysis
In internal/controller/volumepopulator_controller.go, reconcilePVC() creates the prime PVC and then calls rebindPVClaim(). Every "not ready yet" path returns an empty result with no requeue:
:417-420— prime PVC has no volume yet:426-434— PV not found yet:257-263— "Waiting for pv rebind" (bind controller has not updated the claim yet):469-471— after patching the PVclaimRef
The controller therefore relies entirely on a later event (normally the prime PVC update) to run again. If that event is missed/dropped — or if a reconcile runs while the prime is still unbound and the later "bound" update is missed — the PVC is stuck forever. The ReplicationDestination watch at :484 normally masks this while the RD is still updating, but an idle RD produces no events.
There is also a related variable-shadowing bug at :240:
pvcPrime, err := GetVolumePopulatorPVCPrime(...) // outer
...
if !isPVCBoundToVolume(pvc) {
pvcPrime, primeResult := r.reconcilePVCPrime(...) // <- shadows outer pvcPrime
...
}
if pvcPrime != nil { ... } // checks the outer (still nil) valueOn the reconcile that creates the prime, the outer pvcPrime is nil, so the wait-for-rebind block (:258-263) is skipped. The code falls through to the "population done" path, emits a spurious Populator finished event, and calls cleanup(..., nil) (:266-272), which prematurely removes the snapshot's "in use" label.
Proposed fix
- Fix the shadowing at
:240so the code uses the prime PVC returned byreconcilePVCPrime(). - Return a bounded
ctrl.Result{RequeueAfter: ...}(e.g. 5-10s) from every path that waits on external state, so a missed watch event self-heals. This is the key fix. - After patching the PV
claimRef, requeue as well rather than relying on the bind controller's update event.
Improve diagnosability (include with this fix)
Today the failure surfaces as a 15-minute compare-pvcs Job timeout with a generic message, and the populator log is buried in a large dump. Update test-e2e/roles/compare_pvc_data/tasks/main.yml to verify that both PVCs are Bound before creating the compare Job, and include the PVC names/phases in the failure message. The test would then fail in seconds and point directly at the unbound PVC. This also makes non-populator unbound-PVC failures obvious.
Acceptance criteria
- A missed reconcile no longer leaves the target PVC
Pending; with no further object changes, the controller re-reconciles and completes the rebind within a bounded time. - Test coverage: a unit/envtest case where the prime PVC is created out-of-band (simulating the create reconcile) and a subsequent reconcile completes the rebind; assert the requeue behaviour while waiting.
- The
pvcPrimeshadowing is fixed and no spuriousPopulator finishedevent/cleanup occurs on the creation reconcile. -
compare_pvc_datafails fast with a clear message when either PVC is notBound. - Affected e2e tests no longer hit the 15-minute compare timeout.
(AI generated)
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity