Repository navigation
Description
Summary
test_multi_sync_snapshot_rsync.yml intermittently fails its second comparison with Unexpected comparison result (/mnt2/datafile: FAILED). The test populates data-dest-2 from a stale snapshot because its "wait for the 2nd sync" condition can already be satisfied before the 2nd sync runs.
Root cause
The test records first_snap after the first sync, applies trigger.manual: secondsync, and then waits for:
# test-e2e/test_multi_sync_snapshot_rsync.yml:184-196
until: >
res.resources | length > 0 and
res.resources[0].status.latestImage is defined and
res.resources[0].status.latestImage.name != first_snaplatestImage.name != first_snap is not a reliable indicator that the second sync completed: the ReplicationDestination can publish another latestImage between the first comparison and the second trigger. When that happens the wait returns immediately, second_snap is set to the pre-trigger image, and data-dest-2 is populated from data that predates the moredata write.
Evidence
Run (39881084243 / job 120179821974, namespace test-186800-0):
| Time | Event |
|---|---|
| 01:09:06 | RD publishes volsync-test-dst-20261023010906 (first_snap) |
| 01:09:27 | RD publishes volsync-test-dst-20261023010927 (before the 2nd trigger) |
| 01:09:50 | Test applies trigger.manual: secondsync to the ReplicationSource |
| 01:09:53 | VolumePopulator creates the prime PVC for data-dest-2 from ...010927 |
| ~01:09:53 | Wait returns immediately (latestImage != first_snap); second_snap = ...010927 |
| 01:10:07 | RD publishes volsync-test-dst-20261023011007 (the actual 2nd sync) |
| 01:10:08 | Comparison fails: /mnt2/datafile: FAILED (destination has data, source has moredata) |
The same failure occurred in run 39670362191 / job 119470494748.
Note that test_multi_sync_snapshot_rclone.yml does not have this problem because it waits on status.lastManualSync == "secondsync" for both the source and destination.
Additional observation
The RD published two latestImage snapshots (...010906 and ...010927, ~21s apart) while only one source sync had been triggered. I have not determined whether ...010906 was a premature image taken mid-transfer or ...010927 was a duplicate image for the same sync. That is worth investigating separately, but the test race described above exists regardless.
Proposed fix
Anchor the wait on state that can only change as a result of the secondsync trigger:
- Wait for the
ReplicationDestination'sstatus.lastSyncTime(or equivalent) to advance after applying the trigger, similar to the rclone test'slastManualSynccheck, and/or - Capture the current
latestImageimmediately before patching the trigger and wait for thelatestImageto differ from that baseline, and - Assert that the snapshot used to populate
data-dest-2was created after the trigger.
Acceptance criteria
- The "wait for 2nd sync" condition cannot be satisfied by state that existed before
trigger.manual: secondsyncwas applied. - The test verifies that
data-dest-2was populated from the post-trigger snapshot. - Repeated e2e runs of this playbook no longer fail with
Unexpected comparison result.
(AI generated)
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
test-e2e: fix stale snapshot race in rsync multi-sync test
test-e2e: fix stale snapshot race in rsync multi-sync test