Skip to content

e2e: test_multi_sync_snapshot_rsync can populate data-dest-2 from a stale latestImage #2179

Closed

Description

@JohnStrunk

Summary

test_multi_sync_snapshot_rsync.yml intermittently fails its second comparison with Unexpected comparison result (/mnt2/datafile: FAILED). The test populates data-dest-2 from a stale snapshot because its "wait for the 2nd sync" condition can already be satisfied before the 2nd sync runs.

Root cause

The test records first_snap after the first sync, applies trigger.manual: secondsync, and then waits for:

# test-e2e/test_multi_sync_snapshot_rsync.yml:184-196
until: >
  res.resources | length > 0 and
  res.resources[0].status.latestImage is defined and
  res.resources[0].status.latestImage.name != first_snap

latestImage.name != first_snap is not a reliable indicator that the second sync completed: the ReplicationDestination can publish another latestImage between the first comparison and the second trigger. When that happens the wait returns immediately, second_snap is set to the pre-trigger image, and data-dest-2 is populated from data that predates the moredata write.

Evidence

Run (39881084243 / job 120179821974, namespace test-186800-0):

Time Event
01:09:06 RD publishes volsync-test-dst-20261023010906 (first_snap)
01:09:27 RD publishes volsync-test-dst-20261023010927 (before the 2nd trigger)
01:09:50 Test applies trigger.manual: secondsync to the ReplicationSource
01:09:53 VolumePopulator creates the prime PVC for data-dest-2 from ...010927
~01:09:53 Wait returns immediately (latestImage != first_snap); second_snap = ...010927
01:10:07 RD publishes volsync-test-dst-20261023011007 (the actual 2nd sync)
01:10:08 Comparison fails: /mnt2/datafile: FAILED (destination has data, source has moredata)

The same failure occurred in run 39670362191 / job 119470494748.

Note that test_multi_sync_snapshot_rclone.yml does not have this problem because it waits on status.lastManualSync == "secondsync" for both the source and destination.

Additional observation

The RD published two latestImage snapshots (...010906 and ...010927, ~21s apart) while only one source sync had been triggered. I have not determined whether ...010906 was a premature image taken mid-transfer or ...010927 was a duplicate image for the same sync. That is worth investigating separately, but the test race described above exists regardless.

Proposed fix

Anchor the wait on state that can only change as a result of the secondsync trigger:

  • Wait for the ReplicationDestination's status.lastSyncTime (or equivalent) to advance after applying the trigger, similar to the rclone test's lastManualSync check, and/or
  • Capture the current latestImage immediately before patching the trigger and wait for the latestImage to differ from that baseline, and
  • Assert that the snapshot used to populate data-dest-2 was created after the trigger.

Acceptance criteria

  • The "wait for 2nd sync" condition cannot be satisfied by state that existed before trigger.manual: secondsync was applied.
  • The test verifies that data-dest-2 was populated from the post-trigger snapshot.
  • Repeated e2e runs of this playbook no longer fail with Unexpected comparison result.

(AI generated)

Activity

  1. self-assigned this
    on Oct 23, 2026
  2. added 2 commits that reference this issue on Oct 23, 2026
    19c14b4
    a45e7cc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions