Skip to content

e2e: assert sync status, conditions, and resource cleanup in existing playbooks #2188

Closed

Description

@JohnStrunk

Summary

The e2e playbooks are strong at proving data equality and weak at proving what the controllers report and what they clean up. Regressions in status/conditions or temporary-resource leaks pass every playbook.

Current assertion coverage across the 38 test-e2e/test_*.yml playbooks:

Assertion Playbooks
status.latestMoverStatus 22/38
status.lastSyncTime 11/38
status.lastSyncDuration 10/38
status.conditions 1/38 (test_restic_without_trigger.yml)
resource cleanup (state: absent checks) 3/38

So Synchronizing=False with the expected reason, condition message sanity, and "temporary PVC/Snapshot/Job/Service is gone after cleanup" are effectively untested end to end. These are exactly the signals users and downstream operators (ACM/ODF) monitor, and the leaks that show up as support cases.

Proposal

Add one reusable role (e.g. roles/verify_sync_status) and call it from playbooks at points where they already wait for data, so the assertions cost nothing beyond the in-cluster kubectl get they replace:

  1. Source/destination status: status.lastSyncTime advanced past the sync start; status.lastSyncDuration set and plausible; status.nextSyncTime set when a schedule is configured.
  2. Conditions: Synchronizing is False with the expected reason, and no stale True condition is left behind after completion.
  3. Mover status: latestMoverStatus.result == "Successful" (and on a negative-path playbook, the error text/condition reason).
  4. Cleanup: after completion, no mover Job/Pod remains, and temporary PVCs/VolumeSnapshots created for the copy method are gone — except the protected/latest image, which must still exist.
  5. Idempotence spot check: a second reconcile/no-op cycle does not create duplicate or renamed resources.

Keep the existing data-comparison roles as-is; this is additive and runs against the same cluster state.

Acceptance criteria

  • A single shared role implements the status/condition/cleanup assertions; playbooks do not grow bespoke until: blocks.
  • All playbooks assert at least the completion condition (Synchronizing=False) and cleanup of temporary resources.
  • At least one playbook (can be a new negative-path case, see e2e: cover paused, Clone copy method, CLI migration, and a negative path #2190) asserts Synchronizing=False with an error reason.
  • No measurable wall-clock increase: assertions run in-process against objects the playbooks already wait on; no new cluster, job, or image.

(AI generated)

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions