[DO NOT MERGE] eve-k: reclaim the purged generation's PVC

[eriknordmark]

Description

Draft, for comparison — not competing for merge. @andrewd-zededa is preparing
the production version of this work, which handles multi-node clusters and makes
the lifecycle consistent across the designated node and the rest of the cluster.
This branch is the single-node reference implementation that has been running on
a local integration branch, opened so the two can be read side by side. Prefer
his where they disagree.

The leak

On EVE-k a purge deletes the stale generation's VMIRS and its pod, but not its
PVC. The Longhorn volume underneath is never reclaimed, so the space stays
consumed for the life of the device. #6257 fixed the duplicate-generation bug
and merged; sweepStaleGenerations there deletes the workload objects and stops
short of the disk.

An ordinary purge does not leak — it cleans up its own PVC. The orphan appears
when a purge is interrupted such that sweepStaleGenerations is what reaps the
stale generation, so accumulation tracks interruption frequency rather than purge
frequency.

The fix

volumemgr's periodic garbage-collection pass already collects file- and
dataset-backed volumes with no currently-published VolumeStatus. Extend it to
PVC-backed ones on the same rule, running only on EVE-k. A volume this node
still tracks is never collected, including one it tracks only as a failover
candidate for an app running elsewhere in the cluster.

Commits

1. volumemgr: reclaim orphaned PVCs on EVE-k — the fix, plus unit tests for
   the reap decision (which is separable from the Kubernetes delete call, so it
   is testable without one).
2. evetest: assert the purged PVC is reclaimed — TestVMAppPurgeAfterPowerCycle
   checks the old generation's disk is gone on the local-hypervisor path but
   skips the equivalent on Kubevirt, so the leak leaves it green. Enables the
   check on both paths.
3. evetest: purge with the designated node down — a three-node cluster
   test: app deployed with a preferred designated node, that node powered off so
   KubeVirt reschedules the replica, purge issued while it is still down. This is
   the case where gating the teardown on the designated node — or on wherever a
   replica currently happens to be scheduled — deadlocks. It is also the area
   Andrew's version addresses properly; treat it as a statement of the problem
   rather than as the answer.
4. evetest: work around a Longhorn CSI-provisioner stall — marked REMOVE ME
   in its own file. Works around infra, not anything this suite tests. Drop this
   commit if you would rather the test flake than carry it.

PR dependencies

None. #6257 has merged; this applies on top of it.

How to test and validate this PR

Unit — make -C pkg/pillar test. The reap decision is covered directly:
a published volume is never reaped, an unknown one is, a replicated one is not,
and one unparseable candidate is skipped rather than stopping the pass.

Negative control run locally: neutering the "still published" guard makes
TestVolumesToReapSkipsLiveVolume fail on its assertion, and it passes again
once restored — so the test can actually fail.

e2e — EVETEST_HYPERVISOR=kubevirt make evetest NAME=TestVMAppPurgeAfterPowerCycle.
Without the fix this now fails on the assertion enabled by commit 2:

PVC "<uuid>-pvc-0" exists in the cluster but no currently published VolumeStatus
would produce it - orphaned disk left behind by a stale generation

-pvc-0 is generation 0 — the old generation's disk, the object
sweepStaleGenerations leaves behind. With the fix the same scenario passes.

That A/B was run locally, one image per leg differing only in the three
cmd/volumemgr files, both legs driven from the tree where the assertion is
live: power-cycle × no-reclaim failed, power-cycle × reclaim passed, and the
plain-purge baseline passed on both — which is what establishes that a healthy
purge does not leak and that the assertion is able to fire.

TestVMAppPurgeDuringFailover needs a three-node cluster and has not been run
against this branch.

Not verified here: the leak rate under repeated interrupted purges, and
whether accumulation causes Longhorn storage pressure or downstream volume
failures.

Changelog notes

Fixes a bug on eve-k (kubevirt hypervisor) where a purge interrupted by a reboot
could leave the old application generation's disk behind indefinitely, consuming
storage for the life of the device. Adds automated tests for purge storage
reclaim.

PR Backports

• 17.0-stable: To be backported.
• 16.0-stable: No, as the feature is not available there.
• 14.5-stable: No, as the feature is not available there.
• 13.4-stable: No, as the feature is not available there.

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR

And the last but not least:

• I've checked the boxes above, or I've provided a good reason why I didn't
  check them.

Notes on the unchecked boxes: pkg/pillar/docs/volumemgr.md is untouched — say
so if the GC pass gaining a third backend warrants a line there. The amd64 e2e
evidence is the A/B described above rather than a run of this exact branch;
arm64 has not been exercised. Labels are left for whoever picks this up, given
it is a draft.

[codecov]

Codecov Report

❌ Patch coverage is 53.57143% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 25.04%. Comparing base (df6bdd0) to head (783418b).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/pillar/cmd/volumemgr/initialvolumestatus.go	57.69%	9 Missing and 2 partials ⚠️
pkg/pillar/cmd/volumemgr/volumemgr.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6318      +/-   ##
==========================================
+ Coverage   24.67%   25.04%   +0.37%     
==========================================
  Files         514      524      +10     
  Lines       94151    95880    +1729     
==========================================
+ Hits        23231    24014     +783     
- Misses      69057    69814     +757     
- Partials     1863     2052     +189     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eriknordmark]

Closing: every part of this PR now lives elsewhere.

The gcPVCs reclaim is carried by #6406 — cmd/volumemgr/initialvolumestatus.go
and volumemgr.go are byte-identical there, and its unit tests are a superset
(the seven here plus TestVolumesToReapSkipsLiveVolumeWithLocalGeneration and
TestVolumesToReapIncludesSupersededGeneration). #6406 also re-enables the
Kubevirt assertNoOrphanedPVCs check in TestVMAppPurgeAfterPowerCycle, and
does it better than the commit here did: it drops timer.gc.vdisk to its 60s
floor so the assertion does not have to wait out the default hourly GC pass.

The two evetest commits with no counterpart there — TestVMAppPurgeDuringFailover
and the REMOVE ME Longhorn CSI-provisioner workaround it needs — are now
#6478, rebased onto current master as a test-only PR.