evetest: purge with the designated node down - #6478
Open
eriknordmark wants to merge 2 commits into
Open
Conversation
7 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6478 +/- ##
==========================================
+ Coverage 25.95% 26.25% +0.30%
==========================================
Files 513 523 +10
Lines 94167 95883 +1716
==========================================
+ Hits 24437 25174 +737
- Misses 67846 68640 +794
- Partials 1884 2069 +185 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A purge must not gate its teardown on the app's designated node, nor on wherever a replica happens to be scheduled: neither signal is both durable and liveness-aware, so either one deadlocks a purge issued while the designated node is down. Add a three-node cluster test for that case. The app is deployed with a preferred designated node, that node is powered off so KubeVirt reschedules the replica elsewhere, and the purge is issued while it is still down; the surviving workload must be exactly one VMIRS, named for the new generation. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A PVC can stay Pending forever. Its ProvisioningFailed events alternate "volume not found" and "volume already exists" - the provisioner created the Longhorn backend volume once, lost track of that success in its own cache, and keeps retrying a name it no longer recognizes. This hits TestVMAppPurgeDuringFailover at first-ever PVC creation, timing it out before it reaches the failover step it tests. Confirmed live with no EVE or pillar change involved: deleting the csi-provisioner pod forces a fresh leader election and cache, and the next retry succeeds. waitForAppRunningMitigatingPVCStall wraps the app-running wait with a background watcher that restarts csi-provisioner once, and only after a PVC has shown the failure signature continuously for two minutes. A PVC that clears on its own before then passes through untouched, and the wait's own timeout and failure behavior are unchanged. It is wired into the failover test alone, the one test observed to fail this way, because restarting a cluster-wide Longhorn pod is not something to do from every test that creates a volume. Marked REMOVE ME: it works around infra, not anything this suite tests. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark
force-pushed
the
purge-during-failover
branch
from
September 9, 2026 14:27
537df21 to
8bc3c48
Compare
eriknordmark
marked this pull request as ready for review
September 9, 2026 14:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A purge must not gate its teardown on the app's designated node, nor on wherever
a replica happens to be scheduled: neither signal is both durable and
liveness-aware, so either one deadlocks a purge issued while the designated node
is down. Nothing covered that case.
TestVMAppPurgeDuringFailoverdoes: a three-node replicated-storage cluster, anapp deployed with a preferred designated node, that node powered off so KubeVirt
reschedules the replica elsewhere, and the purge issued while it is still down.
The surviving workload must be exactly one VMIRS, named for the new generation.
It makes no volume or guest-level assertion — replicated-storage
VolumeStatussemantics across a node failover are not established for this suite.
The second commit is a
REMOVE MEworkaround for an infra bug, not foranything the suite tests. A first-ever PVC creation can stall forever in
Longhorn's CSI provisioner, whose
ProvisioningFailedevents alternate "volumenot found" and "volume already exists"; restarting the provisioner pod clears it
on the next retry. The mitigation wraps only this test's app-running wait, fires
only after the signature has held continuously for two minutes, and restarts the
pod once. Delete that file and its single call site once Longhorn no longer
needs the restart.
Both commits were drafted as part of #6318, which also carried a
gcPVCsPVCreclaim and re-enabled the Kubevirt
assertNoOrphanedPVCscheck. Both of thosenow live in #6406 — the pillar files are byte-identical there and its unit tests
are a superset — so #6318 is closed and this PR is the test-only remainder.
The test sits in
evetest/tests/appsrather thanevetest/tests/cluster, wherethe other multi-node tests live, because every helper it uses is defined in the
apps package:
vmShimApplication,assertExactlyOneVMIRSAtGeneration,listAppVMIRS,kubectlListItems,singleVIFWithSSHand the purge timeoutconstants. Relocating it would mean duplicating those into a second package;
promoting them to the framework first, then moving the test, is the better
sequence and is left as follow-up.
PR dependencies
None. The end-state assertion is on VMIRS objects, so this test does not need
the
gcPVCsreclaim in #6406.If #6406 lands first, the workaround commit needs a small rebase: it calls the
package-local
kubectlListItems, which #6406 promotes toEdgeDevice.KubectlListItemsalongside a newRunKubectl.How to test and validate this PR
Kubevirt only, and it needs three devices; it is registered last in
TestAppsSuitefor that reason. To run it alone:It skips on any other
HYPERVISOR.EVETEST_TPMandEVETEST_FILESYSTEM(
ext4|zfs, defaultext4) are the other parameters. Thecheckpoints to watch are
app-is-deployed(the app must land on its preferreddesignated node while that node is healthy),
failed-over(the replicarescheduled onto another node with the powered-off one excluded from the
lookup), and
purge-complete— a purge that gates on the dead node hangs hererather than failing an assertion.
To confirm the test can fail: the decisive check is "exactly one VMIRS, named
for the new generation", which a resurrected or stranded generation breaks.
Changelog notes
None — test-only.
PR Backports
Checklist