evetest: add TestDNIDandBackupDNID for DNID backup takeover - #6505
Draft
andrewd-zededa wants to merge 3 commits into
Draft
evetest: add TestDNIDandBackupDNID for DNID backup takeover#6505andrewd-zededa wants to merge 3 commits into
andrewd-zededa wants to merge 3 commits into
Conversation
evetest pinned pkg/pillar at a revision from April, which predates types.KubernetesVmiDescheduleEvents and types.VmiDescheduleEventBoot, so cluster tests that need event-driven descheduling had no way to name that config item other than by its literal string. Scoped to evetest's own go.mod. make bump-eve-pillar is not usable here: it walks every go.mod referencing pkg/pillar -- ten subprojects, pinned at three different revisions -- and runs go mod vendor in each, which would re-vendor eight unrelated subprojects and create vendor trees in evetest and evetest/sdn/vm, neither of which vendors today. Carries transitive upgrades to grpc, otel, k8s.io/api, apimachinery and genproto. go vet passes across the whole evetest tree. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
andrewd-zededa
added
stable
Isolates an app's designated node with a hard power-off, then exercises every app operation backup DNID is meant to cover -- activate, deactivate, purge, and delete -- while that node stays down. vm2 covers activate/deactivate/purge: deactivated before the outage, activated on a peer once its own node is down, deactivated again while still down (activate and deactivate share the same gate, so this checks that symmetry rather than assuming it), reactivated, purged there, then the node comes back and vm2 fails back to it. vm3 covers delete: Required affinity, so it can never fail over and its only exposure to this rework is deletion completing at all. It is deleted outright while its node is still down, and both its app info and its volume info are confirmed gone -- the volume check matters because DestroyVolume silently skips the PVC delete on any node that isn't the app's designated node, leaking the volume for as long as that node stays unreachable. Failback needs the descheduler, and event-driven descheduling is off by default, so kubernetes.vmi.deschedule.events=boot goes into the initial config -- the on-boot trigger is armed once per boot right after zedkube's WaitForKubernetes, so enabling it later would not fire for the boot that matters. Both surviving nodes are checked to have accepted it up front, so a rejected property fails there instead of surfacing as an unexplained failback timeout. The outage threshold override is read back from the device's reported config item status rather than assumed, and the waits that depend on it are sized off whichever value is really in effect: one minute if the override landed, the ten-minute default if it did not. Added to TestNodeClusterSuite, at the end. No pillar changes in this commit. Uses real VM apps (a pinned Alpine Linux cloud-init qcow2), the same fetch-a-real-disk-image pattern tests/apps/vnc_test.go already proves out under Kubevirt, since EVE-K's production apps are predominantly VMs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
Info() reported HALTED as soon as the VMIRS Get returned NotFound. The three objects form an ownership chain: the VMIRS owns the VMI, and the VMI owns the virt-launcher pod. EVE deletes only the VMIRS, so it is the first to go, and Kubernetes then collects its dependents in the background. The pod holds the RWO disk until it goes away. A domain was therefore reported as halted while its workload still ran. A peer that stands in for a downed node reads that state to decide if an app is really down, and a controller reads it to decide the app can move. Both can act too early. Info() now takes the state from the existence of the VMIRS, the VMI and the pod, each found by its App-Domain-Name label. An absent VMIRS with a surviving dependent gives HALTING and keeps a non-zero DomainId, because a zero DomainId is the confirmed-absent token. HALTED with a zero DomainId needs all three to be absent. The dependent lookup runs only on the absent-VMIRS branch, so a steady-state poll makes no more API calls than it did before. A config apply must not wait for this. A kubevirt deactivate now publishes HALTING, deletes the VMIRS, and returns. verifyStatus settles the state and releases the resources when Info reports all three objects absent. Kubernetes owns how long that takes, and on a node that is off it has no end, so no budget on the config path is the correct one. The paths that must finish before they return keep their behavior, because impatient is already true for them: handleDelete unpublishes DomainStatus as soon as it returns, and a purge or a restart advances its counter only after the teardown clears status.Activated. Two other reads of HALTING had to be separated from this one. waitForDomainGone took it for a guest that powered off while its hypervisor process stayed parked and held resources, a qemu -no-shutdown condition, so it stopped waiting and let the caller reap the domain. verifyStatus took it for a teardown to finish now. Under kubevirt HALTING is the reverse of both: nothing is parked, and Kubernetes is still at work. Both short circuits now apply to the other hypervisors only. Start() already waits for a stale generation to disappear, object and pods, before it creates the next one, because generations share the disk and the veth names. That is what keeps an asynchronous deactivate safe beside a create. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
andrewd-zededa
force-pushed
the
eve-k-backup-dnid
branch
from
September 10, 2026 21:29
4f2e524 to
64a45a2
Compare
github-actions
Bot
requested review from
OhmSpectator,
jsfakian,
rene,
rucoder and
shjala
September 10, 2026 21:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds evetest coverage for a backup DNID, where a peer acts for a DNID that
is down longer than a configured threshold. It changes no pillar code. The test
lands first, so you can run it against builds with and without the pillar work.
PR dependencies
Pending commits to this branch to add in support for handing these app operations on the non DNID node.
How to test and validate this PR
The test needs a three-node cluster environment and internet access. The apps
boot a real Alpine image from
dl-cdn.alpinelinux.org. Parameters: TPM (bool)and FILESYSTEM.
make evetest NAME=TestDNIDandBackupDNID EVETEST_LOG_LEVEL=debug make evetest NAME=TestDNIDandBackupDNID EVETEST_PAUSE_ON_FAILURE=true make evetest NAME=TestDNIDandBackupDNID make evetest NAME=TestNodeClusterSuite # the test runs last in this suiteChangelog notes
No user-facing changes. Adds EVE-K cluster test coverage and bumps evetest's own
pkg/pillar pin; no device or runtime behavior is changed.
PR Backports
Checklist
And the last but not least:
check them.
Please, check the boxes above after submitting the PR in interactive mode.