TestVaultZvolTrimReclaimsBlocks always fails on EVE-KVM: /persist/vault is a ZFS dataset, not ext4-on-zvol

[eriknordmark]

TestVaultZvolTrimReclaimsBlocks fails on its first assertion under the default parameters of the suite it belongs to, because on EVE-KVM /persist/vault is a plain ZFS dataset rather than ext4 on a zvol, and fstrim has nothing to discard.

The failure is at evetest/tests/storage/vault_trim_test.go:98, the baseline cleanup fstrim, before the test writes any data:

vault_trim_test.go:98: TEST FAILURE:
    baseline cleanup fstrim failed
    ...
    fstrim: /persist/vault: the discard operation is not supported

Why

SetupDefaultVault in pkg/pillar/vault/handler_zfs.go branches on base.IsHVTypeKube():

• kubevirt: CreateZvolVault(...) — ext4 on a ZFS zvol, so fstrim has a DISCARD path to the zvol and the test's premise holds.
• KVM: zfs.CreateDataset(types.SealedDataset) — a plain ZFS dataset mounted at /persist/vault. There is no ext4 layer, so fstrim fails exactly as above.

TestStorageSuite declares HYPERVISOR with default kvm, so the test takes the second branch every time it runs as part of its suite.

The test's doc comment states the opposite rationale:

Requires a ZFS node; the underlying bug was found on EVE-K (Longhorn replica churn), but the mechanism is purely a ZFS/fstrim concern and applies equally to EVE-KVM, so the hypervisor is a parameter rather than hard-coded.

The fstrim mechanism is indeed generic, but the ext4-on-zvol layout it depends on is not — it exists only on the EVE-k path.

The device also had no TPM (WithTPM unset), so the !etpm.IsTpmEnabled() branch above is the one taken. I have not traced whether the TPM-enabled path (h.setupVault()) produces a dataset or a zvol under KVM, so this report covers the TPM-less case only.

Reproduction

make evetest NAME=TestStorageSuite

Reproduced on three separate EVE images built from master f94089001 and two integration branches based on it, all -kvm-amd64. Identical subtest, identical assertion, identical error, 120-123 s each. The other 6 subtests in the suite pass on all three.

Why it has not been noticed

The test arrived with #6105, an EVE-k fix, where it does pass. grep finds no reference to TestStorageSuite or TestVaultZvolTrimReclaimsBlocks under .github/workflows/, so no CI job runs it.

Possible resolutions

• Skip under a non-kubevirt hypervisor, alongside the existing precedent in tests/diag/diag_test.go and tests/apps/purge_replaces_vmirs_test.go for hypervisor-inapplicable tests.
• Or hard-code WithHypervisor: evetest.HypervisorKubevirt and drop the HYPERVISOR parameter, matching where the test can actually run.

Either way the doc comment's claim about EVE-KVM needs correcting, since it is the reason the parameter is there.

[eriknordmark]

Re-ran with a TPM-backed device, which closes the caveat in the original report ("I have not traced whether the TPM-enabled path (h.setupVault()) produces a dataset or a zvol under KVM, so this report covers the TPM-less case only").

The TPM path behaves the same. Same three images, same suite, devices with WithTPM: true, and the failure is identical:

vault_trim_test.go:99: TEST FAILURE: baseline cleanup fstrim failed
    fstrim: /persist/vault: the discard operation is not supported

	master	resize-allprs	appvol-allprs
TPM-less	FAIL 120.3 s	FAIL 122.7 s	FAIL 121.4 s
with TPM	FAIL 134.4 s	FAIL 171.2 s	FAIL 122.9 s

The TPM was verified in effect on the live device (evetest status reports With TPM: true for edge-dev on this suite), not merely requested in the test source.

So SetupDefaultVault's !etpm.IsTpmEnabled() branch is not the only way to reach a plain dataset at /persist/vault: the h.setupVault() path gets there too under HV=kvm. The defect is a property of the KVM personality, not of whether a TPM is present, and a fix that only addressed the TPM-less branch would be incomplete.

The other 6 subtests in TestStorageSuite pass with a TPM on all three images, as they do without one.

(The line number moved from :98 to :99 because the TPM run adds one line to the file.)

[eriknordmark]

#6480 makes this test pass, but not by changing the reported behaviour -- flagging that so the pass is not mistaken for a fix.

It hard-codes the test to Kubevirt, replacing the HYPERVISOR parameter:

-	evetest.DefineTestParameters(evetest.HypervisorParameter())
-	hypervisor := evetest.GetHypervisorParameterValue()
-	...  WithHypervisor: hypervisor,
+	...  WithHypervisor: evetest.HypervisorKubevirt,

and rewrites the doc comment to say so: "the underlying bug was found on EVE-K (Longhorn replica churn), so the hypervisor is hard-coded to Kubevirt rather than a parameter". That is the second of the two resolutions suggested above. It passes at 301 s on EVE-k, where SetupDefaultVault takes CreateZvolVault and there is an ext4-on-zvol for fstrim to act on.

/persist/vault under HV=kvm is still a plain ZFS dataset -- #6480 states it makes no EVE functional changes, and the EVE image used here is byte-identical to the one that failed. So the KVM-personality behaviour described above is unchanged; the test simply no longer runs in the configuration where its premise does not hold.

#6480 also replaces the logicalused before/after delta with fstrim's own reported byte count, citing k3s/Longhorn churn on the shared dataset -- a separate robustness improvement.

Leaving this open for a maintainer to decide: if scoping the test to EVE-k is the intended resolution, this can be closed as descoped; if the KVM vault layout is itself worth addressing, it stands.