[17.0-stable] zedkube: detect and recover stuck kubelet volume mounts - #6487
Open
eriknordmark wants to merge 5 commits into
Open
[17.0-stable] zedkube: detect and recover stuck kubelet volume mounts#6487eriknordmark wants to merge 5 commits into
eriknordmark wants to merge 5 commits into
Conversation
In cluster mode a Longhorn volume can be attached to the node (block device present, VolumeAttachment reporting Attached) while kubelet never issues NodeStage for it, so the consuming pod sits in ContainerCreating /Init forever with no FailedMount event and the app never runs. The stall is in kubelet's volume manager (shipped via k3s), not Longhorn, CDI or EVE, and only a fresh kubelet clears it. Nothing detected or recovered from this before. Add a zedkube check that flags a pod left Pending past a threshold on this node whose Longhorn PVC is attached-but-unmounted and shows no container/init startup error (image pull, crash, or missing secret/config). When one is found it restarts k3s: it resets cluster-init.sh's restart backoff and sends SIGTERM to the k3s server process, which the supervisor then relaunches, so kubelet returns with a fresh volume manager. Recovery is rate-limited per episode (bounded attempts plus a cooldown) and every restart logs a distinctive MOUNT-WEDGE-RECOVERY marker. A build-time flag disables the action and leaves only detection logging. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 092d7a3)
The mount-wedge detector listed every pod in the EVE namespace and discarded the irrelevant ones in the loop, so on a multi-node cluster it pulled other nodes' pods over the API on every tick. Only a Pending pod scheduled on this node can exhibit the wedge, so ask the apiserver for exactly that set, matching how the drain and SR-IOV device-plugin paths already restrict their LISTs. The per-pod node and phase checks stay as a guard. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit f16bfce)
The detector decides whether to restart k3s -- a disruptive, node-wide action -- from a pod's phase, age, container error reasons and its volume's attach state, and then rate-limits itself per wedge episode. None of that was covered. Add tests for the wedge signature, one per condition that must keep a Pending pod from being called wedged (including the missing-Secret pod, which looks identical from the outside but is not helped by a fresh kubelet), for the per-episode attempt cap and its cooldown, and for the cmdline match that decides which process gets SIGTERM. The episode test also pins that a cleared wedge resets the attempt count, since otherwise the cap would disarm recovery for the lifetime of the process. Reaching that logic requires the cluster and host lookups to sit behind indirections a test can replace, and the tick body to be separable from the clientset it runs against. The /proc scan is likewise split from the signaling, so process matching is exercisable without terminating anything. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 41756b7)
The mount-wedge recovery restarted k3s by touching cluster-init.sh's manual-start flag and sending SIGTERM. On an image where the Go kube-init daemon supervises k3s instead, that flag is read by nobody and the daemon's own pre-restart hooks are skipped, so the recovery would go quietly ineffective while the detector kept reporting wedges. Ask whichever supervisor is present. kube-init exposes a control socket that takes a "restart" verb, so prefer it when the socket exists and fall back to the flag plus SIGTERM when it does not. Choosing per attempt rather than once at start-up keeps one pillar binary correct on both, including across an upgrade that swaps the supervisor underneath a running device. The socket reply is a single line whose ERR prefix, not the connection, reports refusal, so an ERR and a silent daemon are both failures rather than a restart assumed to have happened. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 922d041)
A Longhorn volume left behind by a torn-down app keeps the mount-wedge signature forever: attached to the node, block device present, and a CDI upload pod sitting Pending because nothing will ever send it the image. Restarting k3s cannot change that, so each attempt bought a control-plane outage and no progress -- and because the per-episode attempt cap lives in memory, every reboot re-armed the detector and the device paid for it again. A lab run recorded ten such futile recoveries out of twelve, driven by two stale volumes. Require that some app instance still references the volume before treating it as wedged. The name a VolumeRefConfig resolves to is the same one VolumeStatus already resolves to, and zedkube subscribes to AppInstanceConfig, so this is a local lookup with no apiserver call -- placed ahead of the PVC and VolumeAttachment lookups, it also keeps those off the stale path entirely. A volume nothing references is counted and reported once, rather than every tick, pointing the operator at the stale object instead of at kubelet. An empty subscription makes every volume unwanted, which suppresses recovery instead of triggering it. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit ece6e50)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 17.0-stable #6487 +/- ##
===============================================
+ Coverage 21.38% 21.89% +0.50%
===============================================
Files 500 511 +11
Lines 92650 94541 +1891
===============================================
+ Hits 19813 20697 +884
- Misses 71191 72011 +820
- Partials 1646 1833 +187 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
@andrewd-zededa @naiming-zededa @zedi-pramodh @rene @milan-zededa — review please. Backport of #6197, approved there by @rene. All checks have been green since 2026-09-08 and the PR is mergeable; an approval is the only thing outstanding. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6197 to
17.0-stable.In cluster mode a Longhorn volume can be attached to a node while kubelet never
issues
NodeStagefor it, leaving the consuming pod inContainerCreatingwithno
FailedMountevent and the app never running.zedkubenow detects thatsignature — pod Pending past a threshold on this node, its Longhorn PVC attached
with the
/dev/longhorndevice present but unmounted, no image-pull or crasherror — and restarts k3s so kubelet comes back with a fresh volume manager.
Recovery requires that some app instance still references the volume, so a stale
CDI upload PVC that nothing is waiting for is logged rather than acted on, and is
rate-limited per episode.
Five of the source PR's six commits are cherry-picked with
-x, in order:092d7a326,f16bfce19,41756b7e9,922d041b8,ece6e50ab. The sixth,Makefile: raise the non-k rootfs ceiling to 291MB, is omitted: it liftsROOTFS_MAXSIZE_MBfrom 290 to 291 for a master rootfs baseline that no longerfits under the old ceiling, and this branch has its own baseline. It can be added
if the size check fails here.
One adaptation:
const testNodeName = "this-node"is defined instuckmount_test.go. On master it comes fromcmd/zedkube/vmirsaffinity_test.go,which this branch does not have; the value is master's. Nothing else differs —
every added and removed line matches the source PR's diff byte for byte.
How to test and validate this PR
Covered by unit tests:
go test -tags k ./cmd/zedkube/inpkg/pillar(44 testspass on this branch, 13 of them the stuck-mount and restart-mechanism tests).
Note the
kbuild tag — without it the package reports "no test files".Neutering the app-reference gate fails exactly the tests that guard it
(
TestCheckStuckVolumeMountOrphanNoRestart,TestPodMountWedge/volume_belongs_to_no_app_config), so the suite doesdiscriminate rather than passing vacuously.
On an EVE-k device, a pod stuck
ContainerCreatingon an attached-but-unmountedLonghorn volume for more than ~5 minutes triggers a k3s restart logged with a
MOUNT-WEDGE-RECOVERYmarker at Warn level. On this branch the restart uses thekube-init supervisor socket where that daemon is present and a SIGTERM to
k3s serverotherwise; the mechanism is chosen per attempt. Grep all of/persist/newlogforMOUNT-WEDGE-RECOVERY, not justcollect/, which holdsonly the last ~5 minutes. Hardware validation of both the recovery and the
app-reference gate is in #6197.
Changelog notes
No user-facing changes.
PR Backports
Checklist