Repository navigation
fix(ateom): refuse a restore from the pre-per-volume durable layout - #265
Merged
Merged
Conversation
Both the gVisor and the micro-VM ateom snapshot each durable-dir volume as its own durable-dir-<volume>.tar, and a restore skips a volume whose tar is missing, since it may have been added to the template after the snapshot. A snapshot written before that layout holds every volume in the single durable-dir.tar instead, which nothing reads any more: the restore reports success and the actor comes up with empty durable volumes, losing its data without an error. A workload that keeps its state there fails later with unrelated symptoms (a missing database table, a failed first turn). Refuse such a restore. A checkpoint no longer writes durable-dir.tar, so its presence in the restore directory marks the old layout: the restore now fails with FailedPrecondition, "snapshot written before the per-volume layout; re-snapshot from a running actor or reset the golden snapshot", before anything is extracted, and the actor crashes on the failed Resume instead of running with empty volumes. Signed-off-by: Timo Derstappen <teemow@gmail.com> (cherry picked from commit 11a6bc4, agent-substrate#2462; the micro-VM hunks are dropped: on this line the micro-VM ateom still writes the single durable-dir.tar, which its restore reads)
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
This was referenced Oct 11, 2026
feat(substrate): follow Substrate 1.8.0 (a snapshot reset on the roll)
giantswarm/agent-platform#934
Closed
teemow
added a commit
that referenced
this pull request
Oct 11, 2026
The step from 1.7 to 1.8.1 converts each actor's 1.7-layout durable snapshot on its first resume (#277) and takes every golden written in an older layout again on the new workers, a create during the re-take answering Unavailable and succeeding on retry (#282). It needs no snapshot reset and no manual step. The snapshot reset runbook says so and stays the remedy for a corrupted snapshot store only. The rolling upgrade runbook gains the 1.7 to 1.8 step note with its floor 1.8.1: 1.8.0 refuses a 1.7 restore (#265). FORK.md's layout row points at the step instead of the reset. Refs #271, #272
teemow
added a commit
that referenced
this pull request
Oct 12, 2026
The step from 1.7 to 1.8.1 converts each actor's 1.7-layout durable snapshot on its first resume (#277) and takes every golden written in an older layout again on the new workers, a create during the re-take answering Unavailable and succeeding on retry (#282). It needs no snapshot reset and no manual step. The snapshot reset runbook says so and stays the remedy for a corrupted snapshot store only. The rolling upgrade runbook gains the 1.7 to 1.8 step note with its floor 1.8.1: 1.8.0 refuses a 1.7 restore (#265). FORK.md's layout row points at the step instead of the reset. Refs #271, #272
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the 1.8.0 line (since 1.8.0-rc.6, which carries agent-substrate#2110 through #260), a restore of a gVisor snapshot written in the pre-agent-substrate#2110 layout reports no error and brings the actor up with an empty
/data: the Go ADK runtime then fails withno such table: app_states, the claude Harness withHarness runtime execution failed. The old layout keeps every durable-dir volume in onedurable-dir.tar, which the restore no longer reads; it skips each missingdurable-dir-<volume>.taras a volume added since the snapshot. A missed snapshot reset therefore loses the sessions' data without a word.Proposed solution
The carried, upstream-ready commit of agent-substrate#2462 (
git cherry-pick -x 11a6bc4f):untarDurableVolumesrefuses a restore directory that holdsdurable-dir.tarbefore it extracts anything, withFailedPrecondition"snapshot written before the per-volume layout; re-snapshot from a running actor or reset the golden snapshot". The Resume fails and ate-api crashes the actor with that message (handleAteletError: not Unavailable, Canceled or DeadlineExceeded), so it stays CRASHED with the cause instead of running empty. Adapted to the line: the micro-VM hunks are dropped, since the line's micro-VM ateom still writes and reads the singledurable-dir.tar(agent-substrate#2283 is after the pin). A second commit adds theFORK.mdrow.Acceptance criteria
/data.TestUntarDurableVolumesRefusesLegacyLayoutrestores an old-layout archive (onedurable-dir.tarwith<volume>/...entries) and asserts the refusal, its code and the untouched volume directory;go test -race ./cmd/ateom-gvisor/ ./cmd/ateom-microvm/passes. The e2e suite has no way to plant a snapshot written by an older build, so the unit test at the restore boundary is the closest level.FORK.mdrow.Closes #263
This pull request was written by an agent.