fix(ateom): convert a pre-per-volume durable snapshot on restore

FORK.md

@@ -115,6 +115,7 @@ Everything on `giantswarm` that is not in the pin (`git log v0.4.0-alpha1..giant
 | The snapshot takes one rootfs-upper tar per container and one tar per durable volume; the restore creates the layout before it extracts into it, so a snapshot no longer decides which host paths are mounted or deleted; `tarutil` writes and reads the tree through an `os.Root` (`fs.WalkDir` over `root.FS()`, `root.Lstat`, `root.Readlink`), component by component, never by the full host path (`internal/tarutil`, `cmd/ateom-gvisor/durable.go`, `cmd/ateom-microvm/rootfsupper.go`) | a workspace tree deeper than `PATH_MAX` crashed the actor at its checkpoint and lost the session on every harness: the tar walked and stat'ed each entry by its full path, which the kernel refuses with `ENAMETOOLONG`; the same tar named the host paths a restore mounted and deleted (upstream's agent-substrate/substrate#2083; giantswarm/giantswarm#38054; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 165). **Snapshots written before it do not resume after it**, nor its own on a release before it: the roll needs the snapshot reset of `docs/snapshot-reset.md`, golden and session snapshots, in either direction | `3afd756f` ([#260](https://github.com/giantswarm/substrate/pull/260), rebase-merged 2026-10-10; first release 1.8.0-rc.6): `git cherry-pick -x` of upstream's squash commit `893b1edb`, clean | [agent-substrate/substrate#2110](https://github.com/agent-substrate/substrate/pull/2110) (merged 2026-10-06, after the pin). Falls away at the re-pin onto the first release that carries it |
 | A snapshot extract creates only `0:0` character devices (overlay whiteouts) and sets only `user.*` xattrs and the path-based overlay attributes (opaque, redirect, impure); `Create` skips other device nodes; on the micro-VM host the merged rootfs overlay and the binds into the virtio-fs share are `nosuid,nodev` (`setNosuidNodev`, recursive) | snapshot tars are extracted as root on the host, and the extract made any device node and set any `trusted.overlay.*` xattr the archive named; a stray device could make a snapshot unrestorable (upstream's agent-substrate/substrate#2090; giantswarm/giantswarm#38054; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 166) | `eb04dfc1` ([#260](https://github.com/giantswarm/substrate/pull/260), rebase-merged 2026-10-10; first release 1.8.0-rc.6): `git cherry-pick -x` of upstream's squash commit `8da76493`. Adapted to the line: it still mounts through `mount(8)`, so `nosuid,nodev` joins the overlay's `-o` options and `BindIntoShare` calls `setNosuidNodev` after the rbind; the hunk's `RemountReadOnly` and `Unmount` tests cover helpers the line does not have and are left out | [agent-substrate/substrate#2111](https://github.com/agent-substrate/substrate/pull/2111) (merged 2026-10-06, after the pin). Falls away at the re-pin onto the first release that carries it |
 | A gVisor restore refuses a snapshot in the layout before one tar per durable volume (`durable-dir.tar` in the restore directory) with `FailedPrecondition` "snapshot written before the per-volume layout; re-snapshot from a running actor or reset the golden snapshot", before it extracts anything (`cmd/ateom-gvisor/durable.go`) | since the one-tar-per-volume layout (agent-substrate/substrate#2110, two rows up) a restore skips a missing `durable-dir-<volume>.tar` as a volume added since the snapshot, so a snapshot written before it restored with no error and an empty `/data`: the Go ADK failed with `no such table: app_states`, the claude Harness with `Harness runtime execution failed`, and a missed snapshot reset lost the sessions' data without a word ([#263](https://github.com/giantswarm/substrate/issues/263); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 171). Now the Resume fails and the actor is CRASHED with that message | [#265](https://github.com/giantswarm/substrate/pull/265): `git cherry-pick -x` of `11a6bc4f`, adapted to the line: the micro-VM hunks are left out, its ateom still writes and reads the single `durable-dir.tar` (agent-substrate/substrate#2283 is after the pin) | [agent-substrate/substrate#2462](https://github.com/agent-substrate/substrate/pull/2462) (ours, open; both ateoms). Falls away at the re-pin onto the first release that carries it |
+| A gVisor restore converts a snapshot in the layout before one tar per durable volume instead of refusing it (`convertLegacyDurableLayout` in `cmd/ateom-gvisor/durable.go`: `tarutil.Split` rewrites `durable-dir.tar` into one `durable-dir-<volume>.tar` per declared volume, tar to tar, all or nothing, `durable-dir.tar` left in place; each volume is then extracted like a current snapshot). The refusal of the row above stays for a snapshot that does not convert — an unreadable archive, an entry outside every declared volume, a hardlink across volumes — now with `FailedPrecondition` and the cause. `tarutil.Split` is exported for the ActorTemplate reconciler | the refusal kept an actor from running with an empty `/data`, but every actor suspended on 1.7, and every actor created from a 1.7 golden, still lost its conversation and durable volumes on the step to 1.8, and so did a snapshot reset ([#271](https://github.com/giantswarm/substrate/issues/271)). Now the first resume on 1.8 restores the data and the next checkpoint writes the per-volume layout | [#277](https://github.com/giantswarm/substrate/pull/277): the fork's own commit on top of the refusal's; the micro-VM ateom of the line still writes and reads the single `durable-dir.tar` and needs no conversion | to open: the successor of [agent-substrate/substrate#2462](https://github.com/agent-substrate/substrate/pull/2462) (or folded into it), queued for the team's review of the upstream list. Upstream's `AGENTS.md` asks for no fallbacks for old formats before v1.0.0, so upstream may decline it; then it is carried until a release after 1.0 makes it moot. Falls away with the refusal's row |
 | helm-e2e pins `azure/setup-helm` to its v4.3.1 release by commit and installs Helm v4.3.0 on every run (`with: version: v4.3.0`) | the action ran without a version and installed whatever Helm it resolved that day: a run got Helm 3.18.4 instead of Helm 4, and the step's `helm plugin install … --verify=false`, a Helm 4 flag, failed with `unknown flag: --verify` until a rerun ([#246](https://github.com/giantswarm/substrate/issues/246)). Workflow-only | [#275](https://github.com/giantswarm/substrate/pull/275): `git cherry-pick -x` of `5b47d357`, adapted to the line's indentation | to file: the upstream-shaped patch is branch [`upstream/helm-e2e-pin-setup-helm`](https://github.com/giantswarm/substrate/tree/upstream/helm-e2e-pin-setup-helm) here (`5b47d357`, on kagent-dev/substrate `main` @ `9c4b1fb5`, DCO-signed; its message is the pull-request text). kagent-dev/substrate's own helm-e2e carries the same unpinned step; agent-substrate/substrate has no helm-e2e. A team member opens the kagent-dev/substrate pull request. Falls away at the re-pin onto the first release that carries it |
 
 Twenty-three patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no

cmd/ateom-gvisor/durable.go

@@ -44,15 +44,15 @@ func durableTarFile(volumeName string) (string, error) {
 }
 
 // legacyDurableTarFile is the single tar of every durable-dir volume that
-// snapshots held before the one-tar-per-volume layout. A checkpoint never
-// writes it any more, so its presence marks a snapshot in that layout.
+// snapshots held before the one-tar-per-volume layout, its entries named
+// <volumeName>/... . A checkpoint never writes it any more, so its presence
+// marks a snapshot in that layout.
 const legacyDurableTarFile = "durable-dir.tar"
 
 // errLegacySnapshotLayout refuses a restore from a snapshot in the layout
-// before one tar per durable-dir volume: its data is in legacyDurableTarFile,
-// which is never extracted, so restoring it would bring the actor up with
-// empty volumes.
-var errLegacySnapshotLayout = errors.New("snapshot written before the per-volume layout; re-snapshot from a running actor or reset the golden snapshot")
+// before one tar per durable-dir volume that convertLegacyDurableLayout
+// cannot convert.
+var errLegacySnapshotLayout = errors.New("snapshot written before the per-volume layout does not convert")
 
 // hasDurableVolumes reports whether any container mounts a durable-dir volume.
 func hasDurableVolumes(containers []*ateompb.Container) bool {
@@ -89,14 +89,44 @@ func tarDurableVolumes(ctx context.Context, dir, checkpointDir string, volumes [
 	return files, nil
 }
 
+// convertLegacyDurableLayout rewrites a snapshot directory in the layout
+// before one tar per durable-dir volume into the current one: the entries of
+// legacyDurableTarFile under each of volumes become that volume's
+// durableTarFile, which a restore extracts like any other. It does nothing
+// for a snapshot in the current layout. legacyDurableTarFile stays, so a
+// conversion that did not finish runs again on the next restore; the next
+// checkpoint writes the current layout.
+//
+// A snapshot it cannot convert, one whose archive is unreadable or holds an
+// entry outside every volume, is refused with errLegacySnapshotLayout and the
+// cause, so the actor never resumes with empty volumes.
+func convertLegacyDurableLayout(snapshotDir string, volumes []string) error {
+	legacy := filepath.Join(snapshotDir, legacyDurableTarFile)
+	if _, err := os.Stat(legacy); errors.Is(err, fs.ErrNotExist) {
+		return nil
+	}
+	dsts := make(map[string]string, len(volumes))
+	for _, vol := range volumes {
+		name, err := durableTarFile(vol)
+		if err != nil {
+			return err
+		}
+		dsts[vol] = filepath.Join(snapshotDir, name)
+	}
+	if err := tarutil.Split(legacy, dsts); err != nil {
+		return apierror.FailedPrecondition("%w: %w; re-snapshot from a running actor or reset the golden snapshot", errLegacySnapshotLayout, err)
+	}
+	return nil
+}
+
 // untarDurableVolumes restores each durable-dir volume from the snapshot into
 // its directory under dir, which atelet has already created, empty. A volume
 // with no tar in the snapshot (added to the template since) stays empty. A
-// snapshot in the layout before one tar per volume is refused with
-// errLegacySnapshotLayout.
+// snapshot in the layout before one tar per volume is converted first (see
+// convertLegacyDurableLayout).
 func untarDurableVolumes(dir, snapshotDir string, volumes []string) error {
-	if _, err := os.Stat(filepath.Join(snapshotDir, legacyDurableTarFile)); err == nil {
-		return apierror.FailedPrecondition("%w", errLegacySnapshotLayout)
+	if err := convertLegacyDurableLayout(snapshotDir, volumes); err != nil {
+		return err
 	}
 	for _, vol := range volumes {
 		name, err := durableTarFile(vol)

cmd/ateom-gvisor/durable_test.go

@@ -17,11 +17,12 @@
 package main
 
 import (
-	"archive/tar"
 	"errors"
+	"maps"
 	"os"
 	"path/filepath"
 	"slices"
+	"strings"
 	"testing"
 
 	"google.golang.org/grpc/codes"
@@ -126,46 +127,162 @@ func TestUntarDurableVolumesMissingTarIsEmpty(t *testing.T) {
 }
 
 // writeLegacyDurableTar writes the snapshot file of the layout before one tar
-// per volume: a single durable-dir.tar whose entries are <volume>/... .
+// per volume as that layout's checkpoint did: one archive of the whole
+// durable-dir volumes directory, its entries named <volume>/... .
 func writeLegacyDurableTar(t *testing.T, snapshotDir string, files map[string]string) {
 	t.Helper()
-	f, err := os.Create(filepath.Join(snapshotDir, legacyDurableTarFile))
-	if err != nil {
-		t.Fatal(err)
-	}
-	defer f.Close()
-	tw := tar.NewWriter(f)
-	for name, body := range files {
-		if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(body)), Typeflag: tar.TypeReg}); err != nil {
+	src := t.TempDir()
+	for rel, body := range files {
+		p := filepath.Join(src, rel)
+		if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
 			t.Fatal(err)
 		}
-		if _, err := tw.Write([]byte(body)); err != nil {
+		if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
 			t.Fatal(err)
 		}
 	}
-	if err := tw.Close(); err != nil {
+	if err := tarutil.Create(t.Context(), filepath.Join(snapshotDir, legacyDurableTarFile), src); err != nil {
 		t.Fatal(err)
 	}
 }
 
-// A snapshot in the layout before one tar per volume holds the actor's data in
-// durable-dir.tar, which a restore never extracts: it is refused instead of
-// restoring the volumes empty.
-func TestUntarDurableVolumesRefusesLegacyLayout(t *testing.T) {
+// A snapshot in the layout before one tar per volume restores its volumes:
+// the restore converts it into one tar per volume first.
+func TestUntarDurableVolumesConvertsLegacyLayout(t *testing.T) {
 	dst, snap := t.TempDir(), t.TempDir()
+	mkVolumeDirs(t, dst, "data", "cache", "new")
+	writeLegacyDurableTar(t, snap, map[string]string{
+		"data/app.db":           "state",
+		"data/sessions/s1.json": "turn",
+		"data/.gvisor.filestat": "internal",
+		"cache/c.bin":           "cached",
+	})
+
+	if err := untarDurableVolumes(dst, snap, []string{"cache", "data", "new"}); err != nil {
+		t.Fatalf("untarDurableVolumes: %v", err)
+	}
+	for rel, want := range map[string]string{
+		"data/app.db":           "state",
+		"data/sessions/s1.json": "turn",
+		"cache/c.bin":           "cached",
+	} {
+		if got, err := os.ReadFile(filepath.Join(dst, rel)); err != nil || string(got) != want {
+			t.Errorf("restored %q = %q, %v; want %q", rel, got, err, want)
+		}
+	}
+	if _, err := os.Lstat(filepath.Join(dst, "data/.gvisor.filestat")); !os.IsNotExist(err) {
+		t.Errorf(".gvisor.filestat: Lstat = %v; want it removed", err)
+	}
+	if entries, err := os.ReadDir(filepath.Join(dst, "new")); err != nil || len(entries) != 0 {
+		t.Errorf("new: ReadDir = %v, %v; want an empty directory", entries, err)
+	}
+	for _, name := range []string{"durable-dir-data.tar", "durable-dir-cache.tar"} {
+		if _, err := os.Stat(filepath.Join(snap, name)); err != nil {
+			t.Errorf("converted %s: %v", name, err)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(snap, "durable-dir-new.tar")); !os.IsNotExist(err) {
+		t.Errorf("durable-dir-new.tar: Stat = %v; want no tar for a volume without entries", err)
+	}
+
+	// The conversion is idempotent: a second restore of the same directory
+	// converts again and restores the same data.
+	dst2 := t.TempDir()
+	mkVolumeDirs(t, dst2, "cache", "data", "new")
+	if err := untarDurableVolumes(dst2, snap, []string{"cache", "data", "new"}); err != nil {
+		t.Fatalf("second untarDurableVolumes: %v", err)
+	}
+	if got, err := os.ReadFile(filepath.Join(dst2, "data/app.db")); err != nil || string(got) != "state" {
+		t.Errorf("second restore data/app.db = %q, %v; want %q", got, err, "state")
+	}
+}
+
+// A snapshot in the current layout is restored as it is, the snapshot
+// directory untouched.
+func TestUntarDurableVolumesCurrentLayoutUnchanged(t *testing.T) {
+	src := t.TempDir()
+	mkVolumeDirs(t, src, "data")
+	if err := os.WriteFile(filepath.Join(src, "data/app.db"), []byte("state"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	snap := t.TempDir()
+	if _, err := tarDurableVolumes(t.Context(), src, snap, []string{"data"}); err != nil {
+		t.Fatal(err)
+	}
+	before := snapshotFiles(t, snap)
+
+	dst := t.TempDir()
 	mkVolumeDirs(t, dst, "data")
-	writeLegacyDurableTar(t, snap, map[string]string{"data/app.db": "state"})
+	if err := untarDurableVolumes(dst, snap, []string{"data"}); err != nil {
+		t.Fatalf("untarDurableVolumes: %v", err)
+	}
+	if got, err := os.ReadFile(filepath.Join(dst, "data/app.db")); err != nil || string(got) != "state" {
+		t.Errorf("restored data/app.db = %q, %v; want %q", got, err, "state")
+	}
+	if after := snapshotFiles(t, snap); !maps.Equal(before, after) {
+		t.Errorf("snapshot directory changed by the restore: before %v, after %v", slices.Sorted(maps.Keys(before)), slices.Sorted(maps.Keys(after)))
+	}
+}
 
-	err := untarDurableVolumes(dst, snap, []string{"data"})
-	if !errors.Is(err, errLegacySnapshotLayout) {
-		t.Fatalf("untarDurableVolumes = %v; want %v", err, errLegacySnapshotLayout)
+// snapshotFiles returns the contents of each file in dir by name.
+func snapshotFiles(t *testing.T, dir string) map[string]string {
+	t.Helper()
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		t.Fatal(err)
 	}
-	if got := apierror.Code(err); got != codes.FailedPrecondition {
-		t.Errorf("code = %v; want %v", got, codes.FailedPrecondition)
+	files := map[string]string{}
+	for _, e := range entries {
+		b, err := os.ReadFile(filepath.Join(dir, e.Name()))
+		if err != nil {
+			t.Fatal(err)
+		}
+		files[e.Name()] = string(b)
 	}
-	entries, rerr := os.ReadDir(filepath.Join(dst, "data"))
-	if rerr != nil || len(entries) != 0 {
-		t.Errorf("data: ReadDir = %v, %v; want it untouched", entries, rerr)
+	return files
+}
+
+// A snapshot in the layout before one tar per volume that does not convert is
+// refused with the cause instead of restoring the volumes empty, and the
+// snapshot directory gains no partial conversion.
+func TestUntarDurableVolumesRefusesUnconvertibleLegacyLayout(t *testing.T) {
+	for name, write := range map[string]func(t *testing.T, snap string){
+		"entry outside every volume": func(t *testing.T, snap string) {
+			writeLegacyDurableTar(t, snap, map[string]string{"data/app.db": "state", "stray/x": "x"})
+		},
+		"truncated archive": func(t *testing.T, snap string) {
+			writeLegacyDurableTar(t, snap, map[string]string{"data/app.db": strings.Repeat("s", 4096)})
+			p := filepath.Join(snap, legacyDurableTarFile)
+			if err := os.Truncate(p, 2048); err != nil {
+				t.Fatal(err)
+			}
+		},
+		"not a tar": func(t *testing.T, snap string) {
+			if err := os.WriteFile(filepath.Join(snap, legacyDurableTarFile), []byte(strings.Repeat("garbage", 200)), 0o600); err != nil {
+				t.Fatal(err)
+			}
+		},
+	} {
+		t.Run(name, func(t *testing.T) {
+			dst, snap := t.TempDir(), t.TempDir()
+			mkVolumeDirs(t, dst, "data")
+			write(t, snap)
+			before := snapshotFiles(t, snap)
+
+			err := untarDurableVolumes(dst, snap, []string{"data"})
+			if !errors.Is(err, errLegacySnapshotLayout) {
+				t.Fatalf("untarDurableVolumes = %v; want %v", err, errLegacySnapshotLayout)
+			}
+			if got := apierror.Code(err); got != codes.FailedPrecondition {
+				t.Errorf("code = %v; want %v", got, codes.FailedPrecondition)
+			}
+			if entries, rerr := os.ReadDir(filepath.Join(dst, "data")); rerr != nil || len(entries) != 0 {
+				t.Errorf("data: ReadDir = %v, %v; want it untouched", entries, rerr)
+			}
+			if after := snapshotFiles(t, snap); !maps.Equal(before, after) {
+				t.Errorf("snapshot directory changed by the refused restore: %v", slices.Sorted(maps.Keys(after)))
+			}
+		})
 	}
 }
 

docs/snapshot-reset.md

@@ -37,7 +37,12 @@ claude Harness:
   state: SQL logic error: no such table: app_states`, the claude harness
   with `Harness runtime execution failed`. The Agent stays `Ready`, every
   new session on its golden fails at once, and the checkpoint after the
-  failed turn uploads an empty durable tar.
+  failed turn uploads an empty durable tar. Since the refusal and the
+  conversion FORK.md carries, a gVisor ateom converts such a snapshot's
+  `durable-dir.tar` into the per-volume tars on its first restore, and
+  refuses with `FailedPrecondition` and the cause one that does not
+  convert. Until the forward step is rehearsed again with the conversion,
+  the reset stays the remedy.
 - **Backward (1.8.0 snapshots on 1.7.0):** the restore fails and the actor
   crashes: `while restoring durable-dir volumes … opening tar
   "…/restore-state/durable-dir.tar": no such file or directory`. A turn