Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions FORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ Everything on `giantswarm` that is not in the pin (`git log v0.4.0-alpha1..giant
| `crashActor` terminates the sandbox on atelet, detaches the actor's volumes and only then releases the worker (a failed step leaves the actor `CRASHED` with its worker assignment, for `DeleteActor` or `RevertActor` to retry); `DeleteActor` refuses with `FailedPrecondition` while the sandbox cannot be stopped or a volume released, keeping the assignment for the retry; `CreateActorTemplate` refuses a DATA `on_pause` or `on_commit` scope unless a container mounts a durable-dir volume (`InvalidArgument` naming the field); the existing-volumes e2e suite covers the delete after a failed suspend | a failed checkpoint crashed the actor and released its worker with the sandbox still running on it, so `DeleteActor` found no worker to reach and removed the record: the sandbox, its existing-volume NFS mounts and the worker's capacity leaked until the worker pod restarted, and the PersistentVolume stayed `Released` on `directory not empty` ([giantswarm/substrate#237](https://github.com/giantswarm/substrate/issues/237); giantswarm/giantswarm#37742 row 162) | [#258](https://github.com/giantswarm/substrate/pull/258) (`fix(ateapi): tear a crashed actor's workload down before it is deleted`) | [agent-substrate/substrate#1953](https://github.com/agent-substrate/substrate/pull/1953) (open: the crash tears the workload down before `CRASHED` and keeps the assignment when it fails, the shape adapted here) and [agent-substrate/substrate#1972](https://github.com/agent-substrate/substrate/pull/1972) (open: refuses the DATA scope without a durable dir, for agent-substrate/substrate#1936); falls away at the re-pin onto the first release that carries both |
| The snapshot takes one rootfs-upper tar per container and one tar per durable volume; the restore creates the layout before it extracts into it, so a snapshot no longer decides which host paths are mounted or deleted; `tarutil` writes and reads the tree through an `os.Root` (`fs.WalkDir` over `root.FS()`, `root.Lstat`, `root.Readlink`), component by component, never by the full host path (`internal/tarutil`, `cmd/ateom-gvisor/durable.go`, `cmd/ateom-microvm/rootfsupper.go`) | a workspace tree deeper than `PATH_MAX` crashed the actor at its checkpoint and lost the session on every harness: the tar walked and stat'ed each entry by its full path, which the kernel refuses with `ENAMETOOLONG`; the same tar named the host paths a restore mounted and deleted (upstream's agent-substrate/substrate#2083; giantswarm/giantswarm#38054; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 165). **Snapshots written before it do not resume after it**, nor its own on a release before it: the roll needs the snapshot reset of `docs/snapshot-reset.md`, golden and session snapshots, in either direction | `3afd756f` ([#260](https://github.com/giantswarm/substrate/pull/260), rebase-merged 2026-10-10; first release 1.8.0-rc.6): `git cherry-pick -x` of upstream's squash commit `893b1edb`, clean | [agent-substrate/substrate#2110](https://github.com/agent-substrate/substrate/pull/2110) (merged 2026-10-06, after the pin). Falls away at the re-pin onto the first release that carries it |
| A snapshot extract creates only `0:0` character devices (overlay whiteouts) and sets only `user.*` xattrs and the path-based overlay attributes (opaque, redirect, impure); `Create` skips other device nodes; on the micro-VM host the merged rootfs overlay and the binds into the virtio-fs share are `nosuid,nodev` (`setNosuidNodev`, recursive) | snapshot tars are extracted as root on the host, and the extract made any device node and set any `trusted.overlay.*` xattr the archive named; a stray device could make a snapshot unrestorable (upstream's agent-substrate/substrate#2090; giantswarm/giantswarm#38054; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 166) | `eb04dfc1` ([#260](https://github.com/giantswarm/substrate/pull/260), rebase-merged 2026-10-10; first release 1.8.0-rc.6): `git cherry-pick -x` of upstream's squash commit `8da76493`. Adapted to the line: it still mounts through `mount(8)`, so `nosuid,nodev` joins the overlay's `-o` options and `BindIntoShare` calls `setNosuidNodev` after the rbind; the hunk's `RemountReadOnly` and `Unmount` tests cover helpers the line does not have and are left out | [agent-substrate/substrate#2111](https://github.com/agent-substrate/substrate/pull/2111) (merged 2026-10-06, after the pin). Falls away at the re-pin onto the first release that carries it |
| A gVisor restore refuses a snapshot in the layout before one tar per durable volume (`durable-dir.tar` in the restore directory) with `FailedPrecondition` "snapshot written before the per-volume layout; re-snapshot from a running actor or reset the golden snapshot", before it extracts anything (`cmd/ateom-gvisor/durable.go`) | since the one-tar-per-volume layout (agent-substrate/substrate#2110, two rows up) a restore skips a missing `durable-dir-<volume>.tar` as a volume added since the snapshot, so a snapshot written before it restored with no error and an empty `/data`: the Go ADK failed with `no such table: app_states`, the claude Harness with `Harness runtime execution failed`, and a missed snapshot reset lost the sessions' data without a word ([#263](https://github.com/giantswarm/substrate/issues/263); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 171). Now the Resume fails and the actor is CRASHED with that message | [#265](https://github.com/giantswarm/substrate/pull/265): `git cherry-pick -x` of `11a6bc4f`, adapted to the line: the micro-VM hunks are left out, its ateom still writes and reads the single `durable-dir.tar` (agent-substrate/substrate#2283 is after the pin) | [agent-substrate/substrate#2462](https://github.com/agent-substrate/substrate/pull/2462) (ours, open; both ateoms). Falls away at the re-pin onto the first release that carries it |

Twenty-three patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no
skill-carrying agent of the platform boots, the atelet scheduling knobs, the keep policy on the CRD chart's
Expand Down
19 changes: 18 additions & 1 deletion cmd/ateom-gvisor/durable.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"path/filepath"
"strings"

"github.com/agent-substrate/substrate/internal/apierror"
"github.com/agent-substrate/substrate/internal/proto/ateompb"
"github.com/agent-substrate/substrate/internal/tarutil"
)
Expand All @@ -42,6 +43,17 @@ func durableTarFile(volumeName string) (string, error) {
return "durable-dir-" + volumeName + ".tar", nil
}

// legacyDurableTarFile is the single tar of every durable-dir volume that
// snapshots held before the one-tar-per-volume layout. A checkpoint never
// writes it any more, so its presence marks a snapshot in that layout.
const legacyDurableTarFile = "durable-dir.tar"

// errLegacySnapshotLayout refuses a restore from a snapshot in the layout
// before one tar per durable-dir volume: its data is in legacyDurableTarFile,
// which is never extracted, so restoring it would bring the actor up with
// empty volumes.
var errLegacySnapshotLayout = errors.New("snapshot written before the per-volume layout; re-snapshot from a running actor or reset the golden snapshot")

// hasDurableVolumes reports whether any container mounts a durable-dir volume.
func hasDurableVolumes(containers []*ateompb.Container) bool {
for _, c := range containers {
Expand Down Expand Up @@ -79,8 +91,13 @@ func tarDurableVolumes(ctx context.Context, dir, checkpointDir string, volumes [

// untarDurableVolumes restores each durable-dir volume from the snapshot into
// its directory under dir, which atelet has already created, empty. A volume
// with no tar in the snapshot (added to the template since) stays empty.
// with no tar in the snapshot (added to the template since) stays empty. A
// snapshot in the layout before one tar per volume is refused with
// errLegacySnapshotLayout.
func untarDurableVolumes(dir, snapshotDir string, volumes []string) error {
if _, err := os.Stat(filepath.Join(snapshotDir, legacyDurableTarFile)); err == nil {
return apierror.FailedPrecondition("%w", errLegacySnapshotLayout)
}
for _, vol := range volumes {
name, err := durableTarFile(vol)
if err != nil {
Expand Down
49 changes: 49 additions & 0 deletions cmd/ateom-gvisor/durable_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,16 @@
package main

import (
"archive/tar"
"errors"
"os"
"path/filepath"
"slices"
"testing"

"google.golang.org/grpc/codes"

"github.com/agent-substrate/substrate/internal/apierror"
"github.com/agent-substrate/substrate/internal/tarutil"
)

Expand Down Expand Up @@ -120,6 +125,50 @@ func TestUntarDurableVolumesMissingTarIsEmpty(t *testing.T) {
}
}

// writeLegacyDurableTar writes the snapshot file of the layout before one tar
// per volume: a single durable-dir.tar whose entries are <volume>/... .
func writeLegacyDurableTar(t *testing.T, snapshotDir string, files map[string]string) {
t.Helper()
f, err := os.Create(filepath.Join(snapshotDir, legacyDurableTarFile))
if err != nil {
t.Fatal(err)
}
defer f.Close()
tw := tar.NewWriter(f)
for name, body := range files {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(body)), Typeflag: tar.TypeReg}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write([]byte(body)); err != nil {
t.Fatal(err)
}
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
}

// A snapshot in the layout before one tar per volume holds the actor's data in
// durable-dir.tar, which a restore never extracts: it is refused instead of
// restoring the volumes empty.
func TestUntarDurableVolumesRefusesLegacyLayout(t *testing.T) {
dst, snap := t.TempDir(), t.TempDir()
mkVolumeDirs(t, dst, "data")
writeLegacyDurableTar(t, snap, map[string]string{"data/app.db": "state"})

err := untarDurableVolumes(dst, snap, []string{"data"})
if !errors.Is(err, errLegacySnapshotLayout) {
t.Fatalf("untarDurableVolumes = %v; want %v", err, errLegacySnapshotLayout)
}
if got := apierror.Code(err); got != codes.FailedPrecondition {
t.Errorf("code = %v; want %v", got, codes.FailedPrecondition)
}
entries, rerr := os.ReadDir(filepath.Join(dst, "data"))
if rerr != nil || len(entries) != 0 {
t.Errorf("data: ReadDir = %v, %v; want it untouched", entries, rerr)
}
}

func TestDurableTarFileRejectsBadNames(t *testing.T) {
for _, v := range []string{"", ".", "..", "a/b", "/abs", "../x"} {
if _, err := durableTarFile(v); err == nil {
Expand Down
Loading