Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions FORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ Everything on `giantswarm` that is not in the pin (`git log v0.4.0-alpha1..giant
| The Kubernetes credential provider mints Google access tokens (`ate-secret://google-access-token.k8s.io/default/<namespace>/<secret>/<key>`: the entry must hold a service account key; the provider signs its JWT assertion, exchanges it at the key's https `token_uri` for a `cloud-platform` token, caches the token per key while at least fifteen minutes remain, so the gateway's five-minute cache never serves an expired one, and returns the token, never the key; the chart and the kustomize component route both authorities, bundled names of `substrate.egressCredentialProviders` that `additionalProviders` cannot replace, on the HTTP and HTTPS egress listeners to the one provider Deployment) | Vertex AI takes OAuth 2.0 access tokens only, and minting one means signing with the key, which the egress gateway cannot do and an actor must not hold: no kagent agent with a Vertex `ModelConfig` (Gemini or Anthropic on Vertex AI) ran on Substrate ([#142](https://github.com/giantswarm/substrate/issues/142); giantswarm/kagent-upstream#178; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 119) | [#250](https://github.com/giantswarm/substrate/pull/250): `git cherry-pick -x` of the upstream-ready branch's commit, which is kagent-dev/substrate's `5ac16e7c` adapted to the `k8s.io/default` URI grammar (upstream's commit predates it and names the authority `google-access-token.kubernetes.io`) | [kagent-dev/substrate#47](https://github.com/kagent-dev/substrate/pull/47) (open); the adapted commit on the pin is branch [`upstream/google-access-token-provider`](https://github.com/giantswarm/substrate/tree/upstream/google-access-token-provider) (`4453a6f4`), offered to #47 |
| The egress gateway's ephemeral storage is bounded (`atenetEgress.resources` for the `agentgateway` container and `atenetEgress.extProc.resources`, ephemeral-storage requests `16Mi` and limits `256Mi` by default, CPU and memory left to the operator; `atenetEgress.drainSignal.sizeLimit`, `16Mi` on the `drain-signal` emptyDir the ext-proc writes its drain marker to; `null` renders the field as before; `charts/substrate/tests/atenet_egress_ephemeral_storage_test.yaml`; README rows) | the chart rendered both containers without `resources` and the emptyDir without a `sizeLimit`, so a cluster policy that requires bounded ephemeral storage for a container mounting an emptyDir (Kyverno's `require-emptydir-requests-and-limits`) reported the egress gateway on every install ([giantswarm/agent-platform#880](https://github.com/giantswarm/agent-platform/issues/880)) | [#255](https://github.com/giantswarm/substrate/pull/255) (`fix(chart): bound the egress gateway's ephemeral storage`) | to file: upstream's `atenet-egress.yaml` renders the same unbounded shape and no upstream issue covers it; queued in the upstream engagement list |
| Require `golang.org/x/net` v0.61.0 in every module that had it below (`go.mod`, `hack/tools/{code-generator,controller-gen,go-licenses,ko}`, `internal/plugins/gcp-secret-manager`, `tools/apitool`; `go get golang.org/x/net@v0.61.0 && go mod tidy` in each, then `go mod vendor` for the root module the CircleCI builds compile with `-mod=vendor`; `x/sync`, `x/sys`, `x/term`, `x/text` and `x/tools` move along, and tidy drops the stale indirect requirements `code-generator`'s module file still listed) | CVE-2026-97032 and CVE-2026-78663, two HTTP/2 vulnerabilities in `x/net` below v0.61.0; the platform's Go repositories move together, their CircleCI nancy step refusing the older module | [#252](https://github.com/giantswarm/substrate/pull/252) (`fix(deps): bump golang.org/x/net to v0.61.0 for two HTTP/2 CVEs`) | not for upstream as such: changes versions only and falls away at the re-pin onto the first upstream commit that requires `x/net` ≥ v0.61.0 (upstream's dependency automation moves it) |
| The snapshot takes one rootfs-upper tar per container and one tar per durable volume; the restore creates the layout before it extracts into it, so a snapshot no longer decides which host paths are mounted or deleted; `tarutil` writes and reads the tree through an `os.Root` (`fs.WalkDir` over `root.FS()`, `root.Lstat`, `root.Readlink`), component by component, never by the full host path (`internal/tarutil`, `cmd/ateom-gvisor/durable.go`, `cmd/ateom-microvm/rootfsupper.go`) | a workspace tree deeper than `PATH_MAX` crashed the actor at its checkpoint and lost the session on every harness: the tar walked and stat'ed each entry by its full path, which the kernel refuses with `ENAMETOOLONG`; the same tar named the host paths a restore mounted and deleted (upstream's agent-substrate/substrate#2083; giantswarm/giantswarm#38054; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 165). **Snapshots written before it do not resume after it**: the roll needs the snapshot reset, golden and actor snapshots | [#259](https://github.com/giantswarm/substrate/pull/259): `git cherry-pick -x` of upstream's squash commit `893b1edb`, clean | [agent-substrate/substrate#2110](https://github.com/agent-substrate/substrate/pull/2110) (merged 2026-10-06, after the pin). Falls away at the re-pin onto the first release that carries it |
| A snapshot extract creates only `0:0` character devices (overlay whiteouts) and sets only `user.*` xattrs and the path-based overlay attributes (opaque, redirect, impure); `Create` skips other device nodes; on the micro-VM host the merged rootfs overlay and the binds into the virtio-fs share are `nosuid,nodev` (`setNosuidNodev`, recursive) | snapshot tars are extracted as root on the host, and the extract made any device node and set any `trusted.overlay.*` xattr the archive named; a stray device could make a snapshot unrestorable (upstream's agent-substrate/substrate#2090; giantswarm/giantswarm#38054; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 166) | [#259](https://github.com/giantswarm/substrate/pull/259): `git cherry-pick -x` of upstream's squash commit `8da76493`. Adapted to the line: it still mounts through `mount(8)`, so `nosuid,nodev` joins the overlay's `-o` options and `BindIntoShare` calls `setNosuidNodev` after the rbind; the hunk's `RemountReadOnly` and `Unmount` tests cover helpers the line does not have and are left out | [agent-substrate/substrate#2111](https://github.com/agent-substrate/substrate/pull/2111) (merged 2026-10-06, after the pin). Falls away at the re-pin onto the first release that carries it |

Twenty-three patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no
skill-carrying agent of the platform boots, the atelet scheduling knobs, the keep policy on the CRD chart's
Expand Down
85 changes: 64 additions & 21 deletions cmd/ateom-gvisor/durable.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ package main

import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
Expand All @@ -27,10 +29,18 @@ import (
"github.com/agent-substrate/substrate/internal/tarutil"
)

// durableTarFile is the snapshot file holding the tar of the actor's durable-dir
// volumes. Its entries are <volumeName>/... relative to
// ActorDirs.durable_dir_volume_mounts_dir, so extraction restores the same layout.
const durableTarFile = "durable-dir.tar"
// durableTarFile is the snapshot file holding the tar of one durable-dir
// volume: the contents of <volumeName> under
// ActorDirs.durable_dir_volume_mounts_dir, plus a root entry for the volume
// directory's own metadata. The volume directory itself comes from atelet,
// never from the snapshot: runsc bind-mounts it by path, so a symlink planted
// there would expose whatever it points at to the sandbox.
func durableTarFile(volumeName string) (string, error) {
if volumeName == "" || volumeName == "." || strings.Contains(volumeName, "/") || !filepath.IsLocal(volumeName) {
return "", fmt.Errorf("invalid durable-dir volume name %q", volumeName)
}
return "durable-dir-" + volumeName + ".tar", nil
}

// hasDurableVolumes reports whether any container mounts a durable-dir volume.
func hasDurableVolumes(containers []*ateompb.Container) bool {
Expand All @@ -42,36 +52,69 @@ func hasDurableVolumes(containers []*ateompb.Container) bool {
return false
}

// tarDurableVolumes archives the actor's durable-dir volumes (dir) into the
// checkpoint directory. The caller must have paused the guest first.
// tarDurableVolumes archives each durable-dir volume under dir into the
// checkpoint directory, one tar per volume, and returns the file names. The
// caller must have paused the guest first.
//
// Sockets the workload left behind and gVisor internal files (.gvisor.*) are
// skipped rather than archived.
func tarDurableVolumes(ctx context.Context, dir, checkpointDir string) error {
func tarDurableVolumes(ctx context.Context, dir, checkpointDir string, volumes []string) ([]string, error) {
skip := func(rel string) bool {
base := filepath.Base(rel)
return strings.HasPrefix(base, ".gvisor.")
}
if err := tarutil.CreateFiltered(ctx, filepath.Join(checkpointDir, durableTarFile), dir, skip); err != nil {
return fmt.Errorf("while archiving durable-dir volumes from %q: %w", dir, err)
var files []string
for _, vol := range volumes {
name, err := durableTarFile(vol)
if err != nil {
return nil, err
}
if err := tarutil.CreateFilteredWithRoot(ctx, filepath.Join(checkpointDir, name), filepath.Join(dir, vol), skip); err != nil {
return nil, fmt.Errorf("while archiving durable-dir volume %q: %w", vol, err)
}
files = append(files, name)
}
return nil
return files, nil
}

// untarDurableVolumes restores the durable-dir volumes from a snapshot into the
// actor's host directory (dir, which atelet has already created, empty).
func untarDurableVolumes(dir, snapshotDir string) error {
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("while creating durable-dir volumes dir %q: %w", dir, err)
// untarDurableVolumes restores each durable-dir volume from the snapshot into
// its directory under dir, which atelet has already created, empty. A volume
// with no tar in the snapshot (added to the template since) stays empty.
func untarDurableVolumes(dir, snapshotDir string, volumes []string) error {
for _, vol := range volumes {
name, err := durableTarFile(vol)
if err != nil {
return err
}
tarPath := filepath.Join(snapshotDir, name)
if _, err := os.Stat(tarPath); errors.Is(err, fs.ErrNotExist) {
continue
}
volDir := filepath.Join(dir, vol)
if err := os.MkdirAll(volDir, 0o700); err != nil {
return fmt.Errorf("while creating durable-dir volume dir %q: %w", volDir, err)
}
if err := tarutil.Extract(tarPath, volDir); err != nil {
return fmt.Errorf("while restoring durable-dir volume %q: %w", vol, err)
}
removeGVisorFiles(volDir)
}
if err := tarutil.Extract(filepath.Join(snapshotDir, durableTarFile), dir); err != nil {
return fmt.Errorf("while restoring durable-dir volumes into %q: %w", dir, err)
return nil
}

// removeGVisorFiles deletes gVisor internal files (.gvisor.*) under dir,
// best-effort, through an os.Root so the restored tree's symlinks are never
// followed.
func removeGVisorFiles(dir string) {
root, err := os.OpenRoot(dir)
if err != nil {
return
}
_ = filepath.Walk(dir, func(p string, info os.FileInfo, err error) error {
if err == nil && !info.IsDir() && strings.HasPrefix(info.Name(), ".gvisor.") {
_ = os.Remove(p)
defer root.Close()
_ = fs.WalkDir(root.FS(), ".", func(rel string, d fs.DirEntry, err error) error {
if err == nil && !d.IsDir() && strings.HasPrefix(d.Name(), ".gvisor.") {
_ = root.Remove(rel)
}
return nil
})
return nil
}
129 changes: 129 additions & 0 deletions cmd/ateom-gvisor/durable_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
//go:build linux

// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"os"
"path/filepath"
"slices"
"testing"

"github.com/agent-substrate/substrate/internal/tarutil"
)

// mkVolumeDirs creates the per-volume directories atelet prepares.
func mkVolumeDirs(t *testing.T, dir string, vols ...string) {
t.Helper()
for _, v := range vols {
if err := os.MkdirAll(filepath.Join(dir, v), 0o700); err != nil {
t.Fatal(err)
}
}
}

func TestDurableVolumesRoundTrip(t *testing.T) {
src := t.TempDir()
mkVolumeDirs(t, src, "data", "cache")
for rel, content := range map[string]string{
"data/notes.txt": "kept",
"data/.gvisor.filestat": "internal",
"cache/c.bin": "cached",
} {
if err := os.WriteFile(filepath.Join(src, rel), []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// The volume root's own mode is the mount point's, so it must survive.
if err := os.Chmod(filepath.Join(src, "data"), 0o750); err != nil {
t.Fatal(err)
}
checkpointDir := t.TempDir()
files, err := tarDurableVolumes(t.Context(), src, checkpointDir, []string{"cache", "data"})
if err != nil {
t.Fatalf("tarDurableVolumes: %v", err)
}
if want := []string{"durable-dir-cache.tar", "durable-dir-data.tar"}; !slices.Equal(files, want) {
t.Errorf("tarDurableVolumes files = %v, want %v", files, want)
}

dst := t.TempDir()
mkVolumeDirs(t, dst, "data", "cache")
if err := untarDurableVolumes(dst, checkpointDir, []string{"cache", "data"}); err != nil {
t.Fatalf("untarDurableVolumes: %v", err)
}
for rel, want := range map[string]string{"data/notes.txt": "kept", "cache/c.bin": "cached"} {
if got, err := os.ReadFile(filepath.Join(dst, rel)); err != nil || string(got) != want {
t.Errorf("restored %q = %q, %v; want %q", rel, got, err, want)
}
}
if _, err := os.Lstat(filepath.Join(dst, "data/.gvisor.filestat")); !os.IsNotExist(err) {
t.Errorf(".gvisor.filestat: Lstat = %v; want it skipped", err)
}
if fi, err := os.Stat(filepath.Join(dst, "data")); err != nil || fi.Mode().Perm() != 0o750 {
t.Errorf("restored volume dir: Stat = %v, %v; want mode 0750", fi, err)
}
}

// runsc bind-mounts <dir>/<volume> by path, so the snapshot must not be able
// to replace that directory with a symlink out of the actor's tree.
func TestUntarDurableVolumesCannotPlantVolumeDir(t *testing.T) {
victim := t.TempDir()
planted := t.TempDir()
if err := os.Symlink(victim, filepath.Join(planted, "data")); err != nil {
t.Fatal(err)
}
snapshotDir := t.TempDir()
// An entry named after the volume, as the old single-tar layout had.
if err := tarutil.Create(t.Context(), filepath.Join(snapshotDir, "durable-dir-data.tar"), planted); err != nil {
t.Fatal(err)
}

dst := t.TempDir()
mkVolumeDirs(t, dst, "data")
if err := untarDurableVolumes(dst, snapshotDir, []string{"data"}); err != nil {
t.Fatalf("untarDurableVolumes: %v", err)
}
if fi, err := os.Lstat(filepath.Join(dst, "data")); err != nil || !fi.IsDir() {
t.Errorf("data: Lstat = %v, %v; want a real directory", fi, err)
}
entries, err := os.ReadDir(victim)
if err != nil || len(entries) != 0 {
t.Errorf("victim: ReadDir = %v, %v; want it untouched", entries, err)
}
}

// A volume added to the template after the snapshot has no tar: it restores
// empty rather than failing.
func TestUntarDurableVolumesMissingTarIsEmpty(t *testing.T) {
dst := t.TempDir()
mkVolumeDirs(t, dst, "new")
if err := untarDurableVolumes(dst, t.TempDir(), []string{"new"}); err != nil {
t.Fatalf("untarDurableVolumes: %v", err)
}
entries, err := os.ReadDir(filepath.Join(dst, "new"))
if err != nil || len(entries) != 0 {
t.Errorf("new: ReadDir = %v, %v; want an empty directory", entries, err)
}
}

func TestDurableTarFileRejectsBadNames(t *testing.T) {
for _, v := range []string{"", ".", "..", "a/b", "/abs", "../x"} {
if _, err := durableTarFile(v); err == nil {
t.Errorf("durableTarFile(%q) = nil error, want one", v)
}
}
}
17 changes: 9 additions & 8 deletions cmd/ateom-gvisor/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,8 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec
return nil, fmt.Errorf("while creating checkpoint directory: %w", err)
}

// durableFiles are the durable-dir tars written below: the DATA subset.
var durableFiles []string
// Always take durable-dir snapshot if at least one container has a durable-dir volume mount.
// TODO(dberkov): this is a temporary workaround until gVisor supports taking durable-dir snapshots in a single request with the process snapshot.
switch req.GetScope() {
Expand All @@ -693,7 +695,8 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec
if err := rcmd.cmdPause(ctx, ocispec.PauseContainer); err != nil {
return nil, fmt.Errorf("while pausing pause container: %w", err)
}
tarErr := tarDurableVolumes(ctx, req.GetActorDirs().GetDurableDirVolumeMountsDir(), checkpointPath)
var tarErr error
durableFiles, tarErr = tarDurableVolumes(ctx, req.GetActorDirs().GetDurableDirVolumeMountsDir(), checkpointPath, durableVolumeNames(req.GetSpec()))
// Undoing our own pause must not depend on the caller's context:
// tarutil does not check ctx, so a deadline expiring mid-tar would
// fail the resume instantly and leave the sandbox paused forever.
Expand All @@ -712,7 +715,9 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec
return nil, fmt.Errorf("while checkpointing pause: %w", err)
}
if hasDurableVolumes(req.GetSpec().GetContainers()) {
if err := tarDurableVolumes(ctx, req.GetActorDirs().GetDurableDirVolumeMountsDir(), checkpointPath); err != nil {
var err error
durableFiles, err = tarDurableVolumes(ctx, req.GetActorDirs().GetDurableDirVolumeMountsDir(), checkpointPath, durableVolumeNames(req.GetSpec()))
if err != nil {
return nil, fmt.Errorf("while archiving durable-dir volumes: %w", err)
}
}
Expand All @@ -739,11 +744,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec

s.actorLogger.EmitLifecycleLog(ctx, "Actor checkpointed", attribution)

resp := &ateompb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles}
if slices.Contains(snapshotFiles, durableTarFile) {
resp.DataSnapshotFiles = []string{durableTarFile}
}
return resp, nil
return &ateompb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil
}

// listSnapshotFiles returns the (relative) names of regular files directly under
Expand Down Expand Up @@ -892,7 +893,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore
checkpointDir := req.GetActorDirs().GetRestoreDir()

if hasDurableVolumes(req.GetSpec().GetContainers()) {
if err := untarDurableVolumes(req.GetActorDirs().GetDurableDirVolumeMountsDir(), checkpointDir); err != nil {
if err := untarDurableVolumes(req.GetActorDirs().GetDurableDirVolumeMountsDir(), checkpointDir, durableVolumeNames(req.GetSpec())); err != nil {
return nil, fmt.Errorf("while restoring durable-dir volumes: %w", err)
}
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/ateom-microvm/checkpoint.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec
if scope == ateompb.SnapshotScope_SNAPSHOT_SCOPE_FULL {
g.Go(func() error {
t := time.Now()
err := tarRootfsUpper(gctx, rootfsUpperDir(actorDirs), checkpointDir)
err := tarRootfsUpper(gctx, rootfsUpperDir(actorDirs), checkpointDir, containerNames(req.GetSpec().GetContainers()))
dUpper = time.Since(t)
return err
})
Expand Down
Loading
Loading