Skip to content

ateworker-microvm: serve shared snapshot restores as hard links instead of copies #2948

Open

Description

Summary

The snapshot cache (#2796) serves a shared snapshot's files to a gVisor restore as hard links into the cached entry. A micro-VM restore gets private, hole-preserving copies (GetFileCopyTo rather than GetFileTo), so every micro-VM restore from a tag pays a full copy of the memory image, per actor, even when the bytes are already on the node.

Correctness is not the reason. ateworker-microvm already replaces config.json by writing a temp file and renaming it over (cmd/ateworker-microvm/restore.go, rewriteSnapshotPaths), and MergeDeltaIntoBase (cmd/ateworker-microvm/internal/ch/merge.go) checks the memory file's link count and falls back to MergeSparseOverlay when the inode is shared. A link would not corrupt the cache. It would cost more than it saves, for three reasons:

  1. Pinning. An on-demand restore keeps restore-state/memory-ranges open for the VM's whole running life. A link from there keeps the cache's entry at st_nlink > 1 for as long as the VM runs, so the eviction loop can never take it, however old or large.
  2. Slow suspends. With the inode shared, every suspend leaves the in-place delta merge, O(delta), for MergeSparseOverlay, O(working set), a full copy of the memory image at every suspend instead of once at restore.
  3. Page cache. dropActorSnapshotCacheAsync (cmd/ateworker-microvm/snapshotcache.go) calls POSIX_FADV_DONTNEED on the restore's .tar files, and on memory-ranges for eager restores. On a shared inode that drops the cached pages for every other restore of the snapshot on the node, not just this actor's.

What it would take

To serve micro-VM restores as links, each of the three needs an answer in the worker, not in atelet:

  • a way to release the link, or move to a private inode, before the VM's long-lived use begins, so the entry is not pinned for the VM's life;
  • a delta merge that stays O(delta) on a shared base, or a copy-on-first-suspend that is cheaper than copy-at-restore for actors that suspend rarely;
  • page-cache eviction scoped to the actor's own inode, or skipped for files the cache shares.

With those, cacheModeFor in cmd/atelet/snapshotcache.go returns cacheModeLink for the micro-VM class too, and cacheModeCopy remains only as the fallback for a cache on another mount than the restore directories (EXDEV). The copy mode's tests stay for that path.

Why not now

Each point is a change to ateworker-microvm's restore and suspend paths with its own tests, and the first two trade restore cost against suspend cost in a way that depends on how often actors suspend. #2796 ships copies for micro-VM, one download per snapshot per node and a copy per restore; this issue is the follow-up that could earn the links.

Activity

  1. changed the title [-]ateworker-microvm: stop writing restore files in place so the snapshot cache can serve hard links[/-] [+]ateworker-microvm: serve shared snapshot restores as hard links instead of copies[/+] on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions