Repository navigation
ateworker-microvm: serve shared snapshot restores as hard links instead of copies #2948
Description
Summary
The snapshot cache (#2796) serves a shared snapshot's files to a gVisor restore as hard links into the cached entry. A micro-VM restore gets private, hole-preserving copies (GetFileCopyTo rather than GetFileTo), so every micro-VM restore from a tag pays a full copy of the memory image, per actor, even when the bytes are already on the node.
Correctness is not the reason. ateworker-microvm already replaces config.json by writing a temp file and renaming it over (cmd/ateworker-microvm/restore.go, rewriteSnapshotPaths), and MergeDeltaIntoBase (cmd/ateworker-microvm/internal/ch/merge.go) checks the memory file's link count and falls back to MergeSparseOverlay when the inode is shared. A link would not corrupt the cache. It would cost more than it saves, for three reasons:
- Pinning. An on-demand restore keeps
restore-state/memory-rangesopen for the VM's whole running life. A link from there keeps the cache's entry atst_nlink > 1for as long as the VM runs, so the eviction loop can never take it, however old or large. - Slow suspends. With the inode shared, every suspend leaves the in-place delta merge, O(delta), for
MergeSparseOverlay, O(working set), a full copy of the memory image at every suspend instead of once at restore. - Page cache.
dropActorSnapshotCacheAsync(cmd/ateworker-microvm/snapshotcache.go) callsPOSIX_FADV_DONTNEEDon the restore's.tarfiles, and onmemory-rangesfor eager restores. On a shared inode that drops the cached pages for every other restore of the snapshot on the node, not just this actor's.
What it would take
To serve micro-VM restores as links, each of the three needs an answer in the worker, not in atelet:
- a way to release the link, or move to a private inode, before the VM's long-lived use begins, so the entry is not pinned for the VM's life;
- a delta merge that stays O(delta) on a shared base, or a copy-on-first-suspend that is cheaper than copy-at-restore for actors that suspend rarely;
- page-cache eviction scoped to the actor's own inode, or skipped for files the cache shares.
With those, cacheModeFor in cmd/atelet/snapshotcache.go returns cacheModeLink for the micro-VM class too, and cacheModeCopy remains only as the fallback for a cache on another mount than the restore directories (EXDEV). The copy mode's tests stay for that path.
Why not now
Each point is a change to ateworker-microvm's restore and suspend paths with its own tests, and the first two trade restore cost against suspend cost in a way that depends on how often actors suspend. #2796 ships copies for micro-VM, one download per snapshot per node and a copy per restore; this issue is the follow-up that could earn the links.
Activity
[-]ateworker-microvm: stop writing restore files in place so the snapshot cache can serve hard links[/-][+]ateworker-microvm: serve shared snapshot restores as hard links instead of copies[/+]atelet: serve shared snapshot restores from the node-local file cache (…