Repository navigation
atelet: one eviction loop for every node-local file cache, with one metric family - #2889
Merged
Merged
Conversation
Dmitry Berkovich (dberkov)
force-pushed
the
cachegc-driver
branch
2 times, most recently
from
October 19, 2026 18:36
379de80 to
7af8fb0
Compare
…etric family The sandbox asset cache ran its own copy of the eviction loop, with its own flags, instruments and shortfall backoff, and the snapshot cache was about to get a third copy beside the image cache's. The two are the same filecache store and the same policy: a byte budget, least recently used first, down to 90% of the budget, never an entry a consumer links. Move the loop into cmd/atelet/internal/filecachegc. A cache registers its three flags under its own prefix with NewFlags, so the asset cache's flags do not change, and starts its loop with Start. The instruments are created once per process and shared: ate.filecache.size, ate.filecache.evictions and ate.filecache.gc.passes, each labeled by ate.filecache.name, replace the ate.sandboxassetcache.* instruments, and a second cache is a second value of the label rather than three more registry entries. The image cache keeps its own loop: it measures layers against volume watermarks, not entries against a budget.
NewFlags registered on the global command line, so a second call with one prefix panicked with a redefined flag; the package's own flag test could not run twice in a process. Take the flag set as a parameter: atelet passes the command line, a test passes a fresh set.
The three code anchors still named cachegc.go after the package's file became filecachegc.go. Weaver does not check anchors, so nothing failed.
…e uses Start validated nothing, so each caller had to remember Validate first, and a forgotten call with a negative period would have left the loop off without a line in the log. Start now validates and returns the error; main's separate call goes. The package exported its loop, its constructor, its target formula and its metric names although only Start used them and the tests build the loop directly. The API is now NewFlags, NewInstruments and Start, with the Flags, Instruments and Store types they take.
… family The sandbox asset cache page and the e2e metrics suite named the ate.sandboxassetcache.* instruments. They are ate.filecache.* now, labeled by ate.filecache.name; the suite asserts the size under its new name, and the page says where the label comes from.
Dmitry Berkovich (dberkov)
force-pushed
the
cachegc-driver
branch
from
October 19, 2026 21:23
2866375 to
564a9d5
Compare
Zoe Zhao (zoez7)
enabled auto-merge
October 19, 2026 21:33
Zoe Zhao (zoez7)
approved these changes
Oct 19, 2026
Haowei Cai (Roy) (roycaihw)
pushed a commit
to roycaihw/substrate
that referenced
this pull request
Oct 20, 2026
…agent-substrate#2796) Replaces agent-substrate#1551, which was four commits from September that no longer applied: the restore source message, the download path and the module path all changed underneath it. Same design, ported onto today's main, after agent-substrate#2756 and agent-substrate#2889, in four commits, plus three from review: the sharing derived from the snapshot's owner, the gVisor worker evicting a shared file's pages before unlinking it, and the manifest read through the cache. Milestone M2 of the node-local artifact cache from agent-substrate#690: an actor restoring from a shared snapshot, the one a tag holds and every actor created from it starts on, gets its files from a node-level cache instead of its own download. **Sharing is the control plane's knowledge.** `ObjectStorage` gains `SnapshotSharing`, the property rather than the snapshot's kind, so new shared snapshot kinds need no atelet change. ate-api-server sets it on Restore from the snapshot's recorded owner: SHARED for a tag's snapshot, borrowed at creation and immutable, PRIVATE for the actor's own, which its next suspend replaces, and for an unrecorded owner; the local-copy fallback object carries it too. Neither side infers it from the URI. **The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`, behind `--snapshot-cache-dir` (empty disables), `--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened at startup with crash debris swept before atelet serves. The directory must be under the node state dir: the object-store plugin writes only there, so a cache elsewhere could never be filled, and startup refuses it. Every flag is validated whether or not the cache is enabled, and the help text says which are read only while it is. The image cache's base-path check moved to `nodepath.UnderBasePath`, which both caches use. **Eviction.** The cache runs on the eviction loop every file cache on the node shares, `cmd/atelet/internal/filecachegc` from agent-substrate#2889: `--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and `--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least recently used first, never an entry a restore still links. Its size, evictions and passes appear under the existing `ate.filecache.*` instruments as a second value of `ate.filecache.name`, `snapshot`; no new instrument and no new loop. Every flag is validated whether or not the cache is enabled, as the other flags in atelet are, and the help text says which ones are read only while it is; a disabled cache starts no loop and reports no size. **The restore path.** A restore whose durable snapshot is declared SHARED stages each file through the cache; concurrent restores of one snapshot share a single download, later ones read disk. How a hit is served depends on the sandbox class the request's runtime resolved to. ateworker-gvisor gets hard links. Once the restore's page loading is done the worker evicts the memory and filesystem images from the page cache and unlinks them from its restore dir, keeping only the small descriptors linked: a file with another link is treated like a preserved restore dir, since unlinking it would free nothing and would leave its pages charged to the actor's cgroup, the case agent-substrate#1917 fixed. So a link protects an image's entry through the restore and the min age, not for the actor's life; an entry evicted later is still read to the end through the descriptors runsc holds open, and the next restore reads the images from local disk. ateworker-microvm gets private hole-preserving copies, not because a link would corrupt the cache, it replaces `config.json` by rename and its delta merge copies when the inode is shared, but because a link would cost more than it saves: an on-demand restore keeps `memory-ranges` open for the VM's life, which would pin the entry that long and turn every suspend's in-place merge into a full copy, and the worker's page-cache eviction would drop the cached pages for every other restore; agent-substrate#2948 tracks what links would take. EXDEV on a link, from a cache bind-mounted apart from the restore dirs, degrades to a copy, warned once per process. The manifest of a shared snapshot is read through the cache too, so a restore that finds the snapshot on the node makes no request to object storage; a private snapshot's manifest is read fresh, as the commit marker it is. The cache key is the object's address without its query, so a signed URL's credential reaches neither the key, the cache's errors, `meta.json` nor the logs; and a restore that fails after staging removes its restore-state dir, so the links do not hold the cache's entries until the next lifecycle call. The fetch goes through the object-store plugin as the direct path does, one file into a scratch dir on the cache's filesystem; because the cache then writes the restore dir by path rather than through the plugin's `os.Root`, a symlink waiting there fails the restore instead of being followed. **Not here.** Private snapshots stay uncached by design. Links for ateworker-microvm are agent-substrate#2948, a worker change. The open-and-sweep code the two file caches share in shape is still duplicated, since the snapshot cache can be disabled and degrades to copies across mounts where the asset cache refuses to start. **Tests.** ate-api functional: a golden restore declares SHARED, the actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test from agent-substrate#2852 compares the whole Restore snapshot, so its fixture now puts the snapshot under the actor's own UID, and it expects PRIVATE for the downloaded copy and nothing for a local one. atelet: flag validation, enabled or not, including the eviction flags, the fetch timeout, and a directory outside the node state dir; two shared manifest reads download once and a private one reads fresh; open disabled, creating its root, sweeping debris; the sharing gate and the class modes; through the plugin-backed test atelet, two link restores share one download and one inode and the file is read-only, copy mode gives private inodes and survives a consumer's mutation, a nil cache downloads, a missing file's error reaches the caller with nothing left behind, a plugin error through the cached modes keeps its code and leaks no credential from a signed URI, a restore that fails at the worker after staging leaves no file in restore-state and no link on the cache's entry, four concurrent restores download once, a file in the checkpoint subdirectory lands there, and symlinks in place of the file or the subdirectory fail; on Linux, a restore dir on another mount gets a copy with no second download and one warning for two restores. ateworker-gvisor, on Linux: staged images hard-linked elsewhere are evicted after the restore wait and only then unlinked, with the other links intact. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2756, so that #2796 can give the snapshot cache a budget without a third copy of the eviction loop.
What moves. The sandbox asset cache's eviction loop, flags, instruments, target formula and shortfall backoff move from
cmd/atelet/sandbox_asset_gc.gointo a package,cmd/atelet/internal/filecachegc, written against the filecache store. The loop's behavior is unchanged: immediate first pass, serialized passes, panic recovery, dry run, budget with a 90% low water, shortfall warnings that back off to a periodic reminder.How a cache uses it.
filecachegc.NewFlags("sandbox-asset-cache", "sandbox asset cache")registers--sandbox-asset-cache-gc-period,--sandbox-asset-cache-max-bytesand--sandbox-asset-cache-gc-dry-run, so the asset cache's flags are the same as before. The snapshot cache will call it once with its own prefix.filecachegc.NewInstruments(meter)creates the instruments once per process.filecachegc.Start(ctx, name, store, flags, instruments)reports the cache's size, warns once when there is no budget, and runs the loop when there is a budget and a period.One metric family.
ate.filecache.size,ate.filecache.evictionsandate.filecache.gc.passes, labeled byate.filecache.name(todaysandbox_asset) and, for passes,ate.filecache.gc.outcome, replaceate.sandboxassetcache.size,.evictionsand.gc.passes. The caches run on one store and one loop, so one set of instruments describes them, and "which cache is over budget on this node" is one query grouped by name. The registry,ateattrand the observability guide are updated. The name isfilecacherather than something broader because the family is about the filecache store; the image cache, which measures layers against volume watermarks, is not a member and keeps its loop.Tests. The asset GC's tests move into the package unchanged in what they prove, plus one that puts two stores on the same instruments and asserts two size series under their names.
Docs and e2e. The sandbox asset cache page from #2792 and the e2e metrics suite follow the rename in the last commit: the suite asserts
ate_filecache_size, and the page's metrics table and runbook use the new names and say where the label comes from.Follow-up. #2796 adopts the driver for the snapshot cache and adds the
snapshotlabel value.🤖 Generated with Claude Code