atelet: node-local cache for golden snapshots (OnGolden data resumes re-download the golden every time) #690

Activity

1. Max Smythe (maxsmythe)

   added

   area/storage

   on Aug 4, 2026
2. Aditya Shantanu (aditya-shantanu)

   added

   kind/featureAn enhancement / feature request or implementationAn enhancement / feature request or implementation

   on Aug 12, 2026
3. dberkov commented on Sep 4, 2026

   

   Dmitry Berkovich (dberkov)

   on Sep 4, 2026

   CollaboratorAuthor

   More actions

   Design proposal: filecache — a generic node-local artifact cache in atelet

   Full design written up as docs/dev/filecache-design.md (PR to follow); summary below. It covers this issue's golden-snapshot case and, in a later milestone, folds the sandbox-asset cache (fetchAsset/fetchGVisorRelease) onto the same library — that path has the same check-then-download race and never deletes anything.

   Shape

   A small library at cmd/atelet/internal/filecache, porting the mechanics internal/imagecache already proved for OCI layers (singleflight dedup, mtime-LRU, two-phase retire) to a generic key→file cache. Two independent store instances: a golden store and (M3) an assets store, each with its own root and byte budget.

   func (s *Store) GetFileTo(ctx, key Key, dst string, fetch FileFetcher) error  // golden files
   func (s *Store) GetFile(ctx, key Key, fetch FileFetcher) (Entry, error)       // in-place single files
   func (s *Store) GetDir(ctx, key Key, fetch DirFetcher) (Entry, error)         // extracted trees (gVisor release)

   Fetchers are injected (the store owns dedup/publication/GC; download policy — zstd+sparse for goldens via ategcs.FetchLocalFileFromGCSWithZstd, sha256+auth-fallback for assets — stays at the call site). Keys are identities, not addresses: URIKey(goldenURI, file) for goldens, SHA256Key(hex) for content-addressed assets.

   Golden restore path (this issue)

   GetFileTo is atomic get-and-link:

   • Miss: concurrent callers for one key join a single flight (singleflight.DoChan); the download runs on context.WithoutCancel + timeout, so one canceled restore never kills the download other resumes are waiting on. Fetch lands in tmp/, is chmod'd 0444, and is atomically renamed into the cache — crash-safe, never a partial file.
   • Hit: stat + hardlink into the actor's restore dir + mtime touch. N concurrent DATA_ON_GOLDEN resumes (or golden-fallback boots) of one golden = one download; every subsequent restore on the node is link-time.

   The hardlink answers the demand-paging care point from the issue description directly: the actor's link shares the inode, so even if GC evicts the cache entry mid-VM-lifetime, cloud-hypervisor's mapping stays valid until the actor dir is wiped. Eviction can only ever cost a re-download, never break a running actor. Cross-actor mutation is excluded by design assumption (restore files are consumed read-only) and enforced mechanically: cache entries are published 0444, so an in-place write fails with EACCES instead of poisoning the shared copy.

   GC

   Pressure-driven only — age alone never removes a file. Per-store budget (--golden-cache-max-bytes, absolute or % of disk capacity, resolved via statfs at startup), LRU by entry mtime, a per-store min-age guard for the just-published window, and preference for entries with Nlink == 1 (evicting a still-linked entry frees nothing until the actor dir is wiped). Removal is two-phase like imagecache's retire: rename to .rm-* inside the key's singleflight (an in-flight fetch or fresh hit vetoes), bulk RemoveAll outside all locks. Startup sweep reaps tmp/* and .rm-* crash debris. The assets store (M3) additionally gets a root set recomputed each pass from the on-disk actor sandbox records, mirroring imagecache's InUse().

   Design assumptions (stated, not open)

   1. Golden objects are immutable under their URI — matches "immutable once written" in the issue; changed goldens must get new URIs, no generation suffix in the key.
   2. Restore consumes checkpoint files read-only — enforced by the 0444 tripwire.

   Milestones

   • M1 — library, link-out mode only: GetFileTo + GC core + concurrency/crash tests.
   • M2 — golden integration in Restore (downloadExternalCheckpoint/downloadCombinedCheckpoint golden halves, incl. the LOCAL-pause un-pause path, plus the golden manifest.json fetch) + GC driver loop. Exit criterion: two concurrent resumes from one golden on a node → one download.
   • M3 — GetFile/GetDir + root set; migrate sandbox assets and prewarm onto the assets store.
   • M4 — (conditional, metrics-gated) one shared GC coordinator owning the node-wide disk signal across golden/assets/image caches; v1 deliberately reacts only to per-store budgets.
   • M5 — metrics registry entries, e2e under eviction pressure.

   🤖 Generated with Claude Code
4. Dmitry Berkovich (dberkov)

   self-assigned this

   on Sep 11, 2026
5. Eitan Yarmush (EItanya)

   added a commit that references this issue on Sep 11, 2026

   atelet: node-local file cache library (filecache, M1) (#1517)

   ...

   3280b02
6. Zheng Lammerts (zlammerts-svg)

   added this to Substrate Milestones (go/substrate-milestones)on Sep 17, 2026
7. github-project-automation

   moved this to Todo in Substrate Milestones (go/substrate-milestones)on Sep 17, 2026
8. Zheng Lammerts (zlammerts-svg)

   added this to the M3 milestone on Sep 17, 2026
9. a4-a4s1

   added a commit that references this issue on Sep 26, 2026

   ateletpb: carry the Restore base snapshot as a typed source, dropping…

   ...

   1d7ca8c
10. Haowei Cai (Roy) (roycaihw)

    added a commit that references this issue on Oct 20, 2026

    atelet: serve shared snapshot restores from the node-local file cache (…

    ...

    11afdf6
11. dberkov commented on Oct 20, 2026

    

    Dmitry Berkovich (dberkov)

    on Oct 20, 2026

    CollaboratorAuthor

    More actions

    Closing: what this issue asked for, no re-download of a shared snapshot on a node, is on main, under today's names (tags, sharing) rather than the DATA_ON_GOLDEN the issue was written against, which #2105 removed.

    Milestones from the design comment, as landed:

    • M1, the filecache library with link-out and the eviction core: atelet: node-local file cache library (filecache, M1) #1517.
    • M2, shared snapshot restores through the cache, manifest included, on the eviction loop: atelet: serve shared snapshot restores from the node-local file cache #2796, with atelet: one eviction loop for every node-local file cache, with one metric family #2889 for the loop. Two concurrent resumes from one snapshot on a node make one download; that is a test.
    • M3, sandbox assets on the store: atelet: stage sandbox assets per actor from the node-local file cache #2699, atelet: resolve an actor's sandbox assets from the runtimes its workers advertise #2731, atelet: resolve an actor's sandbox runtime from the request's sandbox_runtime #2788, atelet: prewarm each worker's default sandbox runtime as the syncer plans it #2741, atelet: evict the sandbox asset cache to a budget, each worker linking its default runtime #2756, docs: describe the sandbox asset cache, and assert its size instrument in the metrics suite #2792. Per-actor links, a runtime index, prewarm of each worker's default, per-worker links that keep the default cached, a budgeted pass, and the docs page.
    • M5, registry entries: atelet: evict the sandbox asset cache to a budget, each worker linking its default runtime #2756 and atelet: one eviction loop for every node-local file cache, with one metric family #2889. The e2e under eviction pressure was dropped as more machinery than it proved; the size instrument is asserted on every lane instead.

    Not part of this issue, tracked separately: M4, one eviction coordinator over the node's disk, is #2987, open for discussion and not committed to. Two performance follow-ups that came out of #2796's review are #2948 (links instead of copies for micro-VM restores) and #2974 (keeping shared snapshot images resident outside actor cgroups).
12. Dmitry Berkovich (dberkov)

    closed this as completedon Oct 20, 2026
13. github-project-automation

    moved this from Todo to Done in Substrate Milestones (go/substrate-milestones)on Oct 20, 2026