[Suggestion]: Add index persistence #642

Activity

1. phsd0

   added

   enhancementNew feature or requestNew feature or request

   on Jul 1, 2026
2. dmtrKovalenko commented on Jul 1, 2026

   

   dmtrKovalenko

   on Jul 1, 2026

   Owner

   More actions

   There is a way it is just not surfaced in the bun sdk. I had a hope to surface it at 0.10 but likely I won't be able to so this will only be abe to get to 0.11

   👍React with 👍1Reacted by Scott Nixon
3. dmtrKovalenko

   added

   triaged

   on Jul 1, 2026
4. gaojunran commented on Nov 7, 2026

   

   gaojunran

   on Nov 7, 2026

   More actions

   Posting measured startup data from a VS Code extension built on @ff-labs/fff-node (pinned 0.11.1-nightly.b3b8ced), in case it helps scope this. Happy to test a branch.

   Setup. One FileFinder per workspace root, created on demand, destroyed on window close. Extension host is Electron's Node; the finder lives in a worker thread so the UI is never blocked. Numbers below are waitForScan / waitForIndexReady phase timings, macOS aarch64, SSD, warm page cache unless noted.

   Tree	Files	create	scan (walk)	content index	total
   one TS repo	11,339	1 ms	51 ms	102 ms	154 ms
   same repo, first run after boot	11,339	26 ms	51 ms	567 ms	644 ms
   monorepo checkout	8,280	0 ms	51 ms	102 ms	153 ms
   multi-repo directory	90,471	12 ms	665 ms	4,647 ms	5,324 ms

   So the phase worth restoring is the content/bigram index, not the walk: at 90k files it is ~7x the walk, and the ratio grows with file count.

   What a restore would have to beat. The bigram index for the 11k-file tree adds roughly 13 MB of heap (process.memoryUsage().rss 66 MB after scan → 79 MB after indexing), so an on-disk form is in the same order. Reading that back is tens of milliseconds versus 102 ms warm / 567 ms cold to rebuild — a real win, but only ~0.1–0.5 s per window at this size, and ~4.5 s on the 90k tree.

   Why the walk cannot be skipped. Change detection still needs a stat per file (mtime + size). Restoring the file list does not remove that, unless the snapshot is trusted together with a watcher journal that survives process exit.

   Two things I could not find a path to from Node:

   1. BigramFilter gained serialization accessors (feat: add serialization accessors to BigramFilter #330: lookup(), dense_data(), words(), dense_count(), skip_index(), from_raw_parts()), but nothing in fff-c or packages/fff-node exposes them — I grepped both and found zero references. So an embedder cannot serialize or restore the content index today even though the core primitive exists.
   2. feat(core): restore and layer filename indexes across worktrees #873 restores the filename index (and its numbers look great — 100k records in ~11 ms), but per its description grep still rebuilds its content index per worktree.

   What would be most useful from an embedder's point of view, in rough priority:

   • expose the #330 accessors through fff-c + fff-node so callers can persist the content index themselves, with the caller owning invalidation;
   • or an InitOptions pair like indexSnapshotPath plus a documented staleness contract (fff version + index format version + base path + ignore-rule hash + per-file mtime/size), so the SDK can decide whether to restore or rebuild.

   The caller-owns-invalidation variant is attractive because we already know things the SDK does not: we feed search.exclude / files.exclude in as extraIgnore, so we can hash the effective rule set and drop the snapshot when it changes.

   👍React with 👍1Reacted by Philipp