Auto-scale efConstruction and the search-ef ceiling with graph size for 1M+ node HNSW graphs

DESIGN.md

@@ -791,6 +791,75 @@ graphs being identical, though equal metrics do not prove it. It is the expected
 the upper layers are sparse enough that a greedy walk reaches the same entry point, which is why
 standard HNSW descends this way.
 
+## `efConstruction` and the search-`ef` ceiling both auto-scale with the graph
+
+The connection-building pass selects each node's stored edges from a candidate list of
+`efConstruction` entries. Held at a constant (100) while the corpus grows, edge quality erodes in a
+way no search-side setting can compensate: at 1M nodes (768-dim, int8, calibrated hard corpus)
+recall@10 fell to 0.935 and sweeping the search `ef` from 512 to 1536 only reached 0.957 raw / 0.967
+set at 4.7x the latency — the missing neighbours were not deep in the candidate list, they were
+unreachable. Rebuilding the identical corpus (same seed, same level assignments) with
+`efConstruction` 200 restored 0.985/0.997 and made queries _faster_ at the same `ef` (3,110 nodes
+visited vs 3,948 — better-selected edges route more directly). Quantization contributed ~1.5 points
+(float32 rebuild: 0.952); construction quality was the dominant term. Full sweep in #2180.
+
+So when the schema does not configure `efConstruction`, it scales as `AUTO_EF_BASE * sqrt(nodes /
+AUTO_EFC_REF)`, capped at `AUTO_EFC_MAX`. The healthy write path reads the count directly from the
+shared id counter: one atomic load with no memo lag during bulk ingest. If an update-only worker
+cannot attach that counter, it warns once, falls back to the memoized reverse seek, and retries the
+attach after the memo TTL; a new insert still requires the shared counter rather than risking ids
+from a private counter. Scaling starts at 250K nodes: efC 100 held recall through 500K (0.978), so
+smaller graphs — the common case — build exactly as before. The sqrt shape mirrors the search-side
+scale; the cost is build time (1.77x at 1M for efC 200), paid only by tables that actually grow
+large, and partly returned as cheaper queries.
+
+An explicit `efConstruction` stays authoritative and is structural, so changing it triggers a full
+index rebuild. It also seeds the search `ef`: setting `efConstruction: 100` alone cuts query effort
+to 100. Retaining the former large-graph search default while opting out of build scaling requires
+an explicit `efConstructionSearch` as well (512 after the former auto-scale reached its plateau).
+There is currently no "pinned build, auto search" combination.
+
+The search side scales past its old plateau for the same reason. `AUTO_EF_MAX` (512, pinned from
+~13K nodes) was calibrated when layers above 0 were searched at the full `ef`, which made large efs
+cost seconds; after the greedy-descent fix the same headroom costs tens of milliseconds (ef 1024 at
+5M nodes: ~45ms p50), and holding the pin leaves measured recall on the table — set-recall at a
+pinned 512 on well-built graphs decays 0.997 → 0.955 → 0.935 across 1M/2M/5M. So past
+`AUTO_EF_LARGE_REF` (1M nodes, where 512 was last measured sufficient) the scale resumes from the
+plateau — `512 * sqrt(nodes / 1M)` — up to `AUTO_EF_CEILING` (2048, binding at ~16M). The 5M point
+resolves 1,145, bracketed by the measured ef-1024 sweep there (0.985 set). The default's query
+latency therefore grows as sqrt(N) on large tables; that is the recall-first trade chosen here, and
+apps preferring latency pin `efConstructionSearch` or a per-query `ef`. The filtered-traversal
+budget (`maxVisits`, #1241) deliberately does not follow the second regime: each budgeted visit is
+a synchronous record load plus predicate evaluation, so an auto-scaled ef's budget contribution
+stays capped at `AUTO_EF_MAX` — the recall decision and the filtered-scan bound are separate
+decisions, and an explicit ef (per-query or schema) still raises the budget for callers who own
+the cost. Both ceilings are finite on
+purpose: total build work grows as N^1.5 under sqrt scaling, and past roughly tens of millions of
+nodes per graph, sharded medium graphs beat one huge graph on build and query cost alike — scaling
+the constants further is the wrong tool there.
+
+Two caveats are accepted deliberately, both inherited from the count being a lifetime high-water
+mark of allocated node ids rather than a live count. First, churn: a table that deletes heavily
+(TTL eviction, delete-and-reinsert ingest) reads high forever, so its build-side efC can sit at the
+cap while the live graph is small. The 6–7x build-time extrapolation applies to a comparably large
+graph; it is not a bound for a small rolling window. When efC exceeds the live graph size, the
+candidate list cannot fill and an insert can traverse a large fraction of the graph before storing
+only `M << 1` edges. This wastes throughput without improving recall. The search side accepted the
+same over-count as "slightly generous ef" on an opt-in read path; the write path inherits it as a
+known cost until a live count exists (tracked follow-up). Second, ramp history: nodes indexed before
+the graph crossed a scale threshold keep their original edges — the scale applies to inserts from
+that point on. A reindex in a live process rebuilds roughly uniformly (the id counter keeps its
+high-water mark), but a reindex after a restart re-seeds the counter from the largest id in the
+rebuilding store and therefore repeats the ramp — its first 250K nodes rebuild at the base efC.
+Later inserts add reverse edges to older nodes, but a default-ramp 1M build has not been compared
+directly with the uniform-200 A/B. The larger default-ramp runs reached 0.988 set-recall at 2M and
+0.985 at 5M when searched at ef 1024, which shows that the measured neighbours remained reachable
+at those sizes without proving uniform convergence.
+
+Deletes have a separate tail-latency cost: connectivity repair can synchronously reinsert an orphan
+and up to 256 nodes from a severed island. Those reinserts use the current auto-scaled efC, so the
+per-insert build multiplier can land hundreds of times within one delete.
+
 ## An approximate index returns at most `ef` rows, so `limit` has to reach it
 
 Layer 0 keeps at most `ef` candidates, and ef resolves from the auto-scale, not from the query. A
@@ -803,10 +872,10 @@ request. Any future approximate index needs the same plumbing.
 Two bounds keep that from becoming a new problem. `ef` drives a synchronous traversal that holds
 every admitted candidate in a sorted array with an O(len) insert, so a limit-derived `ef` is capped
 at `LIMIT_EF_MAX`; without it, ordinary deep pagination (`offset` in the millions) would walk the
-whole graph on the event loop, which is worse than the truncation being fixed. And a per-query `ef`
-stays authoritative: it is an explicit cost ceiling, so it bounds the result set rather than being
-raised by the limit. A schema-level `efConstructionSearch` is a default rather than a per-request
-decision, so it does not block the floor.
+whole graph on the event loop, which is worse than the truncation being fixed. And schema-level or
+per-query `ef` values stay authoritative: each is an explicit cost ceiling, so it bounds the result
+set rather than being raised by the limit. Only automatically scaled indexes widen toward
+`LIMIT_EF_MAX` to satisfy a larger bounded request.
 
 `LIMIT_EF_MAX` is the _only_ bound on the widening — deliberately not also the graph size. Clamping
 there is tempting and costs more than it saves: the memoized size reads low while a table grows, so
@@ -815,10 +884,11 @@ puts a store lookup back on every query whose `limit` exceeds the table — the
 whole change removed, reintroduced in miniature. An `ef` above the node count is free anyway: the
 traversal is bounded by the nodes it can reach, so it ends at the graph, not at `ef`.
 
-The filter budget deliberately does not follow a limit-derived `ef`. `maxVisits = ef * filterExpansion`
-(#1241) is what stops a selective filter crawling the graph and loading a record per visit, so it is
-computed from the `ef` the index resolved for itself. Multiplying it by a caller's `limit` would turn
-a filtered vector query into a record-loading scan wearing an index's clothes.
+The filter budget deliberately does not follow a limit-derived `ef`. It is computed from the `ef`
+the index resolved for itself, with an automatically scaled `ef` capped at `AUTO_EF_MAX` before it is
+multiplied by `filterExpansion`; explicit schema or per-query `ef` values remain authoritative.
+Multiplying the budget by a caller's `limit` would turn a filtered vector query into a record-loading
+scan wearing an index's clothes.
 
 Paging a vector search is best-effort, not a stable partition. Each page re-runs the approximate
 search at a different `ef` (`offset 0, limit 250` resolves 250; `offset 250, limit 200` resolves 450),
@@ -828,8 +898,10 @@ empty" defect; it does not make offsets a cursor. Callers who need stability sho
 large enough for the whole result set, or pin an explicit `ef`.
 
 One consumer is still calibrated in index-store keys rather than nodes: `estimateCountAsSort`, the
-planner's cost estimate for a vector sort. It is scaled by `INDEX_KEYS_PER_NODE` so the unit switch
-does not silently shift which condition the planner chooses to lead with.
+planner's cost estimate for a vector sort. It is scaled by `INDEX_KEYS_PER_NODE` so the count-source
+unit switch does not shift the estimate on its own. The ef term remains the configured search value,
+not the runtime auto-scaled value, so the planner increasingly underestimates vector traversal cost
+as an automatically scaled graph grows.
 
 ## Env-config empty objects mean three different things (`config/harperConfigEnvVars.ts`)