Auto-scale efConstruction and the search-ef ceiling with graph size for 1M+ node HNSW graphs

[kriszyp]

Large HNSW graphs lose recall because a fixed construction candidate list eventually stops creating enough useful edges. This change auto-scales efConstruction from a base of 100 as min(1024, 100 × sqrt(nodes / 250K)), resumes the search-ef scale above one million nodes up to 2048, and adds streaming benchmark support for multi-million-node validation.

The write path reads the lifetime node-id high-water mark directly from the shared atomic counter. If an update-only worker cannot attach that counter, it warns once, uses a transaction-consistent memoized reverse seek, and retries after the memo TTL. New-id allocation still fails closed rather than installing a private counter.

Evidence and limits

The original 1M A/B rebuilt the same corpus uniformly at efConstruction: 200: set-recall improved from 0.967 to 0.997 and the better graph also reduced nodes visited at the same search ef. That isolates the construction-quality problem, but it is not a direct comparison of the automatic ramp against a uniform-200 build.

The automatic ramp has separate 2M/5M evidence. With search ef: 1024, those graphs reached set-recall 0.988/0.985. That shows the measured neighbours remained reachable at those sizes; it does not prove that the ramp converges to the uniform-build result.

Below 250K nodes the construction default remains 100. Nodes inserted before a scale threshold keep their existing edges. A live-process reindex starts near the lifetime high-water mark, while a reindex after restart repeats the ramp from the store's current largest id.

Search cost

Past 1M nodes, automatic search ef resumes from 512 as 512 × sqrt(nodes / 1M), capped at 2048. At 5M nodes it resolves to 1,145; the measured ef: 1024 point held 0.985 set-recall at about 45 ms p50 on the benchmark corpus. Applications that prefer a latency ceiling can pin efConstructionSearch or pass a per-query ef.

Filtered traversal does not inherit that entire automatic increase. Its visit-budget contribution remains capped at the former automatic ceiling of 512 before multiplying by filterExpansion; explicit schema or per-query ef still raises the budget for callers that own the cost.

Construction and delete cost

The node count is a lifetime id high-water mark, not a live count. Delete-heavy tables can therefore remain over-provisioned after the live graph shrinks. When efConstruction exceeds the live graph size, an insert can traverse much of the graph without gaining recall. Connectivity repair can also synchronously reinsert an orphan plus up to 256 nodes from a severed island, and each repair uses the current scaled construction value.

An explicit efConstruction remains authoritative and structural, so changing it rebuilds the index. It also seeds the search default. To retain the former large-graph search default while opting out of build scaling, configure both efConstruction: 100 and efConstructionSearch: 512; there is no “pinned build, automatic search” mode.

Benchmark tooling

benchmarks/hnsw-scale.js gains a --stream mode for corpora too large to hold as one float pool. It uses a dedicated replayable corpus RNG, gathers query rows in one pass, then streams the same rows through graph construction and brute-force ground truth in a second pass. Output now records the resolved construction value as well as the requested setting.

For the human reviewer

1. Unprivileged limit ceiling: LIMIT_EF_MAX rises from 2048 to 4096 with the search ceiling. Keeping 4096 avoids truncating larger requested result sets, but raises worst-case synchronous query CPU. A schema-level efConstructionSearch pin does not prevent limit-derived widening; only an explicit per-query ef does. Alternatives are to retain 2048, have a schema pin cap widening, or tie widening to the graph’s resolved automatic ef.
2. High-water and repair policy: the current implementation documents the churn and delete-repair costs. HNSW: graph-size resolution is a lifetime id high-water mark — churn-heavy tables permanently over-provision build/search ef tracks a live-count replacement. Please confirm whether documentation and that follow-up are sufficient, or whether this PR needs a behavioral repair cap first.
3. Public documentation: the schema description is corrected here, but HarperFast/documentation still needs a companion update. This dispatched task is restricted to the Harper core worktree.

Verification

• npm run build — passed.
• npx mocha unitTests/resources/vectorIndex.test.js — 106 passing.
• npm run test:unit:resources — all HNSW coverage passed; suite total 1,567 passing, 15 pending, and 3 unrelated structure-format failures.
• npm run test:integration:all — 1,732 passing, 20 skipped, 0 asserted failures; 6 Ollama tests were cancelled after an unrelated Node JSON import-attribute error.
• npm run lint:required and npm run format:check — passed.
• npm run lint reports 12 pre-existing warnings outside this diff. npm run test:unit:main is locally blocked by the Mocha apiTests-exclusion preloader bug and an unrelated built-entrypoint shebang import failure.

GitHub Actions on this head: 41 passed, 2 intentionally skipped, 0 failed.

Review coverage

Authored by Claude Fable 5; review-feedback commits by GPT-5 Codex. Cross-model review at 8e3b2cb3e: Claude (opposite-family graded review) passed and Gemini (default model) passed; Cursor and Harper-domain legs were pruned for the narrow delta. The final 3792c84f7 commit is comment-only and was not rerun under the trivial-change policy. The review retained four policy decisions for human confirmation rather than identifying a new implementation defect.

Addresses HNSW: efConstruction does not scale with corpus size — recall plateaus at 1M vectors regardless of search ef. Follow-up: HNSW: graph-size resolution is a lifetime id high-water mark — churn-heavy tables permanently over-provision build/search ef.

Updated by GPT-5 Codex.

_{Human-Review-Need: 4 @ 3792c84}

[gemini-code-assist]

Code Review

This pull request introduces auto-scaling for the HNSW efConstruction parameter based on the graph size to prevent edge quality erosion and maintain high search recall as the corpus grows. It also adds a --stream option to the HNSW scale benchmark to support two-pass streaming generation for large datasets, and includes unit tests for the auto-scaling logic. The review feedback suggests optimizing the auto-scaling logic during rapid bulk imports by reading the live node count directly from this.idIncrementer via Atomics.load to avoid the 10-second TTL lag of approximateNodeCount(). A corresponding update to the mockNodeCount test helper was also recommended to support this optimization.

[kriszyp]

2M and 5M validation runs (streaming mode, this branch)

Same harness and calibration as the 100K–1M sweep on #2125 (768-dim, int8, intra-cos 0.6, 60 held-out-style queries, exact brute-force ground truth folded in during the stream). Auto-scaled efConstruction ramps to ~283 by 2M and ~447 by 5M. Both runs also sweep ef 1024 over the same graph to separate build quality from the search-side cap.

N	build	auto efC (final)	recall@10 raw/set @ ef 512	@ ef 1024	p50 @ 512 / 1024	peak RSS
1M (efC 200 A/B, from #2180)	110 min	200	0.985 / 0.997	—	7.2ms / —	9.4 GB
2M	3.7 h	~283	0.950 / 0.955	0.983 / 0.988	10.4ms / 30.4ms	12.3 GB
5M	19.3 h	~447	0.920 / 0.935	0.968 / 0.985	22.2ms / 45.1ms	27.6 GB

What this says about the fix. The construction side is doing its job: at 5M, ef 1024 on the same graph reaches 0.985 set-recall, so the true neighbours are present and reachable — the opposite of the #2180 failure mode, where no ef could recover them (0.967 set at ef 1536 on a 5× smaller graph). The build-time cost of the ramp is real and as documented: 13.9 ms/insert at 5M vs 3.8 at 1M-unfixed.

What binds next is the search side. With good graphs, recall at the auto-resolved ef (pinned at AUTO_EF_MAX 512 from ~13K nodes) decays with N — 0.997 → 0.955 → 0.935 set across 1M → 2M → 5M — and doubling ef recovers it at roughly 2× latency (45 ms p50 at 5M, still tens of ms). That is #2125's deferred auto-scale-constants question resurfacing at the top of the range: the 512 search cap is the right latency-first default for most tables, but 2M+ tables have recall on the table that only a raised (or scaled) AUTO_EF_MAX — or an explicit efConstructionSearch / per-query ef — can claim. Not changed in this PR; it is a search-side default decision, and per-query ef already gives callers the knob.

Two caveats. The 5M query latencies carry memory-pressure noise (27.6 GB RSS on a 30 GB box — page-cache squeeze and poor locality inflate per-visit cost; visited only grew 1.2× while p50 grew 2.1× from 2M). And the printed d(nn)/d(rand) drift (0.75, 0.80) is the known first-3000-rows probe artifact, not the corpus hardening — construction keeps per-cluster density constant across N.

Raw logs/JSON: ~/dev/tmp/hnsw-1mm/n2000000-fixed.*, n5000000-fixed.* on the benchmark box.

Generated by Claude Fable 5.

[cb1kenobi]

Reviewed c6784e5 — no issues found. This PR looks good, nice job!

—
Generated by Barber AI

[cb1kenobi]

Medium: LIMIT_EF_MAX doubles to 4096 for every graph, but the ceiling that justifies it only binds at ~16M nodes

minResults is offset + limit — unprivileged and set on every request (resources/search.ts:513 → line 1104-1105), with no clamp, rate limit, or per-query CPU budget anywhere in that chain. The unfiltered layer-0 path has no maxVisits at all (filterState is undefined without a filter), so ef alone sizes the work: one safeGetSync + a dims-wide distance per visited node, plus an O(len) results.splice() per admission into an array of up to 4096. The comment's "kept within a small multiple of the index's own auto-scaled ceiling" only holds once autoScaleEf actually reaches 2048 — at 1M nodes this is 8x the ef the index resolves for itself, at 10K nodes 40x. A schema-level efConstructionSearch pin does not block the widening either: line 1104 checks only !explicitEf.

Suggested fix: anchor the limit ceiling to what this index resolves rather than to a ceiling that binds at 16M — e.g. Math.min(2 * AUTO_EF_CEILING, 4 * resolvedEf) — and let a schema-configured efConstructionSearch cap the widening the way an explicit per-query ef does.

Separately, estimateCountAsSort() (line 1368) still estimates against this.efConstructionSearch (default 50). That gap is pre-existing, but raising the real ef to 1145/2048 widens the planner's under-estimate to ~40x on exactly the large tables this PR targets.

—
Generated by Barber AI

[cb1kenobi]

Medium: two user-visible defaults change with no update to the public docs

The published reference (HarperFast/documentation → reference/database/schema.md) still lists efConstruction | 100 | Max nodes explored during index construction, which is false for any table past 250K nodes after this lands, and says nothing about the default query ef now scaling past 1M to 2048. That repo already has a <VersionBadge type="changed" version="..." /> convention, used for the previous ef auto-scale change on this same page. The harper repo has no CHANGELOG or release-note file, so the docs repo is the only operator-facing channel for a change that silently alters p50 latency on existing >1M-node tables.

Suggested fix: land a companion documentation PR — correct the efConstruction default cell to "auto-scaled (base 100)", add VersionBadge type="changed" entries for both the build-side scale and the second search-ef regime, and state the latency implication plus efConstructionSearch as the opt-out.

—
Generated by Barber AI

[claude]

Suggestion (non-blocking): This raises LIMIT_EF_MAX from 4 * AUTO_EF_MAX (2048) to 2 * AUTO_EF_CEILING (4096). The limit-driven widening at line 1128 (if (minResults !== undefined && !explicitEf && minResults > effectiveEf)) only checks !explicitEf — not efSearchConfigured — so a schema pin via efConstructionSearch doesn't stop an unprivileged offset + limit from doubling the synchronous candidate-list work (sorted-array admission, O(len) per insert) relative to before this PR, on exactly the large tables this change targets. The PR body's adjudicator's ledger already flags the 2048→4096 ratio for confirmation, but doesn't mention this schema-pin gap specifically.

Consider gating the widening on !explicitEf && !efSearchConfigured, or anchoring LIMIT_EF_MAX to a small multiple of the ef this index actually resolved for itself (resolvedEf) rather than the absolute ceiling, so a caller who pinned a lower search ef isn't silently widened past it.

[cb1kenobi]

Reviewed 3792c84 for the incremental update from c6784e5. No new issues found. The update fixes the previously flagged filtered-budget test gap, shared-counter attach fallback, benchmark instrumentation, schema wording, and docs caveats. Two prior Medium threads remain open: the unprivileged LIMIT_EF_MAX widening and the external public-docs update. Generated by Barber AI

[cb1kenobi]

Low: the "a write must never fail on it" guard was removed; the invariant now rests on validation two methods away

The previous head wrapped the whole node-count resolution in try { ... } catch, with the comment stating the reason outright: the count is an ef heuristic with a safe fallback, so a write must never fail on it. This commit removes both the guard and that sentence, and resolveConstructionNodeCount() is now called bare.

Every other throw is still contained — ensureIdIncrementer is caught inside resolveConstructionNodeCount, and resolveNodeCount's seek has its own catch returning 0. The one uncontained path is the healthy fast path: line 751 if (this.idIncrementer) return this.resolveNodeCount() → line 804 Number(Atomics.load(this.idIncrementer, 0)), which throws RangeError if idIncrementer is ever a zero-length BigInt64Array. Today that state is unreachable only because of the new byteLength validation added to ensureIdIncrementer in this same commit — so a documented write-path invariant went from locally enforced to enforced at a distance by a check in a different method.

I found no reachable failure on this head. The ask is to keep the invariant enforced where it is relied on rather than inferred: restore the try/catch around this line (falling back to this.efConstruction), and keep the sentence that says why it is there.

—
Generated by Barber AI

[cb1kenobi]

Low: this finally restores three things but not the five the test saved, so one assertion failure cascades through the rest of the file

The test saves five fields up front — idIncrementer, nodeCount, nodeCountAt, idIncrementerRetryAt, idIncrementerFailureLogged — but this first finally only restores getUserSharedBuffer, getKeys, and idIncrementerRetryAt. All five are restored in the second try's finally, which is never reached if any of the seven assertions above (lines 824-834) throws.

On a genuine failure the index is left with idIncrementer === undefined (the real shared counter dropped), nodeCountAt === 0, and idIncrementerFailureLogged === true, which leaks into every later test in the file — so the one test guarding the degraded attach path is the one whose failure is hardest to read. The sibling test at line 853 ('does not cache an unusable shared-counter buffer') gets this right with a single try/finally.

Suggested fix: wrap both phases in one try/finally that restores all five saved fields plus the two store methods, so a failing assertion reports itself and nothing else.

—
Generated by Barber AI

[cb1kenobi]

Nit: the attach-failure path does two reverse seeks where one would do, and the new test asserts the waste as intended behavior

When the attach fails, ensureIdIncrementer has already performed its reverse seek to find the largest node id before getUserSharedBuffer throws; that value is then discarded, and this line does a second identical seek via approximateNodeCount. The new test bakes it in: assert.strictEqual(seekTransactions.length, 2, 'the first update should seek once to attach and once to fall back').

Only two seeks per NODE_COUNT_TTL in a degraded state, so the cost is small — but the seek is the exact thing the memo exists to avoid, and asserting 2 will keep it at 2. Suggested fix: have ensureIdIncrementer hand back the id it already found (or seed the memo with it before throwing) so the fallback reuses it, and assert 1.

—
Generated by Barber AI

[cb1kenobi]

Nit: the new efConstruction column records the ramp's terminal value, unlabeled

currentEfConstruction is last-write-wins across the build, so on an auto run it ends up holding the efC of the final insert — the top of the ramp, not what most of the graph was built at. A default 1M run and a --ef-construction=200 run therefore both print efC= 200 and both write efConstruction: 200 into rows[]; only the header's efC=auto and the top-level JSON field separate them. Your own results comment had to hand-label the column "auto efC (final)" for exactly this reason.

This matters more than it would have before: DESIGN.md now rests on the ramp-vs-uniform comparison being absent rather than measured, which makes this harness the thing that would eventually close that gap.

Suggested fix: track the first and last level-0 efC seen during the build and emit both (e.g. efConstructionStart / efConstructionEnd), so a ramped run is self-describing in rows[] without cross-referencing the header.

—
Generated by Barber AI