feat: statistical range key-count estimation (getEstimatedKeyCount range support + CountEstimator)

[kriszyp]

Summary

Adds a statistical range key-count estimate that never iterates, for query planning and pagination reporting (HarperFast/harper#2147). Closes #205, and supersedes the earlier attempt in #311.

Native estimateCount(start?, end?) combines three RocksDB primitives:

• GetApproximateMemTableStats — returns an entry count for the memtable portion directly.
• GetApproximateSizes (files only, 10% error margin) — approximate on-disk bytes covered by the range.
• GetPropertiesOfTablesInRange — per-SST num_entries / num_deletions / block sizes for only the SSTs overlapping the range, giving a range-local live-entry density ((entries − deletions) / file bytes) that converts bytes → count.

Using range-local table properties (instead of #311's whole-CF cached mean with write-count invalidation) keeps the density honest when entry sizes vary across the keyspace and needs no cache or invalidation hooks. Open-ended ranges are estimated as estimate-num-keys minus the complementary range — an empty slice is the smallest key, so it must never be passed as an upper bound (a correctness bug in #311's open-ended path). Inverted, empty, and zero-length bounds are guarded (they would otherwise underflow RocksDB's uint64 offset subtraction).

Public API (shape chosen by Kris: estimates carry a trust signal):

• db.estimateCount(options?: RangeOptions) → { count, confidence } (CountEstimate). confidence is a heuristic 0–1 trust indicator, exactly 1 only when the count is exact — derived natively from the estimate's resolution (data-block/memtable-sampling granularity relative to the count), the tombstone fraction of overlapping SSTs, complement-subtraction error for start-only ranges, and failed statistics calls (a degraded estimate caps at 0.1; a failed estimate-num-keys read returns {0, 0}, not a confident empty). exclusiveStart/inclusiveEnd are honored via the bytewise-successor zero byte.
• db.getEstimatedKeyCount() — unchanged original no-arg signature (cheap estimate-num-keys alias), so existing callers are untouched and one name doesn't cover two cost profiles.
• db.createCountEstimator(options?) → CountEstimator — rides an iterator: advance(lastKey, count) checkpoints progress (e.g. once per page), estimate() returns {count, confidence} = exact traversed + remainder calibrated by the observed actual/estimated ratio (clamped 8×), memoized per checkpoint; confidence is the exactness-weighted blend, capped at 0.999 until finish() declares the traversal complete (then exact with confidence 1). Supports reverse and the range bound flags.

For the human reviewer

• Cost model (the review's one carried major, accepted + documented): a bounded estimate enumerates table properties for every SST overlapping the range, synchronously on the JS thread — table-property reads go through the table cache and can do I/O for cold files, and a start-only range does the work of its complement. Inherent to the range-local-density design (the cached whole-CF mean alternative is what made Initial attempt at getApproximateCount for #205 #311 wrong); an async variant is additive later. Callers should prefer bounded ranges.
• confidence is now API surface: callers will encode thresholds against it, so retuning the formula changes their behavior (review ledger point). The semantics doc deliberately promises only "heuristic ordering signal, 1 = exact" — the formula itself is not contract.
• Whole-DB vs full-range disagreement (ledger): estimateCount() no-bounds uses estimate-num-keys while a bounded full range uses bytes×density; they can disagree. Deliberate — the no-bound path stays O(1) and matches getEstimatedKeyCount().
• Calibration constants (CALIBRATION_MIN_TRAVERSED = 16, 8× clamp) are judgment calls; options can be added later.
• Caller-owned progress contract: advance() trusts the caller (monotonic cursors, no double-reporting); a wrapping-iterator variant would be additive.
• Estimates deliberately ignore transaction state (committed statistics only; covered by a test).

Verification

• test/estimate-count.test.ts (12 tests): flushed / memtable-only / mixed ranges, open-ended both sides, empty DB (confident 0), inverted range (exact 0), zero-length native bounds, uncommitted-transaction exclusion, monotonic scaling with range width, estimator refinement (confidence must increase), reverse iteration, finish() exactness, and a paginated loop driven to completion (pre-finish confidence < 1, exact total after finish()). Full suite green at every commit (latest: 767 passed / 2 skipped, 56 files).
• Accuracy/perf (500k entries, varied value sizes 20–220B, flushed): counts within −0.1% to −5.5% of exact across full/half/tenth/1% ranges at 9–83µs vs 1–85ms exact scans (~1000×). Confidence measured: 0.999 full/half ranges, 0.88 at 1%, 0.21 on a 50-key range (~2× over-report, correctly distrusted), 0.13 on a start-only tail (+39% error from complement subtraction, correctly distrusted). Estimator converged 4.2% → 1.4% error by 50% traversal. (Loose 2× bounds asserted in CI; these are measured local numbers.)
• Downstream validation: the harper planner integration (follow-up PR) was run end-to-end against this branch linked into a harper worktree — real tables, secondary indexes (composite [value, primaryKey] keys through RocksIndexStore), width-ordered estimates confirmed.
• Native GoogleTest target untouched (N-API-layer change; the vitest suite is end-to-end through the real binding).

Review coverage

Generated by Claude (Fable 5). Cross-model pre-push review via prepush-review.mjs, six rounds:

• Round 1 (full, 1c4be8d): Codex (graded) + Gemini + Harper-domain adjudication — cursor-composer failed (output-format rejection), cursor-grok pruned. Fixed: inverted-range guard, exclusiveStart/inclusiveEnd, estimator forward off-by-one, finish(), memoization, tempered cost claims.
• Round 2 (delta, 4481bc4): Codex (resumed) + Gemini. Fixed: zero-length end bound bypassing the guard on the native surface (napi nullptr for empty buffers).
• Round 3 (delta, 0ddaf72): Codex + Gemini — hardening confirmed clean.
• Round 4 (full, b2cb53a, API change): Codex (graded) + Gemini + Harper-domain — cursor-composer failed again, grok pruned. Adjudicated major fixed in round 5: failed statistics reported as confident estimates.
• Round 5 (delta, 0548d42): Codex (Gemini leg returned no output this round). Follow-ups fixed in round 6: a successful zero estimate-num-keys claimed exactness; partial null table-properties collections didn't degrade.
• Round 6 (delta, final head 89d437b): Codex + Gemini — verdict COMMENTS, both fixes confirmed, no new findings.

Accepted, not changed (with rationale): silent-degrade carries low confidence instead of an event; sync-on-JS-thread cost model (documented); caller-owned progress contract; calibration reads current state so concurrent writes behind the cursor can swing a checkpoint (inherent to statistical estimates on a live database, bounded by the 8× clamp); no native GoogleTest for the estimator math (it lives in an N-API translation unit, which cannot link into the gtest target — the vitest suite covers it end-to-end through the real binding).

_{Human-Review-Need: 3 @ 75562fc}

[gemini-code-assist]

Code Review

This pull request implements a non-iterating key-count estimation feature for RocksDB ranges, exposing getEstimatedKeyCount with range options and introducing a new CountEstimator class to progressively refine estimates during iteration. The review feedback correctly identifies that CountEstimator currently discards the exclusiveStart and inclusiveEnd options from CountEstimatorOptions, and suggests storing and utilizing these options in the estimate() method to ensure bounds are correctly respected.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.16K ops/sec	41.39	40.03	2,103.03	0.140	120,808
🥈 rocksdb	2	11.21K ops/sec	89.19	85.74	31,686.934	1.25	56,060

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.33K ops/sec	35.30	34.28	575.312	0.099	141,629
🥈 rocksdb	2	11.09K ops/sec	90.19	86.69	531.131	0.054	55,439

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.72K ops/sec	38.88	36.02	1,852.583	0.297	128,607
🥈 rocksdb	2	16.65K ops/sec	60.06	51.96	1,147.403	0.124	83,249

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	370.98 ops/sec	2,695.569	163.816	27,205.201	9.15	744
🥈 lmdb	2	26.34 ops/sec	37,971.12	439.491	1,187,491.031	136.757	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	40.25K ops/sec	24.84	11.03	13,968.133	0.606	201,260
🥈 lmdb	2	443.71 ops/sec	2,253.707	105.502	10,424.858	1.19	2,219

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	747.71K ops/sec	1.34	1.17	4,919.299	0.209	3,738,548
🥈 lmdb	2	450.20K ops/sec	2.22	1.20	2,766.329	0.332	2,251,013

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	827.51 ops/sec	1,208.45	1,017.924	3,013.469	0.508	1,656
🥈 lmdb	2	1.16 ops/sec	859,069.203	802,988.391	943,726.761	3.47	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	24.11K ops/sec	41.48	29.50	538.594	0.592	48,219
🥈 lmdb	2	821.93 ops/sec	1,216.642	84.22	10,567.907	5.20	1,645

---

Results from commit ba79781

[cb1kenobi]

Reviewed 75562fc2 — no issues found. This PR looks good, nice job!

—
Generated by Barber AI