fix(store): bound FTS search result hydration for oversized rows - #963
Open
Tomodad wants to merge 2 commits into
Open
fix(store): bound FTS search result hydration for oversized rows#963Tomodad wants to merge 2 commits into
Tomodad wants to merge 2 commits into
Conversation
Tomodad
marked this pull request as ready for review
July 16, 2026 09:06
13 tasks
Contributor
Author
|
Fresh Windows/Codex recurrence on the official marketplace build
This is fresh evidence that All five checks on this PR are green and GitHub currently reports it clean/mergeable. @mksglu, would you prefer any further split/rebase or an additional regression fixture before review? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #947.
FTS ranking currently selects full
chunks.contentand materializeshighlight()before the server clips output. A legacy 53 MiB row can therefore turn a millisecond MATCH into a multi-minute synchronous highlight operation.This PR keeps the scope to the search path:
rowid, title, type, timestamp, source, rank, session id);MAX_CHUNK_BYTES;substr()window;It deliberately does not implement Markdown chunking (#878/#898), ingestion budgets (#961), trigram/vocabulary policy (#962), or a whole-handler deadline/worker model.
TDD
RED on
nextafter adding the regression tests:Node 22.5 CI exposed a second RED after the first push:
A focused multi-row regression now asserts that each ranked porter/trigram candidate hydrates its own row.
GREEN:
A complete
tests/core/server.test.tsrun in the managed Windows sandbox also exposed three pre-existing process/path tests that fail withSTACK_TRACE_ERROR/EPERM; the directly affected search/formatting subset is green.Before/after: isolated 1.2 MiB legacy row
Node 22.5 after the rowid-compatibility follow-up: porter 4.977 ms, trigram 4.337 ms, fallback 8.046 ms; all still returned 315/317 chars with the match preserved.
Compatibility
Normal capped chunks retain full-content/highlight behavior. Only legacy rows that already violate
MAX_CHUNK_BYTESreturn a bounded window. Proximity reranking for those legacy rows therefore uses the window rather than the full document, which can change ordering among multiple giant rows. The metadata-first + hydration path adds a small DB round-trip cost for normal rows (~0.045 ms/search in a local 500-iteration microbenchmark).