Repository navigation
[PR] Storage Layer Memory Governance: Five Fixes Consolidated into One (Streaming Fingerprint Saves / Batched Spot-Checks / B | 存储层内存治理：五个修复合成一个（保存指纹流式化 / 抽查分批 / 会话装载有界 / 增量车道只渲增量） - #5198
Merged
Conversation
… reads Ordinary full-mode loads materialized every message row (row + content-CAS read + media hydration) although the view resolves only the tail/pluginSnapshot pointer seqs; dropped metadata and blockContents each rebuilt the whole session through loadSync after every idle save. On a 5,043 message / 236 block session one isolated read measured 15.1 -> 230.1 MiB heap growth (#4739). - UnifiedStore.sessionViewRows: the viewFor bundle from ONE read transaction; messages fetched only at pointer seqs (getMessagesAt, chunked IN, same hydration/retired/order semantics as getMessages). - StorageBackend.sessionViewRows (mysql lane): same pointer-scoped fetch. - UnifiedStore.getSessionMetadata / getBlockContents: single-column / blocks+orphans-only reads mirroring loadLegacyView's replay leg. - SessionStore.lazySource + narrowMetadata/narrowBlockContents: one build-scoped source decision per saved record (index vs envelope), bound-source-first with the other copy as fallback; never imports, never schedules saves, refuse-empty throw semantics preserved. Served bytes unchanged: lastMessages/pluginSnapshot resolution, tailGaps counting, orphan journal merge, media hydration, drift-latch and index/envelope freshness rules all preserved. No schema/config/kernel changes.
The engineLane=server metadata added by f590da9 explains the entire drift: removing that one field reproduces the old metadata digest. Regeneration changes only metadata and the enclosing digest; all wire bodies, state, originals and statistics remain identical.
Compare unified reads with an independent envelope reader plus the documented migration. Persist alignment before rollback/corrupt-index checks, select legacy explicitly, and assert the recovered raw ID under its ledger ref. Keep checkpoint writes deterministic so sampled ingest cadence cannot change the seed between reader legs. All 28 targeted storage tests pass; no product migration or digest assertion is disabled.
…nk seams (#4529) Extend storage-canonical-digest.test.ts with deterministic cases the original suite left uncovered: surrogate pairs landing exactly on the 64 KiB hash-chunk seam (writeString boundary pull-back), an MB-scale retained history spanning many chunks end-to-end, deep nesting (visit/JSON.stringify recursion parity) and wide fan-out (separator + pending-buffer flush). Each asserts canonicalDigest(v) === sha256(JSON.stringify(canonicalize(v))), so any chunking or surrogate-split regression fails loudly rather than drifting silently.
Compare unified reads with an independent envelope reader plus the documented migration. Persist alignment before rollback/corrupt-index checks, select legacy explicitly, and assert the recovered raw ID under its ledger ref. Keep checkpoint writes deterministic so sampled ingest cadence cannot change the seed between reader legs. All 28 targeted storage tests pass; no product migration or digest assertion is disabled.
Reconcile shared storage proofs with upstream citation-normalized restart comparisons, retain legacy/full fork coverage and Windows handle cleanup, and preserve both storage preview and batched verification types.
Preserve background integrity checks and their cache while retaining consistent sync/async read snapshots. Keep both MySQL snapshot and fork-clone contract tests, and combine Windows COMSPEC detection with exact launcher argument checks. Validation: 61 targeted tests passed, 3 optional backend cases skipped; TypeScript, build, and diff checks passed.
…28_pr4532-ci-timeout
Windows SIGTERM terminates the child instead of running its graceful shutdown handler. Trigger the existing loopback shutdown route on Windows so the draining-marker regression exercises the same lifecycle as POSIX. Wait for child close before deleting its database, destroy sockets owned by the silent TCP fixture, and stop health polling when the child exits. Keep both regression scenarios and their behavioral assertions intact. Validation: the isolated CI-preloaded serial test file passes 6/6 and exits naturally in 11.58 seconds; npm run typecheck and npm run build pass.
Windows SIGTERM terminates the child instead of running its graceful shutdown handler. Trigger the existing loopback shutdown route on Windows so the draining-marker regression exercises the same lifecycle as POSIX. Wait for child close before deleting its database, destroy sockets owned by the silent TCP fixture, and stop health polling when the child exits. Keep both regression scenarios and their behavioral assertions intact. Validation: the isolated CI-preloaded serial test file passes 6/6 and exits naturally in 11.58 seconds; npm run typecheck and npm run build pass.
Windows SIGTERM terminates the child instead of running its graceful shutdown handler. Trigger the existing loopback shutdown route on Windows so the draining-marker regression exercises the same lifecycle as POSIX. Wait for child close before deleting its database, destroy sockets owned by the silent TCP fixture, and stop health polling when the child exits. Keep both regression scenarios and their behavioral assertions intact. Validation: the isolated CI-preloaded serial test file passes 6/6 and exits naturally in 11.58 seconds; npm run typecheck and npm run build pass.
Windows CI can reset an aborted probe connection instead of ending it. The silent TCP fixture did not handle errors on accepted sockets, so ECONNRESET became an uncaught test exception after the probe finished. Destroy reset sockets and add a real resetAndDestroy regression against the same fixture. Keep the existing port-owner and ladder assertions. Validation: the new regression reproduces ECONNRESET before the fix; the isolated CI-preloaded serial test file then passes 7/7 and exits naturally. Pure tsc and git diff --check pass.
…tal-lane-per-turn
#4685 landed on master (canonicalize sorts object keys at every depth + shared DIGEST_SKIPPED_KEYS table), but this PR's streaming canonicalDigest still iterated raw Object.keys order with its own literal skip list. The two producers then disagreed for any object whose keys arrive in a different order — the false DIGEST DRIFT of #4392/#4986 again, this time between the streaming digest and canonicalize; tests/issue4392-canonical-key-order.test.ts went red on 8 shards (e.g. run 40909901398). Align visit() with the legacy producer's effective byte order: - skip via Object.hasOwn(DIGEST_SKIPPED_KEYS, k), keeping the __proto__ guard; - iterate array-index keys first in ascending numeric order, then the remaining keys lexicographically. Plain .sort() alone is NOT byte-identical: spec enumeration (Object.keys / JSON.stringify) emits array-index keys ascending numeric regardless of insertion or lexicographic order, so an index-shaped pair like {"10","2"} must serialize as "2","10". The pinned byte-identity test (streaming digest preserves JSON omissions, sparse arrays and property order) catches exactly this. Byte-identity to sha256(JSON.stringify(canonicalize(v))) now holds for any key arrival order, in either merge order relative to #4685. Verified locally: issue4392/issue4986/storage-canonical-digest suites 13/13, storage parity integration (engine-proofs + #3407 + #3919 + corpus) 26/26, typecheck clean.
PR #4532 (fix #4529): stream canonical JSON into SHA-256 instead of sha256(JSON.stringify(canonicalize(view))). Conflict resolution: NONE — merged cleanly against origin/master (6db5906); no manual edits required. Files taken as-is from p4532: - src/storage/canonical.ts: new streaming canonical->hash function; digest byte-values must stay identical to pre-change sha256 path. - src/storage/shadow.ts: shadowDigest switched to the streaming fn. - src/web/blogfeed.ts: digest call-site tweak. - tests/storage-canonical-digest.test.ts: new byte-equality suite.
PR #4750 (fix #4739 half B): batched block decode of ordinary loads (32-row batches, shared incremental decoder preserving inactive/archived/orphan originals, pointer order, repeated-message identity, tail-gap reporting), dedicated async SQLite/MySQL connections + deferred read snapshots, lazy getters reading just the requested indexed fields. Ordering note: p4750 lands SECOND (not last) because its merge-base is exactly origin/master (6db5906) — landing it while HEAD still closely resembles its base keeps this PR's ~254-line ingest.ts + persist.ts delta almost conflict-free and pushes the #4739 two-halves composition into the p4765 merge where both halves are visible together. Conflict resolution: - tests/issue4982-draining-twin-ladder.test.ts (~line 253, silentTcpListener): p4532 (merged earlier) and p4750 both independently ported the #5050 squatter-fixture rewrite (close()-handle return, SIGKILL-escalating killChild, 'silent TCP fixture survives an aborted health probe' test) — identical code on both sides. The only divergence was the error-listener comment block: p4532 collapsed master's leftover no-op sock.on('error', () => {}) into ONE sock.destroy() listener with a full rationale; p4750 kept the no-op plus a second destroy listener. Kept p4532's side: strictly superset behavior/rationale, identical runtime semantics (a redundant no-op error listener changes nothing). Both PRs' new test cases survive. No CONFLICT-FLAG needed. - src/storage/shadow.ts: auto-merged (p4532 digest switch and p4750 lazy read plumbing touch disjoint regions) — verified by auto-merge output.
#4739 half A (pointer-scoped session loads). Conflict resolutions: - src/persist.ts (both hunks): keep HEAD's loadResidentFields(id, m, {...}) lazy-field restore; drop p4765's narrowMetadata/narrowBlockContents/lazySource pair — master-side machinery is a strict superset of its stated behavior (build-scoped source decision + file fallback live in loadResidentFields). - src/storage/shadow.ts (hunk 1 sync loadView): keep HEAD readSnapshot{getMetadataJson+paged getBlockContents+getOrphansJson}->LegacyViewBuilder loop; p4765's full-log getMessages+loadLegacyView loses pointer scoping -> rejected. - src/storage/shadow.ts (async viewForAsync hunk): keep HEAD async readSnapshot builder variant; same rationale. - src/storage/store.ts: UNION — head getMessages gains messageRowReads.count++ counter; theirs adds new methods only (getMessagesAt chunked IN<=500, sessionViewRows single-read-transaction bundle, getSessionMetadata). - src/storage/mysql.ts: reorder-UNION — p4765 sessionViewRows bundle placed above master getMessages/getMessagesOn(opts?.seqs); both survive on one connection. hydrateMediaAsync call corrected to current 2-arg signature (conn arg, per adjacent getMessagesOn idiom). - src/storage/backend.ts: UNION both hunks — interface gains cloneSessionRows(initial)/getMessages(seqs)/getBlocks(limit,offset)+sessionViewRows member; SQLiteSyncBackend wrapper implements all five delegating to store. - tests/storage-backend-contract.test.ts: UNION both legs (mysql lane clone-through-seq leg from master + #4739 pointer-scope/one-connection leg from PR); disjoint sids, no interaction. - src/storage/store.ts getBlockContents name clash (PR map variant vs master bounded page reader): merged into ONE dual-mode accessor dispatched by optional opts arg — single signature + union return so BOTH PRs' pinned test stubs/calls compile untouched; shadow paging call-site Array.isArray narrows; shadow passthrough cast. Verification: pretypecheck+typecheck clean; issue4739-bounded-session-load 4/4 green; storage-bounded-load 11/11 green pre-edit (call-shape unchanged since).
#4538: sampled read-back verification in bounded batches of 32 (headers separately, blocks in persisted order, message hydration only at the verifier's pointer seqs); sync and async share one code path; SQLite and MySQL get bounded LIMIT/OFFSET + seq-IN reads. Conflict resolutions (p4540 base is pre-#5193/#4765/#4750; our side is the superset in every region): - src/storage/backend.ts: keep ours — p4540 added no content beyond ours (its sessionViewRows/readSnapshot/StorageReadView surface already present from #4765/#4750 merges); verified diff shows zero theirs-only lines. - src/storage/store.ts: union — p4540's getBlocks 'ORDER BY ord, block_id' total ordering (stable pagination across generations) onto our gc() batch-sweep constants, viewPointerSeqs, readSnapshot/readSnapshotAsync, activeForeignWriter, formatDbSize; comment kept ours' pluginSnapshotSeqs wording (documents the live field shape). - src/storage/mysql.ts: union — p4540's dedicated-pool reads with hydrateMediaAsync(conn) parameterization onto our v9 stamp + idx_content_refcount probe block + collectZeroRefIn lifecycle scoping; theirs-only lines audited line-by-line: all stale base versions of code our branch already carries (#5193 master state). - tests/plugin-fork.test.ts: keep ours' longer #4599 comment, replicasAvailable() gate (supersedes theirs' storageMode==="full"), and ours' #5140 raw-tail cold-restart test semantics (200, not 409); p4540's storageMode loop + env save/restore already on our side via #4765. - tests/storage-orphan-recovery.test.ts: take p4540's extra assert (alignedRef === "m00001") — pure addition, variable already defined.
#5155/#5144: incremental lane — serialize the wire prefix once per canonical entry and splice only the delta per turn (laneBodyBuffer sentinels with full-restring fallback); recordCanonicalPointers walks only rows appended after the last recorded pointer (windowed rawMessages(fromSeqExclusive)) with cluster-suffix (<base>_<n>) tolerant matching so pointers no longer silently freeze; wrap-up save debounces 500ms via scheduleSave instead of a synchronous writeNow+fsync on every turn. Conflict resolutions (both sides' features unioned): - src/storage/shadow.ts: keep ours' activeForeignWriter() (#4612) AND take p5156's optional fromSeqExclusive param on rawMessages() — the two edits sit on adjacent methods; final file carries both. - src/storage/store.ts: getMessages() opts unioned to { includeRetired?, seqs?, fromSeqExclusive? } — #4765's pointer-scoped seqs fetch + read instrumentation coexist with #5155's windowed scan (seqs takes priority when both are present; params assembled positionally). Async backend paths untouched by p5156, so no interface change required.
…4765/#4750 The two PRs added different getBlockContents call shapes (paged rows for bounded view decode vs full narrow contents map for lazy-field restore); merged as one union return type, which broke caller-side narrowing in the read-snapshot test. Declare the overload pair so paged callers get the row array and no-arg callers get the map. storage-bounded-load's read counter monkey-patch now casts to the implementation signature it already matches at runtime (every counted call passes opts.limit=32).
ranxianglei
changed the title
Contributor
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6960-pr.5198.a6dd1a2This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5198.tgz
npm install -g packageInstall instructions are refreshed on each push. |
Contributor
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.925-master.9d89c97 # this exact build
npm install -g billion-context@master # always the newest merged build
|
This was referenced Oct 30, 2026
ranxianglei
pushed a commit
that referenced
this pull request
Oct 30, 2026
…5188) The #5198 batched view verifier (master) only fetches messages for the tail seqs it needs; this suite's fixture carries none, so the old getMessages seam stopped firing and the three latch integration tests saw zero failures. getBlocks is the verify leg's unconditional first read-back — same catch, same classification path under test.
ranxianglei
pushed a commit
that referenced
this pull request
Oct 30, 2026
…hs (#5188 round 3) #5140's flake survived rounds 1-2 because the cold-restore GET is the LAST consumer touch in its timeline: the ctor open dies on a shard-load spike, the resource clears milliseconds later, but no read-check slot fires before the GET arrives, so replicasAvailable() still answers false at request time. The two user-visible 409 paths (GET snapshot / POST fork) now wait a bounded moment (750 ms budget, 50 ms poll; recovery attempts stay paced by the existing 100 ms read-check slot) for that self-recovery before racing it — only when the sidecar is merely missing and reconstruction is actually needed; latched- disabled and healthy paths return immediately, unchanged. Complements the branch-side fix d4d4c82 (contention-suite fault seam moved to getBlocks after #5198's batched view verifier made the old getMessages seam stop firing for tail-less fixtures — the verified root cause of the suite reds on run 41007166321: GH Actions checks out refs/pull/N/merge, so the run picked up #5198 from master even though the PR head predated it).
This was referenced Oct 30, 2026
Closed
ranxianglei
pushed a commit
to Zhishulo/billion-context
that referenced
this pull request
Oct 31, 2026
…i#4942) Resolution notes: - src/persist.ts single hunk: both sides inserted functions at the same spot. Kept ours' restoreBlockContents extraction (from the ranxianglei#5198 era) as its own function, dropped theirs' duplicate 'function buildSession' line (ours' refactored version below it already calls restoreBlockContents), and kept theirs' new export recordToLegacyView immediately above buildSession. The legacyLoad wiring (persist.ts:512) came through auto-merge intact. - src/web/sessions-data.ts: took d-4987's async load shape mapped onto the current process singleton: 'await getStore().loadAsync(id)' (the branch-local getDiskStore() local is obsolete since ranxianglei#5002 unified the store). - store.ts/sessions-data.ts auto-merge verified by full tsc parse.
ranxianglei
pushed a commit
to Zhishulo/billion-context
that referenced
this pull request
Oct 31, 2026
…ViewRows/readSnapshot) to the stub d-4987's hermetic seam suite predates the C-class narrow-read additions to StorageBackend; satisfy them from the same captured rows so the stub stays interface-complete without changing test behavior.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
存储层内存治理：五个 PR 合并成一个（一个 PR 解决五个存储层的"越大会话越吃内存/越慢"问题）
这一支把讨论 #3916 里"存储层 · 会话数据物化/验证/增量"的五个独立修复整合成一个 PR，一次合并全部落地。每个修复各自独立成立，互不冲突；冲突处按"两个功能都保留"的原则做了并集。
修了什么（人话版）
JSON.stringify再整体哈希，会话越大单次保存峰值越高（合成 329 MiB → 76 MiB 量级）writeNow+fsync<base>_<n>容错匹配，指针不再静默卡死；③收尾落盘改 500ms 防抖scheduleSave，不再每轮同步刷盘整合决策（五稿各自的冲突怎么裁的）
UnifiedStore.getMessages的选项在 [PR] fix: bound ordinary session loading and avoid full reloads for lazy fields #4765（seqs指针级取行）与 [PR] Incremental Lane: Each round only renders newly added messages; no more full-block re-rendering + synchronous full flush | 增量车道:每轮只渲染新增消息,不再整段重渲染+同步全量落盘 (#5155) #5156（fromSeqExclusive窗口扫描）间取了超集：两个选项并存，同时给定时seqs优先，参数位置化拼装；异步后端接口不受影响（[PR] Incremental Lane: Each round only renders newly added messages; no more full-block re-rendering + synchronous full flush | 增量车道:每轮只渲染新增消息,不再整段重渲染+同步全量落盘 (#5155) #5156 只用同步路径）。getBlockContents因 [PR] fix: bound ordinary session loading and avoid full reloads for lazy fields #4765/[PR] fix: bound ordinary session loads and narrow lazy field reads #4750 各加了一种调用形态（分页行 vs 全量窄图），合并后声明为真正的 TS 重载：带{limit,offset}返回行页，不带返回窄内容图。idx_content_refcount探针、billion-context database cleanup records GC bug | billion-context 数据库清理记录 gc bug #4796 批量 GC 等 master 已有能力原样保留）。测试
npm run pretypecheck+npm run typecheck干净。不在本 PR 范围
关联
Fixes #4529, Fixes #4538, Fixes #4739, Fixes #5155
合入后以下原 PR 可关闭（内容已全部包含在本 PR 且经过并集处理）：#4532、#4540、#4765、#4750、#5156