Repository navigation
[PR] fix(storage): persist a schema reader floor + unblock the notice-store migration - #6347
ranxianglei wants to merge 3 commits into
Conversation
… legacy→kv migration (#5761) A dismissal landing before the cli-boot hydration lazily created the receipts cache, which made hydrateNoticeReceipts bail on `cache !== null` — skipping the one-time legacy JSON→kv migration for that process's lifetime (re-import churn on every boot) and hiding sibling lanes' kv rows until restart; a dismissal landing inside the hydrate's await gap was clobbered by the final cache assignment. Twin of the host-plugin-notice fix that landed in the #5757 line (commit 4d74613), ported onto master's API surface (unifiedKvActive/getStore; the kvStore() probe from that line is not merged yet): - hydrate unions into the existing lazy cache instead of bailing; a new hydrated latch (distinct from lazy-init) gates re-entry - the live-cache snapshot moves to just before the publish with no awaits in between, so mid-hydration dismissals survive the final assignment - publish only on success: a failed listKvAsync now leaves the cache null (legacy-file fallback stays live, next call retries) instead of clobbering it with an empty/partial map for the process lifetime Tests: two regression cases mirroring the #5757 shapes — both fail on the pre-fix code and pass with it; adjacent notice suites green. (cherry picked from commit cf85588)
…ntics (cherry picked from commit 0b2c949)
…s a store it cannot read (#5788) An older build opening a store a newer build migrated with a NON-ADDITIVE change would silently write rows it cannot read back (rolling upgrade / rollback). Persist, per store, the oldest schema version the writing build can still read (STORAGE_MIN_READER_VERSION / MYSQL_MIN_READER_VERSION, both 8 — v8 is the only non-additive migration, the identity backfill). The floor and the stamp move in ONE transaction (a torn pair would under-report the floor and let a too-old build in). On open, a build below the stored floor REFUSES TO START (same bucket as CANTOPEN/readonly — the data is newer, not corrupt; degrading would split-brain, #3923). A missing or malformed floor is unknown and passes, so existing stores never false-refuse. sqlite: kv_store(storage, schema_reader_floor) + user_version in one txn; gate in the from>=version no-op branch. mysql: bili_meta schema_reader_floor + schema_version in one txn; gate in init(). shadow.ts: dedicated refusal branches naming the real remediation (upgrade bili / restore a pre-upgrade backup) instead of the generic tail, whose "move index.db aside to force a fresh rebuild" would destroy the newer build's data. Tests: tests/storage-reader-floor.test.ts — sqlite hermetic (invariant, migration writes floor atomically, refuse arm, pass arms: current / missing row / additive future bump) + mysql hermetic (typed driver fake: refuse arm, pass arms, migration writes stamp+floor in one txn).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.9250-pr.6347.738e7a4This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr6347.tgz
npm install -g packageInstall instructions are refreshed on each push. |
ranxianglei
changed the title
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — #5768 是本 PR cherry-pick #5761 一半的来源 PR(patch 逐字节一致、作者信息保留),不是并行工作;本 PR 合入后 #5768 被取代,owner 关闭败方。 方向关(①): 两个根因都真实存在且定位正确。#5761 根因是 hydrate 重入门用 回归审查(②): 无无意回归。触碰路径既有行为逐项核验:
代码审查(③): 两后端门等价性成立(sqlite: floor > version → 拒绝;mysql: current > version && floor > version,功能等价);原子性论证成立(floor ≤ 写入 build 的 stamp,故 floor > runningVersion 蕴含 stamp 高于 runningVersion);torn pair 会低报 floor,所以 stamp+floor 必须同事务——已实现。 测试有效性(④): 真测试、有牙,#3617 双向门——sqlite 6 条(不变量 floor≤version;迁移写 floor+stamp;拒绝臂精确错误字段;3 个通过臂:当前版本+已知 floor / 无 floor 行 / 加性未来 bump),mysql 5 条(typed driver fake;拒绝臂;通过臂含「无迁移时不重写 floor」;迁移臂断言 stamp+floor 在同一 begin..commit 窗口),notice-store 2 条回归测试在旧代码上会失败(预水合 dismissal 阻塞迁移;水合中 dismissal 被最终发布覆盖)。 diff 整洁度(⑤): 6 文件 +591/−35,每行都在范围内;无 CHANGELOG 条目——与近期 fix PR(如 #6326)惯例一致,不阻塞。 八问:
CI: 12 绿 + 16 跳过,0 失败;2 个 test shard 仍在跑(ubuntu-24 shard 1、ubuntu-22 shard 3)——全部 windows shard、mysql-contract 真驱动车道、static gates 已绿。 |
Root-cause batch for the
[根因] storageaggregation (mirror #6267). One PR, two root causes, both in the storage / notice-persistence layer:#5761 — notice-store hydration race (cherry-picked from #5768)
src/notice-store.ts: a dismiss landing between process start and cli-boot hydration bailed the whole-process hydration (thecache !== nullgate), skipping the one-time legacy JSON→kv migration and hiding sibling lanes' kv rows in-process. The fix unions the hydrate into whatever lazy cache exists instead of bailing, with a distincthydratedlatch and a live-cache snapshot just before publish (publish only on success — a failed hydrate leaves the legacy-file fallback live and retries next call).Commits (cherry-picked with
-x, original authors preserved):13e7f93d26ci-check — the fix6ef363c9feework-review — comment alignment#5788 — schema reader floor
The schema stamp only says "how far a store was migrated" — nothing about whether an OLDER build can still read the result. An older build opening a newer-migrated store would no-op (sqlite) or re-run additive DDL (mysql) and then silently write rows it cannot read back.
The fix persists, per store, the oldest schema version the writing build can still read (
STORAGE_MIN_READER_VERSION/MYSQL_MIN_READER_VERSION, both 8 — v8 is the only non-additive migration, the identity backfill):kv_store(storage, schema_reader_floor)+user_versionin ONE transaction; gate in thefrom >= versionno-op branch.bili_meta schema_reader_floor+schema_versionin ONE transaction; gate ininit().A missing or malformed floor is UNKNOWN and passes, so existing (pre-feature) stores never false-refuse. The floor and the stamp move atomically (a torn pair would under-report the floor and let a too-old build in).
Behavior change (disclosed): opening a store whose persisted reader floor is above this build's schema version now REFUSES TO START where it previously opened (and silently wrote). This is the intended fix — the previous open-and-write was the data-corruption footgun. Pre-feature stores (no floor row) and additive-bump stores (floor ≤ this build) are unaffected.
Tests:
tests/storage-reader-floor.test.ts(sqlite hermetic: invariant, atomic floor write, refuse arm, pass arms; mysql hermetic via a typed driver fake: refuse arm, pass arms, stamp+floor in one txn).tests/notice-unified-store.test.tscarries the #5761 regression cases.Relationship to PR #5768
#5768 fixes #5761 alone. This batch PR cherry-picks its two commits (authorship preserved) and adds the #5788 reader floor. When this merges, #5768 is superseded — please close whichever loses. If #5768 merges first, I'll rebase this onto it.