Skip to content

[PR] [storage] Degradation Semantics Bundle: Seven accepted fixes consolidated into one branch — a single merge suffices (inc | [storage] 降级语义全家桶:七个已验收修复合成一个分支,合并一次即可(含撤回损坏库自动改名) - #5213

Merged
ranxianglei merged 69 commits into
masterfrom
2026-10-30_classD-downgrade-consolidation
Oct 31, 2026

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

[storage] 降级语义全家桶:七个已验收的修复合入一个分支,合并一次即可

这七个修复各自都是一个独立的 PR(见文末),但七条分支都停在旧基线上,而 master
在这之后又动了存储层三次(#5193 GC、#5198 内存治理、D2 常驻上限),导致每一条
单独提都会和另外六条 + master 撞在同一批文件上
(store.ts 被四条同时改、
shadow.ts 被四条同时改),谁先合谁干净,后面六个全部要重新解冲突。

这个 PR 把七条分支按依赖顺序全部并到最新 master,冲突一次解完、类型检查过、
全部相关测试跑绿。合并这一个 PR,下面七件事一起落地:

⚠️ 本 PR 撤回了 master 的一个既有行为(owner 裁决,2026-10-30)

#5123 的原始方案沿用了 #4175 的老做法:确认 index.db 损坏后,进程自动把它
重命名成 index.db.corrupt-<时间戳> 再开一个空库
。这个自动改名被撤回了,
理由:改完名并不能证明数据全部可恢复——checkpoint 行只存在 index.db 内部
(JSON envelope 不保证是它的完整副本,BILI_STORAGE_JSON_WRITER=off 模式下
index.db 更是唯一持久副本),把一套无法验证完整的文件自动改名走,风险是
销毁唯一副本却仍然没有换来“已证明的恢复”。

新行为:quick_check 可证伪地失败时,所有文件原地一字不动;服务大声降级到
JSON envelope 路径(不停服、不断流),FAIL 结论写入旁路缓存(下次启动直接
降级、不再重扫),日志与 /__bili/status 给出人工处置指引(先备份再把文件集
挪走);照做后下次启动自动重建空库、经 envelope 补回会话——测试里 e2e 钉住。
分类器不变:只有真正的损坏形状才走这条路径,BUSY/LOCKED/IOERR/CANTOPEN
等瞬时错误照样原样上抛、绝不触发降级换库。

修了什么

# 原来的毛病 修复后
#5123 index.db 损坏时的处置链断断续续,且旧方案的自动改名有毁掉唯一副本的风险 损坏库保持原位一字不动,服务大声降级继续跑,状态页直接告知人工处置步骤;算子挪走坏文件后启动即恢复(见上方“撤回”一节)
#4955 SQLite 连接降级(如"rollback mask"误伤)后整进程永久降级,只能重启才恢复 故障源消失后下一次操作自动探测并恢复,不再需要重启
#4987 MySQL 模式下打开/加载会话是同步阻塞的,大库会卡住事件循环 装载与预热走异步通道,卡点消除(附带一套 hermetic 接缝测试)
#4670 JSONL 导出把 retired(归档/退役)行整个漏掉 导出带上 retired 行并标出 retiredAt
#4672 归档→恢复往返后身份字段/retired_at 丢失或错位 往返保持身份与时间戳完整
#4630 "脑裂"(两个实例抢写)告警一旦弹出就永远挂在面板上,就算事实早已不存在 告警按最新证据自动清除(证据仍存疑时 fail-closed 保留)
#5075 实例列表里"第二写者"警告只看车道产物文件判断,判断依据错了会误报 按 sessions 目录判据,不再误报

冲突怎么解的(要点)

  • union 原则:两边都是真修复的地方一律全留,不做取舍。唯一的实质重叠在
    [PR] fix: stale split-brain notice clears itself once the peer stops contesting (#4626) #4630(脑裂自动清除)与 master 上 [PR] release v0.1.209 #4982 已合的"优雅停机标记":两者给 marker
    加了不同名字的时间戳字段,这里让 dispose() 两个字段都写、读侧任一命中即
    视为优雅退出
    ——新旧构建互相可读,四组相关测试原样通过。
  • persist.ts 一处双方都在同一位置插函数的冲突:保留本侧 restoreBlockContents
    抽取 + 对侧新增导出 recordToLegacyView(MySQL 接缝测试要用),去掉重复的
    buildSession 声明。
  • web/sessions-data.ts 取 d-4987 的异步装载形态,映射到 [PR] fix: web overview rides the persist store singleton instead of opening a second SessionStore #5002 之后的进程单例
    getStore().loadAsync()(d-4987 基线里的局部 getDiskStore() 已被统一掉)。
  • MySQL 接缝测试的 stub 补齐了 C 类之后接口新增的 sessionViewRows/readSnapshot/ retiredAt 成员(stub 行为不变,仅接口对齐)。

验证

备注

追加:第二轮裁决(2026-10-30,提交 f05a405)——连降级也撤回,可证损坏 = 拒绝启动

owner 在"不再自动改名"之后进一步裁决:回退 legacy-only 也不行——回退后 JSON
envelope 与死掉的索引库各走各的、永久分叉、无法调和;宁停勿降。据此:

  • shadow.ts maybeOpen:StorageIntegrityError(quick_check 证明的库损坏,
    白名单分类器判定)不再吞掉降级,原样上抛 → 构造即失败 → 进程拒绝启动,
    ERROR 日志带完整手动处置步骤(备份 index.db{,-wal,-shm} → 挪走 → 重启,
    新库自动从 JSON envelope 重建)。启动承接已核实:server.ts:559 在 listen()
    之前首调 getStore(),抛错直接上溢退出。
  • 瞬时错误照旧降级不变(文件没坏、不存在分叉问题):无引擎 / CANTOPEN /
    只读 / 第二写者 / MySQL 失败。
  • FAIL 判定缓存到 .chk,后续每次启动秒失败不重复扫库(大库后台检查在运行期
    查出损坏时只报警+缓存,强制点在下次启动——这是有意的设计边界)。
  • runtime-status 文案改 "REFUSES TO START…";CONFIGURATION.md QUICK_CHECK 行同步。
  • 四个钉旧行为的测试改写为钉新行为:issue4816 shadow facet、storage-durability、
    storage-review-fixes review 3、engine-proofs 子进程场景(checks 收窄为
    noAutoRename/refusesToStart/midFlowGuard/fdGuard——拒绝后不得泄漏句柄,
    否则 operator 挪文件会被 EBUSY 挡)。原 [Bug][storage] corrupt index.db bricks SessionStore construction in shadow/full mode (violates the sidecar never-breaks-legacy policy) #2771 "必须存活" 被此裁决显式取代,
    仅针对可证损坏;瞬态失败仍存活。
  • 追加验证(合入你推进的 ci-fixes 红修后重跑):typecheck 0 错;
    issue4816/durability/review-fixes/issue4666 24 tests 21 pass 3 skip 0 fail;
    engine-proofs+integrity-cache 18/18;此前一轮:web-banner+issue-cache+
    instance-governance+second-writer+rollback-mask 39/39、persist 三件套 14/14。

Fixes #4816, #4614, #4949, #4942, #4664, #4665, #4626, #5046

Supersedes(合并本 PR 后以下原 PR 请关闭,代码已全部进入):
#5123, #4955, #4987, #4670, #4672, #4630, #5075

追加:第三轮裁决(2026-10-30,提交 5159ce9)——任何情况下都不降级

Owner 裁决:“为啥打开还降级,需要在任何情况下不降级”。所有统一存储打开失败现在都拒启动(listen 前抛错退出),不再有静默 legacy-only:

失败 旧行为 新行为
quick_check 证伪损坏 (第二轮起)拒绝启动 不变
没可用 SQLite 引擎 warn + legacy-only throw → 拒启动
CANTOPEN/只读/OS 级 DIO warn + legacy-only throw → 拒启动
同 lane 第二写者 throw(第二轮起) 不变
MySQL 无 DSN / 连接失败 warn + legacy-only throw → ready() reject → 拒启动
  • 承接点确定化:startServer 在 listen() 前 await store.ready(),失败即 process.exit(1);构造期不炸、就绪期 reject,persist.attachMysql 挂 mysqlReady.catch(()=>{}) 防 unhandled rejection。
  • 刻意不动的选择(不是失败):BILI_STORAGE_UNIFIED=legacy/unset、BILI_ENCRYPTION_KEY 显式 opt-out、BILI_PERSIST=0。要 JSON-only 服务就显式声明。
  • 测试重钉新契约:issue5046(throws)、banner 子进程(构造即拒+wire 点名原因)、engine-proofs、cantopen-hint 三种形态、acp-status(改由 recordSidecarOpenFailure 驱动展示面)、multi-instance-live(B 实例启动即死 exit 1,原双活 interleave 变为 by-construction 不可达,envelope guard 仍在 multi-instance-persist 单测层)、mysql-seam gate facet(seed 先于 env 翻转 + ready() 必须 reject)。
  • 验证:typecheck 干净;受影响面+回归共 33+55+31(+1 skip) = 119 pass / 0 fail(seam/5046/banner/cantopen/multi-instance-live/acp-status/engine-proofs/issue4816/durability/review-fixes/integrity-cache/governance/4949/multi-instance-persist/persist×2/unified-ext/backend-contract/backend-knob)。
  • 文档同步:CONFIGURATION.md QUICK_CHECK 行、driver.ts none 条目、store.ts 类注释与后台检查注释统一为 refuse 措辞。
  • 运行期两个残留点(本 PR 未动,需 owner 另行裁决):①运行中大库后台检查 FAIL 只报警+.chk 缓存,强制点在下次启动;②ingest 连续失败的运行时 breaker 仍会自禁用回 envelope(根因持续则下次启动会被拒)。

追加:第四轮裁决(2026-10-30,提交 b9eb27a)——运行中永远不放弃索引写入;后台检查 FAIL 只报警永不拒启动

第三轮留下的两个运行期残留点,owner 本轮全部裁决:都撤掉阻断/禁用机制。

① 撤掉 ingest 熔断(3-strike self-disable):连续索引写入失败后,旧行为是第 3 次起跳过所有后续索引写(reads 翻回 legacy-only、fork 操作 fail-closed),5 分钟后再探活。新行为:MAX_CONSECUTIVE_FAILURES / REENABLE_PROBE_COOLDOWN_MS / noteProbeFailure / clearBreaker 等熔断机制整体删除——每次 save 无条件重试真实索引写(首次失败带文件名+错误详情的 warn 保留,可见性不降),读路径永远 unified,fork 操作不再因"已禁用"拒绝。约 13 处 stats.disabled 门控行删除(字段本身恒 false 保留,panel/wire API 形状不变)。persist 侧随之删掉 #3135① 的 breaker-中强制 JSON 补写块与 writerOffActive() 的 disabled 条件(jsonWriter=off 即接受 index-only 持久化,语义不变)。

② 完整性检查 FAIL 改纯报警:运行期发现(大库后台 worker 检查 FAIL、扫描抛损坏)只打 ERROR 日志 + runtime-status 展示,绝不写 .chk FAIL 缓存、绝不杀进程、绝不让下次启动因此被拒。每个启动对当前文件集做自己的新鲜 quick_check,用自己的证据决定。旧版本遗留的 .chk FAIL 视同缓存失效→重扫(stale verdict 永不挡 boot)。PASS 缓存保留(跳过重复全扫)。本 boot 自己扫出可证损坏 → 仍拒绝启动(第一/二轮裁决,保持不变)。

  • 文档同步:CONFIGURATION.md QUICK_CHECK 行 + BILI_STORAGE_UNIFIED 条目改写为上述两套新语义;README 存储段同步。
  • 测试重钉:issue4949(facet B/C = 失败不致残+源码钉死熔断常量已除)、issue4982 facet1(stub 恒 disabled 仍尝试 queued fork,backend 真实被调)、storage-integrity-cache facet3(stale FAIL 不挡 boot、新鲜重扫决定)、issue4816(FAIL 永不落缓存/每 boot 新鲜证据计数 +1、worker facet=报警面不落缓存且次启仍由自己的新鲜扫描拒绝)、engine-proofs("self-disable after 3"改为"NEVER self-disables"+源码断言)。
  • 验证:typecheck 干净;定向+回归五批 16+28+19+40+23 = 126 pass / 0 fail(4949/4982/integrity-cache/4816/5103/2432/engine-proofs/durability/review-fixes/banner/cantopen/governance/acp-status/multi-instance-live/multi-instance-persist/persist×2/unified-ext-watermark/backend-contract/backend-knob/mysql-seam)。

追加:第五轮裁决(2026-10-30,提交 1766e2d)——可证损坏也照常启动:只报警、文件一个字节都不动、绝不自动恢复

owner 原话(m02675):"拒绝这样做.不要搬家 不要复制 直接启动 只给警告和修复方案 不要从json恢复 有灾难性后果"。此前矩阵最后一格"boot 自扫证明损坏 → 拒绝启动"被翻掉,最终矩阵:

情况 行为
boot 自扫(PRAGMA quick_check)证明损坏 照常启动:index.db 及 -wal/-shm 原样保留在原地——不搬移、不复制、不自动重建;响亮 error 日志 + status wire 告警指明手工修复步骤(备份后把 index.db{,-wal,-shm} 手工移开再启动,全新索引会从 JSON 信封重建);本进程改由 JSON 信封服务
运行期后台检查 FAIL / 连续 ingest 失败 同上:只报警、永远重试、永不拒启动(第四轮已做)
瞬态打开失败(无引擎 / CANTOPEN / 只读 / 第二写者 / MySQL DSN) 仍拒绝启动并给出可操作错误(环境类问题不允许静默走 legacy)
显式选择 legacy / 加密 opt-out / persist off 不变(是选择不是失败)
  • 为什么不自动恢复(撤回 Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 式 rename+重建的理由):checkpoint 行只存在于 index 内部;BILI_STORAGE_JSON_WRITER=off 时 index 是唯一 durable 副本——自动搬走/重建无法保证完整数据恢复,后果比拒启动更严重。因此告警给的是手工步骤,机器只做报警。
  • 无损安全网:writerOffActive()(src/persist.ts)要求 sidecar 存活才跳过信封写入 → sidecar 没起来时即使 jsonWriter=off,每一轮也照写 JSON 信封,不存在无持久副本的窗口(JSDoc 同步更新)。
  • 改动面:src/storage/shadow.ts(内层 catch:typed IntegrityError → 响亮报警 + return null,其余失败仍 throw 拒启动)、runtime-status.ts wire 文案(left UNTOUCHED + starting WITHOUT the index + 手工修复步骤)、store.ts/sqlite-errors/persist 注释、CONFIGURATION.md(+zh-CN) QUICK_CHECK 行按两类分别重写。UnifiedStore.open 内部仍抛 typed 错误作为层间信号(export/CLI 直面坏库时照样响亮失败)。
  • 测试重钉(全部通过):corrupt-index-child engproof 场景=构造成功 + wire 告警点名修复步骤 + 会话仍能从信封加载 + 文件字节级未动 + 句柄泄漏双守卫保留;engine-proofs 标题/注释;issue4816 shadow facet(maybeOpen→null)+ worker facet 措辞;storage-durability facet(maybeOpen→null);review-fixes review 3(两次启动都成功且文件原地);issue5103 facet(首错即 warn 带路径+"RETRIED on every subsequent save",永不自禁用,同错去重一行)。
  • 验证:typecheck 0 错;定向回归六批共 ~180 tests 全绿(issue4816/durability/review-fixes/integrity-cache 25p、engine-proofs 12p、5103/2432-breaker/banner/cantopen/governance/acp-status-storage-degraded/4949/4982 71p、multi-instance-live/persist×2/unified-ext/backend-contract/knob/mysql-seam 37p 含 1 skip),已知红基线 issue3304×2 + issue3157×2 与本 PR 无关(master 基线同红)。

追加:第六轮(merge origin/master,提交 b8ba5d7)——与 #5188「sidecar 可恢复打开」收敛

master 已合入 #5188(fork/snapshot 门禁):sidecar 在启动时因瞬态故障没起来的话,读路径/保存路径会按冷却窗(默认 30s)驱动重试打开——这套需求驱动的自动恢复保留,但收敛进本 PR 的最终政策:

追加:第七轮(提交 28454a1)——收敛后 CI 红的三处收口

40bcb2f 与 28454a1240bcb2f1 与 28454a1 两个收尾提交修掉了上一推(b8ba5d7f)留给 CI 的确定性红灯:

  1. plugin-fork [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 cold-restore facet:本仓测试基线全局 pin BILI_STORAGE_UNIFIED=legacy(preload-test-env.mjs),legacy 下 sidecar 永不打开 → index-reconstruction gate 永远拿不到 sidecar → snapshot endpoint 稳定 409 replicasUnavailable。facet 现在显式 pin full。
  2. plugin-fork broken-dir refuse-to-start facet + multi-instance-live 子进程:同因——拒启动断言与「活的外部写者」前提在 legacy 基线下不存在,全部 pin BILI_STORAGE_UNIFIED=full。
  3. export-jsonl mysql-without-DSN:第三轮裁决后无 DSN 的 mysql 直接在打开时拒绝,断言改为实际 refusal 消息。
  4. env-only.yaml i18n 修复(28454a12):feat: OpenAI Responses API support (/v1/responses) #4 的 docs-sync 修复删掉了 BILI_STORAGE_QUICK_CHECK 的 desc_zh 行(en/zh 键集失衡,generator 五字段必填 → docs-sync gate 必红)。已按 CONFIGURATION.zh-CN.md 中已提交的终审措辞逐字补回;tools/gen-config-docs.mjs generate 四文件零 diff,check-coverage rc=0。

验证:plugin-fork + multi-instance-live + export-jsonl 本地 63/63 绿(node --test --test-concurrency=1);此前全量回归(六批 ~200 测试)与 typecheck 见上轮数据。

追加:第八轮(merge origin/master,提交 78a0d4a / 7ffda72)——收敛 #5239 + #4599

master 推进了两个与存储相关的修复,已合并进本分支:

  1. [PR] fix: read the checkpoint tip-seq as one MAX(seq) aggregate, not a full-log scan #5239(tipseq 窄读):checkpoint tip-seq 从整日志物化降为单条 MAX(seq) 聚合(sessionTipSeq)。本分支侧的裁决:保留 master 的聚合实现,去掉其中的 breaker 时代 assertLive() 调用(该方法随三轮裁决的熔断器一并删除)。
  2. CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599(master 红面收口):src/persist.ts 的 [Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788 debounce-timer ref/unref 宿主修复——有等待者的 flush 判定期间保持 timer 引用;与本 PR 的 D2 spill 路径(afterNextFlush 等待者)语义兼容,自动合入。
  3. 冲突裁定:shadow.ts 通用打开失败日志保留第三轮 "REFUSING TO START" 政策文案(弃 master 旧 "staying legacy-only");plugin-fork.test.ts 两处 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 pin 取 master 版(pin 内容相同、注释更准)。
  4. 同窗口并行提交的 627c2580(storage gc full VACUUM holds the write lock across the whole O(db) rewrite with no progress — online proxy shadow disabled mid-run | storage gc 全量 VACUUM 全程持写锁且无进度:在线代理影子层被降级 #4612 CLI 边界拒绝 + storage-full-flip 重钉)已并入并验证。

验证:tsc 干净;tipseq+4739+4788+inc-canonical-pointers 12/12;plugin-fork+5140+5188+multi-instance-live 62/62;完整性政策核心六件 29/29;storage-full-flip 10/10。

ci-check added 30 commits October 25, 2026 18:04
#4664: since #4390 the unified store's getMessages filters retired_at IS
NULL by default (tombstone-based rewind history), so the JSONL export
silently dropped every rewound message. The store's own design comment
names export a historian caller that must opt into includeRetired — it
never did.

- widen the StorageBackend contract: getMessages(sessionId, opts?) with
  includeRetired and retiredAt on every row (both backends already
  implemented this natively; SqliteBackend now forwards it)
- renderSessionJsonlFromTables reads with { includeRetired: true } and
  emits retiredAt on tombstoned rows only — live rows omit the field, so
  output for sessions without retirements is byte-identical to before

Regression test in tests/export-jsonl.test.ts drives the real
ingestSession sweep (full ingest -> truncated re-ingest retires row one).
The archive dump SELECT and restore INSERT predated #4390's v8 row-state columns: every restored row landed with identity='' (the full-ingest JOIN key can never match again -> next full save re-mints every message beside the restored copy = duplicate messages) and retired_at=NULL (a rolled-back row resurrects as live). Both backends now carry both fields verbatim; pre-fix dumps lacking them fall back to recomputing identity from the CAS body (the v8 migration backfill formula) and restore retired rows as live (state unrecoverable there).
…testing (#4626)

The surfaced contest was persisted into the storage runtime state at
detection time and replayed verbatim on every /__bili/status read until a
manual ack — but detection fires while the restarting peer's tombstone
marker is still seconds fresh (its graceful-shutdown dispose() rewrites
the marker), so a restart handover left an alarming card behind for hours
after the peer pid was gone.

The stored verdict now re-evaluates against CURRENT evidence at the status
read, with the same never-freeze discipline as the sibling second-process
warning: the server registers the very scan that produced the verdict
(findContestedWriters over its own writers dir), and when that scan no
longer reports the accused peer (pid gone + marker past its freshness
grace, or the contest window lapsed) the entry clears itself with an info
log line. Without a registered check the legacy replay behavior stands
(standalone record/ack use). The fresh-tombstone case keeps persisting:
a dead pid alone is not proof of absence on every platform, so the
notice stays honest while marker freshness says the peer may be alive.
…#4626)

slice(0, 8) of a "session-…" id renders literally "session-" — the ack
log line and the web banner both showed an unusable prefix. Log the full
id (the [split-brain] error line already does) and give the card a 24-char
prefix so distinct sessions are tellable apart.
…ecover the unified-store breaker (#4949)

Two defects in one failure domain, both observed in production on v0.1.206
after host storage exhaustion:

1. Error masking: when an I/O-class failure (disk full / SQLITE_FULL) aborts
   BEGIN or COMMIT, SQLite auto-aborts the transaction, so the follow-up
   ROLLBACK in the node:sqlite emulation throws 'cannot rollback - no
   transaction is active' and replaces the real error before it reaches the
   sidecar breaker's lastError / status wire. The rollback attempt is now
   guarded so the original error always wins. The better-sqlite3 seam gets
   an annotated message at our boundary because that library loses the
   original inside its own wrapper.

2. Permanent circuit breaker: stats.disabled was set on 3 consecutive
   failures but never cleared — the unified store stayed disabled for the
   whole process lifetime even after the disk was freed (only a restart
   recovered it). Both sync and async sidecar paths now run a cooldown-gated
   writability probe (REENABLE_PROBE_COOLDOWN_MS = 5min, one probe per
   window on the next save) through the same transaction path production
   writes use; success re-enables the store and the triggering record
   ingests immediately. Missed sessions converge via the existing envelope
   staleness guard at load plus the next full ingest.
- A: injected engine-level COMMIT abort over a real DatabaseSync proves the
  original error surfaces (pre-fix this failed with exactly the reported
  'cannot rollback - no transaction is active' mask); aborted rows do not
  persist.
- A2: negative control — nested transactions on a constraint failure still
  roll back cleanly and propagate the original (gate both directions, #3617).
- B: full breaker loop on a real store — 3 strikes disable with the true
  cause visible, cooldown gates probe frequency (exactly one probe per
  window), healing + one probe re-enables, triggering record lands, unified
  reads work again, exactly one RE-ENABLED log line.
- C: pins REENABLE_PROBE_COOLDOWN_MS at the documented 5-minute cadence.
…ckends (#4942)

MySQL-backed unified storage had no synchronous read facade: viewFor()
returns null and listSessionIds() returns [] on network backends, so
every hot-path read (cold load after restart, LRU eviction reload, boot
enumeration) fell back to local JSON envelopes. Any deployment that
stops writing envelopes lost every session at every process restart.

Owner decision on #4942: option A (async pre-warm + async reload seam),
keeping the byte-bounded resident pool intact.

- SessionStore.loadAsync: envelope/index-first (zero behavior change
  whenever the sync path serves), DB fallback only when the sync read
  misses AND the sidecar is an async backend; awaits ready() then
  viewForAsync().
- getSessionAsync: single-flight variant of getSession for async call
  sites (concurrent first-touches share one load); main request entry
  in server/handle.ts switched to it.
- boot/bootBounded/loadAll unions now enumerate DB ids via
  sessionSummariesAsync() (newest-first under the existing
  maxSessions/maxBytes budget) when a network full backend is expected;
  sqlite/file paths unchanged.
- resolveForkConversation / publishForkSession / maybeAdoptForkBlocks /
  reconcileDerivedLineage / handlePluginStatus made async (all callers
  already in async contexts).
- UnifiedShadow.fromBackend factory + exported recordToLegacyView for
  hermetic testing against stub backends.

Deliberate sync boundaries left as-is (documented in PR body):
hasProcessedState, resume-inheritance parent probe, decompress chain
walker.

Tested: new hermetic tests/storage-mysql-seam.test.ts (typed stub
implementing the full StorageBackend contract, no DSN required).
Related pre-existing ingest-verify finding filed separately: #4986.
…ane repair)

The new seam suite shipped red at the PR head (verified against the pre-rebase
commit): 8 tsc errors from a top-level 'blocks' access that does not exist on
Session (blocks live under state), and two over-strict reference-identity
assertions between separate cold loads.

- .blocks -> .state.blocks (4 sites): mechanical API alignment; makes the
  typecheck lane pass.
- assert.equal(viaAsync, viaSync) / assert.equal(b, a) -> deepEqual (2 sites):
  every cold loadSync builds a fresh Session object (no cross-call caching), so
  reference identity was never part of the contract - even loadSync-vs-loadSync
  differs. The real contracts stay pinned: envelope precedence, zero extra DB
  reads while an envelope exists (the viewCalls guard is unchanged), and
  no-sidecar parity of loadAsync over loadSync.
resolveForkConversation probed with loadAsync but materialized with the
sync getSession. On a network storage backend a DB-only conversation id
(no local envelope/index copy) then minted an EMPTY resident ghost, and
that ghost poisoned every later getSessionAsync of the same id for the
process lifetime — plugin status/fork/snapshot surfaces operated on an
empty session and fork children inherited zero blocks (#4942 seam).

Materialization now rides the same async seam as the probe. Regression:
the seam suite drives handlePluginStatus for a DB-only corpus session
and asserts the resident pool holds the DB state (fails on pre-fix code).
…4987 CI)

#4942 made handlePluginStatus() and reconcileDerivedLineage() async (and
maybeAdoptForkBlocks()/publishForkSession() in the same seam). Six existing
suites still called them synchronously and asserted on the response/log
before the microtask completed, red across 7 of 9 CI shards:

- tests/issue404-restore-timestamps.test.ts: JSON.parse of the empty mock body
- tests/advisory-panel.test.ts, tests/update-notes.test.ts,
  tests/issue2029-status-baseline-provenance.test.ts: 0 !== 200 on the mock res
- tests/lineage-reconcile-boot.test.ts (AC10): 'not on disk' log asserted before
  it was written
- tests/fork-ccr-adoption.test.ts: same class, passed only because the parent
  stayed resident (sync completion by luck) — aligned too
…ync-reload-seam

# Conflicts:
#	src/plugin.ts
…ry, not lane alone (#5046)

The storage-side second-writer guard keys on the sessions directory while
the [instances] guard keyed on lane overlap — after isolating a second
install via BILI_SESSIONS_DIR, the warn kept claiming 'both processes will
write the same sessions directory', a factually wrong false positive that
drowns the real signal. Registry markers now carry the resolved sessions
dir; when both sides report different dirs the notice drops to info level
(unknown dir on either side stays conservative = warn). Both the startup
warning and the #2401 slow-tick rescan use the same criterion.
…#5046)

Two installs on one machine (e.g. two DSH profiles) default to the same
XDG sessions dir + config; with storage.unified on they double-write one
index.db (SQLITE_BUSY → failed turns). Document the BILI_SESSIONS_DIR +
BILI_CONFIG_FILE isolation recipe in both languages.
…tion; middle-truncate long degrade reasons (#5108)
ci-sync and others added 5 commits October 30, 2026 23:26
- plugin-fork: pin the #5140 cold-restore snapshot test to unified-full —
  under the global legacy test baseline its sidecar never opens, so the
  index-reconstruction gate can never pass (409 replicasUnavailable sidecar-missing)
- plugin-fork: pin the new broken-dir refuse-to-start assertion to full mode;
  legacy never touches sqlite at all so the ctor cannot throw there
- multi-instance-live: spawn children with BILI_STORAGE_UNIFIED=full despite
  the global legacy baseline pin, else instance A holds no index.db.lock and
  B has nothing to refuse over (boot-refusal needs a live foreign writer)
- export-jsonl: mysql-without-DSN now refuses at open time — assert on the
  actual refusal message instead of the pre-refusal #3388 tag
- env-only.yaml: sync BILI_STORAGE_QUICK_CHECK seed with the final alarm-only
  policy wording already shipped in CONFIGURATION.md/.zh-CN (docs-sync gate)
…n parity)

The convergence fix 40bcb2f re-synced desc_en of BILI_STORAGE_QUICK_CHECK
but deleted the desc_zh line — env-only.yaml went 65 zh / 66 en while every
other seed file stays balanced and the generator requires all five fields
(type/default/group/desc_en/desc_zh) on each entry. Result: the docs-sync
gate fails at the generate step before it ever reaches its diff check.

Restores the exact final-form Chinese wording already committed in
CONFIGURATION.zh-CN.md (alarm-only: corruption proves itself by THIS boot's
own scan, files stay in place, service keeps running on envelopes with a
loud alarm; FAIL never cached; transient open failures still refuse).
Verified: tools/gen-config-docs.mjs generate -> zero diff on all four
generated files; check-coverage rc=0.
- storage gc --vacuum full: the no-degradation constructor refusal makes the
  old in-runGc guard unreachable — catch StorageSecondWriterError at the CLI
  boundary and refuse with exit 1 plus the actionable one-line reason (the #4612 pin).
- storage-full-flip: #3818 (iii) repins to refuse-to-start when the full-mode
  sidecar cannot open; #3818 (vi) proves the removed breaker's same-tick JSON
  valve stays gone — under jsonWriter=off no envelope appears and the save
  lands in the index (index-only persistence is the contract).
 red-family fixes

Conflicts resolved in favor of owner policy + master's improvements:
- src/storage/shadow.ts: generic sidecar-failure log keeps the round-3
  'REFUSING TO START' wording (owner policy: no silent legacy fallback);
  sessionTipSeq takes master's MAX(seq) aggregate (#5239/#897756a7) but
  drops the breaker-era assertLive() line (method removed with the 3-strike
  breaker, 2026-10-30 owner policy).
- tests/plugin-fork.test.ts: both #5140 pins keep master's version (same
  BILI_STORAGE_UNIFIED=full pinning, clearer comments).
- src/persist.ts auto-merge: #4788 debounce-timer ref/unref host fix is
  compatible with the D2 spill path (afterNextFlush waiters re-ref).
Verified: tsc clean; storage-tipseq-narrow-read + issue4739 + issue4788 +
inc-canonical-pointers 12/12; plugin-fork + issue5140 + issue5188 +
multi-instance-live 62/62; integrity policy core (issue4816/durability/
review-fixes/integrity-cache/issue5103/issue4949) 29/29.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.932-master.7c9e65b (rolling master tag).

npm install -g billion-context@0.0.932-master.7c9e65b  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Merge-guard closing note: this PR is already merged (state=closed, merged=true, merge commit 7c9e65b92, merged 2026-10-31 00:17 UTC). The repeated "PR is behind" wake-ups are stale — there is no open PR left to sync. No sync, conflict-resolution, or push was performed this round, and no decision request was filed (the shadow.ts degradation-semantics divergence between this branch and master's #5188 was superseded by the final converged merge). Nothing further to do on the guard side.

ranxianglei added a commit that referenced this pull request Oct 31, 2026
…r-no-block

[PR] fix(storage): 活着的第二写者不再阻止启动——共享一个本地盘 index.db 是受支持拓扑(撤回 #5213 的拒绝策略)
ranxianglei added a commit to Zhishulo/billion-context that referenced this pull request Oct 31, 2026
…decision 2026-10-31)

Sharing one local-disk index.db IS the supported topology (ranxianglei#5082/ranxianglei#5091):
SQLite WAL is multi-process safe locally, so a peer holding <db>.lock is
informational-only now (info log + web/status wire) whatever its lane —
inline lanes never register, so 'unknown lane' was routinely mistaken for
risky and bricked resumes (the 2026-10-31 `bili pi -r` boot failure,
ranxianglei#5213's refuse-to-start withdrawn). Withdrawn call sites: UnifiedStore.open
secondWriterPolicy:'degrade' opt-in (option removed; crossLaneSafe math
deleted), shadow.ts open-catch refusal + recordSidecarOpenFailure line,
runtime-status degradeReason verbatim branch, storage-maintenance gc catch
(the full-VACUUM starvation guard in runGc via store.unifiedActiveWriter()
is untouched). StorageSecondWriterError stays exported (@deprecated) so
external import/catch sites keep compiling. Genuine hazard unchanged: two
instances walking the SAME session still alarm via the split-brain contest
(ranxianglei#3923). Tests rewritten to the never-block contract incl. regression
pins for both formerly-refused shapes (same-lane, unknown-lane fleet).
Docs: CONFIGURATION EN/ZH seed + README multi-instance section updated.
@github-actions github-actions Bot mentioned this pull request Oct 31, 2026
ranxianglei pushed a commit to fathonix/billion-context that referenced this pull request Nov 12, 2026
 is resolved by ranxianglei#5213's async reload seam, which is on master; the gate stays sqlite-only as a separate decision (ranxianglei#4936 sync)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite unified storage degraded to legacy-only (bili v0.1.208, node v24.21.0, win32-x64)

2 participants