Repository navigation
[PR] fix: surface real storage errors after failed BEGIN/COMMIT and auto-recover unified-store breaker (#4949) - #4955
ranxianglei wants to merge 6 commits into
Conversation
…ecover the unified-store breaker (#4949) Two defects in one failure domain, both observed in production on v0.1.206 after host storage exhaustion: 1. Error masking: when an I/O-class failure (disk full / SQLITE_FULL) aborts BEGIN or COMMIT, SQLite auto-aborts the transaction, so the follow-up ROLLBACK in the node:sqlite emulation throws 'cannot rollback - no transaction is active' and replaces the real error before it reaches the sidecar breaker's lastError / status wire. The rollback attempt is now guarded so the original error always wins. The better-sqlite3 seam gets an annotated message at our boundary because that library loses the original inside its own wrapper. 2. Permanent circuit breaker: stats.disabled was set on 3 consecutive failures but never cleared — the unified store stayed disabled for the whole process lifetime even after the disk was freed (only a restart recovered it). Both sync and async sidecar paths now run a cooldown-gated writability probe (REENABLE_PROBE_COOLDOWN_MS = 5min, one probe per window on the next save) through the same transaction path production writes use; success re-enables the store and the triggering record ingests immediately. Missed sessions converge via the existing envelope staleness guard at load plus the next full ingest.
- A: injected engine-level COMMIT abort over a real DatabaseSync proves the original error surfaces (pre-fix this failed with exactly the reported 'cannot rollback - no transaction is active' mask); aborted rows do not persist. - A2: negative control — nested transactions on a constraint failure still roll back cleanly and propagate the original (gate both directions, #3617). - B: full breaker loop on a real store — 3 strikes disable with the true cause visible, cooldown gates probe frequency (exactly one probe per window), healing + one probe re-enables, triggering record lands, unified reads work again, exactly one RE-ENABLED log line. - C: pins REENABLE_PROBE_COOLDOWN_MS at the documented 5-minute cadence.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.7189-pr.4955.613a1efThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4955.tgz
npm install -g packageInstall instructions are refreshed on each push. |
ranxianglei
changed the title
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。 open+closed issue/PR 按失败签名检索( 这是 [PR] 接单:开始对 PR #4955 做独立审核(不复读 #4949 侧结论)——方向关(读原 issue)、回归审查(清点触碰路径的既有行为)、代码审查(逐段读 diff)、测试有效性审查。分支已拉取( |
(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: origin/master (c499b14) merged into |
(ai: vllm-qwen/qwen3.8-27b) CI 红判定:继承性 flake(master 自身红家族),非本 PR 引入——分支侧无代码可修,已触发失败分片重跑。 取证(run 40921903750,head e4cd055,3 个红分片逐一拉日志):
处置:按 CI 协调政策先查 #4599——master 当前红(floor 487),哨兵收敛在 八问
为何不开 decision-request issue:auto-fix 车道要求「需设计决策则立项」,但此 flake 家族已在 #4599 完整在档(floors 485–494:证据、复现率、处置方向「修复收敛 #5144 侧」俱全),按 CI 协调政策(禁止为 CI 问题开新 issue)+ 同症状同机制归并纪律,不另立项;owner 可见性由 #4599 评论保证,非静默等待。 收工判定:等 rerun attempt 2 结果——转绿后本 PR 回到「代码审核通过(floor 7)、待 owner 合并」状态;若再现,即继承 flake 第二证,留档 #4599 待 #5144 侧收敛或 admin 重跑。 |
(ai: vllm-qwen/qwen3.8-27b) Synced origin/master (44067a5, incl. #5188 contention gates and #5211 revert) into this branch and pushed f3eb015 — branch is now ahead by 6 / behind by 0, mergeable=true (checks pending). Two conflicts resolved: (1) textual in src/storage/shadow.ts — both declaration blocks kept (this PR's |
|
已由人合并进 master：本修复（回滚掩码保真 + 熔断自恢复）全部并入 #5213，且熔断按 owner 终审政策整体撤销、改为永不放弃索引写入。本 PR 不再单独需要。 |
Closes #4949
Problem
After host storage exhaustion on v0.1.206, the status wire reported
unified store auto-disabled after 3 consecutive failures (last: Error: cannot rollback - no transaction is active)— and the unified store stayed disabled even after space was freed, until process restart.Root cause (two defects, one failure domain)
src/storage/driver.tsthen unconditionally ran ROLLBACK in its catch, which threwcannot rollback - no transaction is active— replacing the real error before it reached the breaker'slastError/ status wire. Constraint-class failures were never masked (verified empirically); only I/O-class aborts are.stats.disabledwas set at 3 consecutive failures and never cleared anywhere in-process — recovery required a restart.Fix
REENABLE_PROBE_COOLDOWN_MS = 5 min, constant — no new config knob): while disabled, the next save after each window elapses runs one real probe through the same transaction path production writes use (UnifiedStore.probeWritable()→ single kv row in a transaction). Success re-enables the store and the triggering record ingests immediately; failure keeps the breaker armed with the real error inlastError. Both sync (node:sqlite/better-sqlite3) and async (mysql backend) sidecar paths covered.persist.ts buildForLoadAll: envelopesavedAtvs index rowupdated_at) re-imports drifted sessions, and the next save does a full ingest with raw tail — sessions missed during the outage converge automatically.Behavior change disclosure (old → new)
cannot rollback - no transaction is active; New: the original error (e.g.SQLITE_FULL: database or disk is full) surfaces. On the better-sqlite3 engine the original is lost inside the library, so the surfaced message carries an annotation naming the masking and the common cause.__bili/writability-probein index.db (single key, replaced per probe). No schema, wire-format, or persistence-version change.Config surface
None added. The cooldown is an internal constant following the
VERIFY_EVERY_INGESTSprecedent (not a user-facing behavior knob).Verification
tests/issue4949-rollback-mask-recovery.test.ts(4 tests): injected engine-level COMMIT abort over a real DatabaseSync proves the original error surfaces — running it against the pre-fix driver fails with exactly the reported mask string (original error must survive: cannot rollback - no transaction is active); negative control pins that constraint failures still roll back cleanly (gate both directions, Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617); full breaker loop (disable → cooldown gate → heal → re-enable → triggering record lands); constant pin.kernel-change: n/a (no changes under kernel/)