Skip to content

[Bug] Unified snapshot flushes return before durability and reuse an expired debounce budget #6178

Closed

Description

@Afloat16

Description *

Affected source

Package version 0.1.213, source revision 33104b1f9545caeb0344193009917ceef9969682. This is a source-level regression report for this exact revision.

Problem

Both prefix-affinity and session-order await persistUnifiedNow during shutdown, but that helper returns void while the KV read/modify/write is still queued. Graceful exit or restart can therefore proceed before the snapshot becomes durable. The unified path also leaves pendingSince set to the first batch forever, so after its max-delay expires later mutations schedule immediate writes instead of respecting the debounce window.

Proposed correction

Return the serialized KV promise to flush callers and reset the pending clock when the current batch takes ownership of its write. Later mutations receive a fresh debounce budget even if an older write is still settling. Both writers retain their existing merge, bounds and failure-demotion behavior.

Regression coverage

The prepared branch includes focused regression coverage in tests/snapshot-unified-lifecycle.test.ts, tests/affinity-persist.test.ts, tests/order-persist.test.ts, tests/affinity-unified-store.test.ts, tests/order-unified-store.test.ts.

Category E (storage / persistence).

Suggested triage labels: p1, scope/storage.

bili version *

0.1.213 — source revision 33104b1

Usage mode *

Launcher (bili pi / bili codex / bili dsh / bili kimi …)

Client and version *

Direct Node.js v24.22.1 regression harness; no external agent client session.

Operating system *

Linux x86_64; deterministic local regression fixtures.

Upstream and model *

No live upstream/model request; deterministic source-level regression fixtures.

Compression-related config (optional)

Log excerpt (strongly recommended)


/acp or acp_status output (optional)


Activity

  1. Afloat16 commented on Nov 11, 2026

    @Afloat16
    ContributorAuthor

    Proposed fix: #6186. Flush callers now receive the serialized write promise, and each batch gets a fresh debounce budget. The independent persistence suite passes 28/28, including shutdown durability and later-batch scheduling regressions.

  2. ranxianglei commented on Nov 11, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重 (dedup): Not a duplicate. Searched open AND closed issues+PRs for this mechanism (unified-snapshot flush durability / debounce-budget reuse): only this thread and its fix PR cover it. Nearest lineage — #5753 (the unified-storage migration that introduced these kv writers), #5757 (its failure-demotion review fix), #6066 (debounce/max-delay knobs) — none reports this defect.

    Bug verified at the pinned revision (33104b1f, v0.1.213 — file-identical to current master for both affected files). Both reported defects exist exactly as described:

    1. Flush returns before durability — persistUnifiedNow() in src/order-persist.ts:164 and src/affinity-persist.ts:166 returns void while queueing the KV read-modify-write onto kvChain. The shutdown path (src/server.ts:1757-1758) and both restart paths (src/restart.ts:488-489, 498-499) await it, so the await is a no-op and process.exit(0) can run while the snapshot write is still queued/in-flight. Affects default installs — unified sqlite storage is the default carrier, and bili's routine self-update restarts go through exactly this flush sequence.
    2. Expired debounce budget reused — pendingSince is cleared only on the legacy sync path (writeSnapshotLegacy's finally) and at flush; nothing resets it after a unified batch fires. Once the first batch's max-delay expires, every later mutation computes a non-positive budget → zero-delay timer → one RMW per mutation instead of the documented ~500 ms quiesce / 5 s cap cadence (/bili/url/重启agent、重启bili、开新会话分支啥的都有概率丢失压缩 #1834).

    Labels applied: bug, scope/storage, p1 — on both this issue and PR #6186. Reporter suggested p2; I labeled p1 because silently losing durable state on a routine graceful exit/restart is data-loss class, which owner policy (2026-10-19) keeps at p1. (An earlier daemon pass reported these labels but could not apply them; they are now applied.)

    Fix: your PR #6186 targets this thread, so per the one-fix-PR-per-thread rule I reviewed it instead of opening a parallel one. Independent verification, redone locally just now at head 851e3bb766 (not taken from the description):

    • Targeted suites — new tests/snapshot-unified-lifecycle.test.ts plus the four adjacent persistence suites: 28/28 pass.
    • Negative control — reverting ONLY the two production files to their parent version: all 4 new regression cases fail (each on its named assertion); restoring the fix: 4/4 pass.
    • tsc --noEmit clean on the PR head tree.
    • Remote CI green: 30/30 checks completed, 0 failed / 0 pending (mirror cache).
    • Base drift: master has moved since the PR base e3220adc; no intervening commit touches the three touched files, so the diff still applies cleanly (GitHub's "behind" flag is base-sha staleness only).

    The full structured review — behavior-change disclosures (old→new), test-quality assessment, §7.4 auto-merge gate verdict, and a merge-ordering coordination note for the open immutable-journal PR #5480 (same two source files, whose helper still returns void) — is on the PR thread: #6186.

    Verdict: ready to merge. Merging itself stays human.


    结构化八问:

    1. 是什么问题:统一存储上的两个快照写者(prefix-affinity / session-order)把 KV 写入排进异步链,但关机/重启路径的 await 对这条队列是空操作——优雅退出可能丢掉最近的状态;同时去抖时钟在批次落盘后不复位,持续流量下之后每次变更都触发立即写库。
    2. 什么原因导致的:同一异步生命周期上的两个独立缺陷:(a) persistUnifiedNow() 返回 void,调用方的 await 没有可等的 promise;(b) pendingSince 只在 legacy 同步路径清除,统一路径永远钉死在第一批的时间戳上,预算算出负数 → 定时器延迟被钳到 0。
    3. 影响范围与严重程度:所有启用统一存储的实例(默认配置即命中)。触发条件:优雅退出/重启与排队写入重叠 → 最近的 affinity/order 变更丢失 → 重启后匿名会话可能孤儿化、以零压缩状态重新分叉(全量重发、缓存未命中,即 0.1.63 应用在DeepSeek Harness超出 固定阈值没进行压缩 #351 模式);session-order 骨架丢失则重演冷启动竞态(Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397 病理)。持续流量场景另有每次变更一次 RMW 的写放大。严重度 p1(数据丢失类;丢的是辅助元数据而非会话正文)。
    4. 怎么修复的:让写者兑现其异步契约——返回串行化的 kvChain promise,使 flush 调用方真正等到全部排队写入落定;批次接管写入时清空 pendingSince,后续变更获得全新的去抖预算。合并/限幅/失败降级逻辑一律不动。
    5. 是否有回归问题:逐项审过——去抖节奏(有意恢复文档化行为,已披露)、flush 持久性(有意变更,已披露;退出去掉最多延长一个在途写入时长,仍受既有 withExitDeadline 约束)、失败降级(不变,catch 仍正常结算链,flush 永不抛错)、合并/限幅(未触碰)、legacy 文件路径(未触碰)、hydrate 流程(未触碰)。无 fail-fast↔吞错反转、无阈值/时序漂移(除已披露项)、无输出格式变化。未发现回归。
    6. 是否破坏缓存:不涉及 prompt-cache / 重试 / 增量游标策略;间接为正(减少重启导致的 affinity 丢失 → 减少冷启动全量重发)。
    7. 是否改了设计方向:不改。同一架构,只是补全既有统一写者的异步生命周期契约(与同一关机序列里 flushResignPending 已有的正确模式对齐)。
    8. 是否引入了新的配置:无。
  3. ranxianglei commented on Nov 11, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    Closing status — full triage + structured review posted above in this thread; outcome:

    • 查重: not a duplicate (details above).
    • Bug confirmed at pinned revision 33104b1f (v0.1.213): both reported defects are real — shutdown/restart awaits on persistUnifiedNow() are no-ops against the queued KV writes, and the unified-path debounce clock never resets after a batch fires.
    • Fix: your PR [PR] fix: await unified snapshot writes and reset each debounce batch #6186 targets this thread — reviewed it, did not open a parallel PR. Independently verified at head 851e3bb766: 28/28 targeted persistence suites pass, negative control 4/4 fail-without-fix → 4/4 pass-with-fix, tsc clean, remote CI green (30/30), no master drift on touched files.
    • Labels applied: bug, scope/storage, p1 (data-loss class per owner policy 2026-10-19) on issue and PR.
    • Verdict: ready to merge — merge stays human. Merge-ordering note vs [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 is on the PR thread.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions