Skip to content

storage-render golden suite is not hermetic: locally regenerated digests cannot be reproduced on CI runners #5402

New issue
New issue
Closed
#5859
Closed
storage-render golden suite is not hermetic: locally regenerated digests cannot be reproduced on CI runners#5402
#5859
Labels
bugSomething isn't workingp3scope/ci

Description

@ranxianglei
ranxianglei
opened on Nov 1, 2026

(ai: vllm-qwen/qwen3.8-27b)

来源: #5398 (PR, #5398) 分析 #5400/#3849 的 CI 失败时发现

查重: 无(open+closed 按 golden / storage-render / hermetic / machine-specific / drift 检索均无同机制条目)

症状(一句话)

storage-render 黄金套件在 agent 开发容器里对纯 master 也红(nudge/tsStamps/stats 摘要机器特定),而同一 commit 在 CI runner 上绿 —— 本地跑 scripts/update-storage-render-goldens.ts 重生成并提交的黄金值,CI 必然复现不了。

影响

  • 任何按测试头注释的正规流程(本地重生成 → review diff → PR 说明理由)操作的 agent/人,都会把本机摘要烧进黄金文件 → 全部 cell 红。[PR] fix: retire persistently-unmatched fold coverage after 3 evidence-less passes (#3849) — relay of stranded branch #5398 差点就这么做,最终改为从 CI runner 观测值手工钉入(见该 PR 正文)。
  • 潜在隐患:runner 镜像变化(setup-node 矩阵 22/24 的 minor 漂移等)可能让 master 全舰队红,且没人能在本地复现定位。
  • 这两条套件(tests/golden-storage-render.test.ts、tests/storage-engine-proofs.test.ts)在容器内没有绿色本地信号,触碰持久化状态的开发只能盲信 CI。

已完成的取证(#5398 工作内,全部可复核)

  • 纯 master(7578e81)在本容器:4 条 wire 全红;分歧键 = nudge、tsStamps(state 段)+ stats 段 + metadata 段一部分;blockContents、blocks 键与 turns[] 可复现(与黄金一致)。
  • 分支×机器确定性:重复运行摘要逐字节相同;TZ=UTC vs GMT+0800 摘要逐字节相同 ⇒ 排除时区。
  • 排除:committed dist 过期(kernel/dist 不入 git,CI 每次现建,ci.yml:351 + hashFiles 键控缓存)、node 版本(本容器 v22.23.2 ≈ runner 矩阵 latest 22.x,无 .nvmrc)、时间戳泄漏(规范化后状态无裸日期/epoch;tsStamps 值哈希前已掩码为 TS,分歧在键集/数值而非格式)。
  • nudge 状态形状为纯数值 {anchors:{}, baselineTokens:0, lastNudgeShownTokens:0, lastPerMessageNudgeTokens:, lastShownByTier:{}} ⇒ 机器因子必然作用于其上游的某个估算/校准输入。
  • stats 含 calibratedEstimate/calibrationRing/localInputEstimate/contextTokens;harness 用 setRegistryForTest({}) 固定空注册表 —— 值得查估算回退路径是否在容器与 runner 间取到不同输入(如捆绑注册表快照、模型窗口回退)。
  • CI 侧对照:两个连续 run 的全部 4 个 runner cell 产出逐字节相同的摘要 ⇒ runner 本身确定,分歧纯粹来自「容器 vs runner」环境差。

建议下一步

  1. 给 harness 加一个 dump 模式:对失败键输出哈希前的原始 nudge/tsStamps/stats 字段值,在容器与 runner(GH Actions 一次性 job 即可)各跑一次,逐字段 diff 定位第一个分叉点。
  2. 沿分叉点查 token 估算/校准是否读了环境相关输入(locale/ICU/fs 时序/模型窗口查找回退)。
  3. 定位后二选一:harness 内归一化(哈希前掩码掉易变类,像 STAMP_ATTR_RE 那样)或钉死依赖;并把结论写进测试头注释,防止下次再踩。

标签建议: bug / scope/ci / p3(内部测试基建,无用户面影响;有绕行——从 CI 观测值钉黄金)

Activity

  1. ranxianglei
    added
    bugSomething isn't working
    p3
    scope/ci
    on Nov 1, 2026
  2. ranxianglei
    added a commit that references this issue on Nov 1, 2026

    test: pin storage-render goldens to post-retire persist contract (#3849)

    73b7e19

  3. Louie0806 commented on Nov 1, 2026

    @Louie0806
    Louie0806
    on Nov 1, 2026
    Contributor

    (ai: deepseek/deepseek-v4-flash)

    Root cause found — it is not "machine-specific estimation input". It is a missing #3 parity pin in the regeneration script, and it is deterministic.

    The decisive experiment

    All on upstream/master c78ecb4b5, container node v22.23.4, same machine, same commit.

    regen command then run tests/golden-storage-render.test.ts under the CI preload chain
    node --import tsx scripts/update-storage-render-goldens.ts (the committed golden's own expected command, no env) 1 pass / 4 fail
    BILI_MESSAGE_TIMESTAMPS=0 node --import tsx scripts/update-storage-render-goldens.ts 5 pass / 0 fail

    The only difference is one environment variable at regen time.

    Why

    tests/preload-test-env.mjs:53 pins it unconditionally:

    process.env.BILI_MESSAGE_TIMESTAMPS = "0";

    and scripts/run-ci-tests.mjs:185's PRELOAD_ARGS loads that preload into every per-file child process. So the suite always hashes with message timestamps off.

    scripts/update-storage-render-goldens.ts already documents two parity pins for exactly this class of bug:

    // Parity with tests/golden-storage-render.test.ts (#3400): ...
process.env.BILI_IMAGE_RESCALE = "0";
// Parity #2: the test suite also disarms the shadow observability lane (#3756) ...
process.env.BILI_SNAPSHOT_SHADOW = "0";

    There is no Parity #3 for BILI_MESSAGE_TIMESTAMPS. A bare node --import tsx has no preload, so the timestamp flag resolves through the production default (ON), and every nudge / tsStamps digest is computed against a different render mode than the test hashes. The regenerated golden is stale by construction — exactly your symptom.

    This also explains why the divergence lands precisely on those keys rather than on blockContents / blocks, which do not depend on the timestamp flag.

    Consequence: master itself is red from this right now

    #5415 (c91ef1a81, "kernel: abort-heavy sessions stop starving the nudge cadence") added nudge.absorbedDebt? / nudge.absorbCount? to kernel/src/types.ts (+14) and booked them in src/session.ts (+17), but did not regenerate the goldens — its 9-file diff contains no tests/golden/ file.

    tree golden-storage-render.test.ts tests/storage-engine-proofs.test.ts
    7578e8184 (#5415's direct parent) 5 / 0 12 / 0
    c78ecb4b5 (master tip, after #5415) 1 / 4 8 / 4

    CI on master c78ecb4b5 fails 4 of 11 test shards on exactly these two families (ubuntu-latest,22,3, ubuntu-latest,24,3, windows-latest,22,2, windows-latest,22,4). Failure text matches the state-shape change:

    state-key divergence: nudge=75d1956dd2bc409a (golden 1875424e64e16bc5)
sections: {"state":"6a3fe0d7d15b45c1",...} vs golden {"state":"8d09f6115ebffb9d",...}

    Suggested fix (5 files, verified bidirectionally)

    1. Add the missing parity pin to the regen script:
    // Parity #3: the suite runs under the CI preload chain, whose
// tests/preload-test-env.mjs pins BILI_MESSAGE_TIMESTAMPS=0. A bare
// `node --import tsx` regen has no preload and renders with the production
// default (timestamps ON), so every nudge/tsStamps digest differs from what
// the suite hashes and a freshly regenerated golden reds immediately.
process.env.BILI_MESSAGE_TIMESTAMPS ??= "0";
    1. Regenerate tests/golden/storage-render/{chat,anthropic,responses,google}.json with the now-hermetic documented command.

    Then the diff is the real #5415 state-shape drift only: 16 changed lines across the four files, versus 208 when regen runs under the wrong mode.

    $ node --import tsx scripts/update-storage-render-goldens.ts
$ node --import ./tests/preload-storage-baseline.mjs \
       --import ./tests/preload-test-env.mjs \
       --import ./tests/preload-zone-slot.mjs \
       --import tsx --test tests/golden-storage-render.test.ts
# pass 5  # fail 0
$ ... --test tests/storage-engine-proofs.test.ts
# pass 12 # fail 0

    Negative control: git checkout -- scripts/update-storage-render-goldens.ts (drop the pin), regenerate with the documented command again → back to 1 / 4 fail and a 208-line golden diff. Re-adding the pin restores 5 / 0.

    Note on the hypothesis in the issue body

    Token estimation / calibration reading environment-dependent inputs (locale, ICU, model-window fallback) is ruled out by the matrix above: the divergence is fully explained by a boolean render flag, and it reproduces byte-identically on repeated runs on the same machine exactly as observed — the determinism comes from the env being fixed per process, not from machine identity. The conclusion that "the runner is deterministic and the delta is purely container-vs-runner environment diff" stands, and the delta is now identified.

    I have not opened a PR for this because it touches the golden contract; say the word and I will.

  4. ranxianglei
    added a commit that references this issue on Nov 1, 2026

    test: self-pin BILI_MESSAGE_TIMESTAMPS in both storage-render suites (r…

    45d0889
  5. Louie0806
    added a commit that references this issue on Nov 3, 2026

    fix(test): pin BILI_MESSAGE_TIMESTAMPS in the storage goldens' own te…

    15cea3f
  6. ranxianglei
    added 2 commits that reference this issue on Nov 7, 2026

    fix(test): pin BILI_MESSAGE_TIMESTAMPS in the storage goldens' own te…

    8412fff

    fix(test): pin BILI_MESSAGE_TIMESTAMPS in the storage goldens' own te…

    0ff29bd
  7. ranxianglei
    closed this as completedin #5859on Nov 8, 2026
  8. Louie0806
    added a commit that references this issue on Nov 8, 2026

    Merge upstream/master (848f59b): adopt the CI test-signal batch (guar…

    0b0e35c

  9. ranxianglei commented on Nov 10, 2026

    @ranxianglei
    ranxianglei
    on Nov 10, 2026
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Resolved the merge conflict and pushed a clean fast-forward (no force-push). The dirty conflict is cleared — the PR is now mergeable; state advanced dirty → blocked, i.e. it is no longer a conflict, just waiting on the normal pre-merge gates (CI / approval).

    Conflicts found (5 files) and how each was resolved:

    Master had advanced ~2454 commits and touched several of these files, so I re-synced to the newest master after the first resolution: it auto-merged cleanly.

    Local verification (targeted, on the final tree): npm run typecheck clean; the PR's own tests/issue5300-session-compress-optout.test.ts passes 7/7. Full-suite verdict is left to remote CI on the pushed head — I did not confirm the fresh CI outcome yet (the mirror check cache still shows the pre-push sha; CI is re-running on the new head 91ee0543ad).

    八问（中文结构化结论）

    1. 是什么问题：PR [PR] feat: per-session compression opt-out via /__bili/session/<id>/compress (#5300) #5404（对应 issue [Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300「每会话压缩开关」）与 master 产生合并冲突（mergeable_state=dirty），无法合入。
    2. 什么原因导致的：分支 11-01 切出，master 在这 9 天里前进了约 2454 个提交，双方都改了同一批文件（plugin.ts / admin.ts / handle.ts / session.ts / CONFIGURATION*）；且 master 把 CONFIGURATION.* 迁到了 docs/ 下、又把通知系统重构成单一注册表模型，与 PR 的改动撞在同一区域。
    3. 影响范围与严重程度：只影响这个 PR 能否合入；冲突不解决则 PR 一直卡在 dirty（完全无法合并）。不影响任何已发布/线上行为。
    4. 怎么修复的：把当前 master 合进 PR 分支，逐处按「导入取符号并集 / 保留两个互不排斥的独立块 / 跟随文档迁移」解决 5 个冲突；随后对最新 master 再做一次同步（自动合并干净）。全程用非 force 的 fast-forward push。
    5. 是否有回归问题：过了一遍触碰路径的既有行为——(a) 通知系统：丢掉的 PCK_NOTICE_ID/pckNoticeActive 在合并后正文中 0 引用，非回归；(b) handle.ts 交叉点：[Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300 早退放在 WorkBuddy tier aliases (fast-model / balanced-model) are absent from bili's built-in window table, so the window silently falls back to 200000 and the effective wall collapses to 152000 — mid-session, after a model switch #5490 告警之前，意味着「已关闭压缩的会话」不会再触发 WorkBuddy tier aliases (fast-model / balanced-model) are absent from bili's built-in window table, so the window silently falls back to 200000 and the effective wall collapses to 152000 — mid-session, after a model switch #5490 的未解析模型告警——这是两个新特性的组合语义，符合 [Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300「关闭即纯透传、不做额外记账」的设计意图，不是既有功能的回归（两个特性此前从未共存）。typecheck + [Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300 回归套件 7/7 通过佐证。行为变更披露：旧＝两特性各自独立存在；新＝二者共存且关闭压缩的会话跳过 WorkBuddy tier aliases (fast-model / balanced-model) are absent from bili's built-in window table, so the window silently falls back to 200000 and the effective wall collapses to 152000 — mid-session, after a model switch #5490 告警。
    6. 是否破坏缓存：不涉及（纯合并冲突解决，未触碰缓存/重试/增量游标/去重键逻辑）。
    7. 是否改了设计方向：不涉及（未改架构/边界/数据流；仅把 PR 新增文档段落到 master 已确立的 docs/configuration*.md）。
    8. 是否引入了新的配置：不涉及（无新 env/config/默认值）。

    收工判定：冲突已全部解决并推送（head=91ee0543ad，non-force fast-forward），PR 现为 mergeable（dirty→blocked，剩余是 CI/审批门禁而非冲突）。自动修复 lane attempt 1/3 的目标（接冲突）已完成；后续 CI 结果以远程为准。

  10. ranxianglei commented on Nov 11, 2026

    @ranxianglei
    ranxianglei
    on Nov 11, 2026
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Verified live before touching anything: the dirty/conflict the trigger reported did not exist this run — the PR was mergeable: True, state behind (up-to-date lag), never an actual conflict.

    Root cause of the drift: the branch sat on an older base while master kept moving. Two waves landed since the last sync — (1) the compressAtFoldPoint removal (#6132), then (2) a further ~26-commit busy window (blockOps gate #5765 [default off], snapshot-write-lifecycle #6186, anthropic trailing-system-tails #5441, …).

    What I did:

    • Merged current master into the branch twice (both non-force; dry-run predicted 0 conflict hunks each time, and both real merges were clean).
    • Effective PR diff stays exactly the 8 #5300 files (src/plugin.ts, src/session.ts, src/server/admin.ts, src/server/handle.ts, src/incremental-lane.ts, docs/configuration.md, docs/configuration.zh-CN.md, tests/issue5300-session-compress-optout.test.ts). All kernel / fold-point / guard changes converge to match base, so they add nothing to this PR's own diff (no kernel-guard surface introduced).
    • Local verification after both syncs: npm run typecheck exit 0 + tests/issue5300-session-compress-optout.test.ts 7/7 pass. Full-suite verdict left to remote CI.
    • Result: branch is now fully up-to-date with master tip 31684f368e (ancestry-confirmed), so the behind gate cleared; merge-state advanced behind → blocked. blocked here is the normal pre-merge gate (checks running / approval), not a conflict.
    • Mirror check cache: failed: 0, only in-progress/pending (test shards were running on the prior head 46544a94a4; they re-run on the new head). No red-CI lane is actionable right now.

    Final head: 9fe8c21344. Note: master is in a fast-moving window, so behind can reappear transiently if more commits land before merge — that is expected up-to-date lag, not a conflict, and needs no design decision; a final trivial sync at merge time suffices if the repo enforces up-to-date-before-merge.

    八问（中文结构化结论）

    1. 是什么问题：触发器报告 PR [PR] feat: per-session compression opt-out via /__bili/session/<id>/compress (#5300) #5404（[Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300 每会话压缩开关）与 master「冲突（dirty）」无法合入。但本轮实测全程没有真实冲突——live 状态一直是 behind（落后于 master），不是冲突。
    2. 什么原因导致的：分支停在旧基线，master 持续前进。两轮增量：①compressAtFoldPoint 特性移除（Remove the compressAtFoldPoint feature — it reorders resent history and caused an authorship-misattribution incident in a live session #6132）；②随后约 26 个提交的合并高峰（blockOps 门控 Request two capabilities: compressed block "retirement" and summary "external rewrite". | 请求两个能力：压缩块「退役」与摘要「外部重写」 #5765、snapshot-write-lifecycle [PR] fix: await unified snapshot writes and reset each debounce batch #6186、anthropic trailing-system-tails [PR] fix: insert Anthropic ephemeral tails before trailing client systems (#5432) #5441 等）。本质是「上游在动、分支没跟上」的落后，而非双方改到同一处的冲突。
    3. 影响范围与严重程度：仅影响此 PR 能否合入；behind 会被「合入前须最新」的保护规则挡住合并按钮。不影响任何已发布/线上行为。
    4. 怎么修复的：把当前 master 合进分支（两次 non-force 同步，dry-run 均预测 0 冲突），每次同步后本地验证（typecheck + [Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300 回归套件 7/7）再推。有效 PR diff 始终收敛为恰好 8 个 [Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300 文件——kernel/fold-point/guard 变更与 base 一致，不进入本 PR 自身 diff。
    5. 是否有回归问题：无。同步只是把已合入 master 的提交带进分支，未改本特性语义；[Feature] Per-session opt-out from compression (scope today is only global → provider → model) #5300 早退先于 WorkBuddy tier aliases (fast-model / balanced-model) are absent from bili's built-in window table, so the window silently falls back to 200000 and the effective wall collapses to 152000 — mid-session, after a model switch #5490 告警的组合（上一轮已披露）保持不变。typecheck + 7/7 通过佐证。
    6. 是否破坏缓存：不涉及（纯分支同步，未触碰缓存/重试/游标/去重键逻辑）。
    7. 是否改了设计方向：不涉及。
    8. 是否引入了新的配置：不涉及（无新 env/config/默认值；Request two capabilities: compressed block "retirement" and summary "external rewrite". | 请求两个能力：压缩块「退役」与摘要「外部重写」 #5765 blockOps 默认关是 master 既有内容，非本次引入）。

    收工判定：冲突不存在（触发器的 dirty 为过期信息）；分支已同步到当前 master 尖端并本地验证绿，head=9fe8c21344；状态由 behind→blocked，剩余 blocked 属正常合入门禁（CI 检查运行中 / 待审批），非冲突。镜像检查缓存 failed=0，当前无可修的红灯车道。自动修复 lane attempt 1/3 的目标（接冲突 + 修红 CI）已达成；若新 head 上 CI 转红，将作为独立的红 CI 触发处理。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingp3scope/ci

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions