Repository navigation
Emergency fold path can write latch/summary text into tool_calls.arguments, permanently poisoning sessions for strict-JSON upstreams #5534
Description
Environment
- bili (billion-context)
0.1.209, commit39a0ada5, Windows x64, DSH lane (--host 127.0.0.1 --port 18787) - Upstream: infernix-serve (OpenAI-compatible; it strictly JSON-parses
tool_calls[*].function.argumentsof every history message) - Model window 262144, proxy
maxContextLimit: "75%"-> effective 196608;nudgeGrowthTokens: 120000,emergencyThresholdPercent: "95%"
Symptom
Once a session goes through the emergency path near ~99% usage, outbound requests start carrying an assistant message whose tool_calls[0].function.arguments begins with fold/checkpoint/tag-injection text instead of JSON — in our two incidents the first rejected character was T (the persisted state.emergencyTruncateLatch contained a DSH hypercompact checkpoint starting with "This is an automatically generated checkpoint...") and < (the per-message <acp tokens=...> tag injection text, visible in ACP_DUMP_BODY=1 REQ dumps). The upstream answers:
500 {"code":null,"message":"[json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: 'T'","param":null,"type":"internal_error"}
The session is then permanently poisoned: every "continue" retry re-sends the dirty history and 500s forever (one window burned 6 identical 500s in a row; retries cancelled by the user).
Why the proxy is at fault (client and engine exonerated)
- Repro probe against the same live engine: any history message with
tool_calls[*].arguments = "True..."-> identical 500; valid JSON -> 200. (The engine's strict parse is expected behavior.) - Client-side session logs on the box: all sessions scanned, 16591 tool-call argument payloads, 0 non-JSON.
- bili's own persisted session store (
sessions/openai/*.json) for the poisoned sessions: 0 dirty tool-call entries (all clean). - Rebuilding the wire body from the persisted store using the proxy's own serializer logic and POSTing it directly to the engine: 200 OK.
=> The dirty arguments exists only in the transient outbound request assembled by the proxy under memory pressure; it never lands in either durable ledger (client log or bili store). Whatever writes it lives in the fold/emergency/tag-injection render path at request-assembly time.
Timeline (one incident, bili.log + upstream request log)
18:14:51nudge line shows usage at 99% (194405/196608) -> emergency fold fires the same second an inbound DSH hypercompact checkpoint ("This is an automatically generated checkpoint...") enters; afterwardsstate.emergencyTruncateLatchcontains that checkpoint text verbatim.- Same second, the upstream request log records
request_rejectedwithparse_error.101 ... last read: 'T', 6 consecutive times (automatic retries). - A second window hit the same path later with
last read: '<', matching the<acp ...>tag text.
Suggested fixes
- Guard the fold / tag-injection renderers so summary or tag text can never be written into
tool_calls[*].arguments(they should only ever touchcontent). - Last line of defense: validate every outbound message before forwarding — each
argumentsmust parse as JSON; on violation repair (drop the affected message pair or coerce to{}) and log, instead of letting one bad assembly permanently poison the session. - Consider a self-heal on session load: scrub non-JSON
argumentsfound in memory, so already-poisoned sessions recover.
Workaround applied locally (reduces exposure, not a fix): route-level nudgeGrowthTokens lowered 120000 -> 30000 so normal folds happen early and the emergency path is rarely reached. After this change, an already-poisoned session self-recovered once an early fold dropped the poisoned range (first clean 200 after six 500s).
diagnostics.dumpBody raw dumps are kept enabled and can reproduce the "transient dirty argument on the wire, clean in both ledgers" observation; the dumps contain private conversation data, so I can attach a scrubbed excerpt if that helps.
Activity
ranxianglei commentedon Nov 2, 2026
(ai: vllm-qwen/qwen3.8-27b)
查重: 不重复 — 按失败签名(出站
tool_calls[*].function.arguments出现非 JSON 文本、strict-JSON 上游 parse_error.101、会话永久中毒)检索了 open+closed 的 issue 与 PR,无既有条目。标签:
bugdshscope/kernelp1(缺陷类;DSH 车道;机制在 kernel 的 emergency-truncate latch 重放;会话在绕行前永久不可用,按 owner 2026-10-19 修订归 p1)。审核三步结论(对含 v0.1.209 的 master 代码逐路径核实过):
1. 复现/查证 — 机制成立。 两处 ref-keyed 文本重放都「只认 ref、不认消息身份」:
kernel/src/compress.ts:1346-1367(emergencyTruncateNode):把emergencyTruncateLatch里{ref → 裁剪后文本}写到当前持有该 ref 的任意存活消息上——没有 contentType 校验,ref 之外零身份验证。latch 条目只有{ref, text}两个字段(kernel/src/types.ts:51-54),不带原始消息 id/指纹。kernel/src/view-latch.ts:88-134(applyViewLatch)同模式(ccr/absorb/crush 开启时)。latch 条目如何指向错消息:冷加载 ledger 对齐重编号。
src/persist.ts:1605(maybeRestampToLedger→restampRefsToLedger,kernel/src/refs.ts:217-287)在统一存储首次冷加载时把 ref 整体对齐到 ledger seq+1、被挤占者换新号;而 latch 随 kstate 持久化(src/state-hydration.ts:57)原样活过重启——旧条目从此指向重启前属于别的消息的编号。若该编号现在落在一条 assistant tool-call core 上,它的text就是 arguments JSON,被整段覆盖成裁剪文本;coreToOpenai忠实把它发成function.arguments→ strict-JSON 上游 500。为何永久:latch 存在 state 里且每轮重放自持(重放在水位检查之前执行);500 让模型永远答不上来 → 永远没有成功的 fold → compress.ts:812 的 latch 重置不可达——直到别的机制(如更激进的 nudge fold)把中毒范围折掉。与你观察到的「连续 6×500、调小 nudgeGrowthTokens 后自愈」完全吻合。你排查出 store 干净也对:毒只存在于瞬态出站装配,bili 从不持久化它。
两次事故的字节签名都符合:第一次首字符
T= 被裁剪的 DSH hypercompact checkpoint 原文("This is an automatically generated checkpoint…");第二次首字符<= 同机制,被裁源文本恰好带渲染过的 ACP tag 前缀(plugin 模式回送历史里 text 携带<acp …>)。精确到你们那台机器上触发重编号的具体时刻(重启/更新时序)我无法从日志外重建,但机制族是同一个。2. 层次定位 — 你的描述在症状层准确,指认的生产者家族(fold/latch 渲染器写进 arguments)也正确。根因层:latch 条目缺消息身份,replay 单信 ref。表面现象是「arguments 变脏」,病因是「无身份验证的 ref 重放 + ref 可被冷加载重编号」两层叠加。
3. 方案评估 — 你的三条建议都打到了核心,我分两层落地:
tool_calls[*].function.arguments必须是合法 JSON,违规则替换为合法占位对象 + warn 日志(仓库已有同类先例:src/loop/adapter-google.ts:269、src/loop/core.ts:182)。只对已经非法的字节动手,合法流量零改动;同时覆盖 Responses 线的function_call.arguments(同型字段)。Anthropic 线结构安全(input 恒为对象,safeParse兜底),Google 线已有守卫。这层让已中毒会话当场可用(客户端回送的毒每轮被修复),比 (c) 的加载期清洗更强——它顺带覆盖运行中新增的中毒。kernel/改动,按仓库 Kernel Boundary 属 owner 审批域;我会另发一条 🔶 需要决策 评论列选项,不擅自开修。出站守卫 PR 开工中,链接稍后贴这里。
xiaoqiangexcite-lab commentedon Nov 2, 2026
Root cause located (measured on v0.1.209, Windows)
Reproduced the
tool_calls[i].argumentscorruption from my original report end-to-end withdiagnostics.dumpBodycaptures. The corrupted argument string is the emergency-truncation excerpt format, stamped verbatim onto multiple messages:\n\n ...[truncated for context space] — original ~8061 tokens]... \n\n+ last 2000 chars of a 17,237-char instruction message. This matchesTRUNCATION_MARKER+DEFAULTS={keepPrefixChars:2000, keepSuffixChars:2000}and theapplyTo()generator in thetruncatepass (applyTo(toolResults); applyTo(textMessages);→replacements.set(candidate.id, replacement)→messages.map(m => replacements.has(m.id) ? {...m, text: replacements.get(m.id)} : m)).assistant.tool_calls[0].arguments(breaking JSON) and arole=toolmessage. One poisoned message grew from 632 → 4064 chars, so it is not a legitimate fold — the same replacement blob is being mis-associated by id.stats.contextTokensstays pinned at a stale overestimate (409,126 estimated vs 167,734 actual usage grade from the upstream), so every subsequent request re-enters the truncation pass and re-poisons deterministically. ClearingemergencyTruncateLatch,rules, and fold blocks does NOT heal it — the pass regenerates the poison on every render.Decisive experiment: the exact same (clean) request body POSTed directly to the upstream engine returns 200 OK; via bili it returns 500 (
invalid literal; last read: '<'). Client-side store/history are clean (0 dirty args across 1,659 stored tool-call records; INC captures are clean; only REQ differs at exactly the stamped positions).Local workaround applied here (with user consent): no-op the two
applyTo(...)calls in the truncate pass. All sessions immune after that. This is of course not a fix — the real fix is to make the id association in the write-back exact (and/or never apply truncation replacements to messages carryingtool_calls/role=toolcontent, and recomputecontextTokensfrom the post-truncate body instead of keeping the overestimate).Happy to provide the de-identified INC/REQ diff and per-message fingerprint table if useful.
ranxianglei commentedon Nov 2, 2026
(ai: vllm-qwen/qwen3.8-27b)
你的补充证据我逐条对 v0.1.209 源码核实了，全部吻合。结论先行：根因在 bili 内核侧「按 ref 回放截断文本」缺乏身份校验；出站第一道防线已修好（PR #5537），根治（给 latch 条目加身份校验）涉及内核变更，需要你拍板——选项见下一条 🔶 评论。
证据核对（每条对应代码位置）
TRUNCATION_MARKER（[truncated for context space]）+ 默认keepPrefixChars:2000 / keepSuffixChars:2000+ The "shortened" result of emergency truncation is longer than the original: a 4001-character tool output became 4064 cha | 紧急截断的"缩短"结果比原文还长:4001 字符的工具输出截断后变 4064 字符 #3935 严格变短上限（kernel/src/truncate-tools.ts:29、47、109-111）。你测到的 4064 字符 blob 与该生成器逐字吻合。{ref, text}（kernel/src/types.ts），回放时按「当前谁持有这个 ref」直接盖文本（kernel/src/compress.tsemergencyTruncateNode），不做任何身份校验。冷启动加载时的账本 restamp（src/persist.tsmaybeRestampToLedger→restampRefsToLedger）会重排 ref，旧条目就贴到不相干的消息上。源指令消息在会话内稳定 → 每次 clip 字节相同 → 多条陈旧条目携带同一 blob → 正是你看到的「一份替换命中五条」。effectiveTokenCount（src/server.ts:2554）正常用 usage-grade 读数，但当 Bypass requests overwrite the main session's usage anchor, causing long sessions to permanently stop compressing. | 旁路请求覆盖主会话 usage 锚点，导致长会话永久停止压缩 #4805 尺寸守卫（原始重发 ≥ 8× 已确认用量且绝对差 > 25K）或 cycle-stale 守卫触发时，回落到本地估算（原始历史字符数上界）。长会话里原始重发远大于折叠后用量时该守卫长期成立 → meter 每轮都高于阈值 → truncate pass 每轮重跑 → 重新 clip、重新 latch、重新盖章。这解释了为什么清 latch/rules/块治不好：门一直热着，污染每轮再生。（你会话具体命中哪个守卫分支，需要污染时刻的 stats 快照确认；两个分支指向同一机制。）applyTo）= 整条 truncate 最后手段关掉，确实免疫，但也丢了省上下文的能力——同意你说的「不是修复」。PR #5537（出站第一道防线，无决策依赖，先行交付）
#5537
tool_calls[*].function.arguments与 Responsesfunction_call.arguments的 JSON 合法性；非法 → 替换为合法占位对象 + 一条[wire-guard]warn 日志（带前 64 字节供取证）；custom_tool_call.input是自由文本不动；Google wire 已有同类守卫，Anthropic 结构性安全（对象类型）；行为变更披露：旧行为 = bili 自身 egress 故障产生的非 JSON arguments 原样发出 → 上游 500 + 会话永久中毒；新行为 = 替换为占位符 + warn 日志，请求可继续。仅对已非法的字节生效。
结构化八问
tool_calls.arguments被写成折叠/checkpoint 散文，此后该会话每个请求都 500，且只要 latch 存活污染就自我延续。{ref,text}）；冷启动账本 restamp 重排 ref 使旧条目贴到不相干消息（恰为 tool-call core 时其 text 即 arguments JSON）；叠加 meter 回退（尺寸守卫/cycle-stale）让 emergency 门长期热着，每轮再生污染。判定：主机侧防线可合并（等 CI）；内核根治等你拍板。另外若方便，给我一份污染时刻附近的 stats 快照（contextTokens / contextTokensSource / localInputEstimate / lastInputTokens / lastUsagePayloadUpper），我可以把粘滞触发精确钉到具体守卫分支。
标签：bug / dsh / scope/kernel / p1（已在首条回复打好，理由：用户不可用的永久中毒类，owner 修订一律 p1）。
ranxianglei commentedon Nov 2, 2026
🔶 需要决策
背景一句话：内核的 emergency-truncate latch 与 view-latch 回放「截断文本」时只认 ref 编号、条目不带身份（仅
{ref, text}），ref 重排（冷启动账本 restamp 是已证实的触发面）后旧文本会贴到不相干的消息上——若那条恰好是 tool-call core，其 arguments JSON 被散文覆盖，严格 JSON 上游的会话从此永久中毒（本 issue / GitHub #5534）。出站守卫已在 PR #5537 交付（毒不再上线），但内核侧根因仍开着。待决问题：是否批准内核变更，给 latch 条目加身份校验。
选项：
EmergencyTruncateLatchEntry与 view-latch 条目增加 rawId/内容指纹，回放时仅当 ref 当前持有者与源消息身份匹配才盖文本，否则视同 retired 丢弃。改动面：kernel/src/compress.ts（emergencyTruncateNode）、kernel/src/view-latch.ts（applyViewLatch）、kernel/src/types.ts、序列化/hydration。代价：内核变更（Kernel Boundary：PR 需kernel-change声明 + 人工评审 + kernel 版本号 bump + 双层测试）；旧格式条目（无身份字段）按无效丢弃，天然安全。残余：无——错贴在结构上不可能发生。maybeRestampToLedger重排 ref 时同步丢弃受影响 ref 的 latch 条目。代价：改动小（persist.ts + 条目操作），但只覆盖已知触发面（restamp）；其他潜在重排路径（未来无 floor pin 的 map-shrink）不覆盖。推荐：A（根治，结构性消灭这一类「按裸 ref 回放文本」的病理，view-latch 同族一并修掉）；实现 A 时顺带 C（过渡期双保险）。不决定的默认行为 = B 保持有效：守卫已随 PR #5537 交付，毒无法上线，内核侧残余降级为「模型看到占位散文」的体验问题，不再是可用性问题——所以这个决策可以慢慢拍，不阻塞合并。
test(#5534): rename the guard suite to its actual issue number
fix(#5534): extend the egress args guard to the nudge-decide side bodies
fix(#5534): correct a bad identifier introduced by the merge conflict…
ranxianglei commentedon Nov 2, 2026
看看master修复了吗
ranxianglei commentedon Nov 2, 2026
(ai: vllm-qwen/qwen3.8-27b)
master 还没修——修复在 PR #5537，目前还是 open（未合并）状态：
dec393f43（最新发版 v0.1.212），我 grep 确认上面没有守卫代码；v0.1.209 之后合入的 PR 也没有任何一条碰到这个机制（latch 回放 / truncate 写回 / meter）。所以跑在 v0.1.212 上的会话仍然会中毒。issue5536改成了正确的 GitHub 编号issue5534；补了一个我原修法的盲区——nudge-decide 侧调用（useDecide 分支用主守卫运行前的前缀副本构建 side call，中毒会话里 decide 往返同样会 500），现在两条 wire 的 side 前缀也过同一守卫。5eded5390）：全绿——59 success / 31 skipped / 0 failure。