Repository navigation
[PR] fix: compress separator guard for CJK range words + rejected-payload preview (#6071) - #6074
ranxianglei wants to merge 4 commits into
Conversation
Field evidence (session-9dd7b2f0, v0.1.213): a CJK model joined two refs with the Han range word 到. The #4253 guard class [^\p{L}\p{N}\s] only matches punctuation runs, so "m07890到m07950 t" fell through as a SINGLE ref m07890 with topic "到m07950 t" and silently folded one message of a many-message range — exactly the outcome the guard's own comment says it exists to prevent. Extend UNRECOGNIZED_REF_SEP_RE so the unrecognized-separator run may mix punctuation with Han-script letters; such headers now reject loudly with the actionable reason instead of narrowing. No new accepted forms. Latin topics are untouched (the extension is \p{Script=Han}-only), pinned both directions: Han-word+ref rejects; Han topic without a following ref still parses as a single ref.
The rejected warn carried only kind/keys/reasons, so a field occurrence of an unrecognized separator had to be guessed from the model's next retry (session-9dd7b2f0: the failing bytes were never in the log). The line now carries preview= with up to 300 chars of the offending content (bare string or the string entries of a content array). Server log only via the existing note() helper — quiet mode (#5161) still suppresses it, and it is never echoed into the model-facing receipt (#1454). Object-only rejections log no preview (nothing string-shaped to show).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.9305-pr.6074.baa347aThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr6074.tgz
npm install -g packageInstall instructions are refreshed on each push. |
ranxianglei
changed the title
Review fix for the red lock-sync lane on this PR head: the kernel version bump commit updated kernel/package.json alone, while the established bump pattern (e.g. 5ac2983 for 0.0.191) also moves the recorded version in kernel/package-lock.json (top-level + packages[""]) and the root package-lock.json packages["kernel"] entry. Regenerated with npm install --package-lock-only in both trees; diff is version lines only.
(ai: vllm/qwen3.8-27b) 查重: 不重复 — searched open+closed issues and PRs: nothing else touches the unrecognized-separator guard in 标签: bug + dsh + scope/kernel + p2(理由:field 上 dsh plugin 车道发现的静默窄化缺陷——多消息范围被折成一条消息;核心修在 kernel;模型侧有换分隔符重试的绕行路径,属"能用但有明显问题",故 p2 而非 p1)。 Review in progress — independent read of the diff done, preliminary findings:
Awaiting the CI re-run on 6d246c0 for the full-suite verdict, then the structured 8-question merge call follows in-thread. |
ranxianglei
left a comment
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
Review — findings only (kernel-change: human tier gates merge)
Direction, correctness, tests, regressions, conventions all walked against the head source (not just the diff). Verdict: the change is correct and complete for its stated scope; I am NOT approving because it touches kernel/ and carries a kernel-change: declaration — that merge sits at the human-only review tier (Kernel Boundary rule 3). A 🔶 decision-request follows in a separate comment.
What it does (verified)
- kernel
kernel/src/parse-compress-input.ts:522—UNRECOGNIZED_REF_SEP_REwidened from[^\p{L}\p{N}\s]{1,3}to(?:[^\p{L}\p{N}\s]|\p{Script=Han}){1,3}. CJK range words (到/至, which are\p{L}) now hit the guard instead of slipping through. I replicated the exactparseLineEntrydecision path standalone and confirmed:- OLD:
m07890到m07950 t→ parsed as SINGLE ref m07890 (silent one-message fold of a multi-message range — the reported latent defect). NEW: rejects loudly. ✓ - Accepted pair forms (en-dash / hyphen /
to/ dots) still matchREF_PAIR_IN_LINEbefore the guard is ever consulted → unchanged. ✓ - Latin topics and Han topics without a trailing ref (
到期的任务) still parse as single ref → no over-triggering. ✓ - Punctuation separators (
--,→,〜) still reject as before. ✓
- OLD:
- host
src/compress-tool.tsrejectedContentPreview()+ thepreview=field on the rejected log line. Server-log only via the existingnote()helper (quiet mode #5161 still suppresses; never echoed into the model-facing receipt #1454 — the receipt is built frominvalidReasons, which is untouched). Bounding verified: per-piece 150 chars / total 300; object-only rejections log no preview.
Regression audit (existing behaviors on touched paths)
- Guard: punctuation-reject preserved; accepted-pair forms unaffected (they short-circuit before the guard); Latin topics unchanged. Only intended delta = Han/mixed separators now reject instead of silently narrowing (disclosed in the PR table).
- Log line: purely additive optional field; every existing assertion on that line is
includes-based (no exact-match breakage); model-facing format unchanged. - kernel version 0.0.191→0.0.192: required for a
kernel/srcchange; discipline correct. - High-risk patterns checked: no fail-fast→swallow, no threshold/timing drift, no wire/output-format change to the model, no cache/retry/cursor impact.
Test validity (both directions, teeth present)
- kernel: positive (
到/至/mixed punct+Han →ranges.length===0+no-valid-ranges+ reason match) would fail if the guard regressed to silent-narrow; negative (Han topic w/o trailing ref + Latin topic → single ref) would fail on over-trigger. Real. - host: rejected LINE entry logs
preview=with the exact offending bytes; 800-char payload truncated (asserts no 200×x); rejected OBJECT entry logs no preview. Both directions pinned per #3617. CI is green on the head.
Conventions
Kernel Boundary: kernel-change: declared ✓, version bumped ✓, reproducing regression tests added ✓, mechanical-bug exemption plausibly met (provably-incomplete guard class) ✓. Diff clean (5 files, +124/-3, every line on-purpose), no dead code, naming clear, hex-escape rule n/a (no ACP tags introduced).
Residuals — routed to the pending acceptance-set decision (NOT new debt on this PR)
Two things surfaced while probing edges; both belong to the already-open 🔶 on #6071 (whether to expand the ACCEPTED separator set), so I note them for that call rather than parking them here:
- The
{1,3}run bound is preserved, so a CJK connector LONGER than 3 Han chars (e.g.一直走到) still silently narrows — pre-existing, same mechanism family as the acceptance question. - A single-ref header whose topic contains
<han> <bare-ref>(e.g.m00001 见 m07890) now rejects where it previously parsed — safe direction (loud > silent), consistent with how punctuation-divided topics were already treated, and contrived.
Neither blocks this PR; both are inputs to the owner's A/B/C acceptance decision.
结构化八问（中文，供人工评审）
- 是什么问题：compress 行式头解析器的 #4253 守卫只认标点类分隔符，CJK 汉字区间词（到/至，属
\p{L}）漏过守卫，m07890到m07950被静默解析成单条 m07890——把多消息区间收窄成一条且无报错；另外被拒内容原文不落日志，现场只能靠猜。 - 什么原因导致的：根因在守卫正则字符类
[^\p{L}\p{N}\s]{1,3}只覆盖标点、汉字是字母类直接漏过（疏而非堵）；观测缺口是拒绝日志只记 kind/keys/reasons、不记 payload。 - 影响范围与严重程度：所有用非标分隔符写行式头的会话，中文模型高频触发 到/至。报告症状＝可见报错＋浪费一轮（重试自愈）；静默误压子缺陷一旦发生是数据完整性级（区间缩水、模型与用户都以为压了 N 条实际只压 1 条、无报错）。本 PR 把后者从静默收窄改为显式拒绝。
- 怎么修复的：①kernel 守卫字符类扩为「标点或 Han 字」混合串也算未识别分隔符（不新增任何接受形式），双向回归测试钉死；②host 拒绝日志加有界预览（每段≤150/总≤300 字符，server log only、不进模型回执、quiet 仍抑制）。是否把 到/至/→/--/dash 家族加进接受集合刻意不在本 PR，留给 owner 拍板（已在 #6071 线程挂 🔶）。
- 是否有回归问题：逐项过——标点分隔符（--/→/〜）仍显式拒绝✓；已接受 pair 形式（en-dash/hyphen/to/dots）走 REF_PAIR 先于守卫、不受影响✓；拉丁主题不变✓；模型回执由 invalidReasons 构建、未变（#1454 守住）✓；quiet(#5161) 仍抑制✓；既有日志断言均 includes、无精确匹配被破坏✓。唯一行为漂移＝「Han/混合分隔符从静默收窄→显式拒绝」（有意、已披露）＋「server 拒绝日志新增可选 preview 字段」（纯增量）。两处已核实残留（>3 字 CJK 连接词仍会静默收窄；单条头主题含「汉+裸 ref」现被拒）归入待决接受集合决策，非本 PR 机制缺陷。
- 是否破坏缓存：不涉及——纯解析器＋日志改动，不触碰缓存/重试/游标/去重键。
- 是否改了设计方向：无架构转向；同一失败域两件事一个 PR，符合机制归并；接受策略扩展刻意剥离为独立决策。
- 是否引入了新的配置：不涉及——无新 env/config/默认值。
判定：代码正确、范围内完整、测试双向有牙、diff 干净、内核边界合规（声明＋版本纪律＋复现测试齐备，机械 bug 豁免成立）。因触及 kernel/ 且带 kernel-change: 声明，按硬规则不自批——转人工层，请 owner 授权合并该 kernel 变更（见下方 🔶 评论）。
|
🔶 需要决策 (ai: vllm-qwen/qwen3.8-27b) 请求 owner 授权：PR #6074 触及
请在此回复授权或给意见；我不会自行合并 PR，也不开新 issue。 |
[bot] 🏷 处置：触及
结论：代码正确、范围内完整、测试双向有牙、diff 干净、内核边界合规（机械 bug 豁免四要件齐备）。独立复刻 等你：人工授权内核合并即可；我未合并 PR、未开 issue。 |
[bot] 🏷 (ai: vllm/qwen3.8-27b) CI interim (verified live, head |
[bot] 🏷 (ai: vllm/qwen3.8-27b) Final review verdict: 可合 (mergeable) — human merge; CI state below verified live at head Independent read confirms both changes sit at the right layer:
结构化八问
判定:可合——代码层面零阻塞项; |
[bot] 🏷 (ai: vllm/qwen3.8-27b) Last lane landed green — CI now fully verified. The stalled |
|
该PR已按根因归类纳入 [docs] 根因汇总 #6337，由汇总单统一跟踪处理；本单不再单独跟进，故关闭。如认为归类有误或问题仍未解决，请直接回复，会重新打开。 |
|
状态更正(自动化脚本误操作): 本单此前被「纳入根因聚合」流程误关闭, 现重新打开。机器人工作已停, 本单由根因聚合线程统一跟踪, 根因修复落地后请手动关闭或在本单回复「close」。 |
|
重新打开:此前被归类聚合脚本误关。按 owner 口径,纳入根因聚合的 issue 只停机器人、不关闭,由根因聚合线程统一跟踪,根因修复落地后统一关闭。 |
|
（更正）此前按"清理"口径误关闭；按 owner 口径纠正：被根因聚合纳入的 issue 只停机器人、不关闭。本 issue 重新打开，机器人工作已停。 |
…-6074 # Conflicts: # kernel/package-lock.json # kernel/package.json # package-lock.json
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Conflict already resolved by a concurrent run — I did NOT push (my local merge is stale). While I was resolving the conflict locally, the remote PR branch moved from 七问
Side note (out of scope, not blocking): while building locally I hit a pre-existing drift on master — Bottom line: the merge conflict is resolved on the remote by a concurrent run (verified correct — current master, kernel |
Closes #6071
kernel-change: close the silent-narrowing hole in the #4253 unrecognized-separator guard — Han-script (CJK) range words like 到/至 between two refs bypassed the punctuation-only class and folded ONE message of a many-message range with no error. Mechanical-bug exemption per owner policy 2026-11-06: provably-incomplete guard + reproducing regression tests + this declaration + in-thread report; human review tier still applies.
What happened (field)
session-9dd7b2f0 on v0.1.213 (deepseek-v4.1-flash-1017, dsh plugin lane): the model's T2 distill compress call was rejected with "refs separator not recognized" (the exact separator is unrecoverable from the log — see observability change below), the model retried with en-dash block ranges and succeeded one round later. No data loss, but:
m07890到m07950 t→ parsed as SINGLE ref m07890, topic "到m07950 t" → silent one-message fold of a multi-message range. Same for 至. This violates the guard's own stated invariant (kernel/src/parse-compress-input.ts).--,→,->,,,〜, U+2010/2011/2012/2015 all reject loudly (the guard works for punctuation — those are the reported symptom's candidates).Changes
kernel (acp-kernel 0.0.191 → 0.0.192) — kernel/src/parse-compress-input.ts:
=>stays rejected,toword form, dash variants) untouched.host — src/compress-tool.ts:
[acp-compress-input] rejectedwarn now carriespreview=with up to 300 chars of the offending content (bare string or the string entries of a content array), so the next field occurrence pins the exact shape on first sight. Server log only via the existing note() helper — quiet mode ([Bug][external summary] every compress re-parses the visible past compress calls and logs each summary-less one asrejected- ~550 false warn lines a day #5161) still suppresses it, and it is never echoed into the model-facing receipt (fix: replay connect-phase transport failures and report committed compression when the re-request dies (#1453) #1454). Object-only rejections log no preview.Behavior changes (old → new)
m07890到m07950 t(Han word between refs)Deliberately NOT included (pending owner decision in-thread)
Expanding the ACCEPTED separator set (到/至, →, ->, --, 〜, dash family U+2010–U+2015) so CJK models don't need the retry round-trip at all. That is a leniency/policy choice (precedent: #4255 dash variants); options + recommendation are posted on the issue thread. If approved, it lands as follow-up commits on this branch (one PR per thread).
Verification