Skip to content

[PR] fix: keep only CA certificates in combined-ca.pem (#5802) - #5809

Merged
ranxianglei merged 3 commits into
masterfrom
2026-11-06_ca-bundle-ca-only-filter
Nov 10, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-11-06_ca-bundle-ca-only-filter

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #5802

What was broken

On macOS, bili codex failed on every connection: Codex refused to build its HTTP client against SSL_CERT_FILE (~/.local/share/billion-context/ca/combined-ca.pem). The darwin OS-store export (security find-certificate -a -p over System + login keychains) copies every keychain entry into the bundle — including non-CA leaf identities such as com.apple.kerberos.kdc (which also carries a duplicate EKU extension, violating RFC 5280 §4.2). Clients with replace-semantics trust stores (codex's rustls stack) parse every entry of the file and refuse to start when any single one fails. Net effect: model transport + all MCP clients dead, bili tools never appear.

Fix

  • writeCombinedBundle() keeps only entries that parse as X509 certificates AND carry basicConstraints CA:TRUE or the keyCertSign key-usage bit (legacy pre-basicConstraints anchors). Applied at the single merge choke point to ALL sources: system bundle, OS store, Mozilla roots, MITM root.
  • The bundle gains a leading marker comment line. A fresh file missing the marker (a pre-upgrade bundle) is rebuilt once immediately on startup, so the fix takes effect right after upgrading instead of waiting out the 24 h freshness gate.
  • Availability floor: if the filter ever drops everything (certificate parsing broken), fall back to the unfiltered bundle with a loud error log rather than shipping an empty trust store. Unreachable in practice — bili's own MITM root always passes.

The keyUsage check scans raw DER for OID 2.5.29.15 because Node's X509Certificate.keyUsage getter returns undefined on current runtimes (v22) and node-forge cannot parse non-RSA subjects (using it would silently drop ECDSA anchors).

Behavior changes (old → new)

old new why
Bundle content verbatim union of system bundle + OS store (darwin: ALL keychain entries incl. leaf identities / S-MIME / app-signing certs) + Mozilla roots + MITM root only CA-like entries from all sources; unparseable entries dropped strict replace-semantics clients parse every entry; one RFC-violating leaf bricks them all (#5802)
Bundle header none one comment-line marker PEM parsers skip non-block text; enables the one-time post-upgrade rebuild gate
Post-upgrade refresh stale bad bundle lingers until ≤24 h freshness expiry rebuilt once immediately (marker missing) otherwise the fix looks inert for up to a day
Filter result empty n/a unfiltered fallback + error log never ship an empty trust store

Residual: a self-signed NON-CA cert someone relies on via OpenSSL's lenient anchor quirk loses its anchor (rare; such certs are misissued). Windows/Linux exports were already CA stores in practice; the filter is a uniform safety net there too and also covers user-set SSL_CERT_FILE overrides.

No config surface change (no new config/env vars).

Tests

  • New regression tests in tests/ca-combined.test.ts ([Bug][codex][macOS] bili codex fails on every connection: combined-ca.pem includes non-CA keychain certs, and com.apple.kerberos.kdc makes Codex reject the whole bundle #5802 rows): predicate keeps real roots + legacy keyCertSign anchors + ECDSA roots, drops serverAuth leaves / unparseable entries (both gate directions); written bundle contains only CAs + marker; missing-marker fresh file force-rebuilt; marked+rooted fresh file untouched (steady state).
  • tests/ca-combined.test.ts 19/19 locally; the other src/ca.ts suites (graceful-close, mitm, issue3776-tls-mitm-diagnostics) 32/32; typecheck clean. Full-suite verdict = CI on this head.
  • Reporter verified the same filter shape against their 190-cert bundle: drops exactly the 6 non-CA keychain certs (incl. com.apple.kerberos.kdc), keeps 184 CAs incl. login-keychain user installs; Codex then builds its client cleanly.

Touches CA/trust territory → human review per §7.4 (not auto-merge eligible).

darwin's 'security find-certificate -a -p' keychain export includes
non-CA leaf identities (e.g. com.apple.kerberos.kdc, which also carries
a duplicate EKU extension in violation of RFC 5280 4.2). Strict
replace-semantics clients such as codex's rustls HTTP stack parse every
entry of SSL_CERT_FILE and refuse to build their client when any entry
fails, so a single such cert broke every connection through bili codex.

writeCombinedBundle now keeps only entries that parse as X509
certificates AND carry basicConstraints CA:TRUE or the keyCertSign
key-usage bit (legacy pre-basicConstraints anchors), applied to ALL
sources (system bundle, OS store, Mozilla roots, MITM root). The bundle
gains a leading marker comment; a fresh file missing it (pre-upgrade
bundle) is rebuilt once so the fix takes effect immediately instead of
waiting out the 24 h freshness gate. If the filter ever drops everything
(broken certificate parsing), it falls back to the unfiltered bundle
with a loud error log rather than shipping an empty trust store.
@ranxianglei ranxianglei changed the title fix: keep only CA certificates in combined-ca.pem (#5802) [PR] fix: keep only CA certificates in combined-ca.pem (#5802) Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-06_ca-bundle-ca-only-filter (9f9c073)

Latest build — install this exact version

npm install -g billion-context@0.0.8916-pr.5809.177af75

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
9f9c073 billion-context@0.0.8916-pr.5809.177af75 build
d72c1f2 billion-context@0.0.8368-pr.5809.ec0f77b build
ad4a97a billion-context@0.0.8362-pr.5809.ddbb277 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5809.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 全部开放 [PR] 行中只有本线程针对 combined-ca.pem 过滤(#5802);相关但机制不同的已有条目:#5739(zcode 车道 CA 路径指向 combined-ca.pem)、#5740(node-forge/undici 依赖升级)——均非同一机制,不冲突。

开始审核:① 原 issue #5802 方向已核对(报告者带完整复现 + 二分证据:bisect 钉死 com.apple.kerberos.kdc、X509Certificate.ca 过滤在真实 190 证书包上验证过——根因定位准确,修复打在 producer 层即合并收口点 writeCombinedBundle(),不是症状拦截)。正在逐行读 diff(基于当前 master、无冲突),随后本地定向跑 ca 测试套件 + 看 CI,结论按结构化八问跟进。

Review finding on PR #5809: the raw-DER KeyUsage scan expected the
extnValue OCTET STRING immediately after OID 2.5.29.15, but per
RFC 5280 s4.1 a critical-marked extension carries a BOOLEAN between
the two. KeyUsage is RFC-recommended critical on CA certificates, so
exactly the legacy pre-basicConstraints anchors the fallback branch
exists for were silently dropped whenever they carried the marking.
Empirically verified: a forged keyCertSign-only cert with the
critical flag parsed isCaLike=false before this change, true after.
Adds the matching regression row with ground-truth assertions.
@ranxianglei ranxianglei added bug Something isn't working codex p1 scope/wiring labels Nov 6, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 完成:发现并直接修复 1 处真实缺陷(已推 labeled commit d72c1f210),其余通过。CI 旧 head 有 1 个 shard 红,判定与本 PR 无关,新 head 全量重跑中。

审核中发现的缺陷(已修,commit d72c1f210)

hasKeyCertSign 的 raw-DER 扫描假定 OID 2.5.29.15 之后紧跟 extnValue OCTET STRING,但 RFC 5280 §4.1 规定 critical 标记的扩展在两者之间有一个 OPTIONAL BOOLEAN。keyUsage 在真实 CA 证书上按 RFC 建议普遍带 critical 标记(DER 实测:06 03 55 1d 0f | 01 01 ff | 04 04 …)——也就是说这个 fallback 分支声称服务的「pre-basicContracts 遗留锚点」恰恰在带 critical 标记时会被静默丢掉。本地实证:伪造 keyCertSign-only + critical 证书,修复前 isCaLike=false,修复后 true。修复 = 扫描时跳过可选的 critical BOOLEAN(3 字节)+ 补一条带 ground-truth 断言的回归行。实际影响面极小(现代 OS 信任库里 pre-basicConstraints 锚点已绝迹,#5802 主修复完全靠 .ca 路径,不受此影响),但注释声称的覆盖与代码事实不符,按「小问题直接修」处理。

顺带实证了 PR 的两条关键论断(Node v22 实测):X509Certificate.keyUsage getter 确实对每张证书返回 undefined(raw-DER 扫描的必要性成立);node-forge 不能解析非 RSA subject(ECDSA 根证书测试行钉住)。

结构化八问

  1. 是什么问题:macOS 上 security find-certificate -a -p 把钥匙串里所有条目(含非 CA 叶子身份如 com.apple.kerberos.kdc,其重复 EKU 违反 RFC 5280 §4.2)原样并入 combined-ca.pem;codex 这类 replace-semantics 信任库(rustls)逐条严格解析,一个坏条目就拒绝整份文件 → bili codex 每次连接都失败。
  2. 什么原因导致的:bundle 合并点(writeCombinedBundle)对 OS store 来源零过滤,darwin 导出形状(全量钥匙串)与其他平台(CA 存储)不一致,坏字节在源头进入 bundle。
  3. 影响范围与严重程度:macOS + codex/dsh/hermes 等 SSL_CERT_FILE 车道,完全不可用(模型传输 + MCP 全死);Windows/Linux 导出实践上已是 CA 存储,实际未触发,过滤是统一安全网。
  4. 怎么修复的:在唯一合并咽喉点对全部来源(system bundle、OS store、Mozilla roots、MITM root、用户 SSL_CERT_FILE 覆盖)统一施加「可作信任锚」谓词(basicConstraints CA:TRUE 或 keyCertSign);bundle 头部加格式版本 marker,升级后首个启动一次性强制重建(不等 24h 新鲜度窗口);过滤器若清空一切则回退未过滤 bundle + 响亮 error log(MITM root 恒过过滤,空结果只可能是解析器坏了)。
  5. 是否有回归问题:逐项清点既有行为——① 24h 新鲜度门:保持,新增 marker 条件 = 披露过的有意变更(升级后一次立即重建,稳态 marker+root+fresh 复用不变,有 mtime 相等性测试钉住);② promptCache.routing is ignored by relay-affinity stamping; no way to persist per-endpoint prompt_cache_key abatement (0.1.209) #5103 写拒绝复用路径(EACCES→warn+沿用现有 bundle):未动,与用户 [Bug][codex][macOS] bili codex fails on every connection: combined-ca.pem includes non-CA keychain certs, and com.apple.kerberos.kdc makes Codex reject the whole bundle #5802 workaround(chmod 444)交互正常(升级后启动尝试重建→EACCES→warn+沿用,语义同旧);③ 指纹去重:未动;④ bundle 消费方(codex/dsh/hermes/zcode 的外部 PEM 解析器)均忽略 BEGIN/END 块外文本,marker 行安全;仓内无进程内解析该文件的读方;⑤ 高危模式逐项:无 fail-fast 变吞错(回退 floor 是 loud error)、无阈值/时序漂移(除披露的一次立即重建)、无输出格式变化(除披露的 marker 行)。结论:无意外回归;我修的 critical-BOOLEAN 缺口是新分支内的潜在欠覆盖,非既有行为回归。
  6. 是否破坏缓存:不涉及(无缓存/重试/游标/去重键变更;PEM 指纹去重逻辑未动)。
  7. 是否改了设计方向:无——同一合并咽喉点架构,只是 bundle 文件多了格式版本标记。
  8. 是否引入新配置:无(PR 声明无 config surface 变化,已核实:无新 env/config key)。

测试有效性

4 条新行断言双向门(保留真根/ECDSA 根/遗留锚点[含 critical 变体];丢弃 serverAuth 叶子/不可解析条目),且每条先对 X509Certificate 本身做 ground-truth 断言(防恒真);「写出的 bundle 只含 CA + marker」行在每个 CI 车道对真实 OS store + Mozilla 快照生效;marker 强制重建 vs 稳态不复写成对区分新旧行为。fixture 隔离在 tmp HOME/XDG_DATA_HOME 且 env 还原,无端口/环境运气。本地:ca-combined 19/19 + mitm/issue3776/issue3632 合计 62/62 绿,typecheck 干净。

CI 状态(旧 head ad4a97ae5)

12 个 test shard 中 1 个红(windows-latest/node22/shard 2,2052 测 2 失败),失败签名:

  • tests/msgid-lane-flow.test.ts:197(msgid 增量车道 delta append 断言,期望 [a1,u1,u2] 实得 [u1])
  • tests/wire-exit-matrix.test.ts:334(abort×proxy-json 客户端中止后上游 socket 未销毁,等待 21.5s)

两者都在本 PR 完全不触碰的子域(增量车道账本 / abort-teardown),且在 PR head 上单独本地运行 14/14 全绿;PR base be611a856 的 master 全套 CI 当时为绿,其后 master 仅移动了 docs-only 的 #5803。判定:runner 负载下的 flake(时序敏感测试),非本 PR 回归。新 head d72c1f210 正在全量重跑,结果出来我再回来确认——同一签名复现则会按 #4599 流程继续排查。

判定:代码侧可合(§7.4 CA/trust 域留人工评审);等 d72c1f210 的 CI 全绿即可由 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 自动修复车道(attempt 1/3)收工:零代码改动、零推送,重跑后全矩阵转绿。 分支 head 保持 ad4a97a 不变。

七问:

  1. 是什么问题:本 PR 唯一红项 test (windows-latest, 22, 2)(run 42220608106 / job 127903301148),该 shard 内两枚测试失败——tests/msgid-lane-flow.test.ts:197(delta id 追加断言,expected ['a1','u1','u2'] actual ['u1'])与 tests/wire-exit-matrix.test.ts:334("abort x proxy-json" upstream socket 销毁断言 closed=0);其余 29 个 check 全绿/门控跳过。
  2. 什么原因导致的:两者均为 Windows runner 负载下的在档 flake 家族,与本 PR 代码无关——① msgid-lane-flow settle-race:settle() 只等一次 afterNextFlush 判决,负载下两次 debounced write 重叠,判决落在 pre-turn 快照,行读丢新行(与在档 :139「3 !== 4」同机制,本次表面在 :197 cell);② wire-exit-matrix abort-teardown 时序:closed=0 且耗时 21580ms,与在档 Windows 家族签名(floor 1042/[PR] fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (#5782) #5783 @21.5s、floor 1047 今日 ×2 cells、floor 1032/[PR] kernel: fold tool-call/result pairs by exact id at any distance (fixes the split-pair upstream-400 / folding-stuck family) #5758)逐字吻合。
  3. 影响范围与严重程度:仅 CI 信号可靠性,用户面零影响;PR 自身功能(combined-ca.pem CA-only 过滤)不受牵连——其专属 ca-combined 套件在同 run 内通过,静态门禁(static-gates/gate/lock-sync/version-guard 等)全绿。
  4. 怎么修复的:按在档 flake 处置口径——排除链走完后触发 rerun-failed-jobs(HTTP 201),不改码不推送。排除链:PR diff(src/ca.ts + tests/ca-combined.test.ts + 文档两行)与两条失败路径零交集;本地对同一树定向验证 msgid-lane-flow 6/6×3 连跑 + wire-exit-matrix 8/8 全绿;PR base = master tip be611a8(当期全绿,非继承红,无需 merge ci-fixes)。
  5. 是否有回归问题:无引入——本轮未改任何代码;既有行为审计 = 不适用(零 diff)。
  6. 是否破坏缓存:不涉及。
  7. 是否引入新配置 / 改变设计方向:不涉及。

终态:镜像缓存确认 30 checks failed=0 / pending=0,全矩阵绿。PR 可进入评审/合并(合并动作归 owner)。备注:若 :197 或 closed=0 签名在干净 runner 再复发,按在档口径升档(settle() 收紧属 owner 拍板项,#5791 线已有测试侧修复 e278f03 待合);诊断已同步至 #4599。

@ranxianglei ranxianglei added the ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR label Nov 6, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 补充:d72c1f210 的首次派发出现了重复 dispatch 互相取消(同一秒内同一分支多个 workflow run 被创建,concurrency cancel-in-progress 把主 CI 的两套矩阵全取消在排队阶段,jobs 数为 0),head 上没有有效裁决。已按仓库既有机制用 ci:full 标签触发一次干净的全量重跑(run 42228317787,13:47Z 起),结果出来我给最终确认。代码侧结论不变。

master moved CONFIGURATION{,.zh-CN}.md to docs/configuration{,.zh-CN}.md
(docs layout ruling #5920; root keeps redirect stubs). Resolution: keep
master's stubs at the root and relocate this PR's one-line CA-bundle
note (#5802 CA-only narrowing of combined-ca.pem) onto the same
paragraph in its new home, en+zh. Code commits (ad4a97a, d72c1f2)
are untouched.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Merge-conflict resolution — done, full CI green. New head: 9f9c0733e3 (fast-forward push, no force-push, PR body untouched, no authorization gates involved).

What happened & why it conflicted. Master's docs reorganization (#5920 ruling) moved CONFIGURATION.md / CONFIGURATION.zh-CN.md to docs/configuration.md / docs/configuration.zh-CN.md, leaving one-line redirect stubs at the root. This PR's commit ad4a97ae5 edited the old root files' CA-bundle paragraph — both sides modified the same region, so git could not auto-merge (merge state dirty). Pure timing, not a code disagreement: the PR's two code commits (ad4a97ae5, plus the concurrent second commit d72c1f2108 another agent landed on the same branch during this window) never touched anything conflicting.

How it was resolved. Chose merge over rebase deliberately: the concurrent second commit meant rebasing would have rewritten someone else's work; merging preserves both commits untouched. Exactly 2 files conflicted (CONFIGURATION.md, CONFIGURATION.zh-CN.md). Resolution: keep master's root stubs, and relocate this PR's one-line CA-bundle note (#5802 — combined-ca.pem narrowed to CA certificates only) verbatim onto the same paragraph in its new home, docs/configuration.md:2414 (en) and docs/configuration.zh-CN.md:2381 (zh) — en/zh parity preserved. Pre-push fetch confirmed the remote head was still d72c1f2108, so the push was a clean fast-forward d72c1f2108..9f9c0733e3.

Verification (all live, this run). On merged head 9f9c0733e3: npm ci clean (611 pkgs), tsc --noEmit clean, tsup build clean, tests/ca-combined.test.ts 19/19 pass. Final diff vs master is exactly the PR's own four files (src/ca.ts, tests/ca-combined.test.ts, the two doc paragraphs) — nothing else dragged in. Mirror cache (polled ~60 s cadence until settled): mergeable_state flipped dirty → clean; full CI matrix on the new head = 30 checks, 29 success + 1 skipped, 0 failed, 0 pending (runs 42782430088 / …30303 / …30435 / …30158 / …30066).

七问

  1. 是什么问题:PR 分支与 master 冲突(merge state dirty),阻塞合并——master 把配置文档搬去了 docs/,本 PR 还在改旧位置的根目录文件。
  2. 什么原因导致的:PR 基于 be611a8 开出时根目录还是完整文档;master 其后按 [PR] docs: batch-migrate 14 root-level documents into docs/ (consolidation + master index + redirect placeholders) | docs: 14 个根级文档一次性迁入 docs/(集中 + 总目录 + 重定向占位) #5920 文档布局裁定把 CONFIGURATION{,.zh-CN}.md 迁到 docs/configuration{,.zh-CN}.md、根目录换成单行重定向存根。双方对同文件同区域都有改动,git 无法自动合并。纯时序问题,非代码分歧。
  3. 影响范围与严重程度:仅阻塞本 PR 合并;冲突全部在两个 markdown 文档,不影响任何运行行为,无用户面影响。
  4. 怎么修复的:merge(而非 rebase)origin/master 进分支——因另一 agent 刚在同分支并发落了第二个提交,merge 不动既有提交、保住其工作。两处冲突均按「保留 master 存根 + 本 PR 的一行 CA-bundle 说明原样落到新位置同段落(en+zh)」解决;推送前 fetch 确认远端未动,fast-forward 推上,无需 force-push。
  5. 是否有回归问题:无。合并后树本地验证全干净(typecheck/build/ca-combined 19/19);与 master 的最终 diff 只剩本 PR 自己的 4 个文件,零夹带;文档 en/zh 两份同步落地,parity 保持;CI 全矩阵 0 失败。
  6. 是否破坏缓存:不涉及——纯文档位置调整 + PR 原有代码,未触任何缓存/会话数据路径。
  7. 是否改了设计方向 / 引入新配置:都不涉及。文档只是跟随 [PR] docs: batch-migrate 14 root-level documents into docs/ (consolidation + master index + redirect placeholders) | docs: 14 个根级文档一次性迁入 docs/(集中 + 总目录 + 重定向占位) #5920 的新布局走,PR 原有行为(CA-only 过滤)一字未改。

判定:冲突已解、CI 全绿、mergeable_state=clean —— PR #5809 ready for review/merge(合并动作归 owner)。

@ranxianglei
ranxianglei merged commit 01725f8 into master Nov 10, 2026
56 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1169-master.01725f8 (rolling master tag).

npm install -g billion-context@0.0.1169-master.01725f8  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR codex p1 scope/wiring

Projects

None yet

1 participant