Skip to content

[Bug][codex][macOS] bili codex fails on every connection: combined-ca.pem includes non-CA keychain certs, and com.apple.kerberos.kdc makes Codex reject the whole bundle #5802

Closed

Description

@SnoopyGOGO14

Summary

On macOS, bili codex cannot make any HTTPS request. Codex refuses to load the SSL_CERT_FILE bundle bili writes (~/.local/share/billion-context/ca/combined-ca.pem):

Failed to build HTTP client while using CA bundle from SSL_CERT_FILE (~/.local/share/billion-context/ca/combined-ca.pem): builder error

This breaks both the model transport (Reconnecting... 5/5, workspace routing discovery failed) and every MCP client (rmcp::transport::worker: worker quit with fatal), so the bili tools never appear either.

The culprit is a single certificate that bili copies from the System keychain: CN=com.apple.kerberos.kdc, O=System Identity. It is a leaf identity cert (no basicConstraints), not a trust anchor, and it carries two Extended Key Usage extensions (TLS Web Server Authentication and Signing KDC Response). RFC 5280 §4.2 forbids repeating an extension. OpenSSL accepts the cert, but Codex's Rust HTTP client fails to build with it in the bundle. The duplicate EKU is the most likely reason; I have not traced the exact rejection inside Codex.

bili claude is unaffected (base-URL rewrite, no CA).

Environment

  • billion-context 0.1.212 (daa082e), installed with npm install -g --prefix ~/.local
  • codex-cli 0.162.1
  • macOS 15.6 (24G84), x86_64
  • Node v22.17.0

Reproduce

bili codex -- exec --skip-git-repo-check "hi"

You can reproduce it without bili running. Point Codex at the bundle and send traffic to a dead port; the builder error appears before any connection:

SSL_CERT_FILE=~/.local/share/billion-context/ca/combined-ca.pem \
HTTPS_PROXY=http://127.0.0.1:9 \
codex exec --skip-git-repo-check "hi"

Bisect results

The generated bundle had 190 certs. I tested subsets, each with bili's root CA appended, using the same SSL_CERT_FILE check:

Bundle Codex builder error
bili's combined-ca.pem as written (190) yes
root-ca.pem only no
combined, CRLF stripped yes (CRLF in root-ca.pem is not the cause)
combined minus the two negative-serial certs yes (not the cause)
com.apple.kerberos.kdc + root only yes
combined minus com.apple.kerberos.kdc (189) no: Codex connects and routes through bili

Of the 190 certs, 9 came from the keychains (2 from System, 7 from login). Only 3 of those 9 are CAs. The rest are leaf certs: Apple system identities, application content-signing certs, and personal email/S-MIME certs. None of them belong in a trust bundle.

Cause in code

On darwin, collectOsStorePems() exports every certificate in both keychains:

execFileSync("/usr/bin/security", ["find-certificate", "-a", "-p", k2], …)
// k2 ∈ ["/Library/Keychains/System.keychain", "~/Library/Keychains/login.keychain-db"]

writeCombinedBundle() then merges these unfiltered into combined-ca.pem. find-certificate -a returns identities and leaf certs as well as trust anchors.

Suggested fix

Keep only CA certificates from the OS-store export. Node's built-in X509Certificate is enough:

import { X509Certificate } from "node:crypto";

const caOnly = pems.filter((pem) => {
  try { return new X509Certificate(pem).ca; } catch { return false; }
});

I tested this against the same 190-cert bundle. It drops exactly the 6 non-CA keychain certs, including com.apple.kerberos.kdc, and keeps the 184 CAs, including user-installed CAs from the login keychain. Codex then builds its HTTP client with no error.

Optionally, apply the same filter to the whole merged bundle, so a malformed non-CA entry from any source cannot break replacement-semantics clients again.

Workaround

Remove the com.apple.kerberos.kdc cert from combined-ca.pem, then make the file read-only (chmod 444). writeCombinedBundle() hits EACCES on its 24 h rebuild, logs [ca] cannot rebuild … continuing with the EXISTING bundle, and keeps the filtered file. The downside is that keychain CAs added later are not picked up.

Activity

  1. ranxianglei commented on Nov 6, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复。 combined-ca.pem 家族里已有条目都是不同机制:#5725/#5736 是 zcode GUI 把 CA 路径指错了文件(root-ca.pem vs combined-ca.pem),#2439 是 Windows 重建时闪 PowerShell 黑窗,#1807/#1808 是引入 OS 信任存储合并的功能本身(即本缺陷的源头)。没有针对「keychain 导出混入非 CA 叶证书」的既有 issue/PR。

    标签: bug · codex · scope/wiring · p1(macOS 上 bili codex 完全不可用——Codex 拒绝整个信任包,模型流量与 MCP 全断)。

    审核结论(动手前已核实三步):

    1. 复现/查证:你报的 v0.1.212(daa082e)与当前 master 的 src/ca.ts 逐字节一致——问题真实存在,不是版本滞后。darwin 分支确实用 security find-certificate -a -p 全量导出 System + login 两个 keychain(身份证书、应用签名、S-MIME 都在内),writeCombinedBundle() 零过滤直接并入;你的 bisect(去掉 kdc 一张证书即恢复)与代码路径完全吻合。
    2. 层次判断:表面现象是 Codex builder error;根因是 bili 把「OS 存储里的全部条目」当成「信任锚点集合」写进一个替换语义的信任包——而替换语义客户端(codex/rustls)会对包里每一条做严格解析,一条 RFC 5280 违规(双 EKU)就废掉整个 HTTP client。修在决策点(合并时只收 CA),不是在下游删某张具体证书。
    3. 方案评估:你的建议(X509Certificate.ca 过滤)打到了核心,我在此基础上略扩两处:① 过滤放在最终合并落盘前的单一收口处,覆盖所有来源(keychain、文件系统 bundle、Mozilla 快照、用户 SSL_CERT_FILE),不只 darwin 导出;② CA 判定除 basicConstraints CA:TRUE 外兼容 keyUsage 含 Certificate Sign 的老式锚点,避免误伤少数不带 basicConstraints 的用户自装 CA。另加一个收敛标记行:升级后首次启动强制重建一次 bundle——否则你机器上现有的坏包要等 24h 才过期重建,修复体感上像没生效。

    开始实现,PR 稍后贴这里。

  2. ranxianglei commented on Nov 6, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    修复完成 → PR #5809: #5809 (Closes #5802,分支 2026-11-06_ca-bundle-ca-only-filter)

    并行 PR 核对:镜像 open [PR] 行里没有指向本线程的既有修复 PR(5810 行的 #5805 是 GitHub 侧另一个存储性能帖,与本主题无关),本 PR 是唯一在途修复。

    结构化八问:

    1. 是什么问题:macOS 上 bili codex 完全发不出 HTTPS 请求——Codex 拒绝加载 bili 生成的 combined-ca.pem(Failed to build HTTP client ... builder error),模型连接(Reconnecting... 5/5)和全部 MCP client 一起死掉,bili 工具永远不出现。
    2. 什么原因导致的:darwin 导出 OS 信任库用 security find-certificate -a -p 把 System + login 两个钥匙串的全部条目(含叶子身份证书、S-MIME、应用签名证书)无过滤合并进 bundle。其中 com.apple.kerberos.kdc 是叶子证书(无 basicConstraints)且带两个 EKU 扩展(违反 RFC 5280 §4.2)。Codex 的 rustls 栈对 SSL_CERT_FILE 是替换语义、逐条解析全部条目,任何一条解析失败就拒绝构建客户端——一张坏证书废掉整份 bundle。OpenSSL 系客户端(Node/curl)容忍这种证书,所以只有 codex 被废。
    3. 影响范围与严重程度:bundle 为 codex/dsh/hermes 三条注入 CA 的车道共用;实际失效只发生在严格逐条解析 TLS 栈的客户端(codex rustls)上,Node 系客户端不受影响;触发条件 = macOS + 钥匙串里存在非 CA 证书(报告者 190 张里有 9 张来自钥匙串、6 张非 CA)。触发即完全不可用 → p1。Linux/Windows 不受影响(其导出本就是 CA 存储);bili claude 与 NODE_EXTRA_CA_CERTS 车道不受影响。
    4. 怎么修复的:在唯一合并点 writeCombinedBundle() 加一道 CA 判定门——只保留可解析为 X509 且 basicConstraints CA:TRUE 或 keyUsage 含 keyCertSign 位(覆盖 BC 引入前的遗留锚点)的条目,对所有来源统一生效(system bundle、OS store、Mozilla roots、MITM root)。配套两件事:① bundle 头部加一行 marker 注释,升级后遇到没有 marker 的新鲜文件(= 旧版坏 bundle)立即重建一次,不用等满 24h 新鲜期,否则修复看起来像没生效;② 兜底:若过滤把一切都丢了(解析能力异常),回退未过滤 bundle + error 日志,绝不发布空信任库(MITM 根恒过判定,实际不可达)。keyUsage 判定走原始 DER 扫描 OID 2.5.29.15,因为 Node v22 的 X509Certificate.keyUsage getter 对所有证书都返回 undefined,node-forge 又解析不了 ECDSA 证书(用它会静默丢掉 EC 根)。
    5. 是否有回归问题:逐项过过既有行为——① bundle 主用途(MITM 根 + OS 信任超集 + 公共根,保 pip/git/curl 直连)不变:报告者实测同型过滤保留 184 张 CA(含 login 钥匙串用户自装 CA),Codex 正常建连;② EACCES 回退(promptCache.routing is ignored by relay-affinity stamping; no way to persist per-endpoint prompt_cache_key abatement (0.1.209) #5103)、指纹去重、24h 新鲜期机制原样;③ combined-ca.pem 的其他消费者(launcher/hermes 插件/启动失败页)全是路径引用、无人解析文件内容,头部加注释行安全(PEM 解析器普遍跳过非块文本);④ 已声明残留:若有人依赖自签非 CA 证书靠 OpenSSL 宽松锚点行为工作,会失去该锚点——极罕见且此类证书本身属误发。本地定向测试:tests/ca-combined.test.ts 19/19(含 4 个新回归钉)、其余引用 src/ca.ts 的三个套件 32/32、typecheck 干净;全量回归以 CI 为准。
    6. 是否破坏缓存:不涉及(无缓存/重试/增量游标/去重键;bundle 文件本身按 mtime+marker 判定重建,稳态 = 不重写,缓存稳定)。
    7. 是否改了设计方向:不涉及——同一管道、同一文件、同一刷新周期,只在合并点加一道过滤。
    8. 是否引入新配置:无。无新 config/env/默认值。

    行为变更披露(旧→新):

    • bundle 内容:旧 = 各来源逐字并集(darwin 含钥匙串全部条目,含叶子身份证书);新 = 仅 CA 证书(对用户设置的 SSL_CERT_FILE 覆盖源同样生效),不可解析条目丢弃。原因:严格替换语义客户端逐条解析,一张 RFC 违规叶子证书废整份文件。
    • 文件头:新增一行 marker 注释(用于上面的一次性升级重建判定)。
    • 升级后刷新:旧 bundle(无 marker)立即重建一次,而非等 ≤24h。

    本 PR 触碰 CA/信任面,属 §7.4 人工评审类,请评审后自行合并:#5809

  3. added 3 commits that reference this issue on Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions