Skip to content

[PR] fix(launcher): make the spawn health budget configurable (#4947) - #4948

Merged
ranxianglei merged 7 commits into
ranxianglei:masterfrom
zxmqq1234:2026-10-27_launcher-spawn-budget-env
Oct 30, 2026
Merged

ranxianglei merged 7 commits into
ranxianglei:masterfrom
zxmqq1234:2026-10-27_launcher-spawn-budget-env

Conversation

@zxmqq1234

Copy link
Copy Markdown
Contributor

Fixes #4947.

What

Resolve SPAWN_WAIT_MS and SLOW_START_GRACE_MS through the existing envMillis contract (#1365) as BILI_SPAWN_WAIT_MS / BILI_SLOW_START_GRACE_MS. Defaults unchanged (20s + 40s = 60s total).

Why

Reproduction and timings in #4947: a 1.7 GiB shared index.db plus two lanes cold-starting together pushed pre-bind startup past the fixed 60s budget; the launcher SIGKILLed the mid-storage-open child three times in a row (attempts 2-3 produced zero output) and the lane stayed down ~9 minutes until a retry landed in a lock gap. #2187 already widened the total once (26-45s Windows starts); this makes the budget tunable without patching dist for the cases that exceed it 鈥?large stores (#4939's whole-DB quick_check cost) and cross-lane write-lock contention.

Changes

  • src/launcher.ts: both knobs via envMillis 鈥?invalid values fall back to the built-in defaults, same contract as every other ms knob
  • CONFIGURATION.md: both knobs documented in the env table and in the detailed health-wait section

Tests

…#4947)

SPAWN_WAIT_MS (20s) and SLOW_START_GRACE_MS (40s) were compile-time
constants. A 1.7 GiB index.db plus two lanes sharing one store
cold-starting together pushed pre-bind startup past the fixed 60s
budget, and the launcher kill-spawn looped into the same contention
(three SIGKILLs, zero-output children, lane down for ~9 minutes until
a retry landed in a lock gap).

Resolve both knobs through the existing envMillis contract (ranxianglei#1365) so
operators can widen the budget without patching dist; defaults are
unchanged and invalid values fall back. Documented in CONFIGURATION.md
next to the other health-wait knobs.

Fixes ranxianglei#4947.
@ranxianglei ranxianglei changed the title fix(launcher): make the spawn health budget configurable (#4947) [PR] fix(launcher): make the spawn health budget configurable (#4947) Oct 27, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 与 #4939 / #4609 / #4515 属同一症状家族(大共享库冷启动超过 60s spawn 预算被 SIGKILL),但机制与诉求不同:那几个针对「降低 open 路径自身成本」(#4619 把 quick_check 推迟到 listen 之后、#4539 去掉 pre-listen 的冗余语料解码),本 PR 是让 launcher 的健康等待预算可用 env 调节——属于「成本降下来后仍可能超预算」场景(跨车道写锁竞争、超大库)的逃生阀。两者互补,本 PR 与在途根因修复不重叠。

标签: bug、dsh、scope/wiring、p2(缺陷;dsh 车道复现;launcher 接线层;特定条件下可用但需绕行)。

初审判断:方向正确,代码小而干净(仅把两个常量改为经 envMillis 解析,默认值不变,派生量 STARTING_MARKER_TTL_MS、pi-native 的 spawnWaitMs 均自动跟随)。但 CI 有 3 个红门禁:docs-sync / docs-coverage / lifecycle-ledger-guard——都是本 PR 引入新 env var 但文档面没配齐:website/config-reference/env-only.yaml 缺种子条目(索引表是生成的)、LIFECYCLE.md 未登记、zh-CN 未同步。我会在 PR 分支直接补 commit 修齐并本地复验,随后给出结构化最终结论。

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/wiring labels Oct 27, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 三个红项全部由本 PR 新引入的两个 env 变量(BILI_SPAWN_WAIT_MS / BILI_SLOW_START_GRACE_MS)自身触发,master 上不存在这两个名字,非继承性失败;已对照 #4599 最新评论,无同签名条目,无需另立。

已直接在 PR 分支修复并推送:ce73191 → 0c6405256(单 commit,仅文档/台账登记,零代码改动)。

八问

  1. 是什么问题:PR 把 launcher 的 spawn 健康等待预算从硬编码改成两个可调 env 后,三条文档/台账门禁全红——docs-coverage 报这两个变量「读了但没文档」、docs-sync 报手改的生成表与种子再生成结果漂移、lifecycle-ledger-guard 报这两个变量不在 LIFECYCLE.md 台账里。
  2. 什么原因导致的:PR 在 src/launcher.ts 加了 env 读取、并手改了 CONFIGURATION.md 的表格行,但没动真正的源头——CONFIGURATION.md 的环境变量表是由 website/config-reference/*.yaml 种子生成的(文件头明示「edit the seed」),且 LIFECYCLE.md 台账也没登记。三条门禁检测的正是这个遗漏,属同一机制缺口的三个面。
  3. 影响范围与严重程度:仅 CI 层面,无运行时影响;功能本身可用(默认值未变),但 PR 在红状态下无法合并。严重度:合并阻塞级体验问题。
  4. 怎么修复的:在规范源头登记而非打补丁——① 两个条目加入 website/config-reference/env-only.yaml(英文描述与作者手写表格逐字一致,中文为新译);② 跑 gen-config-docs generate 重新生成(英文表再生成结果与作者手写行字节一致,中文表补齐两行);③ LIFECYCLE.md §2.6 永久区补一行(与 attach 宽限/截止窗同族)。
  5. 是否有回归问题:无。零代码改动;审计过既有行为——英文文档内容与作者所写完全一致(非改写),中文文档只增两行(此前缺失),台账只增一行,种子数 307→309;生成器其余区块未动(工作树仅 3 个预期文件变更)。无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:否——完全沿用既有的「种子→生成」文档纪律与生命周期台账纪律。
  8. 是否引入了新的配置:本 commit 不引入新配置,仅为 PR 已有的两个旋钮补登记:BILI_SPAWN_WAIT_MS(默认 20000ms)、BILI_SLOW_START_GRACE_MS(默认 40000ms),总 spawn 预算 = 两者之和 = 60000ms——与改前硬编码值相同,无行为漂移。

本地已按 CI 同款命令验证三门禁全绿(coverage ok:190 env names 全覆盖;generate+diff 在提交树上干净;ledger OK:170 条在册)。改动文件:website/config-reference/env-only.yaml(+16)、LIFECYCLE.md(+1)、CONFIGURATION.zh-CN.md(+2)。收工判定:等远程 CI 重跑即绿,无需其他动作。

ci-check added 3 commits October 27, 2026 18:22
…red state dir

The starting marker is one global file per state dir; concurrent launcher
activity (other suites, other processes) made this file wait behind it and
burn its budget ticks past the assertions (~40% local failure rate). Private
XDG roots follow the existing test convention (auto-restart.test.ts); the
private port additionally isolates the shared per-port log file (ranxianglei#4811).
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核完成,结论:代码可合(等 owner 合并,agent 不代合)。八问逐项:

1. 是什么问题
launcher 拉起 proxy 子进程后有一个固定 60s 的健康等待预算。#4947 实测:Windows 上 1.7 GiB 共享 index.db + 两条车道同时冷启动时,pre-bind 启动超过 60s,launcher 把健康子进程连杀三次,车道瘫了约 9 分钟。本 PR 把这个预算拆成两个可配置 env(默认值不变)。

2. 什么原因导致的
固定预算 vs 大库冷启动成本:整库 quick_check 的打开成本(#4939)叠加跨车道 SQLite 写锁竞争,能把 pre-bind 推过 60s;#2187 已为 26-45s 的 Windows 启动加宽过一次,但对超出场景没有逃生门,此前只能手改 dist。

3. 影响范围与严重程度
大共享库 / 多车道并发冷启动、且 pre-bind > 60s 的用户;条件性 p2——能用但会反复掉线,重试终会在锁间隙恢复,无数据丢失。

4. 怎么修复的
两个 knob 走既有 envMillis 契约(#1365):非法值回退内置默认,与其他所有 ms knob 一致;派生常量(starting-marker TTL、pi-native spawnWaitMs)自动跟随两者之和,日志行格式保留(数值模板化)。文档面:我首条回复指出缺登记后作者自行在 0c64052 补齐(yaml seed + LIFECYCLE §2.6 + CONFIGURATION en/zh),我本地复验四道门禁(docs-sync / docs-coverage / lifecycle-ledger / config-surface)全绿,未重复动手。

5. 是否有回归问题
无。逐项审过既有行为:默认值不变(20000 / 40000 / 和 60000,与现行为逐字节一致);非法值回退契约与其他 ms knob 相同;派生消费者自适应;既有 #2187 套件不改通过。另在 PR 分支推了两个独立标注的小修 commit:

  • fa2077091:zh 文档措辞修正——“LIVE 模式”是误译(此处 LIVE 指子进程仍存活,不是某种模式),改为“仍存活但慢启动”;
  • 846fc3dcd:新测试改用私有 XDG 根隔离状态目录(沿用 tests/auto-restart.test.ts 既有惯例),避免新测试卷入下面提到的并发竞态。

6. 是否破坏缓存
不涉及——未触碰任何缓存 / 重试 / 增量游标面。

7. 是否改了设计方向
没有——沿用现有 envMillis 模式,无新配置形状。

8. 是否引入新的配置
BILI_SPAWN_WAIT_MS(默认 20000)与 BILI_SLOW_START_GRACE_MS(默认 40000),env-only,已登记 CONFIGURATION.md en/zh 与 LIFECYCLE §2.6;迁移方式:不需要(默认行为逐字节不变)。

审核中发现的新问题(已单独立项):多个 launcher-spawn 测试文件并发运行时共享同一个全局 starting-marker 文件,假时钟测试进程可能“等”住外部真实进程一个完整预算,使 #2187 套件的 tick 断言间歇失败(本地三文件并发复现率约 40%,单文件稳定)。该机制 master 上即存在,与本 PR 代码改动无关(PR 只是多了一个并发参与者)。已按机制归并原则单独开 issue:#4953(bug / scope/ci / p3),本 PR 自己的新测试已在 846fc3d 做隔离。

CI:0c6405256 时 11 个 test lane 全绿;我的两个 commit 后 846fc3d 重跑中——文档/守卫类门禁(docs-sync、docs-coverage、config-surface 等)目前全绿,test lane 运行中,出结果后补一行。

判定:方向正确、代码干净、测试有牙(never-healthy 场景同时断言错误文案与预算实际被消费)、未发现回归——可合,请 owner 合并:#4948

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Merge-guard re-sync (second pass): master had advanced again after your two new commits (fa2077091 docs wording + 846fc3dcd test isolation), so I merged current master into the branch → clean auto-merge, zero conflicts (only overlapping files were CONFIGURATION.md / .zh-CN.md, different hunks); your launcher/test changes are byte-identical and untouched, targeted suites green 8/8 on the merged head. Pushed as a65a8ec25; verified the PR is no longer behind (master fully contained in head, mergeable=true, no conflicts — blocked state is just waiting on required CI checks).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 状态更新:作者又同步了一次 master(新头 a65a8ec25,带入 #4910/#4911/#4912/#4923/#4924 的已合并工作),我之前的两个小修 commit(fa2077091 zh 措辞、846fc3dcd 测试隔离)都在新树里;刚才那批 cancelled 的 check run 是旧头被取代所致,不是失败。我在新头上本地复验过:docs-sync / lifecycle-ledger / config-surface / typecheck 全绿,目标测试矩阵(三个 launcher 文件并发跑)3/3 通过。a65a8ec25 的 CI 正在跑,判定不变:可合,等 CI 绿后请 owner 合并。

@abworks-dev

Copy link
Copy Markdown
Contributor

Windows DSH field feedback: large-index startup budget vs. integrity-check cost

Adding a real-environment data point after checking the current release and existing work, rather than opening a duplicate PR.

Environment and observations

  • Windows DSH desktop, installed billion-context 0.1.208 (not yet upgraded/tested on 0.1.209).
  • A separate local read-only diagnostic reported an index.db size of 2,996,285,440 bytes and PRAGMA quick_check verdict ok after 64,663 ms. This is one diagnostic run, not a repeatability benchmark or proof of the current live database's integrity. We did not rerun the scan for this feedback.
  • The installed launcher was locally changed from 20,000 + 40,000 ms to 20,000 + 160,000 ms (180 seconds total), preserving the other local patches. This is a bundled-constant workaround, not a test of this PR's environment-variable implementation. Subsequent normal desktop restart attempts still encountered startup failures, so increasing this one budget alone was not sufficient in those attempts. This does not identify the outer desktop/host deadline or establish a single cause for all failures.
  • A separate local recovery flow subsequently restored native functionality and documented bypassing the per-start quick_check with BILI_STORAGE_QUICK_CHECK=0, later moving that setting into a persistent configuration script and restoring the native bundle to its pre-skip baseline. The script is inspectable, but a safety guard denied access to the sensitive configuration file itself; we did not bypass it and therefore have not independently confirmed its current value. The current native bundle has only the 180-second constant change relative to its saved baseline.
  • A fresh read-only runtime snapshot shows one matching proxy/parent/lock owner, full storage without degradation, secondProcess=null, splitBrain=null, and zero reported active conflicts. However, storage.integrity=null. Functionality being available and conflict counters being clear must not be presented as a fresh startup integrity-check pass. An earlier split-brain record involved an exited predecessor; it is not evidence of current simultaneous writes.

Release / existing-work cross-check

Contribution / validation scope

I am not proposing a new PR that defaults integrity checking off, duplicates #4852/#4948, or mixes unrelated DSH plugin compatibility fixes into this launcher change. The useful follow-up is a protected Windows 0.1.209 validation with checks enabled: record bind/health timing, background check completion/cache reuse, and the outer desktop host deadline. If that still exposes an independent defect, it should get a minimal reproducer against that version before a separate code PR. This comment reports the older-version field evidence and its limitations; it does not claim an end-to-end test of this PR or of the latest release.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Thanks for the field data points — a useful real-environment anchor for this PR, and your cross-references check out against current source. Confirmations plus one connection worth making explicit:

Verified against current master

The connection your numbers actually highlight
The ~64.7 s quick_check you measured on a ~3 GB index.db is exactly the cost #4852/#4609 moves OUT of the startup critical path ("no longer blocks open"). On 0.1.208 that scan ran synchronously during open, so on a large store it alone could consume most of the startup window — which fits your observation that bumping the launcher budget to 180 s did not by itself stop the failures: if the child spends ~65 s in a synchronous integrity scan before it ever reports healthy, extra spawn-health grace time can't rescue it, and the outer desktop/host deadline can still cut it off first.

So the two PRs are complementary in a specific way:

Agree with your scope call: no duplicate PR, no default-off, no unrelated DSH plugin compat mixed in. The right next step is the protected Windows 0.1.209 validation you outlined (checks enabled; record bind/health timing, background-check completion + cache reuse, and the outer host deadline). If that surfaces an independent defect, give it its own minimal reproducer + issue/PR rather than folding it into #4948 (one-issue-one-scope here). Nothing in this comment changes #4948's plan.

(For the thread: the #4948 branch is now synced to current master after the second merge-guard pass — clean auto-merge, no conflicts, targeted suites green; CI is the remaining gate.)

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.923-master.27b3155 (rolling master tag).

npm install -g billion-context@0.0.923-master.27b3155  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 非新问题,不开新 issue — 本轮三个 test 分片红与 #4599 在档的 master flake 家族(#5140,修复收敛于 #5144)逐字同签名;按 CI 协调政策 CI 问题统一进 #4599、已有在档条目不另立项。

本轮判定:继承性失败(master 自身红),PR 侧无可修点。当前 head 3855caf639 = 作者在我上轮文档修复后又推了措辞澄清 + 测试隔离两个 commit 并两次同步 master——这些都没问题,三个红分片与它们无关。

八问

  1. 是什么问题:test (windows-latest, 22, 4) / test (ubuntu-latest, 24, 3) / test (ubuntu-latest, 22, 3) 三片各挂同一个用例——HTTP a cold-restored non-retained parent serves its intact raw tail immediately (#5140),与本 PR 的 launcher spawn-budget 改动无关。
  2. 什么原因导致的:master 近期落地的 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 新用例是单 cell flake(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 floor 485 起在档,多个 PR 同签名继承),master 自身在这三个分片即红;重型 test 车道只在 PR head 全量运行,所以表现为「每个 PR 都红」。因果链:flake 落 master → required checks 覆盖全分片 → PR head 全量跑测继承红。
  3. 影响范围与严重程度:所有跑全量 test 车道的 PR head(舰队级合并信号噪音),非用户功能问题;本 PR 功能零影响。
  4. 怎么修复的:PR 侧不改代码(政策明令禁止在自己分支重复修 master 级失败,修复必须收敛到 ci-fixes/[PR] fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) #5144)。已做四步核实+处置:① 日志逐字比对——三处均为 tests/plugin-fork.test.ts:1473:16 AssertionError: 409 !== 200,与在档签名字节一致;② origin/ci-fixes tip(cf99d935f)已整体包含在本分支内(merge 无增量可合);③ PR diff(launcher env 旋钮 + 文档/台账)与 plugin-fork/snapshot 路径零交集;④ 对 run 40922066646 触发 rerun-failed-jobs(HTTP 201,attempt 2 排队中)。
  5. 是否有回归问题:本轮零 push,分支 head 未动(仍是作者的 3855caf);上轮文档修复(0c6405256)后三条文档门禁持续绿,未被触碰。无阈值/时序/输出格式变化。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:否。
  8. 是否引入了新的配置:否——本轮无任何改动;PR 的两个旋钮(默认值 20000/40000ms,总预算 60s 不变)维持上轮已登记状态。

收工判定:本 PR 侧工作已完成(代码 + 文档 + 台账全部就绪),当前唯一阻塞是继承的 #5140 flake——rerun 已触发等结果;若同签名再现,归哨兵 ci-fixes 收敛或 admin 重跑处置(已在 #4599 留一行结论),PR 侧不再重复排查。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][launcher] 60s spawn budget not configurable - SIGKILLed children on cold start with a 1.7 GiB shared index.db

3 participants