diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 0bd1b8a3f1..300fe7b925 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -378,6 +378,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `BILI_CLAUDE_UPSTREAM` | string | unset | — | Upstream pin for the claude lane. | | `BILI_CODEX_STATE_MODE` | string | auto | — | codex state handling: auto (shared db via CODEX_SQLITE_HOME on modern codex, legacy overlay copies otherwise), shared, or legacy. Shared mode runs up to 16 concurrent sessions, one per-session overlay slot each; legacy keeps one exclusive overlay at a time. | | `BILI_ATTACH_HEALTH_DEADLINE_MS` | number | (built-in) | — | Deadline for attach health checks when joining an external daemon. | +| `BILI_SPAWN_WAIT_MS` | number | (built-in) | — | Initial health-wait window (ms) for a freshly spawned proxy child before the launcher extends the wait. | +| `BILI_SLOW_START_GRACE_MS` | number | (built-in) | — | Extra grace (ms) for a still-alive child that is slow to bind (slow store, AV scan, shared-store lock contention); the total spawn budget is the sum of the two. | | `BILI_ATTACH_EVIDENCE_GRACE_MS` | number | (built-in) | — | Grace period for attach ownership evidence. | | `BILI_PROVIDER_REWRITES` | string | unset | — | Provider URL rewrite rules applied before routing. | | `BILI_MCP_PROXY` | string | unset | — | Spawn channel: MCP proxy target for plugin-hosted tools. | @@ -2000,6 +2002,8 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_INHERITED_HTTP_PROXY` / `BILI_INHERITED_HTTPS_PROXY` / `BILI_INHERITED_ALL_PROXY` / `BILI_INHERITED_NO_PROXY` | Not user-facing — set automatically by the launcher when it spawns the proxy (#1012). The launcher strips the shell's proxy vars from both the client and the proxy child (clients must send to bili; the proxy must not have its model egress hijacked by a shell proxy), but it forwards the user's pre-strip proxy under these names so the proxy's **auxiliary egress** (MITM blind tunnels — client-side MCP/web traffic) can still ride the user's VPN. They feed only the blind-tunnel fallback tier: explicit routes / global `proxy` / `BILI_UPSTREAM_PROXY` / explicit `"upstreamProxyMode": "direct"` all still win, and a value pointing at bili's own port is dropped. Model egress is unaffected (stays direct unless explicitly configured). | | `BILI_ATTACH_HEALTH_DEADLINE_MS` | Health-wait deadline (ms) for dsh/opencode attach verification when the attach target is down but this process's model channel is **pinned** to it (routed `/bili/…` model traffic was observed against it): bili waits for the target to come back instead of spawning a second instance — a spawn would split the session (model traffic stays pinned, bili tools would 404 against the other instance). When the deadline elapses it fails loudly and keeps re-checking on every model request until the target returns (default `15000`). See #1365. | | `BILI_ATTACH_EVIDENCE_GRACE_MS` | Grace window (ms) during which a dsh/opencode attach verification probing a dead target waits for routed-channel evidence to appear before falling back to the legacy spawn path (covers the decision-before-first-request race: the probe fails at t≈0 while the first model request lands at t≈1s) (default `5000`). See #1365. | +| `BILI_SPAWN_WAIT_MS` | Initial health-wait window (ms) for a freshly spawned proxy child before the launcher logs `still starting up` and extends the wait (default `20000`). Widen it when the proxy's pre-bind open path runs long: a large `index.db` opens whole-DB before bind (integrity check, WAL recovery — see #4939), and two lanes sharing one store cold-starting together contend for the SQLite write lock (observed 28s+ to first log line, 60s+ with zero output, on Windows with a 1.7 GiB store). Non-numeric or non-positive values fall back to the default (same `envMillis` contract as #1365). | +| `BILI_SLOW_START_GRACE_MS` | Slow-start grace (ms) beyond `BILI_SPAWN_WAIT_MS` for a spawned child that is still ALIVE and starting (#2187: Windows spawn windows of 26-45s from AV / first-run disk activity; a large or contended shared store can push total pre-bind time past 60s). The total spawn budget is the sum of the two knobs; a child that EXITS still fails fast (default `40000`). | | `BILI_PERSIST` | Set `0` to disable session persistence (in-memory only, lost on restart). | | `BILI_PERSIST_DEBOUNCE_MS` | Debounce window for persistence writes to disk, in ms (default `500`). | | `BILI_PERSIST_TAIL_TOKENS` | Token budget for the persisted conversation snapshot (#401). The disk record stores the **folded view** (block summaries in place of compressed ranges) truncated to the newest messages within this budget — never the raw full history. Default `16384`; `0` disables message persistence entirely (block summaries and compressed originals still persist; `bili export` falls back to block-only rendering). Live in-memory sessions are unaffected — `bili export` of a live session is always complete. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index fe342bfe2c..facb3afca2 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -378,6 +378,8 @@ | `BILI_CLAUDE_UPSTREAM` | string | unset | — | claude 车道的上游钉死。 | | `BILI_CODEX_STATE_MODE` | string | auto | — | codex 状态处理方式:auto(新版 codex 走 CODEX_SQLITE_HOME 共享库,旧版回退 overlay 副本)、shared、legacy。shared 模式最多 16 个并发会话,每会话一个独立 overlay 槽位;legacy 保持一次仅一个独占 overlay。 | | `BILI_ATTACH_HEALTH_DEADLINE_MS` | number | (built-in) | — | 挂接外部守护进程时健康检查的截止时间。 | +| `BILI_SPAWN_WAIT_MS` | number | (built-in) | — | 新派生的代理子进程在启动器延长等待之前的初始健康等待窗口(毫秒)。 | +| `BILI_SLOW_START_GRACE_MS` | number | (built-in) | — | 对仍存活但仍在启动(慢速存储、杀毒扫描、共享存储锁竞争)的子进程的额外宽限(毫秒);总启动预算为两者之和。 | | `BILI_ATTACH_EVIDENCE_GRACE_MS` | number | (built-in) | — | 挂接归属证据的宽限期。 | | `BILI_PROVIDER_REWRITES` | string | unset | — | 路由前应用的 provider URL 重写规则。 | | `BILI_MCP_PROXY` | string | unset | — | 派生通道:插件宿主工具的 MCP 代理目标。 | diff --git a/LIFECYCLE.md b/LIFECYCLE.md index fdb7f323a4..9564227eb1 100644 --- a/LIFECYCLE.md +++ b/LIFECYCLE.md @@ -161,6 +161,7 @@ | `BILI_ZONE_PORT` | USER zone 基址覆盖(#1660) | | `BILI_CLAUDE_NATIVE_PORT` / `BILI_ZCODE_PORT` | 显式 zone 端口覆盖(strict-port 语义,#1660) | | `BILI_ATTACH_EVIDENCE_GRACE_MS` / `BILI_ATTACH_HEALTH_DEADLINE_MS` | native attach 证据宽限窗 / 健康截止窗 | +| `BILI_SPAWN_WAIT_MS` / `BILI_SLOW_START_GRACE_MS` | launcher spawn 初始健康等待窗 / LIVE 子进程慢启动宽限;总 spawn 预算 = 两者之和(#4947) | | `BILI_DSH_BIN` / `BILI_PI_BIN` | dsh / pi 可执行文件覆盖 | | `BILI_PLUGIN_AGENT` | `--plugin` 目标 agent 选择(默认 claude) | | `BILI_LOG_MASK_HOSTS` | 日志主机名掩码(默认开;`=0` 本地调试退出) | diff --git a/src/launcher.ts b/src/launcher.ts index 15f471813b..4e28117e4a 100644 --- a/src/launcher.ts +++ b/src/launcher.ts @@ -58,7 +58,7 @@ import { import { TWIN_ADOPT_EXIT_CODE } from "./listen-policy.js"; import { tcpConnectable } from "./liveness.js"; import { selfPackageRoot, isBiliPiEntry, ompPluginLoadedFrom, dshNativeInstalled, claudeNativeInstalled, piPackageStrings } from "./plugin-install.js"; -import { applyOmpFirstEventTimeout } from "./agent/native-bootstrap.js"; +import { applyOmpFirstEventTimeout, envMillis } from "./agent/native-bootstrap.js"; import { logHostEvent } from "./agent/host-log.js"; import { compactionDisablePatchRows } from "./agent/dsh-compaction-disable.js"; import { log as teeLog, logTimestamp } from "./logger.js"; @@ -171,12 +171,17 @@ type BaseClientName = "claude" | "codex" | "pi" | "omp" | "opencode" | "hermes" const HEALTH_PATH = "/__bili/health"; const HEALTH_POLL_INTERVAL_MS = 200; -const SPAWN_WAIT_MS = 20000; +// Env knobs (envMillis contract, #1365): large stores (whole-DB pre-bind open, +// #4939) and two lanes sharing one index.db (write-lock contention on cold +// start) have been observed exhausting the fixed budget, kill-spawn looping +// into the same contention. Widen via env instead of patching dist; bad +// values fall back to the defaults. +const SPAWN_WAIT_MS = envMillis(process.env, "BILI_SPAWN_WAIT_MS", 20000); // #2187: grace beyond the initial wait for a LIVE child still starting. // Spawn-window startups have been observed taking 26-45s on slow Windows // machines (AV / first-run disk activity); the one-shot 20s budget misread // that as failure and left the lane degraded for the whole session. -const SLOW_START_GRACE_MS = 40_000; +const SLOW_START_GRACE_MS = envMillis(process.env, "BILI_SLOW_START_GRACE_MS", 40_000); // Total wait for a live child: initial window + grace. A child that EXITS or // fails to spawn still fails fast (the poll loop breaks on exit/error). export const SPAWN_BUDGET_MS = SPAWN_WAIT_MS + SLOW_START_GRACE_MS; diff --git a/tests/issue4947-launcher-spawn-budget-env-defaults.test.ts b/tests/issue4947-launcher-spawn-budget-env-defaults.test.ts new file mode 100644 index 0000000000..b97c4a9986 --- /dev/null +++ b/tests/issue4947-launcher-spawn-budget-env-defaults.test.ts @@ -0,0 +1,16 @@ +// Invalid spawn-budget env values must fall back to the built-in 60s budget +// (envMillis contract, #1365): a bad knob must never shrink or zero the +// budget. Separate file so this node --test process resolves the module +// constants with only the bad values set. + +process.env.BILI_SPAWN_WAIT_MS = "not-a-number"; +process.env.BILI_SLOW_START_GRACE_MS = "0"; + +import test from "node:test"; +import assert from "node:assert/strict"; + +const { SPAWN_BUDGET_MS } = await import("../src/launcher.ts"); + +test("invalid BILI_SPAWN_WAIT_MS / BILI_SLOW_START_GRACE_MS fall back to the built-in budget", () => { + assert.equal(SPAWN_BUDGET_MS, 60000); +}); diff --git a/tests/issue4947-launcher-spawn-budget-env.test.ts b/tests/issue4947-launcher-spawn-budget-env.test.ts new file mode 100644 index 0000000000..ad7ed9f220 --- /dev/null +++ b/tests/issue4947-launcher-spawn-budget-env.test.ts @@ -0,0 +1,90 @@ +// Spawn-budget env knobs (#4939 and the cross-lane shared-store variant): a +// large index.db opens whole-DB before bind, and two lanes sharing one store +// contend for the write lock on cold start — pre-bind startup past the fixed +// 60s budget makes the launcher kill-spawn loop into the same contention. +// The knobs must resolve at module load, and the resolved budget must actually +// be consumed by the wait loop. This file runs in its own node --test process +// so the module-level constants pick up the env set below. + +process.env.BILI_SPAWN_WAIT_MS = "5000"; +process.env.BILI_SLOW_START_GRACE_MS = "3000"; + +import test, { after } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import type { SpawnChild, SpawnFn } from "../src/launcher.ts"; + +// Private XDG roots BEFORE importing the launcher: the starting marker is one +// GLOBAL file per state dir, so any concurrent launcher activity in the shared +// dir (other suites, other processes) makes this file wait behind it and burn +// its budget ticks past the assertions below. +const xdgRoot = fs.mkdtempSync(path.join(os.tmpdir(), "bili-issue4947-env-")); +process.env.XDG_CACHE_HOME = path.join(xdgRoot, "cache"); +process.env.XDG_CONFIG_HOME = path.join(xdgRoot, "config"); +process.env.XDG_DATA_HOME = path.join(xdgRoot, "data"); +process.env.XDG_STATE_HOME = path.join(xdgRoot, "state"); + +const { SPAWN_BUDGET_MS, ensureProxyRunning } = await import("../src/launcher.ts"); + +function makeFakeChild(pid: number): SpawnChild { + return { + pid, + unref() {}, + kill() { + return true; + }, + on() {}, + }; +} + +function fakeClock() { + let ticks = 0; + return { + now: () => ticks * 1000, + sleep: () => { + ticks += 1; + return Promise.resolve(); + }, + ticks: () => ticks, + }; +} + +test("BILI_SPAWN_WAIT_MS / BILI_SLOW_START_GRACE_MS resolve the spawn budget at module load", () => { + assert.equal(SPAWN_BUDGET_MS, 5000 + 3000); +}); + +test("the resolved env budget is consumed by the wait loop, not just exported", async () => { + const clock = fakeClock(); + const spawnImpl: SpawnFn = () => makeFakeChild(42470); + await assert.rejects( + ensureProxyRunning( + // Private port: the port log (.tmp/bili-proxy-.log) is shared + // per port across processes and rotates under writers (#4811). + { host: "127.0.0.1", port: 39787, passthrough: false, debug: false }, + { + fetchImpl: async () => ({ ok: false }), + fetchHealthInfo: async () => ({ ok: false }), + spawnImpl, + now: clock.now, + sleep: clock.sleep, + readInstanceFile: () => undefined, + }, + ), + new RegExp(`did not become healthy within ${SPAWN_BUDGET_MS}ms`), + ); + assert.ok(clock.ticks() >= 8, `the env budget (8s) must be fully consumed, got ${clock.ticks()}s`); + // Upper bound vs the #2187 baseline: with the built-in 60s budget this same + // never-healthy scenario consumes >=59 ticks; a much smaller total proves + // the env budget (not the default) drove the deadline. + assert.ok(clock.ticks() < 59, `the wait must stay below the built-in 60s budget baseline (>=59 ticks), got ${clock.ticks()}s`); +}); + +after(() => { + delete process.env.XDG_CACHE_HOME; + delete process.env.XDG_CONFIG_HOME; + delete process.env.XDG_DATA_HOME; + delete process.env.XDG_STATE_HOME; + fs.rmSync(xdgRoot, { recursive: true, force: true }); +}); diff --git a/website/config-reference/env-only.yaml b/website/config-reference/env-only.yaml index ea4cf403db..984a4f2ee0 100644 --- a/website/config-reference/env-only.yaml +++ b/website/config-reference/env-only.yaml @@ -186,6 +186,22 @@ entries: group: envonly desc_en: "Deadline for attach health checks when joining an external daemon." desc_zh: "挂接外部守护进程时健康检查的截止时间。" + - key: BILI_SPAWN_WAIT_MS + type: number + default: "(built-in)" + kind: env-only + audience: user + group: envonly + desc_en: "Initial health-wait window (ms) for a freshly spawned proxy child before the launcher extends the wait." + desc_zh: "新派生的代理子进程在启动器延长等待之前的初始健康等待窗口(毫秒)。" + - key: BILI_SLOW_START_GRACE_MS + type: number + default: "(built-in)" + kind: env-only + audience: user + group: envonly + desc_en: "Extra grace (ms) for a still-alive child that is slow to bind (slow store, AV scan, shared-store lock contention); the total spawn budget is the sum of the two." + desc_zh: "对仍存活但仍在启动(慢速存储、杀毒扫描、共享存储锁竞争)的子进程的额外宽限(毫秒);总启动预算为两者之和。" - key: BILI_ATTACH_EVIDENCE_GRACE_MS type: number default: "(built-in)"