The admin product list — org optional (#259) suite (tests/cli/product-list-all-orgs.test.ts) intermittently red-flags CI with a per-test timeout (observed ~5.3s on the --help case) even though every assertion passes on rerun and locally. It flaked #366's CI and had to be re-run to merge.
Cause
Each it(...) shells out via runCli([...]) (tests/utils.ts), spawning a fresh Node/Bun process that loads the whole commander tree. Five spawns in one file, under a loaded CI runner, occasionally push a single spawn past bun's default per-test timeout. The tests are correct — it's process-startup variance, not logic.
Options (pick one)
- Raise the timeout for this suite:
it("…", () => {…}, 15_000) (or a describe-level default) — smallest change, removes the false red.
- Collapse the five
runCli spawns into fewer processes where the assertions allow (e.g. one --help spawn asserted against multiple substrings is already single-spawn; the three validation cases each need their own argv, so this only partially helps).
- Warm/one-shot the CLI in-process instead of spawning, if
runCli can expose an in-process entry (larger change; check whether other suites depend on the subprocess boundary for env isolation).
Recommend option 1 — a generous per-test timeout on the subprocess-spawning CLI suites is the pragmatic fix; the assertions themselves are fast and deterministic once the process is up.
Context: surfaced while landing the stub org get work (#366); CI is the merge gate (no pre-push test hook by design), so a flaky suite directly costs rerun cycles.
The
admin product list — org optional (#259)suite (tests/cli/product-list-all-orgs.test.ts) intermittently red-flags CI with a per-test timeout (observed ~5.3s on the--helpcase) even though every assertion passes on rerun and locally. It flaked #366's CI and had to be re-run to merge.Cause
Each
it(...)shells out viarunCli([...])(tests/utils.ts), spawning a fresh Node/Bun process that loads the whole commander tree. Five spawns in one file, under a loaded CI runner, occasionally push a single spawn past bun's default per-test timeout. The tests are correct — it's process-startup variance, not logic.Options (pick one)
it("…", () => {…}, 15_000)(or adescribe-level default) — smallest change, removes the false red.runClispawns into fewer processes where the assertions allow (e.g. one--helpspawn asserted against multiple substrings is already single-spawn; the three validation cases each need their own argv, so this only partially helps).runClican expose an in-process entry (larger change; check whether other suites depend on the subprocess boundary for env isolation).Recommend option 1 — a generous per-test timeout on the subprocess-spawning CLI suites is the pragmatic fix; the assertions themselves are fast and deterministic once the process is up.
Context: surfaced while landing the stub
org getwork (#366); CI is the merge gate (no pre-push test hook by design), so a flaky suite directly costs rerun cycles.