[one-soft-engineer]

Summary

OnlyCueTests was not flaky — the self-hosted runner shares the maintainer's Mac mini, and the two sides were destroying each other's test runs. This adds a pre-flight contention guard that both sides consult before starting.

Closes #816

Root Cause

Not a test bug. The runner executes as user chuan on the same machine as local development, so nothing separates a CI run from a local one except path. Two measurements on 2026-09-15:

• CI → local. Two local full-suite runs failed on two different unrelated tests. ps then showed OnlyCue.app --ui-test-seed=… at 220% CPU under a foreign DerivedData hash plus a live Runner.Worker — the runner was executing dev run 34967315810 through both. With the machine verified idle, the same suite passed: 1822 tests, first try.
• local → CI. dev failed three consecutive runs (34947557702, 34952854230, 34967315810), every one on UI tests (behavioral), every one with the same signature: 6–10 × Lost connection to the application — the app process dying, not an assertion. Local xcodebuild / dotnet test was running during all three. Run 34969558846, the first where the machine was deliberately left alone, passed first try with no retry.

Each is n=1 on its own and neither is proof; together they say contention, not code. The damage is that the three red dev runs invite exactly the wrong conclusion ("something in #835 broke UI tests" — nothing did), and every future flake costs a debugging session to re-derive that.

Fix Description

scripts/ci/machine-busy.sh answers "is another OnlyCue test run using this machine?", and both sides call it. The two sides ask different questions, which is the part that took two attempts to get right:

Side	Question	Flags
local	is a CI job running?	--runner (+ the runner work roots)
CI	is anything running out of the maintainer's working copy?	--under "$HOME/Projects"

The correction, recorded because it would have hurt CI on every run. The first design had each side pass its own checkout and treat everything else as foreign (--self). DerivedData lives in ~/Library/Developer/Xcode/DerivedData, not under $GITHUB_WORKSPACE, so a stale OnlyCue.app from an interrupted previous job reads as foreign — and CI would burn its entire 900 s budget waiting for a process the very next lines of Reset stale test state were about to kill. Asking positively cannot make that mistake. Case 3c in the test suite pins it.

CI warns, never fails. A red dev for a reason that has nothing to do with dev is precisely what this guard exists to stop, so it must not create one. Bounded 900 s wait, then ::warning:: and continue.

No load-average gate, reversing something I floated in the issue: process detection is the direct signal, load average is a lagging proxy for it, so gating on load would add a failure mode (an unrelated heavy process blocking tests indefinitely) to catch nothing the process check misses. Load is printed for diagnostics only.

Changes

• scripts/ci/machine-busy.sh — the detector. Classifies on the executable (xcodebuild, OnlyCue.app/Contents/MacOS/OnlyCue, XCTRunner, Runner.Worker) and matches watched roots against the arguments. MACHINE_BUSY_PS seam so it can be tested.
• scripts/ci/machine-busy.test.sh — 17 cases, plain bash per the repo's existing *.test.sh convention.
• scripts/dev/test.sh — local wrapper: guard, then the killall -9 testmanagerd reset (chore(ci): self-heal the testmanagerd control-session wedge on the runner #471/ci: daemon reset no-ops (sudo -n killall) so testmanagerd wedge never clears #595), then xcodebuild test with ad-hoc signing. Deliberately thin; it is not a build system.
• .github/workflows/ci.yml — guard as the first action of Reset stale test state (before the pkills, which would otherwise reap a live local run's processes); a zero-wait annotating re-check at UI tests (behavioral); timeout-minutes 90 → 105; machine-busy.test.sh added to CI script tests.
• docs/superpowers/specs/816-machine-contention-preflight.md — the spec, including the correction above.

How to Test

• scripts/ci/machine-busy.test.sh — 17 passed, 0 failed
• Mutation-tested, because a first-try-green suite proves nothing. Five mutants, all killed: inverted root match (8 fail, incl. the stale-own-app case), ungated Runner.Worker (1), whole-line classification (7), no-predicate guard removed (1), app/XCTRunner arms removed (2). My first mutation run reported all five surviving — the perl substitutions had silently failed to match; the rerun asserts the file actually changed before trusting the result.
• Real process table, not just canned fixtures — this is what caught the one real bug in the detector (see below) and one vacuous check of my own: a live fake xcodebuild under ~/Projects → CI-side guard exits 1 and names the pid; local-side guard stays 0; after it exits, 0.
• Local refusal path: with a live fake Runner.Worker, scripts/dev/test.sh --no-wait exits 1, names the worker, and no xcodebuild starts.
• Positive path: scripts/dev/test.sh → ** TEST SUCCEEDED **, 1827 tests, 0 failures.
• scripts/ci/run-with-watchdog.test.sh still 9/9; shellcheck clean on all three scripts; ci.yml parses.

A real bug the canned fixtures could not catch. Running the detector for real flagged the very shell that invoked it, because the command I had typed mentioned xcodebuild. Any shell, editor, grep or CI step naming the binary would have held the guard busy forever — a guard that manufactures the phantom failures it exists to prevent. Fixed TDD-style (failing case 3b first), by classifying on the executable rather than the whole command line.

Regression Risk

Moderate and concentrated in ci.yml, since a guard that misfires blocks every run — which is why the CI side can only ever warn.

Two known gaps, stated so they are known rather than silent:

• Mid-job starts. A local run beginning after the guard passes is not caught. Only a mutual flock would catch it, and a fixed shared path is a known trap in this repo (bug(ci): CIRuntime guards stop firing on the UI-test retry attempt #789) — a stale lock from a kill -9'd local run would hold CI to the job timeout. The local wrapper is the mitigation, and it covers the direction that actually caused the three red runs.
• A local run's app host lives under DerivedData and so is not attributable to ~/Projects. CI catches the local xcodebuild instead, which is alive for the whole local run; because the guard runs before the pkills, a live local run is waited out rather than killed.

Worst realistic cost of a misfire: 15 min of extra wall-clock on a job whose real work is already watchdog-capped.

---

OnlyCue verification (required)

Spec link: docs/superpowers/specs/816-machine-contention-preflight.md#input--output
Closes: #816

• New tests added for every behavior (TDD: red→green committed — 8a694e8 pins the contract, fd86a27 implements it; the later d3063f0 corrects both together as one contract change, with the mutants proving the new cases can fail)
• Gherkin scenarios from the issue mapped to UI tests where applicable — n/a, this is CI infrastructure with no user-visible behavior
• Spec updated if behavior diverged from docs/ — the --self → --runner/--under correction is recorded in the spec
• CI green — pending

Note on the template: #816 carries both bug and type:test. bug.md is used here because the change is a bug fix with a root cause and a regression surface; type:test describes the symptom's domain, not this change.

Note on the check that will run: UI tests are gated on push-to-dev, so this PR's green check does not cover XCUITests — and it does not exercise the CI-side guard either, since that lives in Reset stale test state on the same gated path. The guard's CI behavior is first observable on the post-merge dev run.