Skip to content

test(macos): make the macOS test job deterministic - #257

Merged
akonwi merged 4 commits into
mainfrom
macos/test-stability
Oct 22, 2026
Merged

akonwi merged 4 commits into
mainfrom
macos/test-stability

Conversation

@akonwi

@akonwi akonwi commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Makes the macOS test job deterministic. Fixes #234 and #256.

Root causes

CI diagnostics (temporary instrumentation, since dropped) on the 3-core macos-latest runner showed:

  1. Saturated executors at suite start. Swift Testing starts ~970 tests at once. The main actor was blocked for 6.2 s in one stretch, and the cooperative pool for up to 2.7 s. Tests that gave a condition a 0.5–5 s budget failed on correct code:
  2. A shared SwiftUI field editor. SwiftUI edits every text field in the process with one field editor. When another test focused a text field in its own window, it took that editor, and the launcher's window became its own first responder. The diagnostics showed the search field gaining focus and then losing it about 120 ms later. Focus never came back, even after 145 s. The arrow keys and Return then went nowhere (opened == [], renamed == ["a"]). This explains the consistent renderLauncherForReview failures (Flaky macOS CI tests: SessionLauncherTests.renderLauncherForReview and TileArrangementTests #234). Adding windowed tests such as fix(macos): shrink and scroll tile tabs without a system scroller #254's makes the overlap more likely. Opening a second focused window reproduces it locally every time. Activation and key-window status don't matter: the app was never active on CI, and focus works without either.

Fixes

  • eventually (TestTiming.swift): one polling helper whose 60 s limit only stops a broken build from hanging. It replaces every short-budget polling loop and ad-hoc until/eventually/wait helper (about 60 call sites). Tests that pressed a key a fixed number of times, or after a fixed sleep, now press until they see the effect.
  • Injected debounce sleeps: SessionWindowModel and ScratchpadState take a Sleeper. With ManualSleeper, the layout-capture and autosave tests wait for the delay to start, check that nothing happened yet, end the delay, and check the result. They no longer measure wall-clock time; they assert which delay was requested.
  • Windowed tests wait for rendered state: the diff, inline annotation, and transcript history tests slept a fixed 150–1500 ms between steps. They now wait for the layout, scroll, or focus state they assert on. diffPaneShowsServerFileAndTargetInBothAppearances had failed on CI because of these sleeps.
  • FocusIsolatedWindow (TestWindows.swift): edits with its own instance of SwiftUI's field editor. The launcher test uses it, and no longer calls NSApp.activate() or makeKey().
  • The SSH fakes keep the app's own ssh idle timeout (30 s) instead of 5 s. No test waits that timeout out, and the shorter one could end a fake master while a busy runner was relaying a question.

Validation

  • CI: 3 consecutive green runs of the final commit (run 39845097592, attempts 1–3). The suite took 31.7 s, 35.2 s, and 29.0 s. Timings that would have exceeded the old budgets:
    • closingAHostWithdrawsItsQuestion: 5.2 / 8.1 / 3.9 s (old budget 5 s per wait)
    • layoutChangesAreCapturedAfterADelay: 10.2 / 9.9 / 6.7 s (old budget 5 s)
    • renderLauncherForReview: 9.0 / 9.9 / 6.6 s
  • Local (M-series, 12 cores): the full suite passed 9 times in a row on the final test code, 5 of those with 16 yes processes saturating the CPU. Before the last DiffTests fix, a loaded run caught one more fixed-sleep race, which is now fixed. Earlier, the affected suites passed in repeated isolated loops.
  • Reproduction: a second window that focuses a SwiftUI text field makes the original launcher test fail locally with exactly CI's assertions. With FocusIsolatedWindow, it passes.
  • code-reviewer subagent: no blocking issues. Its findings are addressed: the VCS cancellation race is fixed, the remaining polling loops are converted, and the property is renamed to sleeper.

Follow-up for the remaining fixed sleeps that haven't been seen failing: #260.

Closes #234. Closes #256.

@akonwi
akonwi force-pushed the macos/test-stability branch from 53958ab to a27373f Compare October 22, 2026 19:11
The suite runs its tests in parallel, and on CI's three-core runner the
main actor was measured blocked for 6 s, and the cooperative pool for
2.7 s, while the suite starts. Waits that gave a condition 0.5-5 s before
failing therefore failed on correct code (#234, #256).

Add one `eventually` helper whose limit only keeps a broken build from
hanging, and use it for every condition the tests polled with a short
budget. Tests that pressed a key a fixed number of times, or after a
fixed sleep, now press until the effect is seen. The SSH fakes keep the
app's own ssh idle timeout, which no test waits out.
The layout capture and scratchpad autosave tests waited real time for
their delays and then for the result, so a busy runner could miss the
deadline (#234). SessionWindowModel and ScratchpadState now take the
sleep they debounce with, and the tests pass a ManualSleeper: they wait
for the delay to begin, check nothing happened yet, end it, and check
the result, including that quitting cancels a pending capture.
SwiftUI edits every text field in the process with one shared field
editor. When a test running at the same time focused a text field in its
own window, it took the editor from the launcher's window, leaving that
window as its own first responder, so the launcher test's arrow keys and
Return went nowhere (#234). CI diagnostics showed the search field
focused and then losing focus about 120 ms later; a second focused window
reproduces it locally every time.

The test now hosts the launcher in a FocusIsolatedWindow, which edits
with its own instance of SwiftUI's field editor. It no longer activates
the app or makes the window key, since focus never depended on either,
and it waits for the field and the key presses' effects rather than
sleeping.
Windowed diff, inline annotation, and transcript history tests slept a
fixed 150-1500 ms after each step and then asserted on layout, scroll
position, or focus. On CI's busy runner that wasn't always enough:
diffPaneShowsServerFileAndTargetInBothAppearances failed when the pane
hadn't laid out for its shorter window before the wheel event. Each step
now waits with `eventually` for the state its assertions check. The two
diff tests that interrupt a slow read wait for the read to start.
@akonwi
akonwi marked this pull request as ready for review October 22, 2026 21:23
@akonwi
akonwi merged commit b45be33 into main Oct 22, 2026
8 checks passed
@akonwi
akonwi deleted the macos/test-stability branch October 22, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant