Repository navigation
Eliminate macOS cold-refresh 30-second tail latency #504
Description
Activity
- added a commit that references this issue
on Aug 10, 2026 Diagnostic run
31423551324on headf9485d0established the root cause with phase, locator, and operation-count evidence:- First three iterations each emitted exactly two interpreter-probe timeouts, both classified as
usrBinPython3. - Iterations: ~30.3s, ~30.2s, ~30.3s; each found only 9 environments.
PathP95: 30,308 ms.MacCommandLineToolsP95: 15,185 ms.- Later iterations had no probe timeouts and completed in 182-294ms with 10 environments.
The two
/usr/bin/python3calls share the same interpreter cache entry.MacCommandLineTools::try_from()starts a 15-second spawn to enrich the alias while PATH discovery asks to resolve the same shim. The cache-entry mutex serializes them: PATH waits for the locator probe, then performs its own 15-second probe after the first timeout, producing the ~30-second phase.The product fix should therefore:
- Resolve Apple
/usr/bin/python*shims from the active developer directory using filesystem state (DEVELOPER_DIR//var/db/xcode_select_link/ standard fallbacks). - Let MacXCode or MacCommandLineTools identify the selected shim without spawning it.
- Skip spawning unresolved Apple system shims; a non-selected or unavailable shim is not a usable Python environment.
- Remove the duplicate
/usr/bin/python3alias-enrichment spawn from the macOS locators. - Add operation-count tests proving zero interpreter spawns for these paths, plus cross-platform behavior tests.
- First three iterations each emitted exactly two interpreter-probe timeouts, both classified as
- added a commit that references this issue
on Aug 10, 2026 First post-fix validation run
31425828087onb923035passed all platform gates.macOS results:
- 0 interpreter probe timeouts (previously 2 per cold iteration).
- Iteration 1 found all 10 environments in 2,436ms (previously 9 environments in ~30s).
- Full refresh P95: 1,437ms (previously ~30,308ms).
PathP95: 1,436ms (previously ~30,308ms).MacCommandLineToolsP95: 137ms (previously ~15,185ms).- All ten iterations found the complete inventory.
Linux and Windows also passed, retained identical environment inventories, and had no interpreter probe timeouts. A second identical run is in progress to distinguish normal macOS median variance from code drift before opening the PR.
Repeat validation run
31426630835on the same commit also passed every platform gate:- macOS iteration 1 found all 10 environments; every iteration retained the complete inventory.
- 0 interpreter probe timeouts.
- Full refresh P95: 2,056ms.
pathP95: 2,055ms.MacCommandLineToolsP95: 130ms.- macOS P50 remained within the enforced platform budget; Linux and Windows inventories/performance gates also passed.
Across both fixed runs, the deterministic 15s/30s tail did not recur. The remaining cold-run variance is ordinary runner startup/build-host noise rather than interpreter probing.
Final quality conclusion for PR #506 at
c0f026e:- Full CI, lint, CodeQL, Linux/Windows coverage, and enforced performance gates pass.
- Exact-head Copilot review covered 7/7 files with 0 comments and 0 unresolved threads.
- macOS repeat: P50 120ms vs 99ms baseline; P95 1,256ms vs 30,320ms baseline (95.9% lower); time-to-first P50 84ms vs 99ms.
- Windows repeat: P50 138ms vs 124ms; P95 1,224ms vs 2,533ms. Linux P50 53ms vs 55ms.
- Every measured iteration retained the complete inventory (macOS/Windows 10, Linux 5); interpreter probe timeout maps are empty on all platforms.
- Untimed diagnostics still retain 10 phase/locator samples; macOS
MacCommandLineToolsP95 is 35ms. - Coverage increased: Linux lines/functions +0.121pp/+0.015pp; Windows +0.142pp/+0.066pp.
Hosted cold-start variance remains visible (notably first-iteration startup), but two final-head-equivalent gated runs and the exact-head repeat show no material median regression. The deterministic 15s/30s interpreter-probe tail does not recur.
Summary
PET's macOS performance baseline has a deterministic cold-refresh tail near 30 seconds that is hidden by healthy P50 values.
Evidence
Across 11 successful main-branch performance baselines from July 24 through August 6, 2026:
8bd5fb4, run31071711968): P50 142 ms, P95 30,291 ms, mean 8,642 msThe same cache directory is reused while each iteration starts a fresh PET server. The first three processes are slow and initially miss one environment, after which the disk cache is warm. Environment inventory is otherwise stable (10 environments, 1 manager), so this is not explained by changing runner inventory.
Investigation requirements
RefreshProgress/RefreshPerformancedata or equivalent privacy-safe instrumentation.Acceptance criteria
Related