Local-lane jobs die at the 30-minute cap inside 'playwright install --with-deps' — apt hangs on the Azure mirror and the retry loop cannot fire

[TortoiseWolfe]

CAUSE FOUND 2026-08-18 — it is apt hanging inside playwright install --with-deps, and the retry loop cannot fire

The cancelled jobs are not slow tests. They never run a test at all. In every capped job
sampled, Running N tests appears zero times.

.github/workflows/e2e-local.yml:336:

for i in 1 2 3; do
  pnpm exec playwright install --with-deps chromium ${{ matrix.browser }} && break
  echo "Retry $i: playwright install failed, retrying in 10s..."
  sleep 10
done

--with-deps shells out to apt-get as root. On run
32088862298, apt
could not reach azure.archive.ubuntu.com and logged 18–26 Ign: lines per job, then went
silent — and apt hangs rather than exits, so the job sat in that one step until the
30-minute cap killed it:

job	last apt line	cancelled	silence	tests run
webkit-msg-iso 1/1 (attempt 2)	03:08:39	03:34:52	26m 13s	0
chromium-gen 5/6 (attempt 1)	01:41:16	02:07:30	26m 14s	0
firefox-gen 3/6 (attempt 1)	01:48:18	02:14:22	26m 04s	0
webkit-gen 4/6 (attempt 1)	01:41:20	02:07:30	26m 10s	0

The retry loop is dead code in this failure mode, and that is the part worth keeping.
&& break retries on a non-zero EXIT. A hang never exits, so i never reaches 2 — the
three attempts written to survive a bad install cannot run. This is exactly the shape of
#765: an escape hatch that could never fire, so the job died on the line written to save it.

What this settles, and what it costs

• The 30-minute cap is not the bug. It is the only thing that ends the hang. Raising it
  makes the same outage cost 45 minutes instead of 30 — which is what this issue already
  suspected, now with a reason rather than an inference.
• The ">7x spread on identical work" is explained. It is not the shard doing more work;
  it is a shard doing NO work for 26 minutes.
• It is not the shard, the browser, or the branch. Capped jobs spanned chromium-gen,
  firefox-gen, webkit-gen and webkit-msg-iso — projects sharing no spec files — because
  the hang is in a step every job runs before any test.
• The E2E lane captures traces, screenshots and videos on failure — then uploads everything except them #766 is no longer needed to diagnose this. The evidence was in the plain job log the
  whole time. Traces would not have shown it, because the failure is before Playwright starts.

Candidate fixes, cheapest first

1. Bound the command so a hang becomes a failure the existing retry can act on —
   timeout 300 pnpm exec playwright install --with-deps .... One word, and it makes the
   three retries real instead of decorative.
2. Bound apt itself — -o Acquire::http::Timeout=15 -o Acquire::Retries=3, so the mirror
   is abandoned rather than waited on.
3. Question --with-deps at all. GitHub's ubuntu-latest image already ships Playwright's
   system libraries; the browsers themselves come from Playwright's CDN, not apt. If the deps
   are already present, this step is doing an apt round-trip on every one of 24 jobs for
   nothing — and taking the whole lane down when a mirror stalls.

Fix 1 alone converts a merge-blocking 30-minute hang into a ~5-minute retry that usually
succeeds. Do not raise timeout-minutes instead.

RECURRED 2026-08-18 — EIGHT jobs at once, and the "one stalling shard" model does not fit it

Run 32088862298
(PR #794, attempt 1) had 8 of 26 jobs cancelled at exactly 30 minutes: chromium-gen
1/6 and 5/6, firefox-gen 3/6, 4/6 and 6/6, webkit-gen 3/6 and 4/6, and
webkit-msg-iso 1/1. Zero tests failed anywhere in the run. E2E (local) result went
red purely on the cancellations, and the PR could not land.

Capped jobs per run, last ten local-lane runs, newest first:

8  0  0  0  0  0  0  0  0  1

Whole-run shape versus the nine runs before it: median job 12m against 8–9m, max 30m
(capped) against 18–20m.

What is new here is the correlation. Comparing the same job names against the run
immediately before, on the same lane:

job	this run	previous	delta
firefox-gen 3/6	30.2m cancelled	7.1m	+23.1m
chromium-gen 5/6	30.2m cancelled	10.3m	+19.9m
webkit-msg-iso 1/1	30.2m cancelled	19.1m	+11.1m
chromium-gen 6/6	16.7m success	9.0m	+7.7m
chromium-msg 1/1	12.4m success	6.0m	+6.4m
chromium-gen 2/6	~unchanged	—	~0

These jobs share no spec files — webkit-msg-iso and chromium-msg are messaging
isolation, *-gen are the general sweep. They cannot stall each other through anything in
this repo. Yet most of the run slowed together while a few jobs were untouched, which does
not match "a shard stalls" and does not match a uniform infrastructure tax either.

Consequence for the cap. The existing note reasons from a >7x spread on one shard.
Across a whole run the honest figure is narrower and worse: typical jobs run 8–9m and the
worst NORMAL job runs 18–20m, so a 30m cap carries only ~1.5x headroom over ordinary
variance. A run-wide slowdown of the size measured here will therefore cap several jobs
before it caps any single outlier — which is exactly what happened.

Still not an argument for simply raising the number. It is an argument that the cap is
being asked to absorb variance nobody has explained, and that #766 (the lane discards
the traces it captures) is the thing standing between us and an explanation.

RECURRED 2026-08-17 — and it blocked a merge, twice over

webkit-gen 2/6 and webkit-gen 4/6 were both cancelled at the 30-minute cap on
run 32077716593
(started 22:49:17, killed 23:19:32 — 30:15 each), on PR #790.

Consequence: E2E (local) result — a REQUIRED check — failed with
shards reporting: 22/24 and failed 0. No test failed. The aggregate
correctly refuses to score partial data, so a cancelled shard is indistinguishable
from a broken PR at the merge gate, and the PR could not land.

A second, different flavour hit the same PR's earlier run: 23 of 24 shards
reporting with zero failures because one shard succeeded but uploaded no
artifact. Both produce the same symptom — a red required check with no failing
test — and both are resolved by re-running, which means the signal a human acts on
is "re-run and see", not "read the failure".

Frequency measured across the last 8 local-lane runs: 2 runs with cancellations
(this one, and b8fe294f with 1). That is not rare enough to shrug at once the
lane is required.

Why this is worth more than a bigger timeout. #762 already records 8m typical,
30m observed, 4m on re-run — a >7× spread on identical work says the shard is
stalling, not doing more work. Raising the cap hides it; the same stall then costs
45 minutes instead of 30. Worth capturing a trace on the next cancellation before
tuning the number — which is what #766 is for (the lane captures traces,
screenshots and videos on failure and then uploads everything except them).

webkit-gen 3/6 hit the lane's 30-minute timeout-minutes and was cancelled, blocking a merge
on a docs-only branch (#761 — five files: three docs, one role file, one script test; zero
files under src/ or tests/).

It was not hung. The log shows it at test 71 of 85, still completing tests at ~2s each,
when the cap fired. It simply ran out of budget.

The margin is thinner than it looks

Same shard, recent runs:

branch	duration
main	8m
fix/392	8m
fix/668	9m
fix/569	cancelled at 30m
fix/569 (re-run, same commit)	~4m

A ~4x spread on identical work, against a fixed 30-minute cap. The re-run passing on the same
commit is what identifies this as runner speed rather than anything in the diff.

Why it matters now

E2E (local) result became a required check on 2026-08-15. Before that, a cancelled shard
was an annotation someone could ignore; now it blocks the merge — which is the point, and also
means infrastructure variance has a direct cost it did not have last week. This is the first
occurrence since the gate went on.

What NOT to do

Do not raise timeout-minutes as the reflex. The cap is also what stops a genuinely hung
shard burning a full hour, and this repo has a documented history of resizing a number against
a quantity nobody measured (#751 — the retention window, mis-sized twice the same way). Any new
value needs the distribution behind it, not a guess.

Worth considering, roughly in order of appeal:

1. Measure first. Collect per-shard durations across ~20 runs and look at the actual spread
   before touching anything. If p95 is 12m, a 30m cap is fine and this was an outlier; if p95 is
   22m, the cap is the problem.
2. Rebalance the shards. webkit-gen splits 6 ways and the slowest is ~2x the fastest
   (6m vs 13m observed). Better balance buys headroom without weakening the hang guard.
3. Per-shard timeout rather than one global. The -gen shards and the -msg-iso shards do
   very different work and do not need the same cap.

Related

• E2E flaky watch: the standing flake list + the FLAKY_GATE_MODE annotate→block decision #300 — the flaky-gate annotate → block decision. This is a data point for it: the failure
  was infrastructure, the re-run was clean, and a human had to make that call by reading a log.
• E2E costs 2.22 runs per merge — merge discipline alone cannot reach the budget #643 — E2E run economics.