smoke shard 2 has never completed: smoke:opencode-events-release hangs into the 30-minute job timeout, and 13 suites behind it have never run

[davidfarah2003]

Shard 2 of the smoke gate has not completed on any recent run. It is not being cancelled by
anything — it hangs on one suite and hits the 30-minute job timeout, and GitHub badges a
timed-out job as cancelled, which reads as "superseded by a newer run" rather than "this quarter
of the gate did not finish". As a result 13 suites have never executed.

The measurement

Three consecutive runs, shard 2's duration against the other shards:

run	sha	shard 2	shard 0	shard 1	shard 3
32958247575	28f9bf427	cancelled after 30m16s	—	—	—
32960622394	a67e704b7	cancelled after 30m15s	10m46s	0m58s	13m01s
32964962658	47b7d025d	cancelled after 30m	10m43s	1m33s	12m11s

timeout-minutes: 30 is set at .github/workflows/ci.yml:67. Both completed cases land within
16 seconds of it. fail-fast: false is set at ci.yml:70, so sibling shard failures do not cancel
it, and cancel-in-progress is ${{ github.event_name == 'pull_request' }} (ci.yml:21), so a
push to main does not cancel it either. Nothing cancels shard 2. It runs out of time.

Where it hangs, and that it is deterministic

Both timed-out runs whose logs are retrievable started exactly 83 of 96 suites and stopped on
the same one:

smoke:lang-pins                 11:19:37
smoke:runtime-journal-store     11:19:37
smoke:runtime-mesh-wait         11:19:39
smoke:runtime-fork              11:20:03
smoke:agui-opencode-source      11:20:04
smoke:opencode-events-release   11:20:05   <- still running at 11:27:03 when the job was killed

Its five predecessors each took between under a second and 24 seconds. smoke:opencode-events-release
(extensions/connector-opencode/smoke/events-release.smoke.ts) had been running 7 minutes and
had not finished. The same suite, at the same position, on run 32958247575 as well. This is a
deterministic hang, not a slow shard: shard 2 reached suite 83 in 23 minutes and then one suite
consumed the rest.

On the kill, the runner terminates orphans including two nats-server processes, so the suite is
leaving a live broker behind when it wedges.

What has never run

The 13 suites after the hang point on shard 2, none of which has executed in any observed run:

smoke:agui-codex-map           smoke:hermes-launch-env       smoke:overflow-churn-bound
smoke:web-snapshot             smoke:history-flood           smoke:empty-id-ingest
smoke:core-history-limit       smoke:remote-exchange:live    smoke:control-transport-dial
smoke:lang-engine              smoke:jcode-provider-disconnect  smoke:no-implicit-general
smoke:int2-revoke:live

Arithmetic cross-checks against CI's own numbers: the file parses to 384 suites, shard 2 holds 96,
and the hang suite is #83 — matching the job's shard 2/4 — 96 of 384 smokes header and its 83
started.

Two of those are worth calling out because they guard behaviour that shipped recently and were
presumably added to prevent exactly the regressions they cannot currently catch:
smoke:no-implicit-general (the implicit general channel floor removal) and
smoke:control-transport-dial (the transport-aware control dial).

Why this stayed invisible

A hang and a cancellation produce the same badge. Anyone reading the checks list sees three shards
with real verdicts and one greyed-out cancelled, which is what a superseded run looks like. There
is no signal anywhere in the UI that a quarter of the gate timed out, and the log that would show it
is only published once the job ends — so an in-flight investigation cannot see it either.

Suggested scope

1. Fix or quarantine the hang in smoke:opencode-events-release. It should fail loudly on its own
   budget rather than consuming the shard's.
2. Give suites an individual timeout so one wedged suite cannot eat the shard, and make the shard
   runner report which suite it was in when killed.
3. Treat a cancelled smoke shard as a failure rather than as noise — as it stands, the gate cannot
   distinguish "did not run" from "was superseded", which is the same class as a check that cannot
   fail.

Item 2 is the one that prevents recurrence; item 1 only fixes today's instance.

[davidfarah2003]

Third consecutive confirmation, on main at 0252eb855 (run 32966759218):

smoke (shard 2/4)  cancelled  30.3 min
smoke (shard 1/4)  failure     8.9 min
smoke (shard 3/4)  failure    11.0 min
smoke (shard 0/4)  failure    10.9 min

Shard 2 durations across three consecutive runs are now 30m16s, 30m15s, 30.3m — every one at the
timeout-minutes: 30 ceiling, while every sibling shard finished in 9-13 minutes on the same runs.
There is no run in this window where shard 2 completed.

[davidfarah2003]

Correcting my own suggestion 2 above ("give suites an individual timeout so one wedged suite
cannot eat the shard"). I proposed it before measuring, and the measurement makes it much weaker
than I implied. There is also a second, independent problem with shard 2 that the hang was hiding.

Measured suite durations

Gaps between consecutive suite banners across four shard logs, 282 samples:

median  2.0s     p95  38.0s     max  649.0s

The slowest, corroborated across two independent runs so these are not one-off readings:

suite	run a67e704b7	run 28f9bf427	shard	pos
smoke:seed	649s	664s	2	45
smoke:user-spawn:live	137s	137s	2	74
smoke:up-tls-routes	260s	—	0	51

Why that undercuts a per-suite timeout

The legitimate maximum is ~11 minutes (smoke:seed). So a uniform per-suite timeout would have to
sit above that, and at ~12 minutes a hung suite still consumes 40% of the shard's 30-minute budget
before being killed. It converts an invisible hang into a visible one, which is worth something, but
it does not stop one suite eating the shard. A per-suite override table would work and is a
materially bigger change than the one line I implied.

The second problem, which the hang was masking

Shard 2 is structurally close to its budget even without the hang. Both runs took 22.3 and 22.6
minutes of legitimate work just to reach position 82, and smoke:seed alone is roughly half of
that and about a third of the entire shard budget.

So fixing smoke:opencode-events-release may not be sufficient. After it, shard 2 would have ~7.5
minutes left for its remaining 13 suites, and nobody knows whether that is enough because those 13
have never executed. It could clear comfortably or it could time out one suite later, and this issue
would look only partially fixed.

Round-robin assigns by count, not duration. bin/smoke/shard.mjs:8-9 says round-robin "balances
runtime better than contiguous slices", which is true as a comparison and still left the two slowest
suites in the gate on the same shard.

Revised scope

1. Fix or quarantine the hang in smoke:opencode-events-release — unchanged, still the immediate cause.
2. Rebalance by measured duration rather than by count, or raise shard 2's budget. Without this,
   item 1 alone may just move the timeout to a different suite.
3. Treat a cancelled smoke shard as a failure — unchanged, and the most valuable of the three,
   because it is what made a reproducible 30-minute timeout invisible for at least three runs.
4. smoke:seed at ~11 minutes is worth a look on its own merits, independent of sharding.

Replacing my earlier per-suite-timeout suggestion with item 2; it was the wrong instrument and I had
not measured before proposing it.

[davidfarah2003]

Heads-up so a changed symptom is not read as a changed defect: shard 2's failure looks completely
different on main right now, and it is not this issue.

Main at a268c8dbe (run 32977375051, shard 2, job 98205295502):

smoke (shard 2/4)  failure  4.0 min   —  39 suites started
✗ shard 2/4 FAILED at: pnpm smoke:suite-ambient-env

Four minutes and 39 suites, versus the 30-minute timeout at 83 suites this issue documents. That is
because a regression I merged in #898 trips smoke:suite-ambient-env at roughly position 38, so the
shard now dies well before it reaches smoke:opencode-events-release at position 82. Repair is up
as #901.

Nothing here changes this issue's evidence. Once #901 lands, shard 2 should resume running to
position 82 and timing out at 30 minutes, and the 13 suites behind the hang remain unexecuted. If
anyone reads shard 2 in the meantime, the 4-minute failure is mine and temporary; the 30-minute one
is the real subject of this issue.

[davidfarah2003]

Shard 2 has completed, which narrows this from a hang to a budget problem.

Run 33002100783, smoke shard 2/4 re-run on the same sha that had just been cancelled at the timeout:

conclusion  success
banner      ✓ smoke:ci shard 2/4 passed (96 smokes)
duration    19:24:44Z -> 19:51:30Z   = 26m46s

All 96 suites ran, including the one named here — pnpm smoke:opencode-events-release started 19:43:05Z and was done by 19:50:03Z, about seven minutes, and passed.

Set against the two cancelled observations from the same day:

main   run 32982811084   shard 2   cancelled   30m16s
PR     run 33002100783   shard 2   cancelled   30m20s
PR     run 33002100783   shard 2   SUCCESS     26m46s   (single-job re-run, same sha)

So the shard is not wedging on a suite that never returns; it finishes in roughly 27 minutes against a 30-minute limit, and ordinary variance pushes it past. That fits the two failures landing within four seconds of each other at exactly 30 minutes — the signature of a job limit, not of a hang, which would have no reason to cluster.

Two consequences worth having:

• A single-job re-run is a real remedy, not just a way to scrape the log. gh run rerun <run> --job <job-id> cleared it here on the first attempt.
• A cancelled shard 2 discards passes that really happened. smoke:flag-inventory and smoke:meshes-registry run about four minutes in and pass; when the job is later cancelled the check-run reports no result for them. The evidence survives in the job log, so if you need to know whether an early shard-2 suite passed on a cancelled run, read the log rather than the conclusion.

The framing in the title — 13 suites behind it that have never run — is right about the impact when it does get cancelled, but they clearly can run, so the fix is likely on the budget side (raise the limit, rebalance the shard, or reduce opencode-events-release's ~7 minutes) rather than on unwedging a stuck process.

[davidfarah2003]

Correction on #897: the hang framing in the title and opening still stands uncorrected, though comments 2 and 4 below already reached the budget reading independently; what is missing from the record either way is the log evidence, which has been in the cancelled jobs all along. Reading the check-run conclusion instead of the job log is what produced the hang framing, so this correction is written entirely from the logs.

What "never completed" actually looks like

Shard 2 has completed, repeatedly. Every number below is from gh api repos/Cotal-AI/cotal/actions/jobs/<id>/logs and the jobs endpoint:

run	sha	shard 2	duration	suites started	events-release
32958348101	2d797e5	success	24m59s	96 of 384	PASSED 10:51:13Z, 20 checks
33002100783 (re-run, attempt 2)	18d7b4b	success	26m46s	96 of 386	PASSED 19:50:03Z, 20 checks
33021101455 (PR #880 head)	93bc596	success	29m22s	96 of 386	PASSED 23:20:09Z, 20 checks

The re-run in the middle is the same sha that had just been cancelled at the 30-minute ceiling in attempt 1 (job 98286307155, 18:52:39Z to 19:22:59Z, 30m20s, killed inside events-release). Same code, two outcomes, 3m34s apart. That is variance, not determinism.

So the claim "has never completed" is refuted three times over, and the "13 suites behind it have never run" claim falls with it: all 96 suites ran in each of those three runs, including the 13 the issue names.

What the cancelled runs actually show

The cancellations are not all the same thing. Three distinct classes exist in the shard-2 history:

1. Timeout cluster: cancelled at 30m14s to 30m20s (measured: 32958247575 at 30m16s, 32960622394 at 30m15s, 32959885577 at 30m15s, 32962722667 at 30m15s, 32966391386 at 30m20s, 32966759218 at 30m17s, 32955315679 at 30m15s, 32955410723 at 30m15s, 32964962658 at 30m16s, plus 32975573992 at 30m16s, 32981233601 at 30m15s, 32981343378 at 30m14s, 32982811084 at 30m16s on main and PR-side, and PR attempt 1 of 33002100783 at 30m20s).
2. Concurrency supersession on PRs (cancel-in-progress is pull_request-only per ci.yml:21): 33019755782 at 3m53s, 33016261562 at 7m20s, 32965489701 at 10m30s, 32948378828 at 8m14s, 33019953057 at 20m36s, and others.
3. Real red failures that finished on their own (32978172063 at 3m54s, 32977996673 at 4m11s, 32977522667 at 2m20s): a merged regression (fix(web): name rejected history reads #898 fallout, suite-ambient-env) that fails the shard outright at position 39. Those are not this issue and are already noted in the thread.

One run in that cluster is worth naming separately because it is the cleanest single refutation: 33016786843, a PR run cancelled at 30m15s in which events-release PASSED at 22:15:24Z and the shard went on to start int2-revoke:live before the ceiling killed it. A cancelled shard whose log contains the named suite passing, followed by five more suites starting.

Class 1 is the one the issue is about. Here is what the logs show inside it, and this is the part a conclusion badge cannot tell you:

• In twelve of those runs the last suite banner is smoke:opencode-events-release and the suite never prints its PASSED line. The issue read that as a hang.
• In four of them the suite PASSED and the shard moved on before the ceiling killed a later suite: 32955410723 (passed 10:24:41Z, four more suites started), 32964962658 (passed 12:15:43Z, four more), 33016786843 (passed 22:15:24Z, five more, through int2-revoke:live), and job 98212054335 in run 32979408803 (passed 14:48:26Z, eight more suites started, through remote-exchange:live). The passes survive in the log even though the check-run reports no result.
• In every one of those cancelled logs, events-release had already printed all 20 of its check marks before the kill. Counted per run: 20 of 20 in each of the twelve killed-inside-it logs, in the four cancelled logs where it passed, and in the green runs alike.

The suite's own runtime is the decisive measurement. From banner to verdict:

• passing runs: 7.0 minutes, with strikingly little spread (417.7s, 417.8s, 418.4s, 418.9s, 418.9s across five CI runs, banner to verdict: 1.2 seconds of total spread).
• wedged runs: killed at 5.8 to 7.0 minutes in, never past the point where the passing runs finish.
• separately, on a developer box rather than CI hardware (own throwaway broker, no runner contention): 6m59.2s, PASSED, 20 checks, exit 0. Not pooled with the CI figures above; it corroborates them from a different environment rather than adding a sixth equivalent sample.

A deterministic hang does not finish in exactly the same seven minutes every time it is allowed to finish. For contrast, this codebase does contain a suite that hangs exactly the way this issue imagined, and it was reproduced on demand while investigating this one: an unbounded await behind an undrained stderr pipe, last cell printed, then silence until an external kill. That is what a hang looks like here, and distinguishing it from a run that merely ran out of clock is exactly why the verdict has to come from the log rather than the badge.

What varies between the green runs and the killed ones is not the suite; it is what came before it. Across the eight timeout-cluster runs with extractable banner timestamps, the first 82 suites took 22.3 to 23.4 minutes in the ones killed inside events-release (smoke:seed at 649 to 702s in them) versus 15.9 to 19.9 minutes in the three green runs (seed at 376 to 539s). The ceiling lands where it lands because the shard arrives at suite 83 with over 22 minutes already spent. The one thing the tail of the suite does after its last check is wait out the teardown of connections it opened on purpose (the logs show periodic DrainingConnectionError stacks from nats idle-heartbeat monitors on connections already draining, at a 30-second cadence, in both green and killed runs; the green ones simply reach the exit before the clock does).

So: budget boundary, not a hang. The issue's own second comment reached the same place independently (rebalance by duration, treat cancelled as failure); this is the log evidence for it, plus one point that comment got wrong: a per-suite timeout is not useless. At 30 minutes for 96 suites the shard is tight regardless of how the timeout is set, but the runner still cannot name the suite it was in when killed, and that gap is exactly why twelve "hung on events-release" runs were actually twelve "arrived at events-release already over budget" runs and nobody could tell. The suggested scope survives in this shape:

• Item 1 (fix or quarantine the hang in events-release): does not survive as written. There is no hang to fix; the suite passes in about 7 minutes wherever it gets them. What might survive is reducing those 7 minutes (they are mostly deliberate bounded waits: a 12s over-the-bound hold, several multi-second drain windows), which is tuning, not a fix.
• Item 2 (per-suite timeout plus naming the suite at kill): the naming half fully survives and is the cheap, high-value fix. The timeout half is weak as a uniform setting (the legitimate max is smoke:seed at up to 12 minutes in the observed window, so a uniform bound set safely above that still lets one slow suite consume 40 percent of the shard budget, exactly as the issue's own second comment argues) but as a diagnostic it converts this whole class from invisible to self-reporting.
• Item 3 (treat a cancelled smoke shard as a failure): fully survives, and this issue is its own best evidence. A gate whose quarter can be silently absent, where cancelled means both "superseded by a newer push" and "ran out of clock mid-suite", cannot distinguish did-not-run from superseded, and a gate that cannot distinguish those cannot fail. Both the original hang diagnosis and the invisibility it complained about come from reading badges: the badge said cancelled, the log said budget, and only one of them was ever load-bearing.

Why the numbers here differ from the issue's

The issue says the file parses to 384 suites and the job banner says 384. At its cited shas that is correct. The count has since moved: 382 at bf7a87c, 385 at ccd9946 (current main), 386 at 93bc596 (PR #880 head). Position 83 in shard 2 is stable across all of them because the additions append at the end; verify against the banner of whichever run you are reading, not against this issue or against main.

That stability is a property of appending, not a guarantee of the format. bin/smoke/shard.mjs partitions by INDEX (all.filter((_, i) => i % count === shard)), so an INSERT anywhere in the chain file reassigns every entry after it to a different runner. Two runs labelled "shard 2" either side of such an insert are not the same set of suites, and nothing in the job output says so. So anyone diagnosing a shard by comparing it against an earlier run has to establish that the chain file did not move between them; otherwise the comparison reads as agreement or disagreement with equal confidence and no basis. That is a second, time-independent way for a shard result to be incomparable, distinct from the budget boundary above.

The issue's detail about the five predecessors ("under a second to 24 seconds") matches what the logs show for the suites immediately before position 83. They are not where the time goes. The suite that actually eats the budget sits much earlier: smoke:seed at position 45, at 649 to 702s in the killed runs versus 376 to 539s in the green ones, roughly a third of the entire shard budget on its own and the single largest source of the arrival-time spread that decides these outcomes.

[davidfarah2003]

This no longer reproduces. Shard 2 completes and passes routinely now, so the 13 suites behind the
hang are executing.

Measured across the last ten completed CI runs, shard 2's conclusion and wall time:

run	conclusion	duration
33220057022	success	26m10s
33219897171	success	33m24s
33219527255	cancelled	22m52s
33219443753	success	28m02s
33217394490	success	33m24s
33216727065	cancelled	40m15s
33216717392	success	34m00s
33216636261	success	32m47s
33216366010	success	33m20s
33215950948	cancelled	6m14s

Seven of ten succeeded. The three cancellations do not share the old signature: two ended well
before any limit and read as supersession by a newer push, and one ran to 40m.

The cause of the change is the job limit, not the suite. timeout-minutes on the smoke job is now
40 (.github/workflows/ci.yml:94), raised in 85d4d421. Shard 2's real cost is 26 to 34 minutes,
which sat above the old 30 and sits below the new 40.

Two things worth separating before this is closed. The reported symptom is gone and the coverage
hole with it, which is the part that mattered, since a quarter of the gate silently not running is
worse than a slow one. But the headroom is thin: a 34 minute shard under a 40 minute limit leaves
six minutes, and the run at 40m15s above shows what happens when a run drifts past it. Whether
shard 2 genuinely needs half an hour, or whether one suite is still spending most of it, is a
separate question from this issue and is worth measuring per suite rather than per shard.

[davidfarah2003]

The headline is resolved; the cause is not, and the margin is about six minutes

Raising the job limit to 40 minutes in 85d4d421 did what it was meant to. Shard 2 now completes:

run	outcome	duration
main @ ae9f6164, run 33220279945	success	33m54s
PR 908 @ b6d5bb19, run 33221141799 attempt 2	success	33m17s
PR 908 @ b6d5bb19, run 33221141799 attempt 1	failure	31m28s

So "has never completed" no longer holds, and the suites behind it do run. Two things are worth
recording before this is closed.

The margin is thin. A healthy shard 2 finishes about six minutes under its ceiling. That is not a
lot of headroom on a runner whose speed varies, and it means the next slowdown anywhere in the shard
puts this issue straight back.

smoke:opencode-events-release still costs seven minutes, and most of it is not work. In the run
above it held the shard from 00:10:50 to 00:17:49. From 00:12:59 onward it emitted this on a thirty
second cadence until the suite ended:

DrainingConnectionError: connection draining
    at NatsConnectionImpl._check (@nats-io/nats-core/build/src/nats.ts:90)
    at NatsConnectionImpl._resub (@nats-io/nats-core/build/src/nats.ts:164)
    at PushConsumerMessagesImpl.reset (@nats-io/jetstream/build/src/pushconsumer.ts)
    at IdleHeartbeatMonitor.maxOut (@nats-io/jetstream)
    at Timeout._onTimeout (@nats-io/nats-core/build/src/idleheartbeat_monitor.ts)

The last real assertion in that suite lands at 00:11:42. Everything after it is a JetStream push
consumer's idle heartbeat monitor firing every thirty seconds and trying to reset() (resubscribe)
on a connection that is already draining, throwing each time. The consumer's heartbeat monitor is not
torn down when the connection drains, so the suite spends roughly five of its seven minutes on a
timer that can only fail.

That is the same shape as the live job's own hang, which turned out to be a passing suite held open
by handles nobody released (#967, fixed in #970). Here it does not hang the shard, because the suite
does eventually end. It just makes shard 2 five minutes more expensive than the work it does, which
is most of the margin this issue is about.

Not a cause, but adjacent: attempt 1 above failed at smoke:jcode-provider-disconnect, not on a
timeout. That is filed separately as #971.

[davidfarah2003]

Closing on the teardown fix landing in #1030 (merged): presence and channel-registry kv.watch iterators are now stored and stopped in both clearConnectionScoped() and stop(), taking the DrainingConnectionError storm from 19 to 0 in the PR's live-broker proof. Together with the earlier describe-timeout raise this closes the two mechanisms this issue reproduced. The remaining observability wishes in the thread were explicitly deferred there and stay deferred.

[davidfarah2003]

Correcting my close: this thread had already established that the 40-minute raise (85d4d42) resolved only the headline, with the root cause open and ~6 minutes of margin. #1030's watch teardown plausibly recovers real margin (it removes the DrainingConnectionError storm during suite teardown), but that is a hypothesis until measured. Reopening until a post-#1030 main run shows shard 2's duration; if the margin has moved materially I will close with those numbers instead.

[davidfarah2003]

Controlled non-reproduction at exact current-main SHA dca9462423fb96c9ac2eea3155fd26195ff5cc29. The original deterministic-hang / never-ran premise no longer holds. Live bounded run: smoke:opencode-events-release passed 20/20, exit 0, in 423s, with 0 DrainingConnectionError lines. Public positive series: seven consecutive completed main shard-2 jobs all contain the suite's PASS marker and the full partition completion banner (105/420, one 105/419). Historical issue comments also corrected the premise from direct logs: identical code completed all 96 suites in 25-29 minutes; the old failures clustered at the former 30-minute job ceiling and varied with earlier-suite runtime. Current CI uses a 40-minute budget, and merged PR #1030 fixed endpoint watch teardown. ci-ok already requires the smoke matrix result to be success, so a cancelled shard fails the required aggregate. A uniform per-suite timeout still does not exist, but current evidence does not show a hanging suite or dark tail. Recommend bug, triage:not-reproduced, area:ci, severity:medium, and close as currently fixed. File any duration-aware balancing/per-suite containment follow-up separately with current acceptance criteria.