Repository navigation
feat(ai,agent,coding): visible model retries and interrupted-stream recovery - #3
Merged
Merged
Conversation
…stalled A stream that dies before its first event is replayed by the model middleware, but nothing reported it: the wait happened silently, and the only way to notice was that the turn looked slow. Announce every re-attempt on the model stream (ai.StreamRetry carrying the attempt about to start, its backoff and a short cause), project it to the model.retry Coding event, keep it as live reducer state, and render it as the activity line "Retrying… · attempt x/y · in Ns · reason". The notice is progress rather than output: Collect ignores it, it never counts as produced content, it cannot close an open text or reasoning part, and it is not durable state — the next event of any kind ends the wait.
A mid-stream truncation (io.ErrUnexpectedEOF from a dropped SSE body) failed the whole interaction even though the turn contributed nothing: no message was committed and no tool ran, so replaying it could neither duplicate content nor repeat an effect. WithStreamRecovery re-issues such a turn, bounded, after emitting CandidateDiscarded so a frontend drops the partial output it already rendered. That retraction is why the loop owns this case and the model middleware does not; a failure that produced nothing stays with the middleware, so the two budgets never stack on one failure. Coding runs (interaction, subagent, team worker, draft and proposal agents) share one policy: ten re-issues, doubling from two seconds up to thirty, which covers a few minutes of outage before reporting it.
… frontends The retry behavior now follows what Claude Code and Grok Build do, with the numbers taken from their documentation, their binary, and live retry payloads: - Coding runs get ten retries for a request that produced nothing (was five), keeping the 500ms base, full jitter, the 30s cap, and a provider Retry-After. - Certificate validation failures are no longer retryable: the caller has to fix them, so they surface on the first attempt, while transient TLS conditions such as a handshake timeout stay retryable. - The notice counts retries rather than tries: attempt is the 1-based retry ordinal and max_retries is the budget, mirroring the fields Grok Build pushes to its frontends; the TUI reads "retry 2/10". - The activity row reports a silent model — "Waiting for the model… · no data for 25s" — once an open turn has carried no model progress for twenty seconds, so a gateway that buffers a response no longer looks like a slow one.
- Give the parent-stream progress switch a default branch, which is how this repository keeps event-type switches forward compatible. - Assert the rejected retry payloads with require, so the table stops at the first failure instead of reporting once per row. - Drop the unused streamRecovery.enabled helper; the loop already spells the same condition out as a remaining-budget check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A model call that dies mid-stream used to kill the whole interaction, and the retries that did
happen were invisible. This branch makes both visible and survivable, following the strategies
Claude Code and Grok Build actually ship.
1. Model retries are announced (
ai,internal/coding, TUI)ai.StreamRetrycarries aRetryNotice{Attempt, MaxRetries, Delay, Reason}; the retry middlewareemits one per replay, and
Collectignores it, so a notice is never counted as output.model.retry, kept as live reducer state (State.Retry) that the next event ofany kind clears; it is not durable state.
Retrying… · retry 2/10 · in 4s · connection error, with a countdowndriven by the existing activity clock.
2. A turn whose stream broke after output is re-issued (
agent,internal/coding)WithStreamRecovery(attempts, base, maxDelay)re-issues such a turn after emittingCandidateDiscarded, so a frontend drops the partial output it already rendered. Only the loop canretract what a consumer has seen, which is why this is not the middleware's job; a failure that
produced nothing stays with the middleware, so the two budgets never stack on one failure. Nothing
from a failed attempt is committed and no tool ran, so a replay cannot duplicate content or repeat
an effect. Coding runs share one policy: ten re-issues, doubling from 2s, capped at 30s.
3. Budgets and counters aligned with the shipped frontends (
ai,coding/model, TUI)the 30s cap, and a provider
Retry-After.attempt/max_retries), mirroring theretry_statepayloads Grok Build pushes to its UI.
Waiting for the model… · no data for 25sreplaces the thinking row once an open turn has carriedno model progress for twenty seconds, so a gateway that buffers no longer looks like a slow model.
Research behind the numbers
Recorded in
.trellis/tasks/10-07-stream-retry-recovery/research/provider-retry-strategies.md:CLAUDE_CODE_MAX_RETRIESdefault 10 (capped at 15), a 20s no-data banner, byte/stream watchdogs,a first-byte deadline retry once, and certificate failures reported on the first attempt.
retry_stateupdate; a real transcript from this machine shows{"attempt":1,"max_retries":15,"reason":"reqwest error stream: Transport error: error decoding response body","error_type":"http"}— the same truncated-SSE class this branch recovers from —plus
models.max_retries/rate_limit_retry_threshold/subagent_rate_limit_max_attemptsknobs and
Retrying (attempt n/m)in the UI.Verification
go test ./...green (89 packages) withTERM=xterm-256color;go vet ./...clean;gofmtclean.non-retryable paths, an end-to-end runtime test (provider truncates after a delta → the run
recovers, reports one notice, discards the candidate, commits the re-issued answer), reducer
lifecycle including the batched delta path, projector mapping, and the TUI retry/stall rows.
golangci-lintcould not run here: the installed binary was built with Go 1.26.4 and panics onthe local Go 1.27.1 standard library, including on untouched packages.
Out of scope
A transcript marker for discarded partial text, journaling retry notices, config knobs for the
budgets, and an aborting idle watchdog (the stall row reports silence without aborting).