fix(memory): cancel a sub-call's request when its timeout fires - #412
Merged
Merged
Conversation
Every memory sub-call wrapper in bootstrap (reflection, link generator, vote, query rewriter, distill) enforced its runner's timeout by racing the completion against the abort signal without handing that signal to llmComplete. The runner gave up while the HTTP request kept running: still billing on a cloud provider, still holding a llama-server slot. The five copies are replaced by abortableSubcall, which forwards the signal into the request and keeps the race as a backstop for a provider that ignores it. Each wrapper still sends exactly the fields it did. Forwarding alone was not enough, for two reasons found on the way: - OpenAI-compatible unary requests stopped listening to the signal once headers arrived (openAiFetch unlinks it because it also opens streams), so a provider that sends headers before the body could not be cancelled. openAiPostJson now reads the body through a reader the signal cancels. - An aborted llama-server request surfaces as a status-null LlamaServerError, which classifies as transport and makes shouldAdvance switch providers immediately. The unary fallback seam now rethrows an aborted request as the signal's reason, so it classifies as cancelled and never trips a breaker or flips the override, the rule completeStream already applied.
This was referenced Sep 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Before: every memory sub-call wrapper in
bootstrap.ts(reflection, link generator, vote, query rewriter, distill) enforced its runner's timeout by racing the completion against an abort promise — without passing the signal intollmComplete. The runner gave up; the HTTP request kept running, still billing on a cloud provider and still holding a llama-server slot. An orphan that later failed also ran through the fallback chain like any live request, so it could trip a breaker or flip the sticky override long after nobody was waiting for it.After: the five copies are one helper,
abortableSubcall(src/runtime/abortable-subcall.ts), which forwards the signal into the request and keeps the race as a backstop for a provider that ignores it. Each wrapper sends exactly the fields it sent before.Forwarding alone was not enough — two links dropped or misfiled the abort:
openAiFetchunlinks the caller's signal oncefetchresolves (it also opens streams, whose consumer owns the signal from then on). A server that sends headers before the body could not be cancelled.openAiPostJsonnow reads the body through a reader the signal cancels; without a signal it still usesres.json().transport. It surfaces as astatus: nullLlamaServerError, andshouldAdvancetreats that as an immediate provider-down signal — so forwarding the signal would have made every timed-out sub-call on a llama link flip the chain. The unary fallback seam now rethrows a request whose caller aborted as the signal's reason, which classifiescancelledand never advances — the ruleOpenAiProvider.completeStreamalready applies on the streaming path.Why
Memory sub-calls time out routinely on hosted reasoning models (measured on v0.6.0: reflection timed out 6 of 8 calls on
z-ai/glm-5.3-flash, the rewriter 16 of 16 onmoonshotai/kimi-k2.6). Each of those kept a request running to completion for a result nobody would read. A timeout should cancel the work, not just stop waiting for it.Not touched: the streaming seam. Aborting a llama stream while it is still opening may have the same
transportclassification; it is not reachable from the memory sub-calls (all unary) and is left for a follow-up.How it was verified
npm run lintcleannpx vitest run src/runtime/abortable-subcall.test.ts src/runtime/abortable-subcall.network.test.ts src/runtime/llm-fallback-seam.test.ts src/llm/provider/openai src/llm/fallback— 25 files / 411 tests greenabortable-subcall.test.ts: the inner request receives the caller's signal; an abort rejects promptly with anAbortErroreven when the inner promise never settles; an already-aborted signal sends nothingabortable-subcall.network.test.ts: the real link-generator runner (150 ms timeout) → helper → fallback seam → real provider → a local server that never answers / answers headers then stalls, for both the OpenAI-compatible and llama-server clients, behind a two-link chain. Each case asserts the runner reportstimeout, the server sees its socket close, the backup link is never called, and the partition has no override