Skip to content

fix(memory): cancel a sub-call's request when its timeout fires - #412

Merged
plombeer31 merged 1 commit into
mainfrom
fix/memory-subcall-cancel
Sep 13, 2026
Merged

plombeer31 merged 1 commit into
mainfrom
fix/memory-subcall-cancel

Conversation

@plombeer31

Copy link
Copy Markdown
Collaborator

What

Before: every memory sub-call wrapper in bootstrap.ts (reflection, link generator, vote, query rewriter, distill) enforced its runner's timeout by racing the completion against an abort promise — without passing the signal into llmComplete. The runner gave up; the HTTP request kept running, still billing on a cloud provider and still holding a llama-server slot. An orphan that later failed also ran through the fallback chain like any live request, so it could trip a breaker or flip the sticky override long after nobody was waiting for it.

After: the five copies are one helper, abortableSubcall (src/runtime/abortable-subcall.ts), which forwards the signal into the request and keeps the race as a backstop for a provider that ignores it. Each wrapper sends exactly the fields it sent before.

Forwarding alone was not enough — two links dropped or misfiled the abort:

  • OpenAI-compatible unary requests went deaf after headers. openAiFetch unlinks the caller's signal once fetch resolves (it also opens streams, whose consumer owns the signal from then on). A server that sends headers before the body could not be cancelled. openAiPostJson now reads the body through a reader the signal cancels; without a signal it still uses res.json().
  • An aborted llama-server request classified as transport. It surfaces as a status: null LlamaServerError, and shouldAdvance treats that as an immediate provider-down signal — so forwarding the signal would have made every timed-out sub-call on a llama link flip the chain. The unary fallback seam now rethrows a request whose caller aborted as the signal's reason, which classifies cancelled and never advances — the rule OpenAiProvider.completeStream already applies on the streaming path.

Why

Memory sub-calls time out routinely on hosted reasoning models (measured on v0.6.0: reflection timed out 6 of 8 calls on z-ai/glm-5.3-flash, the rewriter 16 of 16 on moonshotai/kimi-k2.6). Each of those kept a request running to completion for a result nobody would read. A timeout should cancel the work, not just stop waiting for it.

Not touched: the streaming seam. Aborting a llama stream while it is still opening may have the same transport classification; it is not reachable from the memory sub-calls (all unary) and is left for a follow-up.

How it was verified

  • npm run lint clean
  • npx vitest run src/runtime/abortable-subcall.test.ts src/runtime/abortable-subcall.network.test.ts src/runtime/llm-fallback-seam.test.ts src/llm/provider/openai src/llm/fallback — 25 files / 411 tests green
  • abortable-subcall.test.ts: the inner request receives the caller's signal; an abort rejects promptly with an AbortError even when the inner promise never settles; an already-aborted signal sends nothing
  • abortable-subcall.network.test.ts: the real link-generator runner (150 ms timeout) → helper → fallback seam → real provider → a local server that never answers / answers headers then stalls, for both the OpenAI-compatible and llama-server clients, behind a two-link chain. Each case asserts the runner reports timeout, the server sees its socket close, the backup link is never called, and the partition has no override
  • vacuity: not forwarding the signal in the helper turns the network cases red; removing the seam's abort mapping turns the llama cases red (backup link called)

Every memory sub-call wrapper in bootstrap (reflection, link generator,
vote, query rewriter, distill) enforced its runner's timeout by racing
the completion against the abort signal without handing that signal to
llmComplete. The runner gave up while the HTTP request kept running:
still billing on a cloud provider, still holding a llama-server slot.

The five copies are replaced by abortableSubcall, which forwards the
signal into the request and keeps the race as a backstop for a provider
that ignores it. Each wrapper still sends exactly the fields it did.

Forwarding alone was not enough, for two reasons found on the way:

- OpenAI-compatible unary requests stopped listening to the signal once
  headers arrived (openAiFetch unlinks it because it also opens
  streams), so a provider that sends headers before the body could not
  be cancelled. openAiPostJson now reads the body through a reader the
  signal cancels.
- An aborted llama-server request surfaces as a status-null
  LlamaServerError, which classifies as transport and makes
  shouldAdvance switch providers immediately. The unary fallback seam
  now rethrows an aborted request as the signal's reason, so it
  classifies as cancelled and never trips a breaker or flips the
  override, the rule completeStream already applied.
@plombeer31
plombeer31 merged commit 126283c into main Sep 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant