Skip to content

fix(llm): retry a sub-call without response_format when the endpoint refuses structured outputs - #417

Merged
plombeer31 merged 1 commit into
mainfrom
fix/subcall-response-format-fallback
Sep 13, 2026
Merged

fix(llm): retry a sub-call without response_format when the endpoint refuses structured outputs#417
plombeer31 merged 1 commit into
mainfrom
fix/subcall-response-format-fallback

Conversation

@plombeer31

Copy link
Copy Markdown
Collaborator

What

Before: cloud memory sub-calls (query rewriter, link generator, vote runner, consolidation distill) always sent OpenAI Structured Outputs (response_format: {type: "json_schema"}). An endpoint with no structured-output support refused every one of them — OpenRouter pinned to Z.AI's own endpoint for z-ai/glm-5.3-flash answers 404 No endpoints found after its "Filter by Parameters" routing step; direct vendor APIs without json_schema answer 400. The sub-call failed forever on that provider, and because every cloud OpenAiHttpError classifies transport, each refusal also ran through the provider fallback chain.

After: when a request carrying responseFormat is refused as a structured-output refusal, OpenAiProvider.complete retries it once on the same provider without response_format, returns that result, and remembers the (provider id, model) pair for the rest of the process, so later sub-calls to it skip response_format up front. One warn line the first time. A handled refusal never reaches runWithFallback, so it never advances the chain or trips its breaker. Streaming main-turn requests never carry responseFormat and are untouched.

Why

The sub-call parsers never needed Structured Outputs to get an answer: all four (rewriter, link, vote, distill) try JSON first and fall back to the text format their prompt asks for. Losing decode-time enforcement degrades an answer; a permanent 404 loses it.

The detector is deliberately narrow — an OpenAiHttpError that is not our own timeout, and:

  • 404 whose body says No endpoints found together with requested parameters, a feature word (response_format / json_schema / json_object / structured outputs), or a "Filter by Parameters" funnel step that actually dropped endpoints (the step appears in every OpenRouter funnel, so its count is checked when readable);
  • 400 / 422 whose body names a feature word — except Alibaba's must contain the word 'json' (a prompt problem on an endpoint that does support structured outputs; fix(memory): structured-output sub-calls that OpenAI strict mode and Qwen reject #414 fixes the prompt) and request-size rejections.

401 / 402 / 403 / 429 / 5xx, network failures and untyped errors are never refusals. The pair is remembered only after the retry succeeds; a retry that fails propagates its own error and remembers nothing.

How it was verified

  • npm run lint clean
  • npx vitest run src/llm/provider/openai src/runtime/llm-fallback-seam-structured-output.test.ts src/runtime/llm-fallback-seam.test.ts src/llm/fallback — 26 files / 435 tests green
  • new tests: a 404 refusal sends a second request identical except for response_format and returns its success; at the seam advanceFrom is never called and the local link is never reached; the next call skips response_format with no failed round trip; memory is per provider id and per model; invalid-key, context-length and json-word 400s propagate after a single request; no retry without responseFormat or with tools; a failed retry remembers nothing; completeStream never retries; a control test shows the same 404 without responseFormat still advances the chain
  • vacuity: forcing the detector to return false turns 11 of the 34 new tests red across all three files; removing the must contain the word json exclusion turns 2 red (that 400 must not be read as a refusal)
  • live, this branch's build, z-ai/glm-5.3-flash pinned to Z.AI's endpoint, same scripted 8-turn session as the baseline:
v0.6.0 this branch
rewriter outcomes failed 18 ok 11, timeout 7, failed 0
vote runner 8 fired / 8 failed 8 fired / 8 ok, votes applied
"No endpoints found" log lines 26 0
structured-output warn lines 1
provider switches 0

Merge note: #413 also edits OpenAiProvider.complete (it threads providerPreferences into the buildOpenAiChatBody call this PR moves into a closure) — whichever lands second needs a trivial rebase. #414's JSON instruction is only added when response_format is attached, so the stripped retry sends the prompt unchanged.

The 7 remaining rewriter timeouts are the 3 s rewriter budget on a thinking model — before, the call failed in ~74 ms; now it actually waits for GLM. #416 raises that budget to 10 s and runs the rewriter once per turn.

…refuses structured outputs

Memory sub-calls (query rewriter, link generator, vote runner, distill)
send OpenAI Structured Outputs. Endpoints without the feature refuse the
request: OpenRouter pinned to Z.AI's endpoint for z-ai/glm-5.3-flash
answers 404 "No endpoints found" because its "Filter by Parameters"
routing step drops that endpoint, and vendor APIs without json_schema
answer 400/422 naming the field. The sub-call then failed on every run
(the rewriter failed 18 of 18 times in a live session), and because every
cloud OpenAiHttpError classifies as transport, each refusal also advanced
the provider fallback chain.

OpenAiProvider.complete now recognises that refusal narrowly and sends
the same body once more without response_format. The prompts still ask
for their text formats and every parser reads them when the reply is not
JSON, so only decode enforcement is lost. The retry happens inside the
provider, so a handled refusal never reaches runWithFallback. The
provider and model pair is remembered for the process once the stripped
send is accepted, later sub-calls skip the field up front, and the first
time logs one warn line.

Not treated as refusals: 401/402/403/429/5xx, timeouts, network failures,
size rejections, and the "messages must contain the word json" 400, which
is a prompt problem on an endpoint that does support the feature.
Streamed requests never carry response_format and are unchanged.
@plombeer31
plombeer31 merged commit f9922b5 into main Sep 13, 2026
2 checks passed
plombeer31 added a commit that referenced this pull request Sep 13, 2026
Integrates #411#421 for v0.6.1, including the two rc-only reconciliations (#413×#417, #414×#417).
Bartok9 pushed a commit to Bartok9/atomic-agent that referenced this pull request Sep 14, 2026
 structured-output fallback

AtomicBot-ai#417 moved complete()'s body build into a closure handed to
sendWithStructuredOutputFallback; thread this.providerPreferences through
that closure (after strictTools) so unary sub-calls keep OpenRouter
provider routing on both the first send and the prompt-only retry.
Bartok9 pushed a commit to Bartok9/atomic-agent that referenced this pull request Sep 14, 2026
…mpt-only retry test

AtomicBot-ai#414 appends a JSON instruction to the prompt of any body that carries
response_format; AtomicBot-ai#417's prompt-only retry drops response_format, so its
prompt is the caller's own. The retry test asserted the two bodies were
identical apart from response_format; assert the JSON mention on the
first send and the original prompt on the retry instead. No runtime
change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant