fix: two provider failures that arrive looking like successes

[kenforthewin]

Two provider failures that reach us looking like successes, found from one bug report. Neither raises an error, neither is retried, and both end with work silently missing — an atom with no tags, a wiki article that never updated, duplicate tags that never merged.

---

1. A padded 200 is a truncated transfer, not a parse failure

The report: auto-tagging failing on google/gemma-4-26b-a4b-it with EOF while parsing a value at line 233 column 0, a blank body preview, and nothing at all in the user's OpenRouter dashboard.

What's on the wire

OpenRouter fronts slow upstreams with a Cloudflare Worker that must return a response object immediately, so it commits 200 OK + Transfer-Encoding: chunked before the upstream has produced anything, then keeps the connection alive with insignificant JSON whitespace:

t=0.320s   HTTP/1.1 200 OK   Transfer-Encoding: chunked   X-Generation-Id: gen-…
t=0.487s   16 bytes of whitespace
t=0.912s   16 bytes of whitespace        <- every ~425ms
   …
t=4.721s   the actual JSON, all at once

Whitespace works because RFC 8259 permits it before the top-level value — a compliant parser cannot see it. Once that status is committed it can't be retracted, so a late failure can only be an error object in the body or an ended stream. Ending it leaves a complete, well-formed 200 whose entire body is padding.

That's a transport failure wearing a parse error's clothes. We classified it permanent, so auto-tagging spent one attempt of a three-attempt budget and dropped the atom. The identical event cut a moment earlier arrives as Network and is retried — only the politely-terminated form was fatal.

Evidence

• Padding on every non-streaming call observed (15/15), 6–112 newlines.
• The JSON payload contains zero newlines, so every line in that serde error was padding. line 233 = 232 padding newlines.
• Cadence 4.66 newlines/sec, stable from 4.5s calls to an 8-minute one ⇒ 232 newlines implies ~49.8s. The report's timestamps span 49.13s.
• A real hang delivered 12288 bytes / 2235 newlines / no JSON over 8 minutes.
• Not our 300s timeout: that would have cut at ~1400 newlines.

Why their dashboard was clean. Three deliberately abandoned calls came back finish_reason: stop, cancelled: false, fully billed. A generation record models the upstream generation, not delivery, so this failure class is invisible there and lands as a billable success. An absence of errors is a prediction of this diagnosis, not a contradiction. The failing body also carries no provider field, so the report's upstream_provider="DeepInfra" belongs to the previous, successful call — no upstream is implicated.

The fix

decode_error uses serde_json's Category::Eof as the line: input ran out (empty, all padding, or cut mid-JSON) ⇒ truncated transfer ⇒ retryable Network + Transient for schedulers. Syntax/Data mean bytes arrived and were wrong ⇒ still a permanent ParseError. Applied to both chat and both embedding paths, plus:

• Both streaming parsers returned Ok with empty content when a stream carried no payload, persisting silence as if the model chose it. Guarded on "no SSE payload and no terminator", so a legitimately empty completion is untouched.
• The chat path had no handling for a 200 carrying an error object — the gateway's other exit. choices is required, so it landed as a permanent parse failure. The embedding path's handling is now shared via upstream_error.
• x-generation-id is captured into the error and log line. It arrives in the headers, so it survives the failures the body doesn't, and resolves via GET /api/v1/generation?id=… to provider, timings, finish reason and cost.
• Previews are no longer logged raw — that's why the field was blank. Whitespace bodies are described (<2552 bytes of gateway padding, 232 newlines, no payload>), everything else escaped.
• Tagging sent no max_tokens at all (with_params replaced the struct carrying the default). This changes routing — OpenRouter filters endpoints by max_completion_tokens, so an absent cap widens the pool and an oversized one empties it (pinned to DeepInfra: 32000 → 404 No endpoints found, 8192 → serves). call_structured now guarantees a cap; tagging sets 8192.

---

2. Generative structured calls come off wire-level schema enforcement

Chasing the first bug surfaced DeepInfra returning {"tags": []} for content every other endpoint tagged fine — 8 completion tokens, finish_reason: stop, zero reasoning tokens, no error signal anywhere.

It generalises by the shape of the call, not the endpoint. Where the model decides how much to emit and an empty result is legal, a constrained decoder can take the shortest grammar-valid path and return nothing. That parses, satisfies the schema, fires no fallback, and is indistinguishable from a considered "nothing to do".

Measurements

Pinned to DeepInfra, gemma-4-26b-a4b-it, 6 calls per cell, on inputs whose correct answer is unambiguously non-empty (planted duplicates, planted new source material):

call	wire-level	prompt-only
tag extraction	5/6 empty	0/6
tag merge	2/6 empty	0/6
wiki section ops	3/6 no-op	0/6
tag consolidation	0/6	0/6

Pinning matters: an unpinned run showed almost none of this, because OpenRouter routes around the weak endpoint most of the time. That's also why it reaches users as rare, unreproducible gaps rather than as a bug.

The trade is a decoding guarantee for the tolerant parser, and the parser was measured never to be exercised — across gpt-5-nano, gemma-4-26b-a4b-it, gpt-5-mini and glm-5.2, all 24 prompt-mode replies parsed directly, no fences, no surrounding prose.

The change

SchemaEnforcement makes the transport explicit — Strict and Lenient send response_format, PromptOnly sends none. with_strict remains shorthand over the two wire-level modes so it can't silently reach a different transport. One policy constant, GENERATIVE_CALL_ENFORCEMENT, covers all five generative call sites.

Consolidation never degraded and moves anyway: one rule beats four, prompt-only measured zero cost, and "didn't collapse on one input" isn't immunity. Closed-shape calls are explicitly out of scope — one enum choice, fixed-arity object — where there's no volume for a decoder to shorten and constrained decoding is free.

This also corrects a claim in call_long_form_markdown's docs that short structured calls "genuinely benefit from constrained decoding". True of shaping, not of generating.

The instruction had to be fixed first

Prompt-only originally stated the contract as "reply with a JSON object matching this schema" followed by the JSON Schema. A schema is itself JSON, so that has a literal reading and small models take it: llama3.2 returned the schema, answer nested under properties, in half of replies.

schema_instruction now leads with a concrete example derived from the schema — leaves replaced by <placeholder> text from each field's description, enums rendered as their alternatives — keeping the schema behind it for constraints an example can't express. Placeholders rather than invented values on purpose: invented values get copied, and for an enum whose first variant is the do-nothing case (wiki_update_section_ops) a "realistic" example would point straight at the degenerate answer.

llama3.2, no grammar, first-attempt valid:
  schema document only    3–4 of 8
  example only            8 of 8
  example + schema        8 of 8

This improves every caller, not just the ones that moved: the prompt-based fallback is where wire-level calls land when a primary parse fails.

---

Testing

• 535 tests pass across 11 suites; workspace checks clean.
• Clippy and rustfmt verified per file against main — zero new diffs. structured.rs carries 8 rustfmt diffs and wiki/mod.rs one clippy warning that both pre-date this branch; left alone rather than buried in the diff.
• gateway_truncated_response_tests.rs — 5 tests built from captured wire bytes and real chunked framing, including the padding unit verbatim. Covers: padded body is transient; both framings agree; a payload-less stream fails instead of returning empty; a padded first attempt recovers on retry; every structured call sends a cap.
• Unit tests pin that PromptOnly puts no response_format on the wire, states the shape before the schema, never names a single enum variant, and absorbs a fenced reply without spending the fallback.
• Live end-to-end across all three provider families: gemma-4-26b-a4b-it and gpt-5-nano via OpenRouter, llama3.2 via native Ollama (8/8 first-attempt valid), and the OpenAI-compatible provider against Ollama's /v1 surface (6/6, no empty extractions).

One test-side consequence worth knowing: MockAiServer routed by wire schema name, so removing response_format made 8 tests fall through to {}. It now sniffs the schema stated in the prompt, with consolidation checked before extraction since both carry parent_name.

Not addressed

• The 300s client timeout is untouched — nothing in the evidence implicates it.
• What tipped that specific request over is unknown and not determinable from the log, which is exactly what capturing the generation id fixes for next time.
• An empty extraction still records tagging_status = complete, so if a provider does return nothing the atom is silently untagged and never retried. Change 2 removes the cause we measured, not the blind spot. Worth a follow-up that records empties distinctly — tagging_status is plain TEXT with no CHECK constraint, so no migration needed.

🤖 Generated with Claude Code