Workers AI gemma-4: tool_choice 'required' fails open on long contexts; non-stream response discards generated tokens; named-function forcing leaves arguments unconstrained

[scott-edwards]

Surface: Workers AI serving for @cf/google/gemma-4-26b-a4b-it (REST /ai/run/... and the env.AI binding; observed identically through AI Gateway). Not a workers-ai-provider code issue — filing here because platform-behavior reports for Workers AI/AI Gateway live in this tracker (cf. #547); happy to re-route if there's a better channel.

Summary

Three related serving-side behaviors around tool_choice on @cf/google/gemma-4-26b-a4b-it:

1. tool_choice: "required" is advisory under long contexts. The model schema documents 'required' = must call a tool, and it does force a call on short prompts. On a real 61-message chat (~9.2k prompt tokens, thinking enabled), the model ignores it and emits free prose. Streaming surfaces that prose to the client as normal content deltas.

2. Non-stream response swallows the generated output when no tool call is parsed. Replaying the same long-context request with stream: false returns:

   {
     "choices": [{
       "finish_reason": "tool_calls",
       "message": { "content": "", "reasoning": null, "tool_calls": [] },
       "stop_reason": 106
     }],
     "usage": { "prompt_tokens": 9218, "completion_tokens": 317, ... }
   }

   317 completion tokens were generated and billed, but the response contains no content, no reasoning, an empty tool_calls array, and a contradictory finish_reason: "tool_calls". Whatever the right resolution for behavior (1) is, this response shape looks like the tool-call parser dropping the model's actual output on the floor; an error, or returning the text, would both be more truthful.

3. Named-function forcing constrains the envelope, not the arguments. With tool_choice: { "type": "function", "function": { "name": "read_skill_resource" } } on the same long context, the serving layer does force a call to the named function — but the model can emit free prose inside arguments (we received the model's conversational reply as the arguments string, not JSON matching the declared parameters schema). If guided decoding is in play for the function name, extending it to the parameters schema (or documenting that arguments are unconstrained) would help callers.

Repro sketch

Short context (forcing works — baseline):

POST /accounts/{account}/ai/run/@cf/google/gemma-4-26b-a4b-it
{
  "messages": [
    { "role": "system", "content": "You are a warm coach." },
    { "role": "user", "content": "That went really well! How do you think I did?" }
  ],
  "tools": [{ "type": "function", "function": { "name": "read_skill_resource", "description": "Read a bundled resource file.", "parameters": { "type": "object", "properties": { "name": { "type": "string" }, "path": { "type": "string" } }, "required": ["name", "path"] } } }],
  "tool_choice": "required",
  "max_tokens": 2000,
  "chat_template_kwargs": { "enable_thinking": true, "clear_thinking": true }
}
// → finish_reason "tool_calls", proper forced call. Reproduced twice.

Long context (forcing fails open): identical request shape, but messages is a realistic 61-message coaching conversation (~9.2k prompt tokens) ending in an assistant tool call + tool result. Same single tool, same tool_choice: "required":

• stream: true → plain prose streamed as content deltas, no tool call.
• stream: false → the swallowed-output response shown above (completion_tokens: 317, empty message).
• swap in the named-function tool_choice → forced call envelope, prose in arguments.

I can share exact request IDs / AI Gateway log IDs from our account (Science of People, efa870654588498c88cc1f2a563323fa) with Cloudflare folks privately if useful — the failing long-context request is fully captured there, including the cf-wai-tool-call IDs.

Why it matters

The AI SDK ecosystem maps forced tool choice onto these knobs (today workers-ai-provider maps {type:"tool"} → "required"; proposed fix is the named form — filed as #560). Agentic loops built on prepareStep tool forcing assume "must call a tool" is a hard constraint; on this model it fails open exactly in the conditions where agents need it most (long multi-turn transcripts), and the non-stream response shape hides that it happened.

[threepointone]

Cross-linking some related work, and clarifying which slice of this is now mitigated at the SDK layer vs. still needs a serving-side fix.

Client-side (workers-ai-provider) — done in #575 (refs #560):

Behavior (1) is the motivation for #560. workers-ai-provider previously mapped the AI SDK's forced single-tool choice (toolChoice: { type: "tool", toolName }) onto tool_choice: "required", which — as this issue documents — is advisory and fails open on long contexts. #575 switches that path to the documented named-function form tool_choice: { type: "function", function: { name } }, which the serving layer does enforce.

I reproduced the gemma-4 fail-open from this issue while testing: with "required", the forced call was dropped on long context for @cf/google/gemma-4-26b-a4b-it (and @cf/qwen/qwen3-30b-a3b-fp8), and even on short context for @cf/qwen/qwq-32b; the named-function form forced the call in every one of those cases. So for AI SDK users this sidesteps both (1) and the (2) non-stream swallow, because the forced-single-tool path no longer goes through "required" at all.

Two limits worth noting:

• This only covers forcing a specific tool ({ type: "tool", toolName }). A plain toolChoice: "required" (any-tool) still maps to "required" and still hits behavior (1) — there's no single tool to name.
• It can't help behavior (3): the named form forces the envelope, but the model can still emit prose into arguments. That's exactly what we see and flag as a known limitation.

Still serving-side (no provider-level fix possible):

• (2) the non-stream response that returns completion_tokens: 317 with content: "", tool_calls: [], and a contradictory finish_reason: "tool_calls" — the tokens are dropped before any client sees them.
• (3) arguments being unconstrained under named-function forcing (guided decoding apparently covers the function name but not the parameters schema).

Adjacent serving bug: #574 tracks a different model's different leak — gpt-oss (harmony format) serializes a forced tool call as JSON into content (with empty tool_calls, finish_reason: "stop"), which is distinct from gemma-4's empty-content case in (2) here. #575 also adds a conservative client-side salvage for that gpt-oss case, but the robust fix for both belongs in the serving/harmony parser.

Summary: #575 mitigates (1) for AI-SDK forced-single-tool calls; (2) and (3) remain open serving-side items on @cf/google/gemma-4-26b-a4b-it.