Skip to content

fix(gemini): strip Google model prefix - #979

Open
kywch wants to merge 3 commits into
mainfrom
fix/975-gemini-prefix
Open

fix(gemini): strip Google model prefix#979
kywch wants to merge 3 commits into
mainfrom
fix/975-gemini-prefix

Conversation

@kywch

@kywch kywch commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Normalizes google/gemini-* to the bare Gemini model ID only when configuring the Gemini ACP agent.

Root cause

Gemini CLI expects a bare model name. BenchFlow retained the models.dev google/ prefix, causing Gemini CLI to construct an invalid models/google/gemini-* path.

Impact

Provider-qualified Gemini model IDs work with the Gemini ACP agent; provider/model agents retain their existing prefix behavior.

Validation

  • uv run python -m pytest tests/test_acp.py -k 'TestConnectAcpModelSelection'
  • uv run ruff check src/benchflow/acp/runtime.py tests/test_acp.py
  • git diff --check

Fixes #975.

@kywch
kywch deployed to pypi-internal-preview August 12, 2026 17:18 — with GitHub Actions Active
@kywch
kywch had a problem deploying to pypi-internal-preview August 12, 2026 17:24 — with GitHub Actions Failure
@kywch
kywch deployed to pypi-internal-preview August 12, 2026 17:30 — with GitHub Actions Active
@kywch
kywch marked this pull request as ready for review August 12, 2026 17:42
@kywch
kywch deployed to pypi-internal-preview August 12, 2026 17:43 — with GitHub Actions Active
@Galius5136

Copy link
Copy Markdown

I reproduced the model-prefix behavior from #975 locally and compared the merge-base (29c64e7), the isolated Gemini fix commit (771be99), and the current PR head (bc9a1c3).

On the merge-base, gemini + google/gemini-2.5-pro stays as google/gemini-2.5-pro. With the Gemini fix, it becomes gemini-2.5-pro, and the current PR head behaves identically to the isolated fix commit.

I also checked the surrounding non-regression cases: an already-bare Gemini model stays bare, OpenCode still maps gemini-2.5-pro to google/gemini-2.5-pro, and google/gemma-* is left unchanged for Gemini.

The ACP test suite passed on my setup (94 passed, 1 skipped), and the four files touched by the PR pass both ruff check and ruff format --check.

Everything behaved as expected in the offline path I tested. I didn't run a live Gemini CLI call because I don't have a GEMINI_API_KEY, so I didn't independently verify the bare model ID against the real Gemini CLI.

@kywch

kywch commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @Galius5136! I found other gemini issues as well, so I'd probably update on this thread.

@tulerfeng

tulerfeng commented Aug 20, 2026

Copy link
Copy Markdown

I have Gemini API credit available, so I picked up the piece @Galius5136
explicitly left open here — the live Gemini CLI call against the real provider.
Full write-up and the raw numbers are in #1030; summarising the part that
concerns this PR.

Setup: merge-base 29c64e7 vs this PR head bc9a1c3, with
@google/gemini-cli@0.42.0 (the registry pin) in a container, hitting the live
Google GenerateContent endpoint. The ACP modelId is produced by the real code
path from #975 and handed verbatim to the CLI.

The fix works for the reported case:

requested model tree ACP modelId sent live gemini-cli 0.42.0
google/gemini-3.1-flash-lite-preview 29c64e7 (before fix) google/gemini-3.1-flash-lite-preview 404 ModelNotFoundError
google/gemini-3.1-flash-lite-preview bc9a1c3 (this PR) gemini-3.1-flash-lite-preview 200 ok
gemini-3.5-flash-lite (already bare) both trees gemini-3.5-flash-lite 200 ok

tests/test_acp.py gives 94 passed / 1 skipped here, matching what @Galius5136
saw, and ruff check / ruff format --check are clean on all four touched
files. I also mutation-tested the new case: removing the two added lines makes
TestConnectAcpModelSelection::test_model_id_selection[gemini-google] fail with
Expected: set_model('gemini-3.1-flash-lite-preview') /
Actual: set_model('google/gemini-3.1-flash-lite-preview'), and restoring them
makes it pass, so the test does guard the fix.

One gap. The guard here is bare.startswith("google/gemini-"), so google/gemma-*
is still passed through prefixed and still 404s live:

requested model tree ACP modelId sent live gemini-cli 0.42.0
google/gemma-4-31b-it 29c64e7 (before fix) google/gemma-4-31b-it 404 ModelNotFoundError
google/gemma-4-31b-it bc9a1c3 (this PR) google/gemma-4-31b-it 404 ModelNotFoundError

Bare gemma-4-31b-it returns 200 from the same CLI, so this is the same bug in a
different model family rather than an unsupported model.
_MODELSDEV_PROVIDER_HEURISTICS already maps ("gemma", "google"), so BenchFlow
itself produces that shape in the reverse direction. #1030 matches on google/
instead and normalizes both families; I verified it returns 200 for
google/gemma-4-31b-it.

Worth noting the two PRs are not in conflict: applying this PR's
tests/test_acp.py diff on top of #1030 passes (14 passed in
TestConnectAcpModelSelection). The integration-style assertion here is the
stronger guard of the two, so the combination that seems best is #1030's
normalization with this PR's test.

Separately, bc9a1c3 ("fix(judge): allow scratch edits in native titles") is
unrelated to the Gemini prefix and touches tests/integration/agent_judge.py.
It looks fine on its own — tests/test_judge_robustness.py gives 55 passed —
but it might be easier to review as its own PR.

All of the above is end-to-end validation against the live Gemini API rather
than an offline reproduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemini ACP passes google/ provider prefixes into Gemini CLI model IDs

3 participants