Audit ACP model and reasoning-effort capability mapping across agent adapters

[szysammie]

Summary

We found that the current BenchFlow ACP integration does not correctly map model selection and reasoning-effort configuration across versions of @agentclientprotocol/codex-acp.

The currently pinned codex-acp@0.0.45 is substantially behind the current adapter release and represents reasoning effort as part of the model ID:

<model>[<reasoning-effort>]

For example:

gpt-5.5[high]

By contrast, newer Codex ACP releases expose separate ACP session config options for model selection and reasoning effort:

model = gpt-5.6-sol
reasoning_effort = high

BenchFlow currently does not fully distinguish these two protocols.

Observed problems

1. Legacy Codex ACP does not receive the requested reasoning effort

For legacy Codex ACP, BenchFlow selects a model ID such as:

gpt-5.5[medium]

but does not translate:

--reasoning-effort high

into:

session/set_model("gpt-5.5[high]")

It then attempts to send the effort as a separate config option, which legacy Codex ACP does not expose.

2. Newer Codex ACP cannot accept legacy-formatted model IDs

Newer Codex ACP exposes a model config option, but it expects the public, effort-free model name:

gpt-5.6-sol

If BenchFlow reuses the legacy selection path and sends:

gpt-5.6-sol[medium]

through that config option, the adapter rejects the request as invalid.

Therefore, upgrading the ACP package alone is insufficient: BenchFlow must choose the model representation according to the connected agent's advertised ACP capabilities.

Requested change

Please audit all ACP-based agents, not only codex-acp, for equivalent version-dependent mismatches between:

• Public BenchFlow CLI arguments (--model, --reasoning-effort)
• ACP session capabilities advertised by the running adapter
• Model selection transport (session/set_model versus session/set_config_option)
• Reasoning-effort transport (model ID suffix versus a dedicated config option)

The issue may affect other adapters whose ACP protocol or configuration-option names have changed between releases.

Suggested behavior

Use capability-first dispatch based on the session/new response.

For each agent/version:

1. If the adapter exposes a model config option, send the public model name through that option.
2. If the adapter exposes an effort config option, send the requested effort through that option.
3. If the adapter instead exposes legacy model IDs in the form model[effort], encode the requested effort in session/set_model.
4. Fail with a clear compatibility error when a requested model or effort is not supported, rather than silently selecting a default effort.

This would allow BenchFlow to support both legacy and current ACP adapters without relying on version-specific assumptions in the registry.

[Jason-Tree]

Independent repro of §2 from a FrontierPhysics task submission, in case a concrete data point helps.

Environment: benchflow 0.6.7, --sandbox daytona, --agent codex-acp, OpenAI API key (verified working against /v1/responses directly).

Result: every rollout fails at session/set_model, with three retries each, for every model id tried:

RuntimeError: Failed to set model 'gpt-5.1-codex' via ACP for agent 'codex-acp': ACP error -32603: Internal error
RuntimeError: Failed to set model 'gpt-5-codex'   via ACP for agent 'codex-acp': ACP error -32603: Internal error
RuntimeError: Failed to set model 'gpt-5'         via ACP for agent 'codex-acp': ACP error -32603: Internal error

The credential is not the cause. Recorded token usage is zero on every attempt:

"metrics": {"n_tool_calls": 0, "n_prompts": 1},
"token_usage": {"input_tokens": 0.0, "output_tokens": 0.0, "total_tokens": 0.0},
"stop_condition": "agent_error"

and the same key succeeds directly against the Responses API (gpt-5 → HTTP 200; gpt-5-codex → 404 for this project, i.e. /v1/models listing is not project-scoped access).

Consistent with this issue's diagnosis. The registry entry for codex-acp declares acp_model_format: bare, so BenchFlow sends a bare id, while the pinned codex-acp@0.0.45 expects <model>[<reasoning-effort>]. We have not verified that --model "gpt-5[medium]" works, so treat that as a hypothesis rather than a confirmed workaround.

Why it may be worth prioritising: codex-acp is the agent named in FrontierPhysics AGENTS.md for the documented with/without-skills baseline commands, so a contributor following those instructions cannot produce baselines at all. Switching to --agent opencode --model openai/gpt-5 works on the same key and sandbox (reward 1.00, 10 tool calls), which is what we used instead.

Two small things that cost time and might be worth surfacing in the error text:

• the -32603 Internal error gives no indication that the model id format is the problem;
• repeated bench eval run invocations resume an existing job directory and skip completed tasks, so they return cached results in 0.0min rather than running again — --jobs-dir per run is needed for independent trials. That is easy to mistake for N successful runs.