fix(litellm): bind host proxy to 0.0.0.0 when the docker gateway is unbindable

[manunicholasjacob]

Summary

Every bench eval run --sandbox docker fails on Docker Desktop's WSL2/Hyper-V backend. The host LiteLLM proxy tries to bind the docker bridge gateway directly, but under Docker Desktop that address belongs to the Docker VM rather than to the distro benchflow runs in, so the bind fails and the run aborts before the agent starts.

benchflow already handles this network shape correctly — it is the macOS path (bind 0.0.0.0, advertise host.docker.internal). The bug is only in how the path is chosen: the code assumes any syntactically valid IPv4 gateway is bindable.

Reproduction

bench eval run \
  --tasks-dir ~/skillsbench/tasks/offer-letter-generator \
  --agent gemini --model gemini-flash-lite-latest \
  --sandbox docker --skill-mode no-skill --jobs-dir /tmp/out

Environment: Windows 11, Docker Desktop 4.85.0 (WSL2 backend), Ubuntu 26.04, Docker 29.6.2. The daemon is reachable from the distro and images build fine.

Actual result

ERROR:    [Errno 99] error while attempting to bind on address ('172.17.0.1', 41495):
          cannot assign requested address
RuntimeError: LiteLLM proxy failed to start for model '...':
              LiteLLM exited before becoming healthy.
[ERR] offer-letter-generator (tools=0) (LiteLLM proxy failed to start for model...)
Job complete: 0/1 (0.0%), errors=1

Reproduces on every task, with n_tool_calls=0 — the agent never runs.

Cause

In src/benchflow/providers/litellm_runtime.py:

• _docker_host_address() shells out to docker network inspect bridge and gets 172.17.0.1. That is correct for the daemon, which under Docker Desktop lives in its own VM.
• _host_bind_address() validates it with socket.inet_aton(), which only checks that the string parses as IPv4, then returns it as the bind address. 172.17.0.1 is not configured on any interface inside the WSL distro, so bind() returns EADDRNOTAVAIL.

The existing fallback to 0.0.0.0 only triggers when the gateway is a hostname (host.docker.internal, as on macOS), so this configuration never reaches it.

Fix

Probe whether the address can actually be bound rather than only that it parses, and when it cannot, take the path already used for macOS. _agent_endpoint_for_environment is updated to match, so the container is advertised host.docker.internal rather than a gateway it also cannot reach.

Behaviour after the fix, same machine:

detected gateway : 172.17.0.1
bindable locally : False
bind address     : 0.0.0.0
agent calls      : http://host.docker.internal:41495
health check     : http://127.0.0.1:41495

The proxy then starts and the container reaches it, verified end to end by a task completing with reward 1.0 and errors=0 on the same setup that previously failed on every run.

No behaviour change on native Linux, where the gateway is bindable and the existing branch is taken.

Tests

tests/test_litellm_hardening.py:

• test_host_bind_address_docker_unbindable_gateway_falls_back — new, covers the Docker Desktop case.
• test_agent_endpoint_docker_unbindable_gateway_advertises_host_name — new, covers the endpoint the container is told to call.
• test_host_bind_address_docker_uses_bridge_ip — existing, now pins _address_is_local explicitly. Without this it would depend on whether the CI runner happens to have docker0 configured, which would make it pass on Linux CI and fail elsewhere. The original intent (a bindable gateway is used directly) is preserved.

uv run --with pytest --with pytest-asyncio python -m pytest tests/test_litellm_hardening.py → 53 passed.

Note for anyone reproducing: without pytest-asyncio installed, the async tests in this file surface as PytestUnknownMarkWarning and report as failures rather than being skipped.

Two smaller things found alongside

Happy to open these separately if useful, not included here to keep the diff focused:

1. The gemini agent's default_model is gemini-2.5-flash, which now returns 404 ... This model is no longer available to new users for recently issued AI Studio API keys.
2. bench eval run --help documents --sandbox as accepting docker, daytona, agentcore, while the README also mentions Modal and Apple Container.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.