Feature/n agent support

[Raandom]

Motivation

CooperBench has so far measured how well two agents cooperate on a shared codebase. Many of the questions the benchmark is built for — how coordination overhead scales, how merge complexity compounds, whether teams degrade gracefully — only become visible with N > 2 agents. The runner side (runner/team.py, runner/coop.py) was already written N-generically, but everything around it assumed exactly two agents: task discovery destructured 2-tuples, the eval harness tested exactly two patches, the GCP batch pipeline had feature1/feature2 hardwired end to end, and the dataset had no N-tuple task definitions. This PR removes that ceiling across the whole pipeline: dataset → task discovery → evaluation → reporting.

What is implemented

Dataset & task discovery

• Subset files can declare "groups" (N-element feature tuples) alongside the legacy "pairs"; discover_tasks() validates whole groups instead of destructuring 2-tuples (runner/tasks.py).
• scripts/check_gold_conflicts.py generalises the pairwise gold-conflict checker to arbitrary group sizes via --group-size N, and fixes two pre-existing Modal bugs (container keep-alive, double-terminate).

N-way evaluation (eval/sandbox.py)

• test_merged_n(): sets up one git branch per agent, applies each agent's patch, fold-merges them sequentially (agent2 into agent1, then agent3, …), and runs every feature's test suite against the merged tree. Keeps the identical-patch short-circuit and the lead-alone fallback from the 2-agent path.
• test_solo_n(): one agent's single patch tested against all N feature suites.
• Both return a features dict keyed by feature id plus all_passed, and dual-write the legacy feature1/feature2/both_passed keys when N = 2.

Eval routing (eval/evaluate.py)

• team (any N) and coop/solo with N > 2 route to the N-generic functions; 2-agent coop/solo keep the exact legacy test_merged()/test_solo() paths, so existing results and eval.json schemas are byte-compatible.
• Fixed a latent bug where a passing N > 2 team eval was counted as a failure in multi-run progress mode (only both_passed was checked).

GCP batch backend (eval/backends/gcp.py)

• EvalTask/EvalResult gain N-generic fields (feature_ids, patches, tests_patches, features_passed, all_passed) with __post_init__ normalisation, so legacy 2-field construction keeps working.
• The batch VM script loops downloads over N_FEATURES; the in-container script builds one branch per agent, fold-merges sequentially (keeping the batch path's per-step union-merge fallback), and runs all N suites. The former "skip team runs with > 2 agents" guard is removed.

Reporting — runner/core.py and the eval progress display accept all_passed alongside both_passed, and batch eval.json now carries features_result/all_passed with legacy keys dual-written at N = 2.

Support matrix

Setting	Run (agents)	Eval: docker	Eval: modal	Eval: GCP batch
team	✅ any N	✅ test_merged_n	✅	✅ (skip removed)
coop	✅ any N	✅ N=2 legacy, N>2 test_merged_n	✅	✅
solo	✅ any N	✅ N=2 legacy, N>2 test_solo_n	✅	✅

N = 2 behaviour is unchanged in every cell (legacy code paths and result schemas are preserved exactly).

Benchmarks run with this feature

Gold-patch merge sweeps over the full dataset (via Modal, using the generalised check_gold_conflicts.py) to characterise N-way conflict structure and curate evaluation subsets:

Sweep	Groups analysed	Conflicts	Clean merges	Errors	Wall clock
3-tuples (gold_conflict_report_3tuples*.json)	1,409	594	20	795	~8 min
4-tuples (gold_conflict_report_4tuples.json)	2,219	988	5	1,226	~12 min

From these sweeps, four curated subsets are checked in for N-agent runs (dataset/subsets/):

• team3_clean — 20 groups (all 20 clean-merging 3-tuples) across 4 tasks
• team3_slight — 50 lightly-conflicting 3-tuples across 4 tasks
• team4_clean — 5 groups (all 5 clean-merging 4-tuples)
• team4_slight — 50 lightly-conflicting 4-tuples across 3 tasks

The steep drop in clean merges (20/1,409 at N=3, 5/2,219 at N=4) is itself the headline motivation: multi-way integration is drastically harder than pairwise, which is exactly the regime this PR makes measurable with agents.

Caveat: the sweeps check textual mergeability only — check_gold_conflicts.py attempts the fold merge but does not run the feature test suites on the merged tree. So "clean merge" means git merged without conflicts, not test-verified compatibility; the *_clean subsets are curated by mergeability. Semantic verification (running all N suites on the merged gold patches via test_merged_n) is a pending follow-up — at eval time, however, test_merged_n/test_solo_n always run every feature's suite on the merged tree, so agent results are test-verified.

Agent baseline: Qwen3.5-9B on all four group subsets

Full end-to-end run of the team setting over all 125 groups (430 agent trajectories, ~10h wall clock): mini_swe_agent_v2, Qwen/Qwen3.5-9B (vLLM on Modal H100, 32k ctx), default team harness with --git, concurrency 3, evaluated with the new N-way docker eval.

Subset	Groups	Group pass	Features passed individually	Non-empty submissions	Agent exits (Submitted / LimitsExceeded / Error)
team3_clean	20	0	3/60	4/60	20 / 29 / 11
team3_slight	50	0	11/150	28/150	87 / 39 / 24
team4_clean	5	0	2/20	4/20	9 / 4 / 7
team4_slight	50	0	8/200	14/200	97 / 32 / 71
total	125	0 (0.0%)	24/430 (5.6%)	50/430 (11.6%)	213 / 104 / 113

Reference point: the same model at N=2 (prior internal team run, reduced protocol) passed 23/344 pairs (6.7%). Going from 2 to 3–4 agents collapses group success to zero — the coordination cliff this PR makes measurable. Failure modes, from the trajectories: agents engage the PR flow (242 gh pr calls, 201 pushes across the sweep) but rarely complete open-PR-then-submit within the 100-step budget; leads exhaust steps re-verifying or waiting; 4-agent groups show elevated context-overflow errors at 32k. Best group: 2/3 features passing (llama_index_task/17244, team3_slight). The 0% is the honest headline for a 9B model under the full protocol — a reduced-protocol (--team-no-*) comparison run is a natural follow-up.

Testing

• Unit tests for the N-way merge/solo helpers, error-result schemas, and legacy dual-write invariants (tests/eval/test_sandbox.py)
• Routing tests pinning that N=2 stays on the legacy paths and N>2 routes to the _n functions for all three settings (tests/eval/test_evaluate.py)
• GCP batch unit tests (no credentials needed) for dataclass normalisation, the N-generic VM script, and _run_gcp_batch with a mocked evaluator over 3-agent team, 2-agent coop, and 3-feature solo runs (tests/eval/test_gcp_batch.py)
• Subset groups discovery tests (tests/runner/test_tasks.py)
• Full suite: 421 passed locally; Lint + Test workflows green in CI

Checklist

• Code follows project style
• Tests pass locally
• Documentation updated (if needed)

🤖 Generated with Claude Code