Thanks for maintaining SWE-bench Pro. We are reproducing the public set and
would appreciate guidance on the reference compute and time budgets for two
cases:
- Grading only: apply a submitted/gold patch and run the evaluator.
- Agent + grading: run an agent harness in the task environment, then grade
its patch.
For each case, what CPU, memory, writable disk, and phase-specific time budgets
should be allocated per instance? Is there one supported envelope or do heavier
tasks need per-instance/per-repository overrides? For fanned-out runs, is there
a recommended concurrency or host-sizing rule?
The current Modal evaluator
uses cpu=(1, 4), memory=(5 * 1024, 30 * 1024), and a one-hour sandbox
lifetime, while the
local-Docker path
does not set equivalent limits. Should the Modal envelope be treated as the
reference grading configuration?
For context, our repeated gold-patch runs at 1 vCPU / 4 GB with a 600-second
grading cutoff (T600) produced:
- 593 / 731 rows with every expected F2P and P2P status passing in every
observed replica;
- 4 rows with complete expected-test vectors containing explicit non-pass
results, plus 1 row with both a non-pass result and missing status;
- 35 rows with missing expected statuses but no observed non-pass result; and
- 98 unevaluable rows, 94 of which produced no complete replica at T600.
Across the full 731-row artifact, 133 of the 138 excluded T600 rows were
missing-status-only or unevaluable rather than demonstrated test failures, so
matching the intended resource and timeout contract matters.
Intersecting its stable rows with our public-dev split retains 229 / 294 rows.
Thanks as well for sharing in your recent
announcement that v1.1
is already underway and SWE-bench Pro Verified is coming soon. When convenient,
could you share any update on its status and whether the release will include
updated instance IDs plus reference resource and timeout guidance for
grading-only and agent-harness runs? Any guidance would be very helpful—thank
you.
Thanks for maintaining SWE-bench Pro. We are reproducing the public set and
would appreciate guidance on the reference compute and time budgets for two
cases:
its patch.
For each case, what CPU, memory, writable disk, and phase-specific time budgets
should be allocated per instance? Is there one supported envelope or do heavier
tasks need per-instance/per-repository overrides? For fanned-out runs, is there
a recommended concurrency or host-sizing rule?
The current Modal evaluator
uses
cpu=(1, 4),memory=(5 * 1024, 30 * 1024), and a one-hour sandboxlifetime, while the
local-Docker path
does not set equivalent limits. Should the Modal envelope be treated as the
reference grading configuration?
For context, our repeated gold-patch runs at 1 vCPU / 4 GB with a 600-second
grading cutoff (T600) produced:
observed replica;
results, plus 1 row with both a non-pass result and missing status;
Across the full 731-row artifact, 133 of the 138 excluded T600 rows were
missing-status-only or unevaluable rather than demonstrated test failures, so
matching the intended resource and timeout contract matters.
Intersecting its stable rows with our public-dev split retains 229 / 294 rows.
Thanks as well for sharing in your recent
announcement that v1.1
is already underway and SWE-bench Pro Verified is coming soon. When convenient,
could you share any update on its status and whether the release will include
updated instance IDs plus reference resource and timeout guidance for
grading-only and agent-harness runs? Any guidance would be very helpful—thank
you.