Skip to content

[Feature]: Exclude frozen Red tests from SWE-bench model_patch #24

Description

@GenTang

痛点与背景 / Problem and Context

Problem

In the red_green flow, the frozen Red test is materialized into the workspace. SWE-bench evaluation currently collects the complete Git diff, so model_patch contains both:

production fix + generated Red test

This may conflict with SWE-bench’s official test_patch and affect evaluation validity.

期望的解决方案 / Proposed Solution

Keep both deliverables in the workspace, but submit only the production fix to SWE-bench:

Workspace:         production fix + frozen Red test
SWE-bench patch:   production fix only
Trace artifacts:   full patch + Red test patch

Requirements

  • Exclude only the exact paths recorded in frozen_test.files.
  • Do not broadly filter tests/**.
  • Apply filtering only to red_green SWE-bench submissions.
  • Keep direct_execute behavior unchanged.
  • Fail closed if the frozen test cannot be safely excluded.
  • Record excluded paths and submission patch SHA in the trace.
  • Add tests covering tracked and newly created Red test files.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions