Repository navigation
Stop samples that loop unproductively - #60
Merged
Merged
Conversation
Wrap each agent's on_continue in inspect_boltons' loop limits: 100 consecutive assistant turns without a tool call, or 30 consecutive turns with no tool call and identical text, end the sample with a "custom" limit. Applied to the main prover (react and deep), the red-team prover, and the software tester. The wrapper keeps each agent's existing continue message, played back only on turns without a tool call, exactly as the plain-string form did.
Collaborator
Author
|
Smoke test on this branch (head 916c5e5): claude-opus-5, first 3 Eval set |
This was referenced Sep 21, 2026
Collaborator
Author
|
Same smoke test (head 916c5e5) for claude-fable-5-1: first 3 Eval set |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the unproductive-loop limits from tadamcz/inspect_boltons (already in the pinned commit) to the main prover agent:
NoToolCallLimit(turns=100, unproductive_tools=["resources"]): 100 consecutive assistant turns without a tool call other thanresources.RepeatedTextLimit(turns=30): 30 consecutive turns with no tool call and identical visible text.Either one ends the sample as an Inspect limit of type
custom, so the sample is still scored on what it has and shows up as a limit rather than an error.For the deep agent the limits apply to the parent loop only, not inside
general()subagents.Empirical data
Replaying these limits over the logs of
bloom68-fable51-8t9up8nw9358oyrl(68 samples, $19.9k), they would have fired in 43 of 68 samples, and about 29% of the total spend (~$5.7k) came after the point where a limit would have ended the sample.The typical case is the model answering "Continue working on the problem." with a short "this is open, I will not fabricate a proof" for thousands of turns; the no-tool-call limit is what catches it, since the text varies slightly turn to turn.