fix: guard _classify_completed_outcomes against non-dict rewards - #1054
Open
VaisakhiMishra wants to merge 3 commits into
Open
fix: guard _classify_completed_outcomes against non-dict rewards#1054VaisakhiMishra wants to merge 3 commits into
VaisakhiMishra wants to merge 3 commits into
Conversation
Previously, a truthy non-dict value in the 'rewards' field of result.json
(e.g. bare float 1.0, int, bool, list) caused an uncaught AttributeError
in _classify_completed_outcomes, crashing the entire evaluation run before
any new tasks could execute.
Root cause: the two-step pattern
rewards = r.get('rewards') if isinstance(r, dict) else None
reward = rewards.get('reward') if rewards else None
only guards against falsy values; a truthy non-dict (e.g. 1.0) passes the
'if rewards' check and then blows up on .get().
Fixes:
1. Import extract_reward from benchflow._utils.scoring — it already
handles None, dict, and non-dict values safely.
2. Replace the two-liner in _classify_completed_outcomes with:
reward = extract_reward(r) if isinstance(r, dict) else None
3. Add a logger.warning in _get_completed_tasks when rewards is a
non-dict truthy value, so operators can identify malformed result
files without a crash.
The fix is backwards-compatible: valid dict-shaped rewards are unchanged;
falsy non-dict values were already silently treated as errored and still
are (with a warning now); truthy non-dict values no longer crash.
Author
|
/devin review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, a truthy non-dict value in the 'rewards' field of result.json (e.g. bare float 1.0, int, bool, list) caused an uncaught AttributeError in _classify_completed_outcomes, crashing the entire evaluation run before any new tasks could execute.
How to reproduce the crash (pre-fix)
Can we make this bug fix as part of https://github.com/benchflow-ai/FrontierPhysics/issues/125 or should I open a separate issue to track this fix?
Root cause:
only guards against falsy values; a truthy non-dict (e.g. 1.0) passes the
if rewardscheck and then fails on .get().Fixes:
extract_rewardfrombenchflow._utils.scoring— it already handles None, dict, and non-dict values safely._classify_completed_outcomeswith:logger.warningin_get_completed_taskswhen rewards is a non-dict truthy value, so operators can identify malformed result files without a crash. The fix is backwards-compatible: valid dict-shaped rewards are unchanged; falsy non-dict values were already silently treated as errored and still are (with a warning now); truthy non-dict values no longer crash.uvx ruff check .failure forsrc/benchflow/contracts/user.py