Skip to content

fix: cast subject filtration to their true types - #614

Merged
prabhuteja12 merged 2 commits into
mainfrom
subjectfilterbug
Aug 7, 2026
Merged

fix: cast subject filtration to their true types#614
prabhuteja12 merged 2 commits into
mainfrom
subjectfilterbug

Conversation

@prabhuteja12

@prabhuteja12 prabhuteja12 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Description

  • --task-subjects filtering breaks for tasks whose SUBJECTS are tuples with non-string elements. For e.g.
class MyFancyTask(BaseTask[tuple[str, int, str]])
    SUBJECTS = [("ctx1", 123, "single")]
  • When creating a task with with_overwrite, passing custom_subject="ctx1,123,single" and BaseTask._filter_task_subjects splits that string on , to build a tuple to match against SUBJECTS -> ("ctx1", "123", "single"). Every part is a str, including "123". But the real subject has an int there: ("ctx1", 123, "single"). Python's "123" == 123 is False, so the match fails and you get
AssertionError: Subject part 123 not found in task MyFancyTask
  • This only breaks for tuple subjects with a non-str position. It's been invisible because every other tuple-subject task (e.g. GlobalMMLU) uses all-str tuples like ("EN_US", "topic1"), where str == str just works.

Fix

  • Infer the real type at each tuple position from an actual subject (type(SUBJECTS[0][i])) and cast each comma-separated part back to it before comparing

@prabhuteja12 prabhuteja12 self-assigned this Aug 7, 2026
@prabhuteja12
prabhuteja12 requested a review from pacman82 August 7, 2026 11:43
Comment thread src/eval_framework/tasks/base.py Outdated
Comment thread tests/tests_eval_framework/test_base_task.py
Comment thread tests/tests_eval_framework/test_base_task.py
@prabhuteja12
prabhuteja12 merged commit adaa19e into main Aug 7, 2026
15 checks passed
@prabhuteja12
prabhuteja12 deleted the subjectfilterbug branch August 7, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants