Skip to content

Add opt-in deduplication to the MedDialog scenario - #4342

Open
azrabano23 wants to merge 1 commit into
stanford-crfm:mainfrom
azrabano23:fix-3746
Open

azrabano23 wants to merge 1 commit into
stanford-crfm:mainfrom
azrabano23:fix-3746

Conversation

@azrabano23

Copy link
Copy Markdown

Problem

The upstream med_dialog data contains duplicate entries; the icliniq test split has 3108 entries but only 2069 unique source texts (#3746, confirmed by @yifanmai).

Approach

This implements the deduplication inside HELM, following the review feedback on the previous attempt (#4178), which stalled after changes were requested:

  • MedDialogScenario gains a dedup argument that defaults to False, and get_med_dialog_spec() exposes the same argument — exactly as requested in that review, so previously published runs remain reproducible bit-for-bit. With dedup=False (the default) the emitted ScenarioSpec and RunSpec are identical to today's.
  • When dedup=True, examples whose src duplicates an earlier example are dropped, keeping the first occurrence in the original order, and the run spec name becomes med_dialog,subset={subset},dedup=True so deduplicated runs are distinguishable from existing ones — the "new version of this scenario" suggested in the issue thread.

Tests

New test_med_dialog_scenario.py with two tests:

  • test_med_dialog_scenario_dedup — deterministic, no network (it pre-seeds the split file, so ensure_file_downloaded skips the download; runs in default CI): asserts the default keeps every entry including duplicates, and dedup=True keeps exactly the first occurrence of each source text in the original order, with the matching references.
  • test_med_dialog_scenario_dedup_real_data — marked scenarios: asserts the real icliniq test split yields 3108 instances (2069 unique) by default and 2069 unique instances with dedup=True, preserving first occurrences.
pytest src/helm/benchmark/scenarios/test_med_dialog_scenario.py                # deterministic test
pytest -m scenarios src/helm/benchmark/scenarios/test_med_dialog_scenario.py  # real-data test

Note: at the time of filing, the CodaLab Worksheets REST API is returning HTTP 500 for this bundle (and for bundle queries generally), so the real-data test — and any fresh run of this scenario on main — currently cannot download the data. The counts in the real-data test are the ones reported and confirmed in #3746.

Closes #3746.

The upstream med_dialog data contains duplicate entries: the icliniq
test split has 3108 entries but only 2069 unique source texts (stanford-crfm#3746).

Add a dedup argument, defaulting to False, to MedDialogScenario and
get_med_dialog_spec, as requested in the review of stanford-crfm#4178. When True,
examples whose source text duplicates an earlier example are dropped,
keeping the first occurrence, and the run spec name gains a
dedup=True suffix so deduplicated runs are distinguishable from
previously published ones. The default behavior is unchanged for
backwards compatibility.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate entries in med_dialog dataset

1 participant