Score MedCalc-Bench lab and dosage categories instead of always 0

[azrabano23]

What was wrong

MedCalcBenchMetric.check_in_range only range-checks the categories "dosage conversion" and "physical". The categories that evaluate_generation actually sends to it (everything except risk/severity/diagnosis) are date, lab, dosage, physical in ncbi/MedCalc-Bench-v1.0; "dosage conversion" never occurs. Unrecognized categories reach raise ValueError("Category ... not supported"), which is inside the try whose except ValueError: return 0 catches it. Result: all 354 lab + dosage instances (of 1000) score 0 no matter what the model answered.

Concrete instance from the published claude-3-7-sonnet run: category lab, prediction 303, limits [287.8766, 318.1794], reported medcalc_bench_accuracy = 0.0.

Re-scoring the published MedHELM v3.0.0 display_predictions.json with the range check applied to lab/dosage (same code path physical uses today):

run	lab (n=315)	dosage (n=39)	overall (n=1000)
claude-3-7-sonnet-20250219	0.000 -> 0.292	0.000 -> 0.308	0.210 -> 0.314
gpt-4o-2024-05-13	0.000 -> 0.251	0.000 -> 0.333	0.188 -> 0.280

Every other category is unchanged.

The fix

• Range-check lab, dosage, physical (the reference evaluate.py in ncbi-nlp/MedCalc-Bench range-checks every non-date calculator).
• Move the unsupported-category check before the try, so a genuinely unknown category raises instead of silently scoring 0.

Net change to medcalc_bench_metrics.py is 3 lines. Unparseable predictions and out-of-range values still score 0, as before.

Tests

New src/helm/benchmark/metrics/test_medcalc_bench_metrics.py:

• lab/dosage/physical in-range, boundary, out-of-range and non-numeric predictions via check_in_range;
• unknown category raises ValueError;
• end-to-end evaluate_generation on a lab instance.

On main, 4 of 5 fail (lab and dosage return 0, the raise is swallowed, end-to-end scores 0.0); with the fix all 5 pass.

Verification

python -m pytest src/helm/benchmark/metrics/test_medcalc_bench_metrics.py   # 5 passed
black --check / flake8 / mypy on the two touched files                       # clean

Python 3.12, CPU only. Published MedCalc-Bench numbers produced by this metric understate every model's accuracy by roughly 0.09-0.10 absolute on the two runs above, and the gap is model dependent.

Fixes #4351. Also resolves item 2 of #4340 (the swallowed raise at this site).

🤖 Generated with Claude Code