[azrabano23]

What was wrong

KPIEdgarMetric scored the gold reference as if it were the model's prediction, and the prediction as if it were the reference. _request_states_to_pred_gold_pairs appended (references[0].output.text, completions[0].text), while _compute_stats unpacks each pair as for pred_text, gold_text in pred_gold_pairs and passes them on as pred_entities= / gold_entities=.

Concrete effect, running _compute_stats on one instance with the real TAG_DICT:

gold = "revenue increased[kpi], 100[cy], 90[py]"
pred = "revenue[kpi], 100[cy]"

intended (pred, gold)              word_macro_f1_score=0.416667  adjusted_macro_f1_score=0.416667
what the code built (gold, pred)   word_macro_f1_score=0.416667  adjusted_macro_f1_score=0.450000

Root cause

word_macro_f1_score is symmetric in pred/gold: with adjust=False, F1 = 2*overlap / (num_pred + num_gold), which is unchanged under exchange, so that metric masked the swap. adjusted_macro_f1_score normalizes tp/fn by num_gold and fp by num_pred, so swapping num_pred and num_gold changes its value. The direction of the error depends on whether the model over- or under-predicts per tag, so it does not average out across instances.

The fix

Build the tuple as (completion, reference), the order the consumer and its keyword arguments already assume. One-line change in src/helm/benchmark/metrics/kpi_edgar_metrics.py.

Adds src/helm/benchmark/metrics/test_kpi_edgar_metrics.py with two tests on the asymmetric pair above: one pins the tuple order from _request_states_to_pred_gold_pairs, the other asserts adjusted_macro_f1_score == 5/12 through KPIEdgarMetric.evaluate_instances. Both fail on main (0.45 != 0.416667) and pass with the fix.

Verification

python -m pytest src/helm/benchmark/metrics/test_kpi_edgar_metrics.py src/helm/benchmark/metrics/test_classification_metrics.py
# 6 passed

black --check src/helm/benchmark/metrics/kpi_edgar_metrics.py src/helm/benchmark/metrics/test_kpi_edgar_metrics.py
flake8 src/helm/benchmark/metrics/kpi_edgar_metrics.py src/helm/benchmark/metrics/test_kpi_edgar_metrics.py
mypy   src/helm/benchmark/metrics/kpi_edgar_metrics.py src/helm/benchmark/metrics/test_kpi_edgar_metrics.py
# all clean

Python 3.12, CPU only. Published KPI-EDGAR adjusted_macro_f1_score numbers computed before this change were computed with the orientation swapped.

Fixes #4348

🤖 Generated with Claude Code