feat(fitness): objective ground-truth verifier for skill evolution (reference: arxiv)

[MaxFreedomPollard]

The problem

Phase 1's gate is "at least one skill measurably improved." Right now that measurement can't be trusted, because the metric the optimizer maximizes (and the holdout is scored with) is keyword overlap against the rubric text:

# evolution/core/fitness.py
overlap = len(expected_words & output_words) / len(expected_words)
score = 0.3 + (0.7 * overlap)

This rewards echoing the rubric's vocabulary, not being right. LLMJudge exists but is dead code (imported in evolve_skill.py, never called), and even judge scoring is subjective: it never checks whether an answer is actually true. The Phase 1 validation report is honest about this and lists replacing the heuristic as its own next step.

So every reported improvement so far is unfalsifiable. Nothing downstream (benchmark gates, significance checks like #135/#136, Phase 2's tool-selection accuracy) means anything until the number being optimized measures reality.

What this adds

A third, strongest tier of fitness signal: objective verifiers that grade outputs against checkable ground truth, plus a working reference implementation for the arxiv skill (the same skill the Phase 1 validation report used).

Signal	How it scores	Trust
Objective verifier (new)	Compares output to verifiable facts; pure Python	Not gameable by rubric echoing, reproducible, $0 per grade
LLM-as-judge	Model grades a paraphrase on a rubric	Subjective, circular
Keyword overlap (current)	Lexical overlap with the rubric	Weakest

evolution/core/verifier.py

• Verifier ABC: a verifier owns both halves of the evaluation contract, build_dataset() (tasks whose answers are verifiable facts) and score() (grade an output against those facts). They must travel together, since the grader can only score tasks it knows the truth for.
• Registry (@register_verifier, get_verifier(skill_name)).
• verifier_metric() adapts a verifier to every metric calling convention in the pipeline: plain float for MIPROv2 and holdout comparison, and score-plus-feedback (dspy.Prediction(score=..., feedback=...)) when GEPA asks about a specific predictor. The verifier's feedback text ("expected ID 1706.03762, response said 1706.03799") is exactly the reflection signal GEPA mutates on.

evolution/verifiers/arxiv_verifier.py

Grades four kinds of verifiable facts across 17 landmark ML papers (word2vec through the GEPA paper itself): arXiv IDs, exact titles, first authors, and submission years. Grading is regex plus normalization, no LLM calls.

Score shaping is monotone and anti-gaming:

Output	Composite
Correct and concise	1.00
Correct but hedged across several candidate IDs	0.75
Confidently wrong (an ID was given, it's the wrong one)	0.50
Evasive or empty	0.00

Details that matter:

• Hedging gets half credit. Listing several candidate IDs where one happens to be right is not a usable answer, and full credit would reward shotgunning.
• Conciseness only counts when an answer is present. A brief evasion can never outscore a verbose correct answer (a test locks this in).
• Year tasks are solvable from the ID alone (the YYMM prefix), so they measure whether the skill teaches the arXiv ID format. That's a property a skill edit can genuinely flip, which is exactly the kind of headroom evolution needs.
• ID-shaped substrings are stripped before year matching, so quoting 2005.14165 doesn't get misread as the year 2005.

The embedded facts are historical constants, and every one is re-checkable against the live arXiv API:

$ python -m evolution.verifiers.arxiv_verifier --validate
All 17 papers verified against the live arXiv API.

All 51 facts (17 papers x title/author/year) passed at commit time.

Wiring (evolve_skill.py)

New --fitness auto|verifier|keyword flag, default auto: use the objective verifier when one is registered for the skill, otherwise fall back to current behavior. When a verifier is active it also supplies the eval dataset (deterministic sample, 50/25/25 split, saved to datasets/skills/<skill>-verifier/).

# Objective fitness picked automatically
python -m evolution.skills.evolve_skill --skill arxiv --iterations 10

# See how the grader treats correct, wrong, and evasive answers
python -m evolution.verifiers.arxiv_verifier --demo

Deliberate scope boundaries

• Does not touch GEPA instantiation, evolved-text extraction, or the frontmatter validator; fix(phase1): GEPA-compatible SkillModule + LLM-as-judge metric + constraint fixes #137, fix(skills): extract evolved instruction with overfit/collapse guard #146, fix: GEPA 3.2.x compatibility + symlink resolver + validator false-positive #142, Fix evolve_skill validation and assembly bugs (fixes #119) #140 and others own those fixes, and this composes with whichever of them land. The metric adapter already speaks GEPA's feedback protocol, so it plugs straight into a fixed optimizer loop.
• Does not replace judge or keyword scoring; the --fitness selector leaves room for an LLM-as-judge mode (as in fix(phase1): GEPA-compatible SkillModule + LLM-as-judge metric + constraint fixes #137) for skills where no objective verifier is possible.
• Adding a verifier for another skill is subclass-and-register; tool-selection accuracy (Phase 2) and test suites (Phase 4) are both objectively checkable, so this interface is the substrate those phases need anyway.

Test plan

• 40 new tests (tests/verifiers/test_arxiv_verifier.py, tests/core/test_verifier_registry.py): grading correctness for all four fact kinds, the anti-gaming behaviors, dataset determinism and gradability, metric adapter under all three calling conventions, edge cases (empty output, unknown task, versioned IDs).
• Full suite: 183 passed (143 existing + 40 new), offline; the live API is only touched by the --validate flag.
• Manual: --validate (17/17 papers verified), --demo, and evolve_skill --dry-run in both auto-verifier and forced-keyword modes.

No new dependencies (stdlib urllib + xml.etree for the validation fetch; click/rich already in use).

[enzo-adami]

Independent audit at 448488e: the objective-verifier direction is valuable, but three admission defects are reproducible and should be fixed before merge.

1. Incorrect facts can receive admission-grade score. Synthetic reproduction:

v = ArxivVerifier()
task = task_input_for(PAPER_BY_ID["1706.03762"], KIND_ID)
f = v.score(task, "The paper is arXiv 1706.03799.")
# f.correctness == 0.0, but f.composite == 0.5

The 0.3 procedure + 0.2 conciseness weights exactly compensate for zero correctness. For an objective verifier, enforce correctness == 0 => metric/admission score == 0 (while retaining feedback dimensions for reflection).

2. Paper leakage across all three splits. With the default num_cases=24, seed=13, I mapped each generated task back to its paper:

• train ∩ val: 1810.04805, 2201.11903
• train ∩ holdout: 1810.04805, 2006.11239, 2106.09685
• val ∩ holdout: 1810.04805

Shuffle and partition papers (group IDs), then sample task kinds within each assigned paper group. Add pairwise-disjoint group assertions.

3. The six-example holdout is underpowered. Even in the best-case one-sided paired test with no candidate wins, 4/4 losses gives p=0.0625; five losses are required to cross 0.05 (p=0.03125). A small mean increase is currently enough to report improvement. Admission should use a declared paired non-regression/power rule; feat: complete Phases 2-5 (tool descriptions, prompt sections, code evolution, continuous loop) #162 already contains the shared exact-McNemar/Wilson/power machinery, so please reuse/extract that rather than create a second statistics implementation.

Compatibility warning: a merge-tree simulation of #150 with #168 reports conflicts in evolve_skill.py. Accepting the #150 side mechanically would restore max_steps and the catch-all except Exception. Rebase onto #168 and preserve its _compile_optimizer: build the reflection LM first, catch only AttributeError/TypeError during GEPA construction, keep compile outside the fallback boundary, pass the same valset to both optimizers, and pass metric_fn into that existing helper.

Current branch tests do pass (184 passed), so dedicated tests for the three counterexamples above are necessary.

[enzo-adami]

I turned the reproduced correctness-floor and paper-split leakage fixes into a draft extension against your exact branch: MaxFreedomPollard#1. Full repository: 187/187 PASS; Ruff and diff-check PASS. Functional commit is 2041d57; lint-only follow-up is 0e66121. Statistical power remains deliberately out of scope because #162 should own that layer.

[MaxFreedomPollard]

Thanks for the audit. All three admission defects are fixed, plus the disjointness assertion you prescribed. The compatibility point is not done, and I want to be explicit about why rather than let it look handled.

Your draft went in as-is. Commits 2041d57 and 0e66121 are on the branch with your authorship intact, which closed MaxFreedomPollard#1 as merged. I reproduced both defects before taking it: the confidently wrong ID scored composite 0.5000 with correctness 0.0, and at num_cases=24, seed=13 the leakage was exactly your three sets, down to 1810.04805 appearing in all three splits. I also stress-tested the grouping past your tests, across seeds 1/7/13/99/2026 and num_cases 8/12/24/40/68, and found no leakage and exact split sizes.

One thing I added on top. You prescribed pairwise-disjoint group assertions, and the draft had that as a test rather than an invariant, so build_dataset now raises on leakage directly. The grouping makes it structurally impossible, which is exactly why the check is cheap and worth keeping: a later change to the sampling would otherwise reintroduce it silently.

Defect 3 reuses #162 rather than adding a second statistics implementation. evolution/core/stats.py and its tests are copied byte for byte from that branch, so whichever of the two lands second merges without a conflict. Admission is now declared and paired: the test must find a significant improvement at alpha and must not find a significant regression. Where the verifier supplies ground truth the outcomes are binary and mcnemar_exact decides, otherwise the signed-rank test on the scores does. Power is reported rather than gating, since a result that reached significance did so on evidence; what it does is qualify the non-regression claim, because min_detectable_paired_shift(6) is 5/6, so "no significant regression" on six examples means "no regression above 83.3%" and the run now says so. Your six-example case is a committed test: one win and five ties gives p=0.5 and is refused, while five aligned discordant pairs give p=0.03125 and are admitted. The mean-only improvement claim is gone.

On the rebase onto #168, I did not do it, because with #168 unmerged and #150 based on main it does not stay a topology change. It takes this PR from 8 files and +981 to 19 files and +3815, and pulls six of your commits and +552/-49 of your unmerged work inside my PR. _compile_optimizer also arrives in 6073ab2, the last commit of #168, which depends on the earlier ones, so there is no narrow cherry-pick either.

Your underlying point stands and I verified it rather than assuming it. A merge simulation still conflicts in evolve_skill.py, and the #150 side of that hunk does carry max_steps and the catch-all except Exception, so a mechanical resolution restores both. The resolution is to take the #168 side whole and change one argument to metric=metric_fn: I ran that combination and the full suite passes at 327. Nine hunks conflict, eight of them additive, and only the optimizer block is dangerous. Once #168 is in main this is a rebase and that one edit.

Head 0156342, 312 tests passing, Ruff clean on every file added here. Each of the three counterexamples was checked to fail on the prior code by reverting the source and keeping the tests.