Skip to content

fix(evolution): restore GEPA skill evolution on dspy 3.2 - #155

Open
A-KH17 wants to merge 1 commit into
NousResearch:mainfrom
A-KH17:fix/dspy-32-gepa-compat
Open

A-KH17 wants to merge 1 commit into
NousResearch:mainfrom
A-KH17:fix/dspy-32-gepa-compat

Conversation

@A-KH17

@A-KH17 A-KH17 commented Jul 22, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes seven defects that made the Phase-1 GEPA skill-evolution pipeline unusable on the current dspy release (3.2.1). Depending on where execution stopped, users got a silent fallback to MIPROv2, a hard crash, or — worst — an "evolved" skill byte-identical to the baseline (the optimizer never touched the skill text; see fix 6).

Verified end-to-end with these fixes applied: a 150-rollout GEPA run on github-code-review (hermes-agent @ 9acc4b4, moonshot/kimi-k3 optimizer, moonshot/kimi-k2.7-code eval) completed 21 iterations, improved valset 0.474 → 0.579, and produced a genuinely mutated skill (used downstream in NousResearch/hermes-agent#69641).

The bugs

  1. dspy.GEPA(max_steps=…) no longer exists in dspy 3.2 (budget args are auto / max_full_evals / max_metric_calls) → TypeError → silent MIPROv2 fallback. Now passes max_full_evals=iterations.
  2. GEPA requires an explicit reflection_lm in dspy 3.2; none was passed → construction raised → fallback. Now passes reflection_lm=dspy.LM(optimizer_model, temperature=1.0, max_tokens=32000) (per dspy's own error-message guidance).
  3. Metric signature mismatch. dspy 3.2 GEPA requires a 5-arg metric (gold, pred, trace, pred_name, pred_trace); skill_fitness_metric takes (example, prediction)TypeError → fallback. Wrapped it; the wrapper returns a plain float because dict/Prediction returns break dspy.Evaluate's sum() aggregation.
  4. The MIPROv2 fallback itself was brokenoptuna is required but undeclared. Added to dev extras.
  5. Constraint validation always failed skill_structure because it checked the frontmatter-less body for YAML frontmatter (warned at baseline, hard-stopped the evolved skill). Now validates the reassembled full text.
  6. The core flaw: the skill was never optimized. SkillModule passed the skill text as an input field (skill_instructions=…); GEPA mutates signature instructions and demos, never plain inputs — so the skill body stayed frozen and optimized_module.skill_text returned the original text. Now the skill text is installed as the predictor instruction via with_instructions(...), and extraction pulls the optimized instruction from predictor.predict.signature.instructions.
  7. skill_fitness_metric crashed on list rubrics — dataset generators may emit expected_behavior as a list of bullet strings (observed with moonshot/kimi-k2.7-code), and the keyword-overlap code called .lower() on it. Lists are now coerced to text.

Changes Made

  • evolution/skills/evolve_skill.py — fixes 1, 2, 3, 5, and the extraction half of 6
  • evolution/skills/skill_module.py — fix 6 (skill text as optimizable instruction)
  • evolution/core/fitness.py — fix 7
  • pyproject.toml — fix 4 (optuna in dev extras)

How to Test

  1. pip install -e ".[dev]" on a clean env with dspy 3.2.x.
  2. python -m evolution.skills.evolve_skill --skill github-code-review --hermes-repo <hermes-agent checkout> --optimizer-model <model> --eval-model <model> --iterations 1
  3. Expected: GEPA runs (no "falling back to MIPROv2" message), constraints pass, and output/…/evolved_skill.md differs from baseline_skill.md with a holdout comparison printed.

Notes

  • No behavior change to the intended design — this restores the documented GEPA pipeline (README: "GEPA Optimizer ◄── Execution traces") on current dspy.
  • Fixes were developed against dspy 3.2.1; the metric wrapper and with_instructions approach follow dspy's public API.

Seven defects made the Phase-1 GEPA pipeline unusable on current dspy
(silent fallback to MIPROv2, hard crashes, or a byte-identical 'evolved'
skill). Verified end-to-end: with these fixes, a 150-rollout GEPA run on
github-code-review improved valset 0.474 -> 0.579 and produced a
genuinely mutated skill.

1. dspy.GEPA(max_steps=...) no longer exists in dspy 3.2 (budget args are
   auto/max_full_evals/max_metric_calls) -> TypeError -> silent MIPROv2
   fallback. Pass max_full_evals=iterations instead.
2. dspy 3.2 GEPA requires an explicit reflection_lm; none was passed ->
   fallback. Pass reflection_lm=dspy.LM(optimizer_model, temperature=1.0,
   max_tokens=32000).
3. dspy 3.2 GEPA requires a 5-arg metric (gold, pred, trace, pred_name,
   pred_trace); skill_fitness_metric takes (example, prediction) ->
   TypeError -> fallback. Wrap it; return a plain float (dict/Prediction
   returns break dspy.Evaluate's sum()).
4. The MIPROv2 fallback itself was broken: optuna undeclared. Add it to
   dev extras.
5. Constraint validation checked the frontmatter-less body for YAML
   frontmatter, so skill_structure always failed (warn at baseline,
   hard-stop at evolved). Validate the reassembled full text instead.
6. SkillModule passed the skill text as an INPUT field; GEPA optimizes
   instructions/demos, not inputs, so the skill body was never evolved
   (evolved output byte-identical to baseline). Install the skill text as
   the predictor instruction via with_instructions() and extract the
   optimized instruction from predictor.predict.signature.instructions.
7. skill_fitness_metric crashed on dataset generators that emit
   expected_behavior as a list of bullets (observed with
   moonshot/kimi-k2.7-code). Coerce lists to text.
@TurtleMcTurtle

Copy link
Copy Markdown

Review: PR #155 — Most complete GEPA fix

This is the most comprehensive fix among the overlapping PRs (#153, #155, #159, #167). It addresses all three layers of the GEPA compatibility bug:

1. SkillModule fix (root cause) ✅

  • Moves skill text from InputField to Signature.instructions via .with_instructions(skill_text)
  • This is critical: GEPA optimizes signature instructions, NOT input fields. Without this, GEPA runs but never actually mutates the skill text.
  • Uses ChainOfThought (preserves CoT reasoning) vs fix: make DSPy skill evolution persist optimized instructions #153 which downgrades to bare Predict

2. GEPA constructor fix ✅

  • max_full_evals + reflection_lm with temperature=1.0 and max_tokens=32000
  • 5-arg metric wrapper (_gepa_metric)
  • Removes the silent MIPROv2 fallback (good — it was masking the real failure)

3. Evolved body extraction ✅

  • Reads optimized_module.predictor.predict.signature.instructions instead of optimized_module.skill_text
  • This is the fix for evolved skills coming out byte-identical to baseline

4. Validator fix ✅

5. Bonus: list type coercion in fitness metric ✅

Missing vs other PRs:

Suggestion: This should be the base PR for the GEPA fix. Cherry-pick the LLMJudge metric from #159 and the overfit guard from #146 to make it bulletproof. The custom API support from #167 should come as a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants