Conversation
Seven defects made the Phase-1 GEPA pipeline unusable on current dspy (silent fallback to MIPROv2, hard crashes, or a byte-identical 'evolved' skill). Verified end-to-end: with these fixes, a 150-rollout GEPA run on github-code-review improved valset 0.474 -> 0.579 and produced a genuinely mutated skill. 1. dspy.GEPA(max_steps=...) no longer exists in dspy 3.2 (budget args are auto/max_full_evals/max_metric_calls) -> TypeError -> silent MIPROv2 fallback. Pass max_full_evals=iterations instead. 2. dspy 3.2 GEPA requires an explicit reflection_lm; none was passed -> fallback. Pass reflection_lm=dspy.LM(optimizer_model, temperature=1.0, max_tokens=32000). 3. dspy 3.2 GEPA requires a 5-arg metric (gold, pred, trace, pred_name, pred_trace); skill_fitness_metric takes (example, prediction) -> TypeError -> fallback. Wrap it; return a plain float (dict/Prediction returns break dspy.Evaluate's sum()). 4. The MIPROv2 fallback itself was broken: optuna undeclared. Add it to dev extras. 5. Constraint validation checked the frontmatter-less body for YAML frontmatter, so skill_structure always failed (warn at baseline, hard-stop at evolved). Validate the reassembled full text instead. 6. SkillModule passed the skill text as an INPUT field; GEPA optimizes instructions/demos, not inputs, so the skill body was never evolved (evolved output byte-identical to baseline). Install the skill text as the predictor instruction via with_instructions() and extract the optimized instruction from predictor.predict.signature.instructions. 7. skill_fitness_metric crashed on dataset generators that emit expected_behavior as a list of bullets (observed with moonshot/kimi-k2.7-code). Coerce lists to text.
12 tasks
This was referenced Aug 4, 2026
Review: PR #155 — Most complete GEPA fixThis is the most comprehensive fix among the overlapping PRs (#153, #155, #159, #167). It addresses all three layers of the GEPA compatibility bug: 1. SkillModule fix (root cause) ✅
2. GEPA constructor fix ✅
3. Evolved body extraction ✅
4. Validator fix ✅
5. Bonus: list type coercion in fitness metric ✅Missing vs other PRs:
Suggestion: This should be the base PR for the GEPA fix. Cherry-pick the LLMJudge metric from #159 and the overfit guard from #146 to make it bulletproof. The custom API support from #167 should come as a separate PR. |
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes seven defects that made the Phase-1 GEPA skill-evolution pipeline unusable on the current dspy release (3.2.1). Depending on where execution stopped, users got a silent fallback to MIPROv2, a hard crash, or — worst — an "evolved" skill byte-identical to the baseline (the optimizer never touched the skill text; see fix 6).
Verified end-to-end with these fixes applied: a 150-rollout GEPA run on
github-code-review(hermes-agent @9acc4b4,moonshot/kimi-k3optimizer,moonshot/kimi-k2.7-codeeval) completed 21 iterations, improved valset 0.474 → 0.579, and produced a genuinely mutated skill (used downstream in NousResearch/hermes-agent#69641).The bugs
dspy.GEPA(max_steps=…)no longer exists in dspy 3.2 (budget args areauto/max_full_evals/max_metric_calls) →TypeError→ silent MIPROv2 fallback. Now passesmax_full_evals=iterations.reflection_lmin dspy 3.2; none was passed → construction raised → fallback. Now passesreflection_lm=dspy.LM(optimizer_model, temperature=1.0, max_tokens=32000)(per dspy's own error-message guidance).(gold, pred, trace, pred_name, pred_trace);skill_fitness_metrictakes(example, prediction)→TypeError→ fallback. Wrapped it; the wrapper returns a plainfloatbecause dict/Prediction returns breakdspy.Evaluate'ssum()aggregation.optunais required but undeclared. Added todevextras.skill_structurebecause it checked the frontmatter-less body for YAML frontmatter (warned at baseline, hard-stopped the evolved skill). Now validates the reassembled full text.SkillModulepassed the skill text as an input field (skill_instructions=…); GEPA mutates signature instructions and demos, never plain inputs — so the skill body stayed frozen andoptimized_module.skill_textreturned the original text. Now the skill text is installed as the predictor instruction viawith_instructions(...), and extraction pulls the optimized instruction frompredictor.predict.signature.instructions.skill_fitness_metriccrashed on list rubrics — dataset generators may emitexpected_behavioras a list of bullet strings (observed withmoonshot/kimi-k2.7-code), and the keyword-overlap code called.lower()on it. Lists are now coerced to text.Changes Made
evolution/skills/evolve_skill.py— fixes 1, 2, 3, 5, and the extraction half of 6evolution/skills/skill_module.py— fix 6 (skill text as optimizable instruction)evolution/core/fitness.py— fix 7pyproject.toml— fix 4 (optunain dev extras)How to Test
pip install -e ".[dev]"on a clean env with dspy 3.2.x.python -m evolution.skills.evolve_skill --skill github-code-review --hermes-repo <hermes-agent checkout> --optimizer-model <model> --eval-model <model> --iterations 1output/…/evolved_skill.mddiffers frombaseline_skill.mdwith a holdout comparison printed.Notes
with_instructionsapproach follow dspy's public API.