fix: constraint validation target + GEPA compatibility with dspy 3.3.0 - #173
Open
luckystar2026 wants to merge 1 commit into
Open
luckystar2026 wants to merge 1 commit into
luckystar2026 wants to merge 1 commit into
Conversation
- validate_all() now receives the full SKILL.md (skill['raw'] / evolved_full) instead of the bare body, matching _check_skill_structure's expectation of YAML frontmatter. Without this, every evolution run was rejected as FAILED even though the reassembled skill was valid. - GEPA: replace deprecated max_steps with max_full_evals (dspy 3.3.0 API) - GEPA: wrap 3-arg skill_fitness_metric in a 5-arg adapter to satisfy the GEPAFeedbackMetric protocol (gold, pred, trace, pred_name, pred_trace) - GEPA: pass reflection_lm so the optimizer can propose instruction mutations Verified: GEPA main path runs 9 iterations, all constraints pass, holdout behavior score +30.7% (0.431 -> 0.564) on arxiv skill with DeepSeek.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: fix: constraint validation target + GEPA compatibility with dspy 3.3.0
Branch:
fix/gepa-compat-and-constraint-validationSummary
Four small fixes in
evolution/skills/evolve_skill.pydiscovered while integrating on Windows with dspy 3.3.0 / gepa 1.x:1. Constraint validation was checking the wrong text (always FAILED)
_check_skill_structurevalidates a full SKILL.md (expects---frontmatter + name + description), butevolve_skill.pypassed the bare body:A bare body never starts with
---, so every run — including perfectly valid ones — reportedskill_structureFAILED and the evolved skill was saved asevolved_FAILED.md.2–4. GEPA API drift (dspy 3.3.0)
The repo targets an older GEPA API. On dspy 3.3.0 all three of these throw, the
except Exceptionsilently falls back to MIPROv2 (the fallback message is easy to miss in rich-console logs), and GEPA's main path never runs:max_steps=N→max_full_evals=N(TypeError: GEPA.__init__() got an unexpected keyword argument 'max_steps')(gold, pred, trace, pred_name, pred_trace).skill_fitness_metrictakes 3 — added a thin adapter:reflection_lmis mandatory (AssertionError: GEPA requires a reflection language model…):Verification (Windows, dspy 3.3.0, DeepSeek, arxiv skill)
pytest tests/ -q: 144 passed / 1 failed (test_resolve_expands_user_home— pre-existing POSIX path assumption, unrelated)dspy.teleprompt.gepa.gepa: Iteration N: New subsample score …visible in logsNote
This PR fixes the pipeline so evolution runs and gates correctly, but skill text itself still never changes on disk — that is a separate design issue in
SkillModule(skill_textis an input field, not the optimized instruction). Tracked in issue #172.