Skip to content

fix: constraint validation target + GEPA compatibility with dspy 3.3.0 - #173

Open
luckystar2026 wants to merge 1 commit into
NousResearch:mainfrom
luckystar2026:fix/gepa-compat-and-constraint-validation
Open

luckystar2026 wants to merge 1 commit into
NousResearch:mainfrom
luckystar2026:fix/gepa-compat-and-constraint-validation

Conversation

@luckystar2026

Copy link
Copy Markdown

Title: fix: constraint validation target + GEPA compatibility with dspy 3.3.0

Branch: fix/gepa-compat-and-constraint-validation

Summary

Four small fixes in evolution/skills/evolve_skill.py discovered while integrating on Windows with dspy 3.3.0 / gepa 1.x:

1. Constraint validation was checking the wrong text (always FAILED)

_check_skill_structure validates a full SKILL.md (expects --- frontmatter + name + description), but evolve_skill.py passed the bare body:

# before
baseline_constraints = validator.validate_all(skill["body"], "skill")
...
evolved_constraints = validator.validate_all(evolved_body, "skill", baseline_text=skill["body"])

A bare body never starts with ---, so every run — including perfectly valid ones — reported skill_structure FAILED and the evolved skill was saved as evolved_FAILED.md.

# after
baseline_constraints = validator.validate_all(skill["raw"], "skill")
...
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])

2–4. GEPA API drift (dspy 3.3.0)

The repo targets an older GEPA API. On dspy 3.3.0 all three of these throw, the except Exception silently falls back to MIPROv2 (the fallback message is easy to miss in rich-console logs), and GEPA's main path never runs:

  • max_steps=Nmax_full_evals=N (TypeError: GEPA.__init__() got an unexpected keyword argument 'max_steps')
  • Metric protocol: dspy 3.3.0's GEPA requires a 5-arg metric (gold, pred, trace, pred_name, pred_trace). skill_fitness_metric takes 3 — added a thin adapter:
    def _gepa_fitness_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
        return skill_fitness_metric(gold, pred, trace)
  • reflection_lm is mandatory (AssertionError: GEPA requires a reflection language model…):
    reflection_lm=dspy.LM(eval_model, temperature=1.0, max_tokens=4096),

Verification (Windows, dspy 3.3.0, DeepSeek, arxiv skill)

  • pytest tests/ -q: 144 passed / 1 failed (test_resolve_expands_user_home — pre-existing POSIX path assumption, unrelated)
  • GEPA main path now runs: 9 iterations, 68/75 rollouts, dspy.teleprompt.gepa.gepa: Iteration N: New subsample score … visible in logs
  • All 4 constraint gates pass (size ≤15KB, growth ≤+20%, non-empty, structure)
  • Holdout behavior score 0.431 → 0.564 (+30.7%)

Note

This PR fixes the pipeline so evolution runs and gates correctly, but skill text itself still never changes on disk — that is a separate design issue in SkillModule (skill_text is an input field, not the optimized instruction). Tracked in issue #172.

- validate_all() now receives the full SKILL.md (skill['raw'] / evolved_full)
  instead of the bare body, matching _check_skill_structure's expectation of
  YAML frontmatter. Without this, every evolution run was rejected as FAILED
  even though the reassembled skill was valid.
- GEPA: replace deprecated max_steps with max_full_evals (dspy 3.3.0 API)
- GEPA: wrap 3-arg skill_fitness_metric in a 5-arg adapter to satisfy the
  GEPAFeedbackMetric protocol (gold, pred, trace, pred_name, pred_trace)
- GEPA: pass reflection_lm so the optimizer can propose instruction mutations

Verified: GEPA main path runs 9 iterations, all constraints pass, holdout
behavior score +30.7% (0.431 -> 0.564) on arxiv skill with DeepSeek.
@luckystar2026 luckystar2026 changed the title PR: fix constraint validation target + GEPA compatibility with dspy 3.3.0 fix: constraint validation target + GEPA compatibility with dspy 3.3.0 Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant