feat(guardrails): semantic preservation constraint + holdout integrity (dedup, seeded splits) - #151
MaxFreedomPollard wants to merge 1 commit into
Conversation
Two guardrails that PLAN.md says every candidate must pass existed only on paper. This implements both. Semantic preservation (PLAN.md constraint 4): evolved text must not drift from the baseline's purpose. Measured as content-term cosine similarity between baseline and evolved text, deterministic and free. Calibrated on real hermes-agent skills (arxiv, pixel-art, dcf-model, docker-management, cli): rewrites of the same skill score 0.84 to 0.99 while pairs of unrelated skills score 0.04 to 0.16, so the default 0.4 threshold has wide margin on both sides. Failures report which baseline vocabulary disappeared. Runs whenever validate_all gets a baseline, so the evolved-candidate path picks it up with no caller changes. Setting min_semantic_similarity to 0 disables it. Holdout integrity: LLM generation and mined session history both produce repeated tasks, and nothing stopped two copies of one task from landing in train and holdout, where measured improvement can be pure memorization. New dedupe_examples() removes exact duplicates (normalized text) and near-duplicates (token-set Jaccard, default 0.9) before any split. New split_examples() replaces the three unseeded random.shuffle splits (synthetic, golden, external importers) with one seeded implementation, so a dataset always splits the same way and re-running a build cannot silently move holdout examples into train. No changes to evolve_skill.py, skill_module.py, or optimizer code. New tests live in new files. 26 new tests; full suite passes (169).
|
Closing in favour of #162. The two halves of this land differently there. Holdout integrity is already present: evolution/tools/selection_eval.py seeds its train/val/holdout split and rejects duplicate tasks before splitting, so a memorised train example cannot reach the holdout set. The semantic preservation constraint as written here is incompatible with Phase 2. It scores cosine over content-term frequency vectors, so a description that repeats one clause six times has that clause dominate its vector, and removing the repetition scores 0.29 against the 0.40 floor and is rejected as topic drift. Collapsing bloated descriptions is precisely what Phase 2 exists to do, and twelve descriptions in a stock hermes-agent checkout are already over the 500-char budget. The constraint is worth having, so it is being carried into #162 in a form that compares the set of content terms rather than their counts, and that treats a candidate introducing no vocabulary the baseline did not already have as having kept its subject. |
1 participant
The problem
PLAN.md's Constraints & Guardrails section says every candidate variant "must pass ALL of these" before it can be considered valid. Two of those guardrails exist only on paper:
1. Semantic preservation (constraint 4). The PLAN requires that evolved text is "compared against the original to ensure it hasn't drifted too far in meaning", and the README's guardrail list already advertises it. Nothing implements it.
validate_allchecks size, growth, non-emptiness, and structure; a skill for GitHub code review could evolve into a poem and pass every gate.2. Holdout integrity. The Safety section relies on "Holdout test sets, separate from training data, to catch overfitting". Two things quietly break that separation today:
random.shuffle, so re-building a dataset silently reshuffles holdout membership between runs. An example that was holdout yesterday can be training data today.Both problems corrupt the exact number Phase 1's gate ("at least one skill measurably improved") is judged on.
What this adds
Semantic preservation constraint
semantic_similarity(baseline, evolved)inconstraints.py: cosine similarity over content-term frequencies (4+ char words minus a small stopword list). Deterministic, dependency-free, zero API cost. A newsemantic_preservationcheck runs whenevervalidate_allreceives abaseline_text, which is exactly the existing evolved-candidate call inevolve_skill.py, so no caller changes are needed and baseline-only validation is unaffected.Calibrated against real hermes-agent skills (
arxiv,pixel-art,dcf-model,docker-management,cli):The default threshold of
0.4sits in the middle of that gap with wide margin on both sides. Failures report the score plus the baseline vocabulary that disappeared ("Baseline terms missing from evolved text: review, diff, ..."), which tells a human reviewer at a glance what the mutation destroyed.min_semantic_similarity = 0disables the check.Holdout integrity
Two helpers in
dataset_builder.py, applied to all three dataset sources:dedupe_examples(): removes exact duplicates (normalized task text) and near-duplicates (token-set Jaccard, default 0.9), keeping the first occurrence. The importer path reports how many were removed.split_examples(): one seeded shuffled split (default seed 13, configurable) replacing the three copies of the unseeded shuffle-and-slice logic. The same examples now always produce the same train/val/holdout membership, and the input list is not mutated.New config knobs:
min_semantic_similarity,dedup_jaccard,dataset_split_seed.Deliberate scope boundaries
evolve_skill.py,skill_module.py, or any optimizer code. This PR does not overlap with the open fixes for GEPA compatibility, evolved-text extraction, or validator false positives (fix(phase1): GEPA-compatible SkillModule + LLM-as-judge metric + constraint fixes #137, fix(skills): extract evolved instruction with overfit/collapse guard #146, fix: GEPA 3.2.x compatibility + symlink resolver + validator false-positive #142, Fix evolve_skill validation and assembly bugs (fixes #119) #140, feat(constraints): class-aware, graduated skill-size cap #134); it composes with all of them.test_semantic_preservation.py,test_dataset_integrity.py) rather thantest_constraints.py, to avoid merge conflicts with PRs that edit that file.Test plan
semantic_similarity().