feat(core): run manifests with cost accounting and constraint audit trail - #152
Closed
MaxFreedomPollard wants to merge 1 commit into
Closed
MaxFreedomPollard wants to merge 1 commit into
MaxFreedomPollard wants to merge 1 commit into
Conversation
PLAN.md's deployment process says every evolved artifact ships with before/after scores on all splits, the eval dataset used, the cost of the optimization run, and any constraint violations that were caught. None of that was recorded anywhere: metrics.json held scores and sizes only, cost was invisible, and rejected candidates left no trace beyond an evolved_FAILED.md file. evolution/core/manifest.py adds the evidence layer: - RunManifest: one JSON record per run with a config snapshot, dataset fingerprint (split sizes + content hash, order-independent within a split but sensitive to split membership), content digests of baseline and evolved text, full constraint results, per-example holdout scores, elapsed time, and LLM usage. - UsageTracker: sums calls, tokens, and provider-reported cost from dspy's LM call history across the whole run (dataset generation, optimization, holdout scoring). Calls with unknown cost are counted separately rather than silently treated as free, and eviction from dspy's bounded history is flagged so totals are never silently understated. - Manifests are written for rejected runs too (manifest_FAILED.json), so caught violations become an audit trail instead of vanishing. evolve_skill.py gains additive wiring only: tracker start after the dry-run gate, LLM calls/cost rows in the results table, and manifest writes on both the success and rejection paths. The output directory now reuses the run's own timestamp so the run id and directory match. 19 new tests, all offline (the tracker takes an injectable history). Full suite passes (162).
Contributor
Author
|
Closing in favour of #162. Cost accounting is implemented there as evolution/core/cost.py, and the per-run artifact directory each phase writes carries the manifest role this PR adds. One implementation rather than two. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
PLAN.md's "Deployment via PR" section (the release process for evolved artifacts) requires each evolution to ship with:
Today none of that is recorded.
metrics.jsonholds aggregate scores and character counts; cost is invisible (the README advertises "$2-10 per optimization run" but no run can actually report what it spent); a rejected candidate leaves nothing behind except anevolved_FAILED.mdfile with no record of which constraints failed or why; and there is no way to tell whether two runs were even measured on the same dataset. Whoever assembles the deployment PR has nothing trustworthy to quote.What this adds
evolution/core/manifest.py: one machine-readablemanifest.jsonwritten next to every run's output, deployable or not.RunManifest records, per run:
UsageTracker sums calls, tokens, and provider-reported cost from dspy's LM call history across the whole run: dataset generation, optimization, and holdout scoring. Honesty details:
calls_with_unknown_cost) rather than silently treated as freepossibly_incompleteflag so totals are never silently understatedRejected runs get manifests too. The constraint-failure path now writes
manifest_FAILED.jsonwith the violations on record. That is the "caught and rejected" audit trail the PLAN asks for; today those rejections vanish.evolve_skill.pychanges are additive only: start the tracker after the dry-run gate, two new rows in the results table (LLM calls/tokens and cost), and manifest writes on both exit paths. The output directory now reuses the run's own timestamp so the run id and the directory name match.Why this is the right next step
The pipeline's remaining pillars are already owned: correctness gates (#127), statistical significance (#136), and PR emission (#139) are open, and the optimizer fixes are contested across #137/#146/#142/#140. What every one of those consumers lacks is a trustworthy record to read from. The manifest is that substrate, and it stays useful standalone: it depends on none of those PRs and none of them depend on it.
Deliberate scope boundaries
skill_module.py, the GEPA/MIPROv2 calls, constraints, or datasets;evolve_skill.pyedits are purely additive.manifest.jsonif it lands.metrics.jsonis kept as-is for backward compatibility.Test plan
tests/core/test_manifest.py: digest determinism, config snapshot serialization with Paths, fingerprint properties (counts, order-independence, split sensitivity), usage tracking (summing, pre-start exclusion, unknown-cost accounting, all-unknown reporting None, empty history, finish-without-start, eviction flag), and manifest round-trips including the rejected-run shape.evolve_skill --dry-runverified unchanged.usage,cost, ISOtimestamp, 10,000-entry cap with FIFO eviction) was verified directly against dspy 3.2.1 source.