Skip to content

feat(core): run manifests with cost accounting and constraint audit trail - #152

Closed
MaxFreedomPollard wants to merge 1 commit into
NousResearch:mainfrom
MaxFreedomPollard:feat/run-manifest
Closed

MaxFreedomPollard wants to merge 1 commit into
NousResearch:mainfrom
MaxFreedomPollard:feat/run-manifest

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

The problem

PLAN.md's "Deployment via PR" section (the release process for evolved artifacts) requires each evolution to ship with:

  • before/after scores on train, validation, AND holdout sets
  • the eval dataset used
  • the cost of the optimization run
  • any constraint violations that were caught and rejected during evolution

Today none of that is recorded. metrics.json holds aggregate scores and character counts; cost is invisible (the README advertises "$2-10 per optimization run" but no run can actually report what it spent); a rejected candidate leaves nothing behind except an evolved_FAILED.md file with no record of which constraints failed or why; and there is no way to tell whether two runs were even measured on the same dataset. Whoever assembles the deployment PR has nothing trustworthy to quote.

What this adds

evolution/core/manifest.py: one machine-readable manifest.json written next to every run's output, deployable or not.

RunManifest records, per run:

  • a config snapshot (models, iterations, thresholds; Paths serialized)
  • a dataset fingerprint: per-split counts plus a content hash that is order-independent within a split but sensitive to split membership, so moving one example from holdout to train changes the fingerprint. Two runs with the same fingerprint were measured on the same data.
  • content digests and sizes of baseline and evolved text
  • full constraint results (name, passed, message, details), including failures
  • per-example holdout scores for baseline and evolved, not just the averages, so a later significance check has raw material to work with
  • elapsed time and LLM usage

UsageTracker sums calls, tokens, and provider-reported cost from dspy's LM call history across the whole run: dataset generation, optimization, and holdout scoring. Honesty details:

  • calls whose cost the provider does not report are counted separately (calls_with_unknown_cost) rather than silently treated as free
  • dspy's call history is size-bounded; if entries from this run may have been evicted, the summary carries a possibly_incomplete flag so totals are never silently understated
  • the history and its cap are injectable, so all tests run offline

Rejected runs get manifests too. The constraint-failure path now writes manifest_FAILED.json with the violations on record. That is the "caught and rejected" audit trail the PLAN asks for; today those rejections vanish.

evolve_skill.py changes are additive only: start the tracker after the dry-run gate, two new rows in the results table (LLM calls/tokens and cost), and manifest writes on both exit paths. The output directory now reuses the run's own timestamp so the run id and the directory name match.

Results table gains:
  LLM Calls    142 (81,304 tokens)
  LLM Cost     $0.3126 (+3 unpriced calls)

Why this is the right next step

The pipeline's remaining pillars are already owned: correctness gates (#127), statistical significance (#136), and PR emission (#139) are open, and the optimizer fixes are contested across #137/#146/#142/#140. What every one of those consumers lacks is a trustworthy record to read from. The manifest is that substrate, and it stays useful standalone: it depends on none of those PRs and none of them depend on it.

Deliberate scope boundaries

  • Does not modify skill_module.py, the GEPA/MIPROv2 calls, constraints, or datasets; evolve_skill.py edits are purely additive.
  • Does not format PR bodies from the manifest; that is feat: emit a pull request for an evolved skill #139's job, and it can read manifest.json if it lands.
  • metrics.json is kept as-is for backward compatibility.

Test plan

  • 19 new tests in tests/core/test_manifest.py: digest determinism, config snapshot serialization with Paths, fingerprint properties (counts, order-independence, split sensitivity), usage tracking (summing, pre-start exclusion, unknown-cost accounting, all-unknown reporting None, empty history, finish-without-start, eviction flag), and manifest round-trips including the rejected-run shape.
  • Full suite: 162 passed (143 existing + 19 new), fully offline.
  • evolve_skill --dry-run verified unchanged.
  • The dspy history entry shape (usage, cost, ISO timestamp, 10,000-entry cap with FIFO eviction) was verified directly against dspy 3.2.1 source.

PLAN.md's deployment process says every evolved artifact ships with
before/after scores on all splits, the eval dataset used, the cost of
the optimization run, and any constraint violations that were caught.
None of that was recorded anywhere: metrics.json held scores and sizes
only, cost was invisible, and rejected candidates left no trace beyond
an evolved_FAILED.md file.

evolution/core/manifest.py adds the evidence layer:

- RunManifest: one JSON record per run with a config snapshot, dataset
  fingerprint (split sizes + content hash, order-independent within a
  split but sensitive to split membership), content digests of baseline
  and evolved text, full constraint results, per-example holdout
  scores, elapsed time, and LLM usage.
- UsageTracker: sums calls, tokens, and provider-reported cost from
  dspy's LM call history across the whole run (dataset generation,
  optimization, holdout scoring). Calls with unknown cost are counted
  separately rather than silently treated as free, and eviction from
  dspy's bounded history is flagged so totals are never silently
  understated.
- Manifests are written for rejected runs too (manifest_FAILED.json),
  so caught violations become an audit trail instead of vanishing.

evolve_skill.py gains additive wiring only: tracker start after the
dry-run gate, LLM calls/cost rows in the results table, and manifest
writes on both the success and rejection paths. The output directory
now reuses the run's own timestamp so the run id and directory match.

19 new tests, all offline (the tracker takes an injectable history).
Full suite passes (162).
@MaxFreedomPollard

Copy link
Copy Markdown
Contributor Author

Closing in favour of #162. Cost accounting is implemented there as evolution/core/cost.py, and the per-run artifact directory each phase writes carries the manifest role this PR adds. One implementation rather than two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant