feat(eval): accuracy + latency benchmark harness - #23
Merged
Conversation
Adds packages/eval: a task set, scorers, latency instrumentation, and a report layer, plus an inclave-eval CLI with two backends. Fixture mode replays recorded replies — deterministic, no Ollama, so it gates every PR via --fail-under. Live mode streams from a real model and is the only mode producing real accuracy and latency numbers. Both score through identical logic; only the backend differs. Scorers call the product's own python_blocks_in rather than reimplementing extraction, so a regression in block detection surfaces as an eval failure instead of being masked by a parallel implementation. Latency records time-to-first-token separately from total duration: a run can stream fast overall and still feel slow if the model stalls before the first chunk, and one total-duration number cannot tell those apart. Refusal tasks encode the num_ctx eviction bug's symptom — a model that lost the system prompt fabricated file contents. The guard checks for a fenced block in any language, not just python: an invented YAML config is still an invented file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
packages/eval— the P1 "Eval & benchmarks" item, plus the P2 latencyinstrumentation that feeds it. Closes the roadmap's biggest gap: the repo had
200+ passing tests but zero numbers on whether the tool actually works.
What's here
Two model backends sharing one scoring path:
inclave-evalinclave-eval --liveFixture mode replays recorded replies, so it's deterministic and gates CI via
--fail-under. It verifies the harness and scorers, not model quality. Livemode streams from a real model and is the only mode producing real accuracy and
latency numbers. Only the backend differs — scoring, timing, and reporting are
identical.
Six tasks across four categories: codegen, file_qa, refusal, format.
Notes for review
Scorers call the product's own
python_blocks_inrather thanreimplementing extraction. If block detection regresses, the eval fails instead
of a parallel implementation masking it.
Latency records time-to-first-token separately from total duration. A run
can stream fast overall and still feel slow if the model stalls before the first
chunk; one total-duration number can't distinguish those. Uses
perf_counterso NTP steps don't corrupt the numbers. Fixture runs don't report latency — it
would be measuring string slicing.
Refusal tasks encode the
num_ctxeviction bug from #22. When the systemprompt was evicted, the "never invent file contents" guard vanished and the
model fabricated a config file.
refusal-no-invented-filefails on anyfabricated block.
A bug found while building this. That guard originally used
python_block,but the regressed fixture fabricates YAML — so it silently passed on output
that plainly invents a file. The guard for the symptom was inert. Fixed with a
language-agnostic
code_blockcheck (scoring.py), with a test pinning it.Every task ships two fixtures, well-behaved and regressed. A test asserts the
regressed set fails every task — a harness that only sees passing input isn't
tested.
Verification
Run locally; see the CI caveat below.
pytest— 262 passed, 13 skipped (was 221 on master)ruff check/ruff format— cleanmypystrict — clean across 86 source filesregressed input
GitHub Actions has run nothing since 2026-07-28 — every job fails in seconds
with "the job was not started because your account is locked due to a billing
issue." #22 merged unverified for the same reason. The checks on this PR will
fail identically, and that failure says nothing about this code. Everything
above was verified locally.
Not in scope
The other three P1s — parsed-file cache, RAG, context window management — are
untouched. Eval lands first so they're measurable, per the roadmap.
--livenow gives the TTFT number to prove the cache's speedup rather than assert it.
Also adds
ROADMAP_STATUS.md: P0sessions.pyand P2 streaming cancel werealready fixed (
0566e33band #22); the roadmap predated those merges.