Skip to content

feat(eval): accuracy + latency benchmark harness - #23

Merged
emretheus merged 2 commits into
masterfrom
feat/eval-benchmarks
Jul 31, 2026
Merged

feat(eval): accuracy + latency benchmark harness#23
emretheus merged 2 commits into
masterfrom
feat/eval-benchmarks

Conversation

@emretheus

Copy link
Copy Markdown
Owner

Adds packages/eval — the P1 "Eval & benchmarks" item, plus the P2 latency
instrumentation that feeds it. Closes the roadmap's biggest gap: the repo had
200+ passing tests but zero numbers on whether the tool actually works.

What's here

Two model backends sharing one scoring path:

Mode Command Ollama CI
Fixture inclave-eval no yes — gates every PR
Live inclave-eval --live yes no

Fixture mode replays recorded replies, so it's deterministic and gates CI via
--fail-under. It verifies the harness and scorers, not model quality. Live
mode streams from a real model and is the only mode producing real accuracy and
latency numbers. Only the backend differs — scoring, timing, and reporting are
identical.

Six tasks across four categories: codegen, file_qa, refusal, format.

Notes for review

Scorers call the product's own python_blocks_in rather than
reimplementing extraction. If block detection regresses, the eval fails instead
of a parallel implementation masking it.

Latency records time-to-first-token separately from total duration. A run
can stream fast overall and still feel slow if the model stalls before the first
chunk; one total-duration number can't distinguish those. Uses perf_counter
so NTP steps don't corrupt the numbers. Fixture runs don't report latency — it
would be measuring string slicing.

Refusal tasks encode the num_ctx eviction bug from #22. When the system
prompt was evicted, the "never invent file contents" guard vanished and the
model fabricated a config file. refusal-no-invented-file fails on any
fabricated block.

A bug found while building this. That guard originally used python_block,
but the regressed fixture fabricates YAML — so it silently passed on output
that plainly invents a file. The guard for the symptom was inert. Fixed with a
language-agnostic code_block check (scoring.py), with a test pinning it.

Every task ships two fixtures, well-behaved and regressed. A test asserts the
regressed set fails every task — a harness that only sees passing input isn't
tested.

Verification

Run locally; see the CI caveat below.

  • pytest262 passed, 13 skipped (was 221 on master)
  • ruff check / ruff format — clean
  • mypy strict — clean across 86 source files
  • Gate proven both directions: exit 0 at 100%, exit 1 with named reasons on
    regressed input

⚠️ CI cannot verify this

GitHub Actions has run nothing since 2026-07-28 — every job fails in seconds
with "the job was not started because your account is locked due to a billing
issue."
#22 merged unverified for the same reason. The checks on this PR will
fail identically, and that failure says nothing about this code. Everything
above was verified locally.

Not in scope

The other three P1s — parsed-file cache, RAG, context window management — are
untouched. Eval lands first so they're measurable, per the roadmap. --live
now gives the TTFT number to prove the cache's speedup rather than assert it.

Also adds ROADMAP_STATUS.md: P0 sessions.py and P2 streaming cancel were
already fixed (0566e33b and #22); the roadmap predated those merges.

Adds packages/eval: a task set, scorers, latency instrumentation, and a
report layer, plus an inclave-eval CLI with two backends.

Fixture mode replays recorded replies — deterministic, no Ollama, so it
gates every PR via --fail-under. Live mode streams from a real model and
is the only mode producing real accuracy and latency numbers. Both score
through identical logic; only the backend differs.

Scorers call the product's own python_blocks_in rather than
reimplementing extraction, so a regression in block detection surfaces as
an eval failure instead of being masked by a parallel implementation.

Latency records time-to-first-token separately from total duration: a run
can stream fast overall and still feel slow if the model stalls before
the first chunk, and one total-duration number cannot tell those apart.

Refusal tasks encode the num_ctx eviction bug's symptom — a model that
lost the system prompt fabricated file contents. The guard checks for a
fenced block in any language, not just python: an invented YAML config is
still an invented file.
@emretheus
emretheus merged commit 6302505 into master Jul 31, 2026
0 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant