Add agent evaluations and Gadget durability tests - #268
Closed
AshishKumar4 wants to merge 8 commits into
Closed
Conversation
AgentSession drives the production agent over the same Cap'n Web RPC the browser uses: fresh account and workspace, one chat across turns, waiting through callback-driven restarts, and reading complete paginated history. It owns transport lifecycle only, so evaluation semantics stay with the consuming suite. startHarness() gains enableGadgetExecution, keeping the checked-in Worker Loader so an agent can run executeCode. It defaults to false, leaving the existing observer suites unchanged.
Gives the production agent a prompt in a real workerd Workshop, then verifies the Gadget it built by calling that Gadget's own RPC. Scoring is deterministic and outcome-only: nothing asserts which tools the agent used or in what order, so a working result reached by an unexpected route passes. Tool failures, agent errors, tokens, time and cost are recorded as diagnostics and never decide a verdict. Defining an eval is a task file plus a registry line, verified by a test so the two cannot drift. A task is prompts and checks; the framework owns Gadget lookup, evidence normalization, and isolating a throw to the check that caused it. Repeated trials are summarized with pass rates, Wilson intervals, and per-trial cost, with unmeasured telemetry reported as absent rather than zero. workshop-agent-cli runs one prompt through the same session driver, for iterating on the agent without a browser. The two packages share one lockfile change, so they land together.
Sharded across jobs, required and frontier reported separately, with frontier never gating. Kept out of CI: each trial boots workerd and spends real inference.
The first three tasks were API-shape smoke tests. These are applications people actually build: a double-entry household ledger with cent-exact splitting and settlement, a pantry that turns recipes into a shopping list across mass and volume units, an appointment desk that must not oversell under concurrent booking, flashcards on an SM-2 schedule, and a freelance timesheet with half-open overlap and billing rounding. Grounded in what the platform can really do: no gadget-to-gadget bindings, no network, no client-side frameworks, no wall clock. Every one is self-contained, driven entirely through its own RPC, and told the date it is reasoning about. Verification leans on invariants over hand-computed constants where it can -- balances summing to zero, transfers clearing them, shares totalling the expense -- because an invariant survives an author's arithmetic slip and still catches the bug it describes. The constants that remain were checked against a reference implementation first, which is how a wrong assertion in the ledger was found sitting next to correct values. Tiers now follow measurement. Five tasks passed every trial and gate CI; the timesheet passed one of three, failing the same two things both times, so it is frontier with the observed gaps recorded on it. Two harness fixes came out of running these: an agent that errors without ever calling a tool is unscored rather than scored zero, so an expired token cannot masquerade as a bad model; and the summary refuses to write over the report it is reducing.
The suite asserted an agent-built booking gadget never oversells under concurrent load, but never showed that a bad implementation would be caught. It now does: a seeded gadget written the obvious wrong way -- read the count, check capacity, await, then insert -- oversells here, so the check has teeth rather than passing for free. Seeding is the new capability behind that. AgentSession.seedGadget() writes hand-authored source straight into the workspace, so a test can pin platform behaviour with no model involved. gadget-durability.test.ts uses it to establish what a restart does: storage survives, memory does not, and outstanding stubs are invalidated -- which is the only honest way to prove a restart happened at all. It holds across five consecutive restarts and across one landing on an unfinished write. Deterministic, free, and part of pnpm test. verifier.restart() exposes the same abrupt restart to eval tasks, because on this platform a Gadget's server goes away on every code change rather than only when something breaks. stock-ledger asks whether an app's state was really persisted: it restarts between appends, immediately after one, and mid-write, and re-checks duplicate rejection afterwards -- an id index kept in memory passes until the first restart. It proves the restart was real before trusting any of that. project-doc covers the standard output formats, which nothing exercised before: asked for a doc, the platform should hand back a real Doc from the shipped blueprint, not a bespoke imitation. Verified as an outcome -- the workpiece is presented as a document output and answers the Docs contract. The harness now waits for formats to install before the first prompt, since that install races the session and silently changes what the agent is told. Scoring got more honest on the way. A failing trial whose turn ended with an error is left unscored: the platform posts an error when a turn dies and never for running out of turns or giving up, so it is almost always infrastructure, and such a failure cannot be pinned on the agent. A trial that passed despite one is still scored. Both new tasks start as frontier because neither has a live baseline yet -- repeated attempts hit Workers AI rate limiting, which is also why CI now caps shard concurrency.
Four suites now cover more than one module at a time, and the canonical doc described two: it predates both workshop-backend/__integration__ and the eval package. Omitting them is what makes someone rebuild a capability that already exists, or fight a runtime that cannot reach what they need. The dividing line is in-process versus out-of-process workerd, and it decides what is available. Only the in-process suite has cloudflare:test, so aborting one Durable Object or asserting on a native RPC boundary can only happen there. Only the out-of-process suites get real transport, several bound Workers, and a Worker Loader running Gadget code. That is also why resetting looks different in each and neither approach ports, which is worth stating where someone will read it before deciding the eval harness should have used abortAllDurableObjects.
States the approach to tests above the unit level in one place: why we separate integration tests from evaluations, the properties we want, the four suites and what each runtime can reach, how we score a result rather than a method, and the two gates. Written in Simplified Technical English (ASD-STE100).
Replaces the aspirational framing with the mechanisms: the entry points of the toolkit, the six steps of an evaluation run, the exact conditions that make a trial invalid, the triggers and commands of each gate, and the figures the summary reports. Corrects one error in the first draft: incomplete telemetry does not make a trial invalid. It only removes that trial from the token, duration, and cost figures.
Contributor
Author
|
Replaced by a stack, so the toolkit can be reviewed apart from the suite:
I also dropped the agent CLI from the change. Nothing in the suite used it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an evaluation suite that measures what the Workshop agent delivers, and integration tests that
pin the platform behaviour the suite depends on.
docs/testing-strategy.mddescribes the whole design. This description gives the summary.What this adds
packages/workshop-evals— 8 tasks. Each task gives the agent a prompt, then calls the RPC of theGadget that the agent built. A task scores the result, and never the method.
packages/integration-tests— two new capabilities onAgentSession.seedGadget()writeshand-authored source, so a test can use a known implementation.
restartGadgets()restarts everyGadget server.
packages/integration-tests/__tests__/gadget-durability.test.ts— 4 tests. They run inpnpm test,they need no model, and they finish in 5 seconds.
packages/workshop-agent-cli— runs one prompt locally, without a browser..github/workflows/workshop-evals.yml— runs the evaluations each night and on demand.Scoring
The score is the fraction of checks that passed. Nothing inspects the tools that the agent called.
One task asks for an appointment desk that never sells more places than it has. The agent passed it
with one synchronous SQL sequence and a database trigger. A check for a mutation queue would have
failed that correct answer.
The tool calls, the errors, the turns, the tokens, the time, and the cost are diagnostics. They explain
a result, and they never decide it.
A trial is invalid when we cannot attribute its result to the agent. The summary excludes an invalid
trial from every rate, and counts it in a separate column. A rate limit therefore reads as missing
data, and not as a regression.
Gates
run-evalslabelpnpm build,pnpm test,pnpm lintpnpm eval:required,pnpm eval:frontierA required task fails Gate 2 when its score is below 1. A frontier task records its score and
never fails. We set the state from measurement. A new task starts as frontier.
Verified
pnpm build,pnpm test, andpnpm lintpass. 76 unit tests, plus the 4 durability tests.interrupts a write.
@cf/zai-org/glm-5.2. One task passed one trial in three.Not verified
carry the frontier state, so they cannot fail the job.
restartGadgets()on the mainline branch only. On a chat branch,stock-ledgerprovesthe restart before it trusts it, so a silent failure reports itself.
One run against an account with fresh quota closes both:
WORKSHOP_EVAL_TRIALS=3 pnpm eval:frontier.Draft
I opened this as a draft. It needs a baseline run with 10 trials and 2 models. That run decides the
state of each task.