Add an evaluation suite for the Workshop agent - #271
Draft
AshishKumar4 wants to merge 13 commits into
Draft
Conversation
This was referenced Aug 19, 2026
Maximo-Guk
reviewed
Aug 19, 2026
AshishKumar4
force-pushed
the
evals/suite
branch
from
August 19, 2026 21:07
04f84ae to
16b53b9
Compare
Gives the agent a prompt in a real workerd Workshop, then verifies the Gadget it built by calling that Gadget's own RPC. The score is the fraction of checks that passed; nothing inspects which tools the agent used, so a working result reached an unexpected way still passes. A task is a prompt and a verifier. The framework owns Gadget lookup, evidence normalization, and the isolation of a throw to the check that caused it. A task can restart every Gadget server mid-verification, which is how it tells persisted state apart from state held in a field. Repeated trials are summarized with a pass rate, a Wilson 95% interval, and the tokens, duration, and cost per trial. A trial whose result does not describe the agent is left unscored and counted separately, so a rate limit reads as missing data rather than as a regression. Two tasks ship here as worked examples. docs/testing-strategy.md describes the design, and docs/integration-testing.md now says which of the four suites to use.
Six more applications, each verified through its own RPC: a household ledger with cent-exact splitting and a schema migration across two turns, a pantry that turns recipes into a shopping list across mass and volume units, flashcards on an SM-2 schedule, a freelance timesheet, an org chart from pasted text, and project notes as a standard Doc. Most checks assert an invariant rather than a constant, because an invariant survives an arithmetic slip in the task. The constants that remain match a reference implementation. Five tasks passed a live run and gate the job. Three carry the frontier state: one passes a third of its trials, and two have no measurement yet.
The scope section asserted that unit tests cannot measure the agent without saying why. It now names the three differences -- an English prompt, code that does not exist until the test runs, and a model that is not deterministic -- and the consequence: a distribution instead of a boolean. Records the boundary as a fact rather than a claim. No other suite sends a prompt to a model, and no other suite runs code that an agent wrote.
The agent's webFetch tool is always available to it, so a trial could reach a live third-party site: that makes a result depend on something outside the trial, and sends the prompt to someone else. The harness now installs a network interceptor that passes only the model provider through. A refused request lands in the trial's diagnostics, so an attempt is visible rather than silent. Diagnostics also gain the model turn count, taken from the assistant messages in the transcript. The summary reports turns and tool calls per trial as distributions, and the fraction of trials that tried to reach a blocked host.
Syntax needs no check of its own: every .js file in a Gadget becomes a module in its Worker, so a file that cannot parse stops the server and fails every check. What nothing caught was code that parses, loads, runs, and does nothing, because it calls a web API the Gadget frame does not have. source-checks.ts reports each such use. It found a real one in an earlier generated Gadget, which read localStorage inside an opaque-origin frame, where it always throws. The feature built on it silently did nothing. Reported as a diagnostic, never as a verdict: the code the task asked about worked, so the task's own checks were right to pass.
It auto-provisions its account with no OAuth and keeps its own state, so it starts offline. A probe confirmed that the Workshop discovers it and mints an account, which makes it the candidate for the first connector task.
AshishKumar4
force-pushed
the
evals/suite
branch
from
August 21, 2026 16:56
436e02a to
3f82fd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an evaluation suite that measures what the Workshop agent delivers.
Stacked on #270, which supplies the session driver. Review that one first; this diff shows only its own
commits once #270 merges.
docs/testing-strategy.mddescribes the design. This description gives the summary.How it works
A task holds one or more prompts and a verifier. The harness starts a workerd Workshop, creates a fresh
account, sends each prompt, and then runs the verifier. The verifier calls the RPC of the Gadget that
the agent built.
The score is the fraction of checks that passed. Nothing inspects which tools the agent called.
For example, one task asks for an appointment desk that never sells more places than it has. The agent
satisfied it with one synchronous SQL sequence and a database trigger. A check for a mutation queue
rejects that correct answer.
The tool calls, the errors, the turns, the tokens, the duration, and the cost are diagnostics. They
explain a result and never decide it.
Trials that do not count
A trial is invalid when its result does not describe the agent: the harness did not start, the task
recorded no checks, or the turn ended with an error and the trial also failed. The summary excludes an
invalid trial from every rate and counts it separately, so a rate limit reads as missing data rather
than as a regression.
Gates
ci.ymlworkshop-evals.ymlrun-evalslabelpnpm build,pnpm test,pnpm lintpnpm eval:required,pnpm eval:frontierA required task fails the job when its score is below 1. A frontier task records its score and
never fails. We set the state from measurement, and a new task starts as frontier.
Content
stock ledger, flashcards, a timesheet, an org chart, and project notes as a standard Doc.
State
Five tasks passed a live run against
@cf/zai-org/glm-5.2. One passed a third of its trials, and Irecorded the two causes on it. Two have no live measurement, because repeated attempts hit the Workers
AI rate limit. All three carry the frontier state, so none can fail the job.
Draft
This needs a baseline of 10 trials against two models. That run sets the state of each task.