feat(eval): add agent-eval harness for gqlkit vs codegen/Pothos - #313
Draft
izumin5210 wants to merge 2 commits into
Draft
feat(eval): add agent-eval harness for gqlkit vs codegen/Pothos#313izumin5210 wants to merge 2 commits into
izumin5210 wants to merge 2 commits into
Conversation
Why: §9 of the TSKaigi 2026 talk needs empirical evidence on whether AI
agents can write idiomatic gqlkit (and avoid N+1, hide sensitive fields)
without explicit hints, and how much the bundled gqlkit-guide skill helps.
Summary:
- New @gqlkit-ts/eval workspace package wrapping @vercel/agent-eval.
- 8 evals: {01-crud, 02-relation} x {codegen, Pothos, gqlkit-plain,
gqlkit-skill}. PROMPTs deliberately omit hints about mappers /
objectRef / DataLoader / N+1; EVAL.ts checks those properties anyway.
- shared/ holds the source-of-truth backing types + DataLoader stubs;
sync-fixtures.ts mirrors them into each eval and copies the gqlkit-guide
skill (from `pnpm gqlkit docs` output) into the gqlkit-skill variants.
- experiments/sonnet.ts targets claude-code x claude-sonnet-4-6 (runs:1
for the first pass). All 8 evals pass end-to-end against sonnet.
- biome.jsonc: exclude testdata and *.md so lint stays green on
pre-existing repo state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Why: capture the eval results so the §9 slide deck can reference concrete numbers (40/40 pass, skill-vs-plain deltas, DataLoader/email-hiding rates). Summary: - scripts/summarize.ts walks results/sonnet/<model>/<timestamp>/<eval>/ run-1/ and emits a markdown report. Defaults to the latest 5 iterations (= 40 runs from the standard `pnpm sonnet --force` loop). - REPORT.md captures one such run against claude-sonnet-4-6: 40/40 passed, with the skill variant ~30% fewer turns and ~17-42% faster than the plain variant on the same library. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
§9 of the TSKaigi 2026 talk needs empirical evidence on whether an AI coding
agent can write idiomatic gqlkit (and naturally avoid N+1 / hide sensitive
fields) without explicit hints in the prompt — and how much the bundled
gqlkit-guideskill helps once it ships.This PR builds the harness that produces that evidence and captures one
40-run pass against
claude-sonnet-4-6.Summary
New
@gqlkit-ts/evalworkspace package wrapping [@vercel/agent-eval]:{01-crud, 02-relation}×{codegen, Pothos, gqlkit-plain, gqlkit-skill}.mappers/objectRef/ DataLoader / N+1.EVAL.ts checks those properties anyway by inspecting the agent's output.
shared/holds the source-of-truth backing types + DataLoader stubs;sync-fixtures.tsmirrors them into every eval and copies thegqlkit-guideskill (the output ofpnpm gqlkit docs) into the*-gqlkit-skillvariants.experiments/sonnet.tstargetsclaude-code × claude-sonnet-4-6.scripts/summarize.tswalksresults/and emits a markdown report.REPORT.mdcaptures the latest 40-run result.Initial findings (see
REPORT.mdfor the full breakdown):DataLoader /
User.email-exclusion assertions pass in every run acrossall 40, with no prompt-side hints.
Notes for reviewers
eval/is added as a new workspace package (@gqlkit-ts/eval) viapnpm-workspace.yaml. It isprivate: trueand not part of the publicsurface.
biome.jsonc:**/testdataand**/*.mdare now excluded.testdata/was already partially excluded (__generated__only). Thenew exclusion covers the rest; lint errors that surfaced under
packages/cli/src/gen-orchestrator/testdata/were pre-existing andunrelated to this PR.
*.mdexclusion was needed because Biome 2.4 reports an "unsupportedfeature" for Markdown, and this PR adds many
.mdfiles from thebundled
gqlkit-guideskill snapshot.packages/cli/src/shared/{directive-definition-extractor,directive-detector,interface-validator}.ts:biome
--unsafeauto-fix applieduseOptionalChain(3 small rewrites).Functionally equivalent.
examples/with-ai-sdk/src/scenario.test.ts: one// biome-ignoreline fornoExplicitAnyon aLanguageModelV3mock that needsany[]to bypassstrict stream-part typing.
eval/shared/skill/.claude/skills/gqlkit-guide/referencesis committed asa symlink pointing into
packages/cli/docs/(matches whatgqlkit docs --claudeemits). The sandbox-side copies are materialized asreal directories via
sync-fixtures.ts, so this only matters on the host.How it ran