Skip to content

feat(eval): add agent-eval harness for gqlkit vs codegen/Pothos - #313

Draft
izumin5210 wants to merge 2 commits into
mainfrom
feat/agent-eval-harness
Draft

feat(eval): add agent-eval harness for gqlkit vs codegen/Pothos#313
izumin5210 wants to merge 2 commits into
mainfrom
feat/agent-eval-harness

Conversation

@izumin5210

Copy link
Copy Markdown
Owner

Why

§9 of the TSKaigi 2026 talk needs empirical evidence on whether an AI coding
agent can write idiomatic gqlkit (and naturally avoid N+1 / hide sensitive
fields) without explicit hints in the prompt — and how much the bundled
gqlkit-guide skill helps once it ships.

This PR builds the harness that produces that evidence and captures one
40-run pass against claude-sonnet-4-6.

Summary

New @gqlkit-ts/eval workspace package wrapping [@vercel/agent-eval]:

  • 8 evals: {01-crud, 02-relation} × {codegen, Pothos, gqlkit-plain, gqlkit-skill}.
  • PROMPTs deliberately omit hints about mappers / objectRef / DataLoader / N+1.
    EVAL.ts checks those properties anyway by inspecting the agent's output.
  • shared/ holds the source-of-truth backing types + DataLoader stubs;
    sync-fixtures.ts mirrors them into every eval and copies the
    gqlkit-guide skill (the output of pnpm gqlkit docs) into the
    *-gqlkit-skill variants.
  • experiments/sonnet.ts targets claude-code × claude-sonnet-4-6.
  • scripts/summarize.ts walks results/ and emits a markdown report.
  • REPORT.md captures the latest 40-run result.

Initial findings (see REPORT.md for the full breakdown):

gqlkit-plain gqlkit-skill delta
01-crud mean turns 11.0 7.8 -29%
01-crud mean duration 592s 345s -42%
02-relation mean turns 11.2 7.8 -30%
02-relation mean duration 466s 387s -17%

DataLoader / User.email-exclusion assertions pass in every run across
all 40, with no prompt-side hints.

Notes for reviewers

  • eval/ is added as a new workspace package (@gqlkit-ts/eval) via
    pnpm-workspace.yaml. It is private: true and not part of the public
    surface.
  • biome.jsonc: **/testdata and **/*.md are now excluded.
    • testdata/ was already partially excluded (__generated__ only). The
      new exclusion covers the rest; lint errors that surfaced under
      packages/cli/src/gen-orchestrator/testdata/ were pre-existing and
      unrelated to this PR.
    • *.md exclusion was needed because Biome 2.4 reports an "unsupported
      feature" for Markdown, and this PR adds many .md files from the
      bundled gqlkit-guide skill snapshot.
  • packages/cli/src/shared/{directive-definition-extractor,directive-detector,interface-validator}.ts:
    biome --unsafe auto-fix applied useOptionalChain (3 small rewrites).
    Functionally equivalent.
  • examples/with-ai-sdk/src/scenario.test.ts: one // biome-ignore line for
    noExplicitAny on a LanguageModelV3 mock that needs any[] to bypass
    strict stream-part typing.
  • eval/shared/skill/.claude/skills/gqlkit-guide/references is committed as
    a symlink pointing into packages/cli/docs/ (matches what
    gqlkit docs --claude emits). The sandbox-side copies are materialized as
    real directories via sync-fixtures.ts, so this only matters on the host.

How it ran

cd eval
cp .env.example .env  # ANTHROPIC_API_KEY + DOCKER_HOST for Colima
pnpm sync             # distribute shared fixtures + skill into each eval
for i in 1 2 3 4 5; do pnpm sonnet --force; done   # 5 × 8 = 40 runs
pnpm tsx scripts/summarize.ts 5                    # regenerate REPORT.md

izumin5210 and others added 2 commits May 22, 2026 04:39
Why: §9 of the TSKaigi 2026 talk needs empirical evidence on whether AI
agents can write idiomatic gqlkit (and avoid N+1, hide sensitive fields)
without explicit hints, and how much the bundled gqlkit-guide skill helps.

Summary:
- New @gqlkit-ts/eval workspace package wrapping @vercel/agent-eval.
- 8 evals: {01-crud, 02-relation} x {codegen, Pothos, gqlkit-plain,
  gqlkit-skill}. PROMPTs deliberately omit hints about mappers /
  objectRef / DataLoader / N+1; EVAL.ts checks those properties anyway.
- shared/ holds the source-of-truth backing types + DataLoader stubs;
  sync-fixtures.ts mirrors them into each eval and copies the gqlkit-guide
  skill (from `pnpm gqlkit docs` output) into the gqlkit-skill variants.
- experiments/sonnet.ts targets claude-code x claude-sonnet-4-6 (runs:1
  for the first pass). All 8 evals pass end-to-end against sonnet.
- biome.jsonc: exclude testdata and *.md so lint stays green on
  pre-existing repo state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Why: capture the eval results so the §9 slide deck can reference concrete
numbers (40/40 pass, skill-vs-plain deltas, DataLoader/email-hiding rates).

Summary:
- scripts/summarize.ts walks results/sonnet/<model>/<timestamp>/<eval>/
  run-1/ and emits a markdown report. Defaults to the latest 5 iterations
  (= 40 runs from the standard `pnpm sonnet --force` loop).
- REPORT.md captures one such run against claude-sonnet-4-6: 40/40 passed,
  with the skill variant ~30% fewer turns and ~17-42% faster than the
  plain variant on the same library.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant