ai-backend-benchmark

How well does an AI coding agent build TypeScript backends across popular frameworks? This repo drives Claude Code through a set of realistic backend tasks on each framework, captures every artifact of every run, and scores them on a consistent rubric so the numbers are comparable.

Each run is executed on a fresh exe.dev VM so the environment is identical across frameworks and has no state left over from previous runs. The harness only orchestrates - it doesn't sit in the agent's loop or help it along.

What gets measured

Five frameworks:

ID	Framework
encore-ts	Encore.ts
express	Express
fastify	Fastify
hono	Hono
nestjs	NestJS

Three independent tasks. Each task starts from the clean framework starter

• tasks do not build on each other's solutions.

ID	Task	What the agent has to do
t1-orders-api	Orders API	CRUD endpoints, Postgres schema, input validation, auth
t2-events-and-schedule	Events & schedule	Pub/sub notifications on order creation + scheduled daily aggregation
t3-tracing	Distributed tracing	OpenTelemetry-style trace propagation and a /trace endpoint

Per-cell metrics captured: test pass rate, turns to green, token cost, wall clock, tool-use breakdown, code diff, agent transcript. The final composite score weights test_pass_rate (50%) / turns_to_green (20%) / cost (15%) / wall_clock (15%); edit report/scoring.json to re-rank under your own weights.

Prerequisites

You need a local machine with Node and an SSH client, plus two paid services: exe.dev for the VMs, and Anthropic for the agent.

Local:

• Node.js 20+ and pnpm
• A standard SSH client (preinstalled on macOS / Linux / WSL)

Services (both require paid accounts):

• exe.dev account. The harness spawns one Ubuntu VM per framework x repeat via ssh exe.dev new, so you need:

  • An exe.dev account with a balance / active plan (basic plan at $20/month is enough as you get $100 credits)
  • Your SSH public key registered on your exe.dev account (easy to setup)
  • ssh exe.dev ls should return your (empty) VM list without asking for a password. If it asks for a password or gives a permission error, your SSH key isn't set up yet - fix that on the exe.dev dashboard before running anything.

• Anthropic API key with usage credit. A full 5x3x3 batch on claude-sonnet-4-6 typically costs ~$20 of Anthropic usage. Smaller scopes scale down linearly; a single --repeats=1 smoke on one framework x one task is under $1.

You do not need Claude Code, Postgres, or Docker installed locally. The agent runs on the exe.dev VM; Postgres and every framework dep are provisioned on the VM too.

Setup

git clone https://github.com/ivancernja/ai-backend-benchmark
cd ai-backend-benchmark
pnpm install
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env

# sanity-check exe.dev SSH access before your first run
ssh exe.dev ls

Running

The CLI has one subcommand: bench run. All flags are optional.

pnpm bench run \
  [--framework=<ids>]   # comma-separated, or "all" (default)
  [--task=<ids>]        # comma-separated, or "all" (default)
  [--repeats=<N>]       # default 3
  [--parallel=<N>]      # default 1 (each parallel slot = one VM)
  [--model=<id>]        # default claude-sonnet-4-6
  [--batch=<name>]      # default: auto-generated timestamp
  [--notes=<text>]      # stored in batch.json

One VM is provisioned per framework x repeat and runs all selected tasks sequentially (each task from a clean starter) to amortize the VM setup cost. So --framework=all --repeats=3 with 3 tasks spawns 15 VMs (5 frameworks x 3 repeats), each running 3 tasks back-to-back.

Examples

Smoke test - one framework, one task, one run:

pnpm bench run --framework=express --task=t1-orders-api --repeats=1

All frameworks on T1 and T2, one run each, 2 in parallel:

pnpm bench run \
  --framework=all \
  --task=t1-orders-api,t2-events-and-schedule \
  --repeats=1 \
  --parallel=2 \
  --batch=t1-t2-smoke

Full benchmark - every framework, every task, 3 repeats for variance:

pnpm bench run --repeats=3 --parallel=5 --batch=full-v1

Encore.ts only, all tasks, detailed notes:

pnpm bench run \
  --framework=encore-ts \
  --repeats=3 \
  --notes="baseline before skill update"

Viewing results

pnpm report                       # latest batch
pnpm report --batch=<batch-id>    # specific batch
pnpm report --all-batches         # combined report across every run

The report writes RESULTS.md into the batch directory (or repo root for --all-batches).

Output layout

runs/
  <batch-id>/
    batch.json                    # batch-level metadata
    RESULTS.md                    # rendered report
    <framework>/
      <task>/
        run-<N>/
          metrics.json            # structured per-cell metrics
          diff.patch              # final diff vs framework starter
          transcript.jsonl        # full Claude Code transcript
          vitest.json             # raw test output
          server.log              # framework stdout/stderr

runs/ is gitignored - artifacts stay local unless you explicitly commit a curated subset.

Adding a framework

1. Create frameworks/<id>/ with:
   • framework.json (see an existing framework for the schema)
   • starter/ - the minimal starter project the agent begins from
   • setup.sh - VM setup commands appended to the base script
2. Optionally add .claude/CLAUDE.md inside the starter if the framework ships agent-facing rules (Encore does; the others don't).
3. Run a smoke test: pnpm bench run --framework=<id> --task=t1-orders-api --repeats=1.

Adding a task

1. Create tasks/<id>/ with:
   • task.json - task metadata and required endpoints
   • PROMPT.md - the prompt the agent sees
   • tests/ - black-box vitest test suite the harness runs against the finished server
2. Set order to control when this task runs within a VM relative to other tasks (tasks run in order order, but each from a clean starter).

Other tools

pnpm test         # unit tests for the harness and report
KEEP_VM=1 pnpm bench run ...   # leave VMs running on failure for debugging

Reproducibility notes

• Claude Code is not pinned to a specific version - each run records the version it used in metrics.json. Absolute numbers drift with Claude releases; framework-to-framework comparisons within a single batch stay valid.
• Every framework's shipped agent-facing materials (CLAUDE.md, MCP config, skills) are loaded as-is. The benchmark measures what a user actually gets out of the box, not a normalized agent context.
• All runs are executed on identical Ubuntu VMs provisioned by exe.dev's default image. Framework-specific setup (Encore CLI install, etc.) is in each frameworks/<id>/setup.sh.