feat(or-bench): add Harbor-based agent benchmark with two verified tasks

or-bench/.gitignore

@@ -0,0 +1,2 @@
+jobs/
+trials/

or-bench/README.md

@@ -0,0 +1,79 @@
+# or-bench
+
+A reproducible benchmark for measuring whether coding agents can build working
+OpenRouter integrations using public documentation and tools — in the spirit of
+[Tempo's stable-bench-v1](https://tempo.xyz/developers/blog/introducing-stable-bench-v1).
+
+or-bench measures three things per task:
+
+- **Efficacy** — did the agent produce a working integration? Verified against
+  live side-effects: the verifier queries the OpenRouter
+  [generation endpoint](https://openrouter.ai/docs/api-reference/get-a-generation)
+  to confirm the agent's code actually made the expected API calls (model,
+  streaming, token accounting).
+- **Quality** — does the integration follow current best practices? (LLM-judged
+  rubric; planned for v2.)
+- **Efficiency** — turns, tokens, and cost, read from the harness trajectory logs.
+
+## Architecture
+
+or-bench is built on [Harbor](https://harborframework.com), the open-source
+harness used by Terminal-Bench. Each task is a versioned directory with four
+parts:
+
+| Part | File(s) | Notes |
+|---|---|---|
+| Instruction | `instruction.md` | The prompt a developer would give an agent |
+| Environment | `environment/Dockerfile`, `task.toml` | Runtime, credentials, network access |
+| Oracle | `solution/solve.sh` | Hidden reference solution proving the task is solvable — agents never see it |
+| Verifier | `tests/test.sh`, `tests/verify.mjs` | Independent grader; writes a 0–1 reward to `/logs/verifier/reward.txt` |
+
+The agent runs inside a container with `OPENROUTER_API_KEY` injected and writes
+its submission as a project under `/app` plus an artifact at `/app/out.json`.
+The verifier inspects the artifact and cross-checks it against OpenRouter's
+generation API — the moral equivalent of Tempo verifying deployments on-chain.
+
+## Tasks
+
+| Task | Tests |
+|---|---|
+| [`streaming-chat`](tasks/streaming-chat) | Streaming chat completion with usage accounting via the OpenRouter API |
+| [`structured-outputs`](tasks/structured-outputs) | Strict JSON Schema structured outputs (`response_format: json_schema`) |
+
+## Running
+
+```bash
+uv tool install harbor
+
+export OPENROUTER_API_KEY=sk-or-...   # key used by the task env AND the verifier
+
+# Sanity check: oracle solutions should score 1.0
+harbor run --path tasks/streaming-chat --agent oracle
+harbor run --path tasks/structured-outputs --agent oracle
+
+# Evaluate a real agent
+harbor run --path tasks/streaming-chat --agent claude-code --model anthropic/claude-sonnet-4-5 \
+  --ae OPENROUTER_API_KEY=$OPENROUTER_API_KEY
+```
+
+Use a dedicated, disposable OpenRouter runtime key per run so verifier lookups
+of generation IDs are scoped to that run's traffic.
+
+## Scoring
+
+Each verifier awards partial credit across sub-checks (artifact schema, live
+generation lookup, streamed flag, model pinning, token accounting, result
+correctness) and writes the total to `/logs/verifier/reward.txt`. Compare
+rewards alongside token/turn counts from Harbor's trial output across agents
+and across documentation revisions.
+
+## Roadmap (v2)
+
+- Pinned-docs sidecar: serve a fixed revision of openrouter.ai/docs inside the
+  environment and rewrite egress for known doc URLs, so runs are reproducible
+  across doc changes and doc changes can be A/B tested against bench scores.
+- LLM-as-judge quality rubric via `[verifier.env]` judge keys.
+- More tasks: OAuth PKCE key provisioning, tool-calling agent loops with stop
+  conditions, model routing with `:free`/`:nitro` variants and fallbacks,
+  analytics API queries with a management key.
+- MCP-on/off comparison runs (`[environment].mcp_servers`).

or-bench/tasks/streaming-chat/environment/Dockerfile

@@ -0,0 +1,5 @@
+FROM node:22-slim
+
+RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates jq && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app

or-bench/tasks/streaming-chat/instruction.md

@@ -0,0 +1,31 @@
+Build a minimal Node.js (TypeScript or JavaScript) project in `/app` that
+streams a chat completion from OpenRouter.
+
+Requirements:
+
+1. Use the OpenRouter API. Documentation is available at
+   https://openrouter.ai/docs. An API key is provided in the
+   `OPENROUTER_API_KEY` environment variable.
+2. Use the model `openai/gpt-5-nano` exactly.
+3. The request must use **streaming** (`stream: true`, consuming the SSE
+   stream incrementally) and must enable **usage accounting** so token usage
+   is included in the final stream event.
+4. Send a single user message: `Reply with exactly the word: pong`.
+5. The project must expose an `npm run eval` script that performs the call.
+
+When `npm run eval` finishes, it must have written `/app/out.json` matching
+this schema:
+
+```json
+{
+  "generationId": "<the id returned by OpenRouter for this generation>",
+  "model": "<the model slug echoed by the API>",
+  "content": "<the full assistant message assembled from the stream chunks>",
+  "usage": {
+    "prompt_tokens": 0,
+    "completion_tokens": 0
+  }
+}
+```
+
+Run `npm run eval` yourself so that `/app/out.json` exists when you finish.

or-bench/tasks/streaming-chat/solution/solve.sh

@@ -0,0 +1,84 @@
+#!/bin/bash
+# Oracle reference solution. Agents never see this.
+set -euo pipefail
+
+cd /app
+
+cat > package.json <<'EOF'
+{
+  "name": "or-bench-streaming-chat",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "eval": "node index.mjs"
+  }
+}
+EOF
+
+cat > index.mjs <<'EOF'
+import { writeFileSync } from "node:fs";
+
+const res = await fetch("https://openrouter.ai/api/v1/chat/completions", {
+  method: "POST",
+  headers: {
+    Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
+    "Content-Type": "application/json",
+  },
+  body: JSON.stringify({
+    model: "openai/gpt-5-nano",
+    stream: true,
+    usage: { include: true },
+    messages: [{ role: "user", content: "Reply with exactly the word: pong" }],
+  }),
+});
+
+if (!res.ok) {
+  throw new Error(`OpenRouter request failed: ${res.status} ${await res.text()}`);
+}
+
+const decoder = new TextDecoder();
+let buffer = "";
+let content = "";
+let generationId = "";
+let model = "";
+let usage = null;
+
+for await (const chunk of res.body) {
+  buffer += decoder.decode(chunk, { stream: true });
+  let idx;
+  while ((idx = buffer.indexOf("\n")) !== -1) {
+    const line = buffer.slice(0, idx).trim();
+    buffer = buffer.slice(idx + 1);
+    if (!line.startsWith("data: ")) continue;
+    const data = line.slice(6);
+    if (data === "[DONE]") continue;
+    const event = JSON.parse(data);
+    generationId = event.id ?? generationId;
+    model = event.model ?? model;
+    content += event.choices?.[0]?.delta?.content ?? "";
+    if (event.usage) usage = event.usage;
+  }
+}
+
+if (!usage) throw new Error("no usage in stream; usage accounting not enabled?");
+
+writeFileSync(
+  "/app/out.json",
+  JSON.stringify(
+    {
+      generationId,
+      model,
+      content,
+      usage: {
+        prompt_tokens: usage.prompt_tokens,
+        completion_tokens: usage.completion_tokens,
+      },
+    },
+    null,
+    2,
+  ),
+);
+console.log("wrote /app/out.json");
+EOF
+
+npm run eval

or-bench/tasks/streaming-chat/task.toml

@@ -0,0 +1,35 @@
+schema_version = "1.4"
+
+[task]
+name = "or-bench/streaming-chat"
+version = "1.0.0"
+authors = []
+keywords = ["openrouter", "streaming", "sse", "usage-accounting"]
+
+[metadata]
+difficulty = "easy"
+category = "api-integration"
+tags = ["openrouter", "streaming"]
+
+[verifier]
+timeout_sec = 300.0
+
+[agent]
+timeout_sec = 900.0
+
+[environment]
+build_timeout_sec = 600.0
+cpus = 1
+memory_mb = 2048
+storage_mb = 10240
+gpus = 0
+mcp_servers = []
+
+[environment.env]
+OPENROUTER_API_KEY = "${OPENROUTER_API_KEY}"
+
+[verifier.env]
+OPENROUTER_API_KEY = "${OPENROUTER_API_KEY}"
+
+[solution.env]
+OPENROUTER_API_KEY = "${OPENROUTER_API_KEY}"

or-bench/tasks/streaming-chat/tests/test.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+set -u
+mkdir -p /logs/verifier
+node /tests/verify.mjs > /logs/verifier/verify.log 2>&1
+status=$?
+cat /logs/verifier/verify.log
+if [ ! -f /logs/verifier/reward.txt ]; then
+  echo 0 > /logs/verifier/reward.txt
+fi
+exit $status

or-bench/tasks/streaming-chat/tests/verify.mjs

@@ -0,0 +1,80 @@
+// Verifier for or-bench/streaming-chat.
+// Cross-checks /app/out.json against the live OpenRouter generation endpoint.
+import { readFileSync, writeFileSync, mkdirSync } from "node:fs";
+
+const EXPECTED_MODEL = "openai/gpt-5-nano";
+const checks = [];
+const check = (name, weight, ok, detail = "") => {
+  checks.push({ name, weight, ok: Boolean(ok), detail });
+  console.log(`${ok ? "PASS" : "FAIL"} [${weight}] ${name}${detail ? ` — ${detail}` : ""}`);
+};
+
+let out = null;
+try {
+  out = JSON.parse(readFileSync("/app/out.json", "utf8"));
+} catch (err) {
+  console.log(`could not read /app/out.json: ${err.message}`);
+}
+
+check(
+  "artifact schema",
+  0.2,
+  out &&
+    typeof out.generationId === "string" &&
+    out.generationId.length > 0 &&
+    typeof out.model === "string" &&
+    typeof out.content === "string" &&
+    out.usage &&
+    Number.isFinite(out.usage.prompt_tokens) &&
+    Number.isFinite(out.usage.completion_tokens),
+);
+
+check(
+  "content mentions pong",
+  0.1,
+  out && /pong/i.test(out.content ?? ""),
+  out ? JSON.stringify(out.content) : "",
+);
+
+let gen = null;
+if (out?.generationId) {
+  // The generation record can take a few seconds to become queryable.
+  for (let attempt = 0; attempt < 10 && !gen; attempt++) {
+    const res = await fetch(
+      `https://openrouter.ai/api/v1/generation?id=${encodeURIComponent(out.generationId)}`,
+      { headers: { Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}` } },
+    );
+    if (res.ok) {
+      gen = (await res.json()).data;
+    } else {
+      await new Promise((r) => setTimeout(r, 3000));
+    }
+  }
+}
+
+check("generation exists on OpenRouter", 0.25, gen, gen ? gen.id : "lookup failed");
+check("request was streamed", 0.2, gen?.streamed === true, `streamed=${gen?.streamed}`);
+// The generation record reports the dated permaslug (e.g. openai/gpt-5-nano-2025-08-07).
+const modelMatches = (m) => m === EXPECTED_MODEL || (typeof m === "string" && m.startsWith(`${EXPECTED_MODEL}-`));
+check(
+  "model pinned",
+  0.15,
+  modelMatches(gen?.model) && out?.model === EXPECTED_MODEL,
+  `gen.model=${gen?.model} out.model=${out?.model}`,
+);
+check(
+  "token accounting matches",
+  0.1,
+  gen &&
+    out &&
+    gen.native_tokens_prompt === out.usage.prompt_tokens &&
+    gen.native_tokens_completion === out.usage.completion_tokens &&
+    out.usage.completion_tokens > 0,
+  gen ? `gen=${gen.native_tokens_prompt}/${gen.native_tokens_completion} out=${out?.usage?.prompt_tokens}/${out?.usage?.completion_tokens}` : "",
+);
+
+const reward = checks.reduce((sum, c) => sum + (c.ok ? c.weight : 0), 0);
+mkdirSync("/logs/verifier", { recursive: true });
+writeFileSync("/logs/verifier/reward.txt", `${Math.round(reward * 100) / 100}\n`);
+writeFileSync("/logs/verifier/checks.json", JSON.stringify(checks, null, 2));
+console.log(`reward: ${reward}`);

or-bench/tasks/structured-outputs/environment/Dockerfile

@@ -0,0 +1,5 @@
+FROM node:22-slim
+
+RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates jq && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app

or-bench/tasks/structured-outputs/instruction.md

@@ -0,0 +1,35 @@
+Build a minimal Node.js (TypeScript or JavaScript) project in `/app` that uses
+OpenRouter **structured outputs** to extract data from text.
+
+Requirements:
+
+1. Use the OpenRouter API. Documentation is available at
+   https://openrouter.ai/docs. An API key is provided in the
+   `OPENROUTER_API_KEY` environment variable.
+2. Use the model `openai/gpt-5-nano` exactly.
+3. Use structured outputs: `response_format` with `type: "json_schema"` and
+   `strict: true`, so the model's reply is guaranteed to match your schema.
+4. Extract the fields `name` (string), `email` (string), and `age` (integer)
+   from this text:
+
+   > Maya Chen (reachable at maya.chen@example.com) joined the platform team
+   > last spring. At 34, she is the youngest principal engineer in the org.
+
+5. The project must expose an `npm run eval` script that performs the call.
+
+When `npm run eval` finishes, it must have written `/app/out.json` matching
+this schema:
+
+```json
+{
+  "generationId": "<the id returned by OpenRouter for this generation>",
+  "model": "<the model slug echoed by the API>",
+  "result": {
+    "name": "<extracted name>",
+    "email": "<extracted email>",
+    "age": 0
+  }
+}
+```
+
+Run `npm run eval` yourself so that `/app/out.json` exists when you finish.