Add an agent session driver and Gadget durability tests

packages/integration-tests/README.md

@@ -1,66 +1,75 @@
 # integration-tests
 
-End-to-end tests that drive the real Workshop and a real gatekeeper over the actual RPC API, plus the
-toolkit those tests are built from. Part of `pnpm test`, so CI runs it like any other package.
+End-to-end tests that drive the Workshop and gatekeepers through the public Cap'n Web API. The package
+also exports the small runtime bridge used by live agent evals.
 
-```bash
+```sh
 pnpm --filter @gadgets/integration-tests test:run
 ```
 
-## The toolkit
-
-Three source-only modules, consumed both by the tests here and by per-vendor suites in repos that
-vendor this one as a submodule:
-
-- **`src/harness.ts`** — boots `workshop-backend` and any set of gatekeepers as real Workers under
-  [`wrangler`'s `createTestHarness()`](https://developers.cloudflare.com/changelog/post/2026-07-21-integration-test-harness/),
-  patching their checked-in `wrangler.jsonc` in memory. Parameterised over gatekeepers on purpose: a
-  suite for a new gatekeeper should be "point the harness at the package", not a forked copy.
-- **`src/network-interceptor.ts`** — `NetworkInterceptor`, mechanism only. It patches
-  `globalThis.fetch` (the harness routes Worker subrequests back through the Node process, so that is
-  enough), passes loopback through, and **throws on anything a handler didn't match** — a test cannot
-  reach the real internet. What a given vendor's endpoints answer lives in a handler module you pass
-  in, which is what makes it reusable across gatekeepers.
-- **`src/rpc-client.ts`** — speaks Cap'n Web over a WebSocket to `/api`, the same transport the
-  browser uses: sign-up, reading connected accounts, and `ObserverConfigRecorder`, which records the
-  overseer's `configure()` calls and answers from a scripted queue.
-
-## Writing a test here
-- **No test may assume a clean slate.** Everything in a file shares one harness, `it.concurrent` runs
-  the cases together, and storage is never reset. Take fresh identities from `nextUsernames()` and use
-  per-test resource URLs; account labels are allocated for you, so two tests can't pick the same one.
-- **The escape assertion lives in `afterAll`, not `afterEach`** — an `afterEach` fires while sibling
-  tests are still running, so it would inspect and clear state they are still using.
-
-## The fixture gatekeeper
-
-`fixtures/gatekeeper-test/` is a real Worker speaking the real gatekeeper protocol, whose verification
-outcome the tests set over an HTTP control route. It exists because the overseer cases need a
-gatekeeper that will refuse an observer *on command*, and every shipping one can do that only at a
-cost that would dominate the test:
-
-- The OAuth ones need a whole vendor auth surface mocked before an account exists at all.
-- The Context Library only refuses after an observation has been *recorded*, which takes a gadget read
-  session, a slash command, or an AI-chat catalog snapshot — and it is a singleton, so it cannot
-  produce two simultaneously failing bindings.
-
-Adding a test hook to those workers was considered and rejected: a "mark observed" hook would stub the
-very state the tracker maintains, and an injected dev credential for an OAuth gatekeeper would bypass
-exactly the flow that makes a real vendor worth testing.
-
-Two deliberate departures from a shipping gatekeeper, both to keep the fixture cheap:
-
-- No `capnweb-validate` build step; `main` points straight at source. `@validateRpc()` would require
-  the fixture to carry its own `wrangler types` output — half a megabyte of generated `.d.ts` for a
-  test double. The harness's handling of a generated `main` is covered anyway, by `workshop-backend`.
-- One control knob, `allow`. A settled denial and an expired credential reach the overseer identically
-  — both as a thrown error, which it deliberately cannot tell apart because it treats every failure as
-  repairable — so the reason string is what carries the difference. Tests cover both narratives by
-  choosing reason text.
-
-## Further reading
-
-[`docs/integration-testing.md`](../../docs/integration-testing.md) covers the reasoning behind the
-shape of all this: why fake timers cannot work here, why a fixture gatekeeper rather than a real one,
-how storage isolation works, and the capnweb, wrangler, and workerd traps to expect. Read it before
-changing the toolkit or starting a suite of your own.
+`workshop-backend/__integration__` runs inside workerd and can use `cloudflare:test` internals. This
+package runs from Node and reaches only the same public API as a real client.
+
+## Exported toolkit
+
+- `harness` boots the backend and selected gatekeepers as real Workers with
+  `wrangler`'s `createTestHarness()`.
+- `rpc-client` opens Cap'n Web sessions, authenticates test users, and supplies observer callbacks.
+- `agent-session` drives one production agent chat, waits for it to settle, reads complete history,
+  discovers workpieces, connects to Gadget RPCs, and exposes provider usage metadata.
+- `network-interceptor` supplies explicit HTTP handlers to gatekeeper suites and rejects unexpected
+  external requests.
+
+The toolkit owns transport and lifecycle only. A consuming test or eval owns prompts, scores, and
+behavioral assertions.
+
+## Local agent session
+
+Keep Gadget execution enabled and configure the Workshop with existing model credentials:
+
+```ts
+const harness = await startHarness({
+  enableGadgetExecution: true,
+  gatekeepers: [],
+  patchWorkshop(config) {
+    config.vars = {
+      ...config.vars,
+      CF_AI_GATEWAY: process.env.CF_AI_GATEWAY,
+      CF_AI_GATEWAY_ACCOUNT_ID: process.env.CF_AI_GATEWAY_ACCOUNT_ID,
+      CF_AI_GATEWAY_API_TOKEN: process.env.CF_AI_GATEWAY_API_TOKEN,
+      CF_AI_GATEWAY_PROVIDERS: "cloudflare",
+    };
+  },
+});
+
+using session = await AgentSession.create(harness.url, {
+  modelId: "@cf/zai-org/glm-5.2",
+});
+const result = await session.run("Build a small status page.");
+```
+
+A caller can also add one directly configured user model through `userModel`. This lets local runs use
+existing `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` credentials without changing the backend.
+
+## Preview agent session
+
+A deployed preview uses Cloudflare Access instead of password signup. Supply an Access application
+JWT. The client sends it on the WebSocket handshake and then calls `authenticateFromCfAccess()`.
+
+```ts
+using session = await AgentSession.create(new URL(previewUrl), {
+  accessToken,
+  modelId: "@cf/zai-org/glm-5.2",
+});
+```
+
+The preview supplies its own model catalog and Workers AI binding. The same prompts and Gadget RPC
+verifiers can therefore run against local workerd or a deployed preview.
+
+## Test isolation
+
+The local harness keeps storage for its full lifetime. Tests must create fresh identities and unique
+resource URLs rather than assume a reset. Dispose every returned RPC stub and close the harness.
+
+[`docs/integration-testing.md`](../../docs/integration-testing.md) describes the in-process and
+out-of-process boundaries in more detail.

packages/integration-tests/__tests__/agent-session.test.ts

@@ -0,0 +1,131 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import type { AiChatAuthorInfo, AiChatHistoryPage, AiChatMessage, AiChatMetadata }
+  from "@gadgets/workshop-shared/api";
+import {
+  finalAssistantText,
+} from "../src/agent-session.js";
+import { AgentTurnCompletion, loadAllChatHistory } from "../src/agent-session-internals.js";
+
+function metadata(active: boolean): AiChatMetadata {
+  return {
+    id: 7,
+    title: "test",
+    started: new Date(0),
+    lastActive: new Date(0),
+    ...(active ? { activeAgent: { type: "agent", id: "model", name: "Model" } } : {}),
+  };
+}
+
+function message(sequence: number): AiChatMessage {
+  return {
+    chatId: 7,
+    sequence,
+    timestamp: new Date(sequence),
+    author: { type: "user", id: "user", name: "User" },
+    type: "message",
+    message: String(sequence),
+  };
+}
+
+afterEach(() => {
+  vi.useRealTimers();
+});
+
+describe("AgentTurnCompletion", () => {
+  it("settles after active becomes idle for the debounce period", async () => {
+    vi.useFakeTimers();
+    const completion = new AgentTurnCompletion(7, () => Promise.resolve(), 1_000, 25);
+    completion.metadata(metadata(true));
+    completion.metadata(metadata(false));
+
+    await vi.advanceTimersByTimeAsync(24);
+    expect(completion.settled).toBe(false);
+    await vi.advanceTimersByTimeAsync(1);
+    await expect(completion.promise).resolves.toBeUndefined();
+  });
+
+  it("retains the final chat usage metadata", async () => {
+    vi.useFakeTimers();
+    const completion = new AgentTurnCompletion(7, () => Promise.resolve(), 1_000, 25);
+    completion.metadata(metadata(true));
+    completion.metadata({ ...metadata(false), totalTokens: 123, totalCost: 0.45 });
+    await vi.advanceTimersByTimeAsync(25);
+    await completion.promise;
+
+    expect(completion.lastMetadata).toMatchObject({ totalTokens: 123, totalCost: 0.45 });
+  });
+
+  it("resets idle settlement when a callback restarts the agent", async () => {
+    vi.useFakeTimers();
+    const completion = new AgentTurnCompletion(7, () => Promise.resolve(), 1_000, 25);
+    completion.metadata(metadata(true));
+    completion.metadata(metadata(false));
+    await vi.advanceTimersByTimeAsync(20);
+    completion.metadata(metadata(true));
+    await vi.advanceTimersByTimeAsync(20);
+    expect(completion.settled).toBe(false);
+
+    completion.metadata(metadata(false));
+    await vi.advanceTimersByTimeAsync(25);
+    await expect(completion.promise).resolves.toBeUndefined();
+  });
+
+  it("stops the agent on timeout", async () => {
+    vi.useFakeTimers();
+    let stops = 0;
+    const completion = new AgentTurnCompletion(
+        7, () => { stops++; return Promise.resolve(); }, 100, 25);
+    await vi.advanceTimersByTimeAsync(100);
+
+    await expect(completion.promise).rejects.toThrow("Timed out after 100ms");
+    expect(stops).toBe(1);
+  });
+
+  it("stops the agent when cancelled", async () => {
+    let stops = 0;
+    const controller = new AbortController();
+    const completion = new AgentTurnCompletion(
+        7, () => { stops++; return Promise.resolve(); }, 1_000, 25, controller.signal);
+    controller.abort();
+
+    await expect(completion.promise).rejects.toThrow("Agent turn was cancelled");
+    expect(stops).toBe(1);
+  });
+
+  it("clears timers and abort listeners when disposed", () => {
+    vi.useFakeTimers();
+    const controller = new AbortController();
+    const completion = new AgentTurnCompletion(
+        7, () => Promise.resolve(), 1_000, 25, controller.signal);
+    completion.dispose();
+    controller.abort();
+    vi.runAllTimers();
+    expect(completion.settled).toBe(false);
+  });
+});
+
+it("loads history pages in authoritative ascending order", async () => {
+  const pages = new Map<number | undefined, AiChatHistoryPage>([
+    [undefined, { messages: [message(4), message(5)], compacted: { to: 4, summary: "tail" } }],
+    [4, { messages: [message(2), message(3)], compacted: { to: 2, summary: "middle" } }],
+    [2, { messages: [message(0), message(1)] }],
+  ]);
+  const history = await loadAllChatHistory(before => {
+    const page = pages.get(before);
+    if (!page) throw new Error(`No page for ${before}`);
+    return Promise.resolve(page);
+  });
+  expect(history.map(entry => entry.sequence)).toEqual([0, 1, 2, 3, 4, 5]);
+});
+
+it("returns the final non-empty assistant message from canonical history", () => {
+  const agent: AiChatAuthorInfo = { type: "agent", id: "model", name: "Model" };
+  const history = [
+    message(0),
+    { ...message(1), author: agent, message: "first" },
+    { ...message(2), author: agent, message: "" },
+  ];
+
+  expect(finalAssistantText(history)).toBe("first");
+});
+

packages/integration-tests/__tests__/rpc-client.test.ts

@@ -0,0 +1,42 @@
+import { createServer, type IncomingMessage, type Server } from "node:http";
+import { once } from "node:events";
+import { afterEach, expect, it } from "vitest";
+import { WebSocketServer } from "ws";
+import { connect } from "../src/rpc-client.js";
+
+const servers = new Set<{ http: Server; ws: WebSocketServer }>();
+
+afterEach(async () => {
+  await Promise.all([...servers].map(async ({ http, ws }) => {
+    ws.close();
+    http.close();
+    await Promise.allSettled([once(ws, "close"), once(http, "close")]);
+  }));
+  servers.clear();
+});
+
+it("sends the Access application token and same-origin header on preview WebSockets", async () => {
+  const http = createServer();
+  const ws = new WebSocketServer({ server: http });
+  servers.add({ http, ws });
+  http.listen(0, "127.0.0.1");
+  await once(http, "listening");
+
+  const address = http.address();
+  if (address === null || typeof address === "string") throw new Error("Test server has no TCP port");
+  const baseUrl = new URL(`http://127.0.0.1:${address.port}/preview`);
+  const requestPromise = new Promise<IncomingMessage>(resolve => {
+    ws.once("connection", (socket, request) => {
+      resolve(request);
+      socket.close();
+    });
+  });
+
+  {
+    using _api = connect(baseUrl, { accessToken: "access.jwt.value" });
+    const request = await requestPromise;
+    expect(request.url).toBe("/api");
+    expect(request.headers.origin).toBe(baseUrl.origin);
+    expect(request.headers.cookie).toBe("CF_Authorization=access.jwt.value");
+  }
+});

packages/integration-tests/package.json

@@ -4,6 +4,7 @@
   "private": true,
   "type": "module",
   "exports": {
+    "./agent-session": "./src/agent-session.ts",
     "./harness": "./src/harness.ts",
     "./network-interceptor": "./src/network-interceptor.ts",
     "./rpc-client": "./src/rpc-client.ts"
@@ -17,11 +18,13 @@
     "@gadgets/workshop-shared": "workspace:*",
     "capnweb": "catalog:",
     "jsonc-parser": "^3.3.1",
+    "ws": "^8.21.0",
     "zod": "^4.4.3"
   },
   "devDependencies": {
     "@cloudflare/workers-types": "^5.20260808.1",
     "@types/node": "^26.1.0",
+    "@types/ws": "^8.18.1",
     "typescript": "catalog:",
     "vitest": "catalog:",
     "wrangler": "catalog:"