You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3e187ed
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: .agents/skills/e2e-tests/SKILL.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,6 +7,8 @@ description: Write, run, and debug end-to-end tests for the Braintrust SDK. Use
7
7
8
8
E2E tests run SDK scenarios in subprocesses against a mock Braintrust server. Prefer extending the closest existing scenario over inventing a new pattern.
9
9
10
+
The goal for provider e2e tests is always to exercise actual models with real, valid input data, either live during recording or through cassettes populated with actual model responses. The mock Braintrust server captures telemetry; provider behavior should come from the real provider.
11
+
10
12
Read first:
11
13
12
14
-`e2e/README.md`
@@ -40,6 +42,10 @@ Try not to use specific test narrowing commands unless hunting down a very nasty
40
42
41
43
Cassettes mock provider HTTP responses (OpenAI, Anthropic, ...) so external-provider scenarios replay in CI without provider keys. The harness starts a local `@braintrust/seinfeld` cassette server and points provider SDK base URL env vars at that server, so subprocesses and SDK-launched binaries are covered too.
42
44
45
+
- Record cassettes by calling actual provider models through the real SDK. Do not hand-author model responses, embedding vectors, or usage data, or replace the provider with a synthetic HTTP server or fetch stub to make an e2e scenario pass. Synthetic responses belong in focused unit tests and do not satisfy provider e2e coverage.
46
+
- Use real, valid inputs that the model can process. Multimodal scenarios need decodable images, audio, video, and documents; arbitrary bytes labeled with a media MIME type do not qualify.
47
+
- If recording is blocked by missing credentials, model access, or provider availability, report the blocker and the remaining recording work. Do not fabricate cassette data or silently substitute synthetic provider responses.
48
+
- Default to cheap provider models for real API calls and cassette recordings, unless the user explicitly requests otherwise. Choose a model that supports the behavior under test, and keep prompts, output limits, and media duration small while preserving meaningful coverage.
43
49
- External-provider tests should thread `runContext: { variantKey: "...", originalScenarioDir }` into the scenario runner. In normal replay mode, missing cassette entries fail loudly instead of skipping or falling back to live providers. In `record` / `record-missing` mode, they run so new cassettes can be authored.
44
50
- Thread `runContext: { variantKey: "...", originalScenarioDir }` into `runScenarioDir`/`runNodeScenarioDir`. Cassettes live at `e2e/scenarios/<name>/__cassettes__/<variantKey>.cassette.json` (parallel to `__snapshots__/`). Only set `runContext.cassette` explicitly for unusual cases, such as `cassette: false` on a non-provider mode inside an otherwise provider-backed scenario.
Copy file name to clipboardExpand all lines: .agents/skills/instrumentation/SKILL.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -44,3 +44,5 @@ Map the change before editing:
44
44
## Process
45
45
46
46
Before implementing or changing instrumentation it is advisable to add or adjust the e2e tests for the desired change, make it fail, then implement the new instrumentation until the test passes.
47
+
48
+
When validating instrumentation with real provider APIs or recording e2e cassettes, default to cheap models that support the behavior under test, unless the user explicitly requests otherwise. Keep prompts, output limits, and media duration small while preserving meaningful coverage.
0 commit comments