Skip to content

Commit 3e187ed

Browse files
committed
Merge remote-tracking branch 'origin/main' into lforst/dum-e/london-1aa915af4f
# Conflicts: # e2e/scenarios/mistral-instrumentation/__snapshots__/mistral-v1.span-tree.json # e2e/scenarios/mistral-instrumentation/__snapshots__/mistral-v1.span-tree.txt
2 parents cfaaecd + 7c4a372 commit 3e187ed

307 files changed

Lines changed: 58508 additions & 7682 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/e2e-tests/SKILL.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ description: Write, run, and debug end-to-end tests for the Braintrust SDK. Use
77

88
E2E tests run SDK scenarios in subprocesses against a mock Braintrust server. Prefer extending the closest existing scenario over inventing a new pattern.
99

10+
The goal for provider e2e tests is always to exercise actual models with real, valid input data, either live during recording or through cassettes populated with actual model responses. The mock Braintrust server captures telemetry; provider behavior should come from the real provider.
11+
1012
Read first:
1113

1214
- `e2e/README.md`
@@ -40,6 +42,10 @@ Try not to use specific test narrowing commands unless hunting down a very nasty
4042

4143
Cassettes mock provider HTTP responses (OpenAI, Anthropic, ...) so external-provider scenarios replay in CI without provider keys. The harness starts a local `@braintrust/seinfeld` cassette server and points provider SDK base URL env vars at that server, so subprocesses and SDK-launched binaries are covered too.
4244

45+
- Record cassettes by calling actual provider models through the real SDK. Do not hand-author model responses, embedding vectors, or usage data, or replace the provider with a synthetic HTTP server or fetch stub to make an e2e scenario pass. Synthetic responses belong in focused unit tests and do not satisfy provider e2e coverage.
46+
- Use real, valid inputs that the model can process. Multimodal scenarios need decodable images, audio, video, and documents; arbitrary bytes labeled with a media MIME type do not qualify.
47+
- If recording is blocked by missing credentials, model access, or provider availability, report the blocker and the remaining recording work. Do not fabricate cassette data or silently substitute synthetic provider responses.
48+
- Default to cheap provider models for real API calls and cassette recordings, unless the user explicitly requests otherwise. Choose a model that supports the behavior under test, and keep prompts, output limits, and media duration small while preserving meaningful coverage.
4349
- External-provider tests should thread `runContext: { variantKey: "...", originalScenarioDir }` into the scenario runner. In normal replay mode, missing cassette entries fail loudly instead of skipping or falling back to live providers. In `record` / `record-missing` mode, they run so new cassettes can be authored.
4450
- Thread `runContext: { variantKey: "...", originalScenarioDir }` into `runScenarioDir`/`runNodeScenarioDir`. Cassettes live at `e2e/scenarios/<name>/__cassettes__/<variantKey>.cassette.json` (parallel to `__snapshots__/`). Only set `runContext.cassette` explicitly for unusual cases, such as `cassette: false` on a non-provider mode inside an otherwise provider-backed scenario.
4551
- To re-record after changing a scenario:

‎.agents/skills/instrumentation/SKILL.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,5 @@ Map the change before editing:
4444
## Process
4545

4646
Before implementing or changing instrumentation it is advisable to add or adjust the e2e tests for the desired change, make it fail, then implement the new instrumentation until the test passes.
47+
48+
When validating instrumentation with real provider APIs or recording e2e cassettes, default to cheap models that support the behavior under test, unless the user explicitly requests otherwise. Keep prompts, output limits, and media duration small while preserving meaningful coverage.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": minor
3+
---
4+
5+
feat: Add advanced filters to `Trace.getSpans()`

‎.changeset/custom-view-helpers.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": minor
3+
---
4+
5+
feat: Add utilities for custom views (push + preview)

‎.changeset/deps-bump-deps-and-fix-old-otel-baggage-header-parsing-vulnerability.md‎

Lines changed: 0 additions & 5 deletions
This file was deleted.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": patch
3+
---
4+
5+
fix(eve): Use `any` as return type to ensure eve version compat
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": patch
3+
---
4+
5+
fix: Bump e2e tested versions for HarnessAgent, bedrock and google adk and fix orchestrion query for harness agent
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": patch
3+
---
4+
5+
fix(cli): Don't externalize braintrust package on push

‎.changeset/fix-pi-coding-agent-stream-function.md‎

Lines changed: 0 additions & 5 deletions
This file was deleted.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"braintrust": patch
3+
---
4+
5+
feat: Add instrumentation for google genai multimodal embeddings

0 commit comments

Comments
 (0)