[written by Claude (AI agent) working with Hesham Salman (@Iron-Ham) on Notion's internal trace analytics]
Problem
In a project that ingests coding-agent traces via the plugins, Braintrust's own automation (Topics classification, and the pattern-refresh/search_patterns runs) writes its execution traces back into the same project as ordinary root spans. Downstream analysis then treats this automation traffic as organic coding-agent behavior.
Measured in Notion's internal-coding-agent-traces project over the fixed 30-day window [2026-08-16, 2026-09-15):
- 29 pattern-refresh
search_patterns roots + 10 known probe roots = 39/751 distinct roots (5.2%) of the project's root stream are automation/probe traffic, not coding sessions.
- Topics attached facet spans to all of them and assigned a primary-failure-mode label to most (24/26 in an earlier window) — e.g. intentional probe runs labeled "stuck"/"incomplete", and refresh runs labeled with restatements of the very pattern drafts they were evaluating.
- The refresh roots alone carried 850k prompt / 116k completion tokens of descendant LLM telemetry in-window; facet/classification work on probe roots added 677k prompt / 40k completion tokens with cost telemetry absent.
These rows are created by the same service identity that runs Topics classification (distinct from the per-user plugin API keys), so they are identifiable in principle — but nothing at the row level marks them as automation traffic, and is_root queries, datasets, monitors, and Topics/Patterns sampling all count them as coding sessions.
Ask
- Tag automation-run roots at write time (e.g. a root-level
span_attributes.type = "automation" or a stable traffic_class metadata), so project analytics can exclude them with one predicate.
- Exclude automation roots from Topics/Patterns input sampling by default — automation runs classifying each other produces self-referential labels (a pattern-refresh run getting failure-labeled with the pattern it was refreshing).
- Optionally: route automation execution traces to a separate internal/audit project instead of the analyzed one.
Happy to share the exact BTQL predicates we use to separate this traffic if useful.
[written by Claude (AI agent) working with Hesham Salman (@Iron-Ham) on Notion's internal trace analytics]
Problem
In a project that ingests coding-agent traces via the plugins, Braintrust's own automation (Topics classification, and the pattern-refresh/
search_patternsruns) writes its execution traces back into the same project as ordinary root spans. Downstream analysis then treats this automation traffic as organic coding-agent behavior.Measured in Notion's
internal-coding-agent-tracesproject over the fixed 30-day window [2026-08-16, 2026-09-15):search_patternsroots + 10 known probe roots = 39/751 distinct roots (5.2%) of the project's root stream are automation/probe traffic, not coding sessions.These rows are created by the same service identity that runs Topics classification (distinct from the per-user plugin API keys), so they are identifiable in principle — but nothing at the row level marks them as automation traffic, and
is_rootqueries, datasets, monitors, and Topics/Patterns sampling all count them as coding sessions.Ask
span_attributes.type = "automation"or a stabletraffic_classmetadata), so project analytics can exclude them with one predicate.Happy to share the exact BTQL predicates we use to separate this traffic if useful.