feat(#6458): export eval measurement scores via OTLP

[ascerra]

Summary

• Implements ADR 0087 portable remote score export: after writing eval-measurements.jsonl, newly scored rows also emit gen_ai.evaluation.result span events on the same TraceID when OTEL_EXPORTER_OTLP_* is set (same path as ADR 0050 agent traces).
• Fail-open: OTLP export errors warn via CLI; local JSONL/ledger always win. Does not rewrite run-telemetry.jsonl.
• No vendor score adapters (MLFLOW_* / Assessments) in core — MLflow Assessments UI can be a separate consumer of the OTLP event.

Closes #6458

Test plan

• Unit tests: go test ./internal/evalmeasure/ ./internal/telemetry/
• Local httptest OTLP sink proof against Review artifact (run 32482721216 / trace 84d470ba…)
• Live dogfood MLflow OTLP: score span + event attached to tr-84d470ba2451ffeccfe09022d9b2aebd
• CI green on this PR
• Optional: after merge, confirm dogfood eval-measure posts scores when OTEL is set

Made with Cursor

[qodo-code-review]

PR Summary by Qodo

Export eval measurement scores as OTLP GenAI evaluation span events

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Export newly written eval measurement rows as OTLP GenAI evaluation span events when OTEL is
 configured.
• Keep local eval-measurements.jsonl/ledger as source of truth; remote export is fail-open.
• Add unit tests and update ADRs/docs to reflect implemented remote score export.

Diagram

graph TD
  A["run-telemetry.jsonl"] --> B["MeasureAndExport"] --> C["eval-measurements.jsonl + ledger"]
  B --> D["ExportOTLPScores"] --> E["OTLP HTTP exporter"] --> F{{"OTLP backend"}}
  subgraph Legend
    direction LR
    _file["File"] ~~~ _mod["Module"] ~~~ _ext{{"External"}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Attach events directly to existing spans (no child span)

• ➕ Avoids creating an extra span per score
• ➕ Keeps score data physically on the scored span
• ➖ Not feasible when scoring offline from JSONL (no live span handle)
• ➖ Would require rewriting run-telemetry.jsonl or reconstructing full span payloads, violating 'primary facts' immutability

2. Export scores as OTEL Logs or Metrics signals

• ➕ Potentially lower overhead than spans for high-volume scoring
• ➕ Some backends may have better UX for log/metric-based scoring
• ➖ GenAI semantic conventions and common vendor UIs often expect evaluation results on trace spans/events
• ➖ Harder to guarantee correlation to the exact parent span across backends compared to trace events

Recommendation: The chosen approach (child span remote-parented to the scored span + gen_ai.evaluation.result event) is the best fit for offline scoring: it preserves strict local-source-of-truth semantics (no mutation of run-telemetry.jsonl), keeps vendor neutrality, and guarantees cross-backend correlation via shared TraceID/parent SpanID. Alternatives either require mutating primary telemetry artifacts or weaken correlation/compatibility.

Files changed (11) +662 / -18

Enhancement (5) +241 / -4

evalmeasure.goSurface OTLP score export behavior and warnings in CLI help/output +10/-2

Surface OTLP score export behavior and warnings in CLI help/output

• Extends the eval-measure command help text to describe optional OTLP score export and fail-open semantics. Adds CLI warnings when remote export fails while keeping local JSONL persistence.

internal/cli/evalmeasure.go

export_otlp.goImplement OTLP score export as GenAI evaluation span events +196/-0

Implement OTLP score export as GenAI evaluation span events

• Introduces ExportOTLPScores to emit a short child span (fullsend.eval_measure) remote-parented to the scored span and add gen_ai.evaluation.result events with semconv attributes. Validates OTLP endpoint configuration, respects OTEL_SDK_DISABLED, and captures exporter errors for fail-open warnings.

internal/evalmeasure/export_otlp.go

parse.goTrack remote export warning in ParseStats +3/-0

Track remote export warning in ParseStats

• Extends ParseStats with RemoteExportWarning for propagating OTLP export failures to the CLI without failing scoring. Keeps the existing model where scoring remains successful even with partial parse issues.

internal/evalmeasure/parse.go

run.goWire MeasureAndExport to perform optional OTLP score export +9/-2

Wire MeasureAndExport to perform optional OTLP score export

• Updates MeasureAndExport documentation to reflect implemented OTLP exporting. Calls ExportOTLPScores after local persistence and records any failure as ParseStats.RemoteExportWarning to preserve fail-open semantics.

internal/evalmeasure/run.go

telemetry.goExpose OTLPEnabled and endpoint validation/exporter construction helpers +23/-0

Expose OTLPEnabled and endpoint validation/exporter construction helpers

• Adds OTLPEnabled and ValidateOTLPEndpoints helpers and a NewOTLPExporter constructor so other packages can reuse the same OTLP export path as agent traces. Keeps endpoint validation behavior explicit for callers.

internal/telemetry/telemetry.go

Tests (2) +241 / -0

export_otlp_test.goAdd unit tests for OTLP score event emission and fail-open behavior +227/-0

Add unit tests for OTLP score event emission and fail-open behavior

• Adds a local httptest OTLP sink to assert emitted spans/events and verify correct trace/parent correlation and event attributes. Covers noop behavior when OTLP is unset/disabled, invalid IDs, and MeasureAndExport fail-open warning propagation.

internal/evalmeasure/export_otlp_test.go

telemetry_test.goTest OTLPEnabled and endpoint validation helper behavior +14/-0

Test OTLPEnabled and endpoint validation helper behavior

• Adds a focused unit test for OTLPEnabled and ValidateOTLPEndpoints covering unset, valid URL, and invalid URL cases.

internal/telemetry/telemetry_test.go

Documentation (3) +14 / -14

0087-eval-measurements-online-trace-scoring.mdClarify OTLP score export semantics and event format +5/-4

Clarify OTLP score export semantics and event format

• Updates ADR 0087 to specify that remote score export is implemented via gen_ai.evaluation.result span events using the same OTEL_EXPORTER_OTLP_* configuration as agent traces. Reinforces fail-open behavior and lack of vendor-specific adapters in core.

docs/ADRs/0087-eval-measurements-online-trace-scoring.md

architecture.mdDocument implemented remote score export on shared OTLP path +2/-4

Document implemented remote score export on shared OTLP path

• Removes the 'planned' note and documents that scores can now export over OTLP when configured. Clarifies that eval scores are derived products stored in eval-measurements.jsonl and optionally exported as correlated span events.

docs/architecture.md

eval-measurements.mdUpdate eval measurements guide with OTLP score export details +7/-6

Update eval measurements guide with OTLP score export details

• Updates the guide to reflect that remote score export is now available and describes the correlation model (child span + gen_ai.evaluation.result event). Reiterates that run-telemetry.jsonl is not rewritten and local JSONL remains authoritative.

docs/guides/infrastructure/eval-measurements.md

Other (1) +166 / -0

main.goAdd local OTLP sink proof tool for score export +166/-0

Add local OTLP sink proof tool for score export

• Adds a standalone program that scores a real telemetry file and asserts OTLP requests contain gen_ai.evaluation.result events. Uses an httptest server as an OTLP endpoint and outputs a structured JSON report for inspection.

hack/prove-otlp-scores/main.go

[github-actions]

Site preview

Preview: https://51652581-site.fullsend-ai.workers.dev

Commit: c64055d479640cd7631226c1cc047c29f7562716

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:39 PM UTC · Ended 7:50 PM UTC

Commit: 7260ca8 · View workflow run →

[codecov]

Codecov Report

❌ Patch coverage is 86.17512% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/evalmeasure/export_otlp.go	88.16%	10 Missing and 10 partials ⚠️
internal/telemetry/telemetry.go	71.42%	8 Missing ⚠️
internal/evalmeasure/run.go	81.81%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Unbounded OTLP flush ✓ Resolved 🐞 Bug ☼ Reliability

Description

ExportOTLPScores calls tp.ForceFlush(ctx) using a context that commonly has no deadline (CLI
passes cmd.Context() and MeasureFile uses context.Background()), so eval-measure can hang
indefinitely on OTLP network/retry issues. This contradicts the stated fail-open behavior by
potentially stalling the run even though local JSONL was already persisted.

Code

internal/evalmeasure/export_otlp.go[R85-87]

+	if err := tp.ForceFlush(ctx); err != nil && firstErr == nil {
+		firstErr = err
+	}

Relevance

●●● Strong

Recent reliability precedent accepted making blocking operations respect cancellation or bounded
time.

PR-#6437

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new code flushes using the caller context, and upstream callers provide contexts without
deadlines (background contexts). That combination means flush duration is unbounded from this code’s
perspective, so OTLP export can stall the CLI/run even though export is supposed to be best-effort.

internal/evalmeasure/export_otlp.go[85-87]
internal/evalmeasure/run.go[20-24]
internal/cli/evalmeasure.go[118-123]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ExportOTLPScores` performs `tp.ForceFlush(ctx)` with the caller-provided context. In typical usage that context has no deadline (`cmd.Context()` from cobra, or `context.Background()` via `MeasureFile`). If the OTLP exporter blocks due to network issues/retries, `ForceFlush` can block indefinitely, stalling `eval-measure` despite the feature being documented as fail-open.

### Issue Context
- `MeasureFile` calls `MeasureAndExport(context.Background(), ...)`.
- The CLI passes `cmd.Context()` to `MeasureAndExport` without adding a timeout.
- `ExportOTLPScores` uses that same context for `ForceFlush`.

### Fix Focus Areas
- internal/evalmeasure/export_otlp.go[68-95]
- internal/evalmeasure/run.go[20-24]
- internal/cli/evalmeasure.go[118-123]

### Suggested approach
- Wrap `ForceFlush` in a bounded context, e.g.:
 - `flushCtx, cancel := context.WithTimeout(ctx, otlpFlushTimeout)` (this will respect earlier deadlines if present)
 - `defer cancel()`
 - `tp.ForceFlush(flushCtx)`
- Optionally use the same bounded context when constructing the exporter if exporter creation can block.
- Add a regression test that uses a context with a very short timeout and a non-routable/blocked endpoint, and assert `ExportOTLPScores` returns within the timeout and surfaces a warning (not a hang).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. ADR 0087 decision rewritten ✓ Resolved 📜 Skill insight ≡ Correctness

Description

The PR edits an already Accepted ADR’s Decision content to incorporate new implementation details,
which violates the rule against substantively rewriting accepted ADRs. This risks changing
historical decision records instead of adding an additive note or superseding ADR.

Code

docs/ADRs/0087-eval-measurements-online-trace-scoring.md[R88-91]

+one new measurement row is produced (including `label: skip`). Remote score
+export uses the same `OTEL_EXPORTER_OTLP_*` configuration as ADR 0050
+(`gen_ai.evaluation.result` span events; fail-open) — no vendor-specific
+score adapters in core. `fullsend` owns the parser, scorers,

Relevance

●●● Strong

Recent ADR precedent explicitly accepted reverting substantive edits to accepted ADR content.

PR-#5244

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The compliance rule prohibits substantive rewrites to accepted ADR content. The ADR is `status:
Accepted`, yet the Decision section text is altered to incorporate new implementation-specific
details about OTLP score export events.

docs/ADRs/0087-eval-measurements-online-trace-scoring.md[1-20]
docs/ADRs/0087-eval-measurements-online-trace-scoring.md[71-92]
Skill: writing-adrs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/ADRs/0087-eval-measurements-online-trace-scoring.md` is an `Accepted` ADR, but this PR modifies the Decision text to add new implementation specifics (e.g., `gen_ai.evaluation.result` span events). Accepted ADRs must not have substantive content rewritten.

## Issue Context
Preserve the original accepted decision text and add new implementation details as an explicitly labeled note/cross-reference (or add a superseding ADR if the decision itself changed).

## Fix Focus Areas
- docs/ADRs/0087-eval-measurements-online-trace-scoring.md[86-93]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Skip scores break export ✓ Resolved 🐞 Bug ≡ Correctness

Description

ExportOTLPScores fails the whole export on any result with an empty/invalid SpanID, which is a
normal outcome for some label: skip measurement rows. This will cause persistent
RemoteExportWarning noise and misleading “OTLP score export failed” warnings even when other
scores could export successfully.

Code

internal/evalmeasure/export_otlp.go[R121-124]

+	sid, err := parseSpanID(r.SpanID)
+	if err != nil {
+		return fmt.Errorf("span_id %q: %w", r.SpanID, err)
+	}

Relevance

●●● Strong

Recent eval-measurement precedents accepted handling skipped runs and edge-case result semantics.

PR-#6036

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new OTLP export path hard-requires a valid SpanID and returns an error otherwise; however the
existing fitness scorer can legitimately produce label: skip rows with an empty SpanID (e.g.,
missing root run span). Since MeasureAndExport exports all newly written rows (including skips),
this mismatch will commonly trigger export warnings and mark the export as failed even when it
should just skip those rows.

internal/evalmeasure/export_otlp.go[116-124]
internal/evalmeasure/fitness.go[53-88]
internal/evalmeasure/run.go[82-86]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ExportOTLPScores` currently returns an error when `EvaluationResult.SpanID` is empty or malformed, but some scorers legitimately emit `label: skip` rows with `SpanID == ""` (e.g., when the root run span is missing). Because the OTLP export is meant to be fail-open, these rows should be skipped (or handled specially) without failing the entire export and triggering `RemoteExportWarning`.

### Issue Context
- `exportOneScore` hard-fails on `parseSpanID(r.SpanID)`.
- `ScoreFitnessNamed` returns skip results with an empty `SpanID` when the run span is missing.
- `MeasureAndExport` exports all newly written rows, including skips.

### Fix Focus Areas
- internal/evalmeasure/export_otlp.go[116-161]
- internal/evalmeasure/run.go[82-86]
- internal/evalmeasure/fitness.go[53-88]

### Suggested approach
- In `ExportOTLPScores` (or `exportOneScore`), treat empty `SpanID` (and potentially invalid IDs) as a **soft skip**: do not return an error; just continue.
- Only return an error for exporter/flush failures (network, protocol, etc.).
- Add a unit test where `results` includes:
 - one valid score with TraceID+SpanID
 - one skip score with the same TraceID but `SpanID == ""`
 and assert export succeeds and still emits the event for the valid score, with no `RemoteExportWarning`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View high (1)

4. Guide not in admin/user ✗ Dismissed 📜 Skill insight ⌂ Architecture

Description

A modified guide exists under docs/guides/infrastructure/, but guides are required to live under
either docs/guides/admin/ or docs/guides/user/. This breaks the required documentation directory
structure.

Code

docs/guides/infrastructure/eval-measurements.md[R28-31]

Fullsend does not pick an observability product for scores. The portable
contract is a local JSONL artifact next to telemetry; remote export reuses
the same OpenTelemetry (`OTEL_EXPORTER_OTLP_*`) configuration as agent
-traces when implemented.
+traces.

Evidence

The rule requires every guide under docs/guides/ to be located in either the admin/ or user/
subdirectory. This PR modifies a guide located at docs/guides/infrastructure/eval-measurements.md,
which is outside the allowed directories.

docs/guides/infrastructure/eval-measurements.md[1-20]
Skill: writing-user-docs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The file `docs/guides/infrastructure/eval-measurements.md` is a documentation guide, but it is not placed under `docs/guides/admin/` or `docs/guides/user/` as required.

## Issue Context
Choose the correct audience (likely `admin/` for infrastructure/ops content) and move/rename the file accordingly, then update inbound links (e.g., from `docs/architecture.md` and glossary entries) to the new location.

## Fix Focus Areas
- docs/guides/infrastructure/eval-measurements.md[1-60]
- docs/architecture.md[321-325]
- docs/glossary.md[89-93]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. TraceID undefined in guide ✓ Resolved 📜 Skill insight ✧ Quality

Description

The guide introduces jargon (TraceID, GenAI semconv) without an inline definition or a glossary
link on first use. This reduces clarity for readers unfamiliar with OpenTelemetry terminology.

Code

docs/guides/infrastructure/eval-measurements.md[R45-47]

+  └─ if OTEL_EXPORTER_OTLP_* set → OTLP export of scores as
+       gen_ai.evaluation.result span events on the same TraceID
+       (fail-open; local JSONL always wins)

Relevance

●●● Strong

Team accepted defining domain jargon on first use in guides.

PR-#5778

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The compliance rule requires domain-specific jargon in guides to be defined on first use via a
glossary link or inline definition. The modified guide text uses TraceID and references `GenAI
semconv` without a definition or glossary link at the point of introduction.

docs/guides/infrastructure/eval-measurements.md[36-56]
Skill: writing-user-docs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/guides/infrastructure/eval-measurements.md` uses terms like `TraceID` and `GenAI semconv` without defining them inline or linking to `docs/glossary.md` on first use.

## Issue Context
Compliance requires jargon to be defined on first use in documentation guides. This can be satisfied by adding a short parenthetical definition (e.g., what a TraceID is) and/or linking to an existing glossary entry or authoritative reference.

## Fix Focus Areas
- docs/guides/infrastructure/eval-measurements.md[42-56]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Per-score synchronous export ✓ Resolved 🐞 Bug ➹ Performance

Description

ExportOTLPScores uses NewSimpleSpanProcessor and creates/ends one span per score, which drives
synchronous exporting on each score and can significantly increase runtime for many measurements.
This can make eval-measure unexpectedly slow when OTLP is enabled, even though the local JSONL
work is already done.

Code

internal/evalmeasure/export_otlp.go[R68-71]

+	tp := sdktrace.NewTracerProvider(
+		sdktrace.WithSampler(sdktrace.AlwaysSample()),
+		sdktrace.WithSpanProcessor(sdktrace.NewSimpleSpanProcessor(capExp)),
+	)

Relevance

●● Moderate

Performance concern is plausible, but history lacks a close precedent for rejecting this OTLP
batching design.

PR-#6036

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new exporter is wired with NewSimpleSpanProcessor and a per-result emission loop, which is the
code-level pattern that causes synchronous per-score export behavior rather than batching.

internal/evalmeasure/export_otlp.go[68-84]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The score export path uses `sdktrace.NewSimpleSpanProcessor`, and emits one span per score in a loop. This makes exports effectively synchronous per score and can lead to high latency when many measurement rows are written.

### Issue Context
- `ExportOTLPScores` builds a new `TracerProvider` with a `SimpleSpanProcessor`.
- It iterates through `results` and emits one child span + event per result.

### Fix Focus Areas
- internal/evalmeasure/export_otlp.go[68-87]

### Suggested approach
- Replace `NewSimpleSpanProcessor` with a `NewBatchSpanProcessor` configured for short-lived CLI usage (small batch timeout + `ForceFlush`/`Shutdown` with bounded context).
- If you move to batch processing, make `capturingExporter.err` concurrency-safe (mutex/atomic) because export can happen from a worker goroutine.
- Keep the existing fail-open behavior: exporter errors should surface only as warnings, never as a hard failure of measurement persistence.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 58 rules

✅ Skills: writing-user-docs, writing-adrs

Tip of the day

💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[ascerra]

Review squad follow-up

Addressed verified findings from the review pass:

• Bounded retries / wall budget for post-hoc score export (NewOTLPExporterBounded + 15s export timeout + batch processor) so a flaky collector cannot hang the job.
• Shared BuildResource so score spans keep service.name=fullsend (+ OTEL_RESOURCE_ATTRIBUTES).
• Empty span_id skip rows no longer warn as OTLP failures.
• No Error span status on measurement fail (label lives on the GenAI event only).
• Omit score.value on skip.
• Docs: removed leftover “not wired yet” callouts; ADR 0050 annotation marked done; ledger best-effort remote noted.

Deferred (documented / intentional): separate remote-export ledger for OTLP retry after local success — remote remains best-effort once; Assessments stay a MLflow-side consumer, not core.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:52 PM UTC · Completed 8:07 PM UTC

Commit: 17e3154 · View workflow run →

[fullsend-ai-review]

Review

Findings

Low

• [edge-case] internal/evalmeasure/export_otlp.go:145 — The BatchSpanProcessor queue size is set to exactly len(exportable), which scales with input size and leaves no margin for SDK internal bookkeeping. Mitigated by the 15-second otlpExportBudget timeout, bounded score count, and a dedicated test validating 2049 spans. Minimal practical risk.

• [error-handling] internal/evalmeasure/export_otlp.go:192 — When the BatchSpanProcessor exports in multiple batches and an early batch succeeds but a later batch fails, the error message says "otlp export failed for all N scores." This is conservative by design. The capturingExporter tracks failedSpans count and accumulates errors without clearing on later success. In practice the conservative path requires >512 scores with intermittent transport failure.

• [naming-convention] internal/evalmeasure/export_otlp.go:215 — Function name inboundUnsampledTRACEPARENT uses all-caps for TRACEPARENT, which is a W3C header name (lowercase in the spec), not a Go initialism. Additionally, the local variable tp at line 216 holds the raw TRACEPARENT env-var string, while elsewhere in the file tp is a *sdktrace.TracerProvider.

• [naming-convention] internal/evalmeasure/export_otlp_test.go:78 — clearOTLPEnv is defined identically in both export_otlp_test.go and cli/evalmeasure_test.go, clearing the same ten env vars. Different packages prevent direct reuse without a testutil package.

• [struct-placement] internal/evalmeasure/parse.go:55 — RemoteExportWarning is a field of ParseStats (whose other fields describe parse outcomes), but it is populated exclusively by the OTLP export path in run.go. ParseStats doubles as the MeasureAndExport return carrier; the placement is pragmatic but breaks the struct's parse-scoped semantic.

• [naming-convention] hack/prove-otlp-scores/main.go:175 — eventView.ParentID maps to JSON key parent_span_id, but the Go field name omits the "span" qualifier. In a struct containing TraceID, ParentID is ambiguous.

• [authorization] docs/ADRs/0050-distributed-tracing-instrumentation.md:11 — docs/contributing/adrs.md requires that edits to accepted ADRs be called out in the PR description. The PR body does not explicitly state that ADR 0050 and ADR 0087 are being modified with implementation annotations.

• [incomplete-doc] docs/cli/README.md:30 — The fullsend eval-measure row in the CLI reference table describes the command as only writing eval-measurements.jsonl. The PR ships optional OTLP score export, but the table row doesn't mention it. The linked eval-measurements guide does cover it.

• [missing-doc] docs/guides/dev/tracing.md — The PR adds hack/prove-otlp-scores as a developer verification tool but there is no documentation for it. The guide already documents the analogous hack/upload-traces.sh.

Previous run

Review

Findings

Low

• [edge-case / denial-of-service] internal/evalmeasure/export_otlp.go:141 — The BatchSpanProcessor queue size is set to exactly len(exportable), which scales with input size and leaves no margin for SDK internal bookkeeping. Mitigated by the 15-second otlpExportBudget timeout, bounded score count, and a dedicated test validating 2049 spans. Minimal practical risk.

• [error-handling] internal/evalmeasure/export_otlp.go:175 — When the BatchSpanProcessor exports in multiple batches and an early batch succeeds but a later batch fails, the error message says "otlp export failed for all N scores." This is conservative by design. The capturingExporter clears its error on later success, so a failed early batch followed by a successful later batch returns nil. Both directions stem from storing only the latest ExportSpans error. In practice requires >512 scores with intermittent transport failure.

• [naming-convention] internal/evalmeasure/export_otlp.go:188 — Function name inboundUnsampledTRACEPARENT uses all-caps for TRACEPARENT, which is a W3C header name (lowercase in the spec), not a Go initialism. The existing codebase uses parentSampledProcessor for similar concepts.
  Remediation: Rename to inboundUnsampledTraceparent and update the call site at line 104.

• [naming-convention] internal/evalmeasure/export_otlp.go:189 — tp is used as a local variable holding the raw TRACEPARENT env-var string, but elsewhere in the same file tp is a *sdktrace.TracerProvider. Different scopes but tp is the canonical Go/OTel abbreviation for TracerProvider.
  Remediation: Rename to traceparent or raw.

• [naming-convention] internal/evalmeasure/export_otlp_test.go:78 — clearOTLPEnv is defined identically in both export_otlp_test.go and cli/evalmeasure_test.go, clearing the same ten env vars.

• [struct-placement] internal/evalmeasure/parse.go:53 — RemoteExportWarning is added to ParseStats, but is set exclusively by the OTLP export path in run.go. The struct doubles as the MeasureAndExport return carrier; the field's scope mismatch is an established pattern.

• [api-shape-consistency] internal/evalmeasure/run.go:28 — MeasureFile passes empty serviceVersion to MeasureAndExport, causing BuildResource to record service.version=unknown when OTLP is configured. The GoDoc now documents this explicitly and recommends MeasureAndExport instead.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (2)

Review

Findings

Medium

• [missing-doc] docs/guides/user/how-to-emit-traces.md — This guide covers configuring OTEL_EXPORTER_OTLP_* for agent trace export but does not mention that eval-measurement scores now also export as gen_ai.evaluation.result span events on the same OTLP path. Users following this guide to set up a remote backend should know that scores will also flow to that endpoint when fullsend eval-measure runs.
  Remediation: Add a brief note that when OTEL export is configured, eval-measurement scores also export as span events on the same endpoint. Link to the eval-measurements guide for details.

Low

• [api-shape-consistency] internal/evalmeasure/run.go:23 — MeasureFile passes empty serviceVersion to MeasureAndExport. Inside BuildResource, empty string defaults to "unknown". Callers using MeasureFile will emit OTLP score spans with service.version=unknown if OTLP is configured, silently fragmenting the resource identity from agent-trace spans that carry the real CLI version.
  Remediation: Add a serviceVersion parameter to MeasureFile or document the unknown default prominently in the godoc.

• [stale-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says derived products "sit beside telemetry as sibling files." After this PR, scores also export as OTLP span events when OTEL_EXPORTER_OTLP_* is set. The glossary is the project's canonical term index and this entry now under-describes the concept.
  Remediation: Amend to note that derived products also export remotely when OTLP is configured, e.g., "Derived products sit beside telemetry as sibling files and, when OTLP is configured, also export as span events."

• [edge-case] internal/evalmeasure/export_otlp.go:141 — The BatchSpanProcessor queue size is set to exactly len(exportable). While a dedicated test validates that 2049 spans (exceeding the SDK default of 2048) all arrive, the exact-size match leaves no margin for any future SDK internal bookkeeping. The code comment documents the rationale; this is an observation, not a blocking concern.

• [error-handling] internal/evalmeasure/export_otlp.go:175 — When the BatchSpanProcessor exports in multiple batches and an early batch succeeds but a later batch fails at the transport level, the error message says "otlp export failed for all N scores." This is conservative by design (the comment notes "nothing is known to have landed"), but could mislead debugging in the partial-success case.

• [naming-convention] internal/evalmeasure/export_otlp_test.go:78 — clearOTLPEnv is duplicated identically in export_otlp_test.go and cli/evalmeasure_test.go, both clearing the same 10 env vars. The CLI copy cross-references the pattern. Since these are _test.go files in different packages, sharing requires a new testutil package; the duplication is a reasonable Go testing pattern.

• [struct-placement] internal/evalmeasure/parse.go:53 — RemoteExportWarning is added to ParseStats, but this field is set exclusively by the OTLP export path in run.go, not by parsing. The struct already serves as the MeasureAndExport return carrier (its Incomplete field is also set there), so the pattern is established.

• [scope-boundary] internal/telemetry/telemetry.go — Seven new public symbols exported from internal/telemetry (OTLPEnabled, NewOTLPExporter, NewOTLPExporterBounded, BuildResource, ValidateOTLPEndpoints, SpanLimits, FreeTextAttrValueLenLimit). These refactor agent-trace internals into a shared API for score export. The extraction is well-motivated by ADR 0050/0087 and is internal-only.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (3)

Review

Findings

Medium

• [missing-doc] docs/guides/user/how-to-emit-traces.md — This guide covers configuring OTEL_EXPORTER_OTLP_* for trace export but does not mention that the same env vars now also control eval-measurement score export (gen_ai.evaluation.result span events). Users following this guide to set up OTLP export will unknowingly also enable score export.
  Remediation: Add a brief note that when OTEL export is configured, eval-measurement scores also export as span events on the same OTLP path. Link to the eval-measurements guide for details.

Low

• [edge-case] internal/evalmeasure/export_otlp.go:119 — When every result in the batch is filtered out (all TraceIDs match an unsampled TRACEPARENT, or all are skip-without-IDs), ExportOTLPScores still creates a TracerProvider, BatchSpanProcessor, and HTTP OTLP exporter connection before reaching ForceFlush/Shutdown — all for zero spans. If the collector is unreachable, the error reads "otlp export failed for all 0 scores" and adds unnecessary latency against the 15s budget.
  Remediation: Add an early return after the filter loop when attempted == 0.

• [naming-convention] internal/evalmeasure/export_otlp_test.go:78 — clearOTLPEnv is duplicated identically in export_otlp_test.go and cli/evalmeasure_test.go, both clearing the same 10 env vars. The CLI copy's comment cross-references the pattern, but both copies must be kept in sync manually.

• [api-shape-consistency] internal/evalmeasure/run.go:23 — MeasureFile passes empty serviceVersion to MeasureAndExport. Inside BuildResource, empty string defaults to "unknown". Callers using MeasureFile will emit OTLP score spans with service.version=unknown if OTLP is configured.

• [stale-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says derived products "sit beside telemetry as sibling files." With this PR, scores also export as OTLP span events when OTEL_EXPORTER_OTLP_* is configured. The entry is incomplete but not incorrect — users can follow the linked ADR 0087 and eval-measurements guide (both updated in this PR).
  Remediation: Append a clause noting that when OTEL_EXPORTER_OTLP_* is set, scores also export as span events on the same TraceID.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (4)

Review

Findings

Low

• [naming-convention] internal/evalmeasure/export_otlp_test.go:71 — clearOTLPEnv in export_otlp_test.go clears 9 env vars (includes TRACEPARENT but not TRACESTATE), while the identically-named function in internal/cli/evalmeasure_test.go clears 10 (includes both TRACEPARENT and TRACESTATE). The function inboundUnsampledTRACEPARENT() reads both env vars via the W3C TraceContext propagator. The asymmetry is harmless since the function checks TRACEPARENT first, but keeping them in sync avoids future confusion.
  Remediation: Add t.Setenv("TRACESTATE", "") to clearOTLPEnv in internal/evalmeasure/export_otlp_test.go.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (5)

Review

Findings

Low

• [edge-case] internal/evalmeasure/export_otlp.go:105 — ExportOTLPScores shares the otlpExportBudget (15s) context between ForceFlush and the deferred Shutdown. If ForceFlush consumes most of the deadline, Shutdown may receive an expired context. Harmless since ForceFlush already flushed spans, and the error is additive (errors.Join, not replacing).
• [error-handling] internal/evalmeasure/export_otlp.go:212 — capturingExporter.ExportSpans overwrites c.err on every call. If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 9 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT, OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT). No cli test exercises attribute truncation, so the risk is theoretical.
• [incomplete-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says "Derived products sit beside telemetry as sibling files" without mentioning the new OTLP export path. The sibling "OTEL Primary Facts" entry already describes OTLP export, creating an asymmetry.
  Remediation: Append a clause noting that when OTEL_EXPORTER_OTLP_* is set, scores also export as OTLP span events on the same TraceID (fail-open).
• [incomplete-doc] docs/cli/README.md:30 — The CLI command table describes fullsend eval-measure as "Score wild-run traces into eval-measurements.jsonl" without mentioning the new OTLP export capability. The command's --help text already documents OTLP export.
  Remediation: Update the description to mention OTLP export when configured.

Previous run (6)

Review

Findings

Low

• [edge-case] internal/evalmeasure/export_otlp.go:105 — ExportOTLPScores shares the otlpExportBudget (15s) context between ForceFlush and the deferred Shutdown. If ForceFlush consumes most of the deadline, Shutdown may receive an expired context. Harmless since ForceFlush already flushed spans, and the error is additive (errors.Join, not replacing).
• [error-handling] internal/evalmeasure/export_otlp.go:212 — capturingExporter.ExportSpans overwrites c.err on every call. If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [data-exposure] internal/evalmeasure/export_otlp.go:247 — Score OTLP export sends evaluation metadata (agent name, work_item_id, measurement version, score label, score value, and explanation text) to the operator-configured OTLP endpoint. This is the designed contract (same OTEL_* path as agent traces). The explanation is bounded by FreeTextAttrValueLenLimit (default 8192 chars).
• [naming-alignment] internal/evalmeasure/export_otlp.go — GenAI evaluation event and attribute names are pinned to the OpenTelemetry GenAI semantic conventions reference document (low-stability / reference status). The code includes a clear comment noting measurement versions should be bumped when attribute names change.
• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 9 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT, OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT). No cli test exercises attribute truncation, so the risk is theoretical.
• [incomplete-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says "Derived products sit beside telemetry as sibling files" without mentioning the new OTLP export path. The sibling "OTEL Primary Facts" entry already describes OTLP export, creating an asymmetry. The operational guides (eval-measurements.md, distributed-tracing.md) were updated in this PR.
  Remediation: Append a clause noting that when OTEL_EXPORTER_OTLP_* is set, scores also export as OTLP span events on the same TraceID (fail-open).
• [incomplete-doc] docs/cli/README.md:30 — The CLI command table describes fullsend eval-measure as "Score wild-run traces into eval-measurements.jsonl" without mentioning the new OTLP export capability. The command's --help text already documents OTLP export.
  Remediation: Update the description to mention OTLP export when configured.

Previous run (7)

Review

Findings

Low

• [edge-case] internal/evalmeasure/export_otlp.go:105 — ExportOTLPScores shares the otlpExportBudget (15s) context between ForceFlush and the deferred Shutdown. If ForceFlush consumes most of the deadline, Shutdown may receive an expired context. Harmless since ForceFlush already flushed spans, and the error is additive (errors.Join, not replacing).
• [error-handling] internal/evalmeasure/export_otlp.go:212 — capturingExporter.ExportSpans overwrites c.err on every call. If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [naming-alignment] internal/evalmeasure/export_otlp.go — GenAI evaluation event and attribute names are pinned to the OpenTelemetry GenAI semantic conventions reference document (low-stability / reference status). The code includes a clear comment noting measurement versions should be bumped when attribute names change.
• [data-exposure] internal/evalmeasure/export_otlp.go:247 — Score OTLP export sends evaluation metadata (agent name, work_item_id, measurement version, score label, score value, and explanation text) to the operator-configured OTLP endpoint. This is the designed contract (same OTEL_* path as agent traces). The explanation is bounded by FreeTextAttrValueLenLimit (default 8192 chars).
• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 9 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT, OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT). No cli test exercises attribute truncation, so the risk is theoretical.
• [incomplete-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says "Derived products sit beside telemetry as sibling files" without mentioning the new OTLP export path. The sibling "OTEL Primary Facts" entry already describes OTLP export, creating an asymmetry. The operational guides (eval-measurements.md, distributed-tracing.md) were updated in this PR.
  Remediation: Append a clause noting that when OTEL_EXPORTER_OTLP_* is set, scores also export as OTLP span events on the same TraceID (fail-open).
• [incomplete-doc] docs/cli/README.md:30 — The CLI command table describes fullsend eval-measure as "Score wild-run traces into eval-measurements.jsonl" without mentioning the new OTLP export capability. The command's --help text already documents OTLP export.
  Remediation: Update the description to mention OTLP export when configured.

Previous run (8)

Review

Findings

Low

• [incomplete-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says "Derived products sit beside telemetry as sibling files" without mentioning the new OTLP export path. The sibling "OTEL Primary Facts" entry already describes OTLP export, creating an asymmetry. The operational guides (eval-measurements.md, distributed-tracing.md) were updated in this PR.
  Remediation: Append a clause noting that when OTEL_EXPORTER_OTLP_* is set, scores also export as OTLP span events on the same TraceID (fail-open).
• [error-handling] internal/evalmeasure/export_otlp.go:212 — capturingExporter.ExportSpans overwrites c.err on every call. If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [edge-case] internal/evalmeasure/export_otlp.go:105 — ExportOTLPScores shares the otlpExportBudget (15s) context between ForceFlush and the deferred Shutdown. If ForceFlush consumes most of the deadline, Shutdown may receive an expired context. Harmless since ForceFlush already flushed spans, and the error is additive (errors.Join, not replacing).
• [error-handling] internal/evalmeasure/run.go:62 — The exportScored closure overwrites stats.RemoteExportWarning on error. In the current code exportScored is called at most once per MeasureAndExport invocation (every error path returns immediately after calling it, and the happy path calls it once at the end). Low risk given current structure.
• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 9 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT, OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT). No cli test exercises attribute truncation, so the risk is theoretical.
• [naming-alignment] internal/evalmeasure/export_otlp.go — GenAI evaluation event and attribute names are pinned to the OpenTelemetry GenAI semantic conventions reference document (low-stability / reference status). The code includes a clear comment noting measurement versions should be bumped when attribute names change.
• [incomplete-doc] docs/cli/README.md:30 — The CLI command table describes fullsend eval-measure as "Score wild-run traces into eval-measurements.jsonl" without mentioning the new OTLP export capability. The command's --help text already documents OTLP export.
  Remediation: Update the description to mention OTLP export when configured.
• [data-exposure] internal/evalmeasure/export_otlp.go:247 — Score OTLP export sends evaluation metadata (agent name, work_item_id, measurement version, score label, score value, and explanation text) to the operator-configured OTLP endpoint. This is the designed contract (same OTEL_* path as agent traces). The explanation is bounded by FreeTextAttrValueLenLimit (default 8192 chars).

Previous run (9)

Review

Findings

Medium

• [incomplete-doc] docs/glossary.md:157 — The "OTEL Derived Products" glossary entry says "Derived products sit beside telemetry as sibling files" without mentioning the new OTLP export path. The sibling "OTEL Primary Facts" entry already describes OTLP export, creating an asymmetry.
  Remediation: Add a clause noting that when OTEL_EXPORTER_OTLP_* is set, scores also export as OTLP span events on the same TraceID (fail-open).

Low

• [error-handling] internal/evalmeasure/export_otlp.go:212 — capturingExporter.ExportSpans overwrites c.err on every call. If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [edge-case] internal/evalmeasure/export_otlp.go:105 — ExportOTLPScores shares the otlpExportBudget (15s) context between ForceFlush and the deferred Shutdown. If ForceFlush consumes most of the deadline, Shutdown may receive an expired context. Harmless since ForceFlush already flushed spans, and the error is additive (errors.Join, not replacing).
• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 8 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT and OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT). No cli test exercises attribute truncation, so the risk is theoretical.
• [naming-alignment] internal/evalmeasure/export_otlp.go — GenAI evaluation event and attribute names are pinned to the OpenTelemetry GenAI semantic conventions reference document (low-stability / reference status). The code includes a clear comment noting measurement versions should be bumped when attribute names change.
• [incomplete-doc] docs/cli/README.md:30 — The CLI command table describes fullsend eval-measure as "Score wild-run traces into eval-measurements.jsonl" without mentioning the new OTLP export capability. The command's --help text already documents OTLP export.

Previous run (10)

Review

Findings

Low

• [error-handling] internal/evalmeasure/export_otlp.go:131 — capturingExporter.ExportSpans overwrites c.err on every call (c.err = err). If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. A successful final batch after an earlier failed batch would silently lose the export error. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [edge-case] internal/evalmeasure/export_otlp.go:90 — ExportOTLPScores shares the otlpExportBudget (15s) context between ForceFlush and the deferred Shutdown. If ForceFlush consumes most of the deadline, Shutdown may receive an expired context and produce a misleading otlp shutdown: context deadline exceeded error joined to the return. Harmless since ForceFlush already flushed spans, and the error is additive (errors.Join, not replacing).
• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 8 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT and OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT). No cli test exercises attribute truncation, so the risk is theoretical.
• [naming-alignment] internal/evalmeasure/export_otlp.go — GenAI evaluation event and attribute names are pinned to the OpenTelemetry GenAI semantic conventions reference document (low-stability / reference status). The code includes a clear comment noting measurement versions should be bumped when attribute names change.

Previous run (11)

Review

Findings

Low

• [error-handling] internal/evalmeasure/export_otlp.go:131 — capturingExporter.ExportSpans overwrites c.err on every call (c.err = err). If the BatchSpanProcessor calls ExportSpans multiple times, only the last call's error is retained. A successful final batch after an earlier failed batch would silently lose the export error. The overwrite is intentional (tested by TestCapturingExporter_ClearsErrorOnLaterSuccess) and practical batch sizes (1–5 scores) make multi-batch splits implausible.
• [code-organization] internal/evalmeasure/export_otlp_test.go:383 — hexOf reimplements encoding/hex.EncodeToString. The hack/prove-otlp-scores/main.go in this same PR already uses the stdlib function.

Info

• [naming-conventions] internal/cli/evalmeasure_test.go:22 — clearOTLPEnv in this file clears 6 env vars while the identically-named function in internal/evalmeasure/export_otlp_test.go clears 8 (also blanking OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT and OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT). No cli test exercises attribute truncation, so the risk is theoretical.
• [api-shape] internal/evalmeasure/parse.go:55 — RemoteExportWarning is added to ParseStats, which is otherwise a parse-centric struct. Consistent with how Incomplete was already used as a cross-concern signal surface for MeasureAndExport.

Previous run (12)

Review

Findings

Low

• [edge-case] internal/evalmeasure/export_otlp.go:189 — exportOneScore silently returns nil when TraceID or SpanID is empty. Currently these are LabelSkip results (missing root run span), but a future scorer producing pass/fail with an empty SpanID would be silently dropped with no warning in RemoteExportWarning. Consider logging or counting silent skips.
• [code-organization] internal/evalmeasure/export_otlp_test.go:383 — hexOf reimplements encoding/hex.EncodeToString. The hack/prove-otlp-scores/main.go in this same PR already uses the stdlib function.
• [code-organization] internal/evalmeasure/export_otlp_test.go:25 — scoreOTLPSink duplicates otlpSink from internal/telemetry/otlpsink_test.go. Different packages prevent direct reuse without a shared test utility.

Previous run (13)

Review

Findings

Low

• [edge-case] internal/evalmeasure/export_otlp.go:108 — exportOneScore silently returns nil when TraceID or SpanID is empty. Currently these are LabelSkip results (missing root run span), but a future scorer producing pass/fail with an empty SpanID would be silently dropped with no warning in RemoteExportWarning. Consider logging or counting silent skips.
• [code-organization] internal/evalmeasure/export_otlp_test.go:988 — hexOf reimplements encoding/hex.EncodeToString. The hack/prove-otlp-scores/main.go in this same PR already uses the stdlib function.
• [code-organization] internal/evalmeasure/export_otlp_test.go:678 — scoreOTLPSink duplicates otlpSink from internal/telemetry. Different packages prevent direct reuse without a shared test utility.
• [naming-convention] internal/evalmeasure/export_otlp_test.go:730 — clearOTLPEnv vs pinOTELEnv (in internal/telemetry) naming mismatch for the same env-clearing pattern.
• [naming-convention] internal/evalmeasure/export_otlp.go — Score-specific Attr* constants in export_otlp.go are separate from run-level Attr* constants in types.go. Feature-scoped grouping is valid Go convention.
• [api-shape] internal/telemetry/telemetry.go:42 — newOTLPExporter (test seam var) and NewOTLPExporter (exported func) differ only by case. Primary call site uses NewOTLPExporterBounded, making confusion risk minimal.
• [incomplete-doc] docs/glossary.md:157 — "OTEL Derived Products" definition says derived products "sit beside telemetry as sibling files" — with OTLP score export, they also travel over OTLP as span events. Incomplete but not incorrect.

Previous run (14)

Review

Findings

Medium

• [stale-doc] docs/problems/operational-observability.md:195 — Line says "remote scores reuse OTEL_EXPORTER_OTLP_* when implemented" — this PR implements the feature, so "when implemented" is now stale.
  Remediation: Update to present tense, e.g., "remote scores reuse OTEL_EXPORTER_OTLP_*" (drop "when implemented").

Low

• [error-handling] internal/evalmeasure/export_otlp.go:72 — TracerProvider shutdown error is silently discarded (_ = tp.Shutdown(shutCtx)). Unlikely to matter in practice since ForceFlush already exports spans.
• [naming-convention] internal/evalmeasure/export_otlp.go — Score-specific Attr* constants are declared separately from run-level Attr* constants in types.go. Feature-scoped grouping is valid Go convention but splits attribute definitions across files.
• [naming-convention] internal/telemetry/telemetry.go — BuildResource exports buildResource with an empty-string fallback. Naming asymmetry is idiomatic Go (unexported→exported) but the wrapper is minimal.
• [error-handling-idiom] internal/evalmeasure/export_otlp.go — parseTraceID/parseSpanID use sentence-like "trace_id must be non-zero" for leaf errors rather than the terse fragment style used elsewhere in the package.
• [code-organization] internal/evalmeasure/export_otlp_test.go — hexOf helper reimplements encoding/hex.EncodeToString.
• [code-organization] internal/evalmeasure/export_otlp_test.go — scoreOTLPSink duplicates otlpSink from internal/telemetry. Different packages prevent direct reuse without a shared test utility.
• [api-shape] internal/telemetry/telemetry.go — newOTLPExporter (test seam var) and NewOTLPExporter (exported func) differ only by case, though the primary call site uses NewOTLPExporterBounded which is more distinct.
• [adr-immutability] docs/ADRs/0087-eval-measurements-online-trace-scoring.md — Edits Accepted ADR's Decision/Consequences sections from future to present tense. Borderline but defensible as minor annotation rather than semantic rewrite — the architectural choice is unchanged.
• [incomplete-doc] docs/glossary.md:157 — "OTEL Derived Products" definition says derived products "sit beside telemetry as sibling files" — with OTLP score export now implemented, they also travel over OTLP as span events. Definition is incomplete but not incorrect.

[waynesun09]

Review (MEDIUM+)

Four inline comments on unique issues. Fail-open OTLP path looks solid (empty IDs skipped, export bounded, batch processor). The correctness gap to fix is resource identity.

Already posted (skipped): ADR 0087 Decision rewrite on docs/ADRs/0087-eval-measurements-online-trace-scoring.md — existing thread already covers it. Preferred fix remains a 0050-style annotation rather than editing the Decision.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:33 AM UTC · Completed 11:53 AM UTC

Commit: 911b9bf · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $6.68

[waynesun09]

Additional review pass (8 findings, all verified against PR head 911b9bf and cross-checked against existing threads for duplicates).

Superseded by updated review

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Ended 11:11 AM UTC

Commit: 20ee2ff · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:13 AM UTC · Completed 11:33 AM UTC

Commit: c64055d · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.65

Superseded by updated review

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 2:53 PM UTC · Completed 3:17 PM UTC

Commit: c64055d · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.32

[fullsend-ai-retro]

Retro: PR #6459 — Export eval measurement scores via OTLP

This was a human-authored PR (ascerra) implementing ADR 0087 portable remote score export — 19 files, ~1,500 lines, merged after 19 days and 23 commits. The PR went through 7 rounds of human review (waynesun09) and 15 successful automated review runs before merging.

Timeline

• Aug 21: PR opened. First automated review and first human review (4 findings: resource identity, TRACEPARENT sampling, span limits, semconv pin).
• Aug 21–29: Six more human review rounds with 17 additional findings spanning correctness bugs, concurrency issues, test hermeticity, and documentation gaps. Each round drove a fix commit.
• Sep 2: /fs-fix triggered → run 33683583135 failed ($3.82 wasted). A duplicate dispatch also fired → run 33683731364 ($3.66 wasted). Both rejected by post-fix script: "Agent commit contains a Signed-off-by trailer."
• Sep 3: /fs-fix do not use a sign-off by trailer → run 33745590114 failed ($3.66 wasted). Same error — even though the agent's own commit was clean, the validation script scanned human-authored commits on the branch that had legitimate DCO trailers.
• Sep 8: Final human review (2 findings: multi-batch error accumulation, prove-otlp-scores prints PASS despite warning). ralphbean approved.
• Sep 9: ascerra addressed remaining findings manually. Bot approved. Merged.

Evidence for existing issues

All potential proposals map to existing open issues. The concrete evidence from this retro is summarized below for each.

fullsend-ai/agents#1138 (validate-code-output.sh scans all branch commits instead of agent-only): This bug caused all three fix agent failures on this PR. The validation script uses MERGE_BASE..HEAD instead of PRE_AGENT_HEAD..HEAD, so human-authored commits with legitimate Signed-off-by: Adam Scerra <ascerra@redhat.com> trailers triggered false positives. Even when the agent produced a clean commit (attempt 2), validation still failed. Total waste: $11.14 across 3 runs, plus human time for 2 manual trigger attempts. The user gave up and fixed the code manually. This is a regression introduced by agents#999 (closed Aug 27), which fixed in-sandbox detection but scoped the check too broadly.

fullsend-ai/agents#1092 (incremental review mode for re-reviews): Of 15 successful review runs on this PR, only 2–3 produced new findings (13% hit rate). The remaining 12 ran the full Opus pipeline ($7.45 each) and self-dismissed. Total review cost: **$112**, of which ~$89 produced zero actionable output. Rapid-fire patterns on Aug 27 (3 reviews in 53 min) and Aug 29 (3 reviews in 48 min) were particularly wasteful — each small fixup commit triggered a full review that found nothing new.

fullsend-ai/agents#1106 and #1051 (review agent self-dismissal / contradictory findings): Across 14 review sweeps, the bot produced 29 distinct findings — all rated [low] severity, many containing self-dismissal language ("intentional and tested," "harmless," "minimal practical risk"). Meanwhile, the human reviewer found 21 findings including 5+ correctness/concurrency bugs (mid-loop persist error skipping OTLP, hack tool data race, capturingExporter dropping batch errors, prove-otlp-scores reporting PASS despite warnings, attribute truncation ignoring operator limits). The bot's flat severity calibration means genuine concerns near real bugs (e.g., capturingExporter overwrite) were indistinguishable from cosmetic naming nits.

fullsend-ai/agents#545 (correctness sub-agent should trace caller chains): The bot's analysis appears function-local — it never traced cross-function control flow. This caused it to miss the "mid-loop persist error silently skips OTLP export" bug (requires following a non-obvious code path across multiple functions) and the hack tool data race (requires reasoning about concurrent access patterns). The human reviewer caught both.

fullsend-ai/agents#552 (agent definitions should prohibit git commit -s): In fix attempt 1, the agent added a Signed-off-by trailer despite skill instructions. The agent's behavioral tendency to mimic human DCO patterns persists even after #6651 was closed (Aug 26). Stronger prohibition in agent definitions is still needed.

fullsend-ai/fullsend#6688 (DCO checks block bot/agent commits): The broader DCO problem is confirmed — fix agent cannot operate on PRs with human DCO-signed commits until bot vs. human commit identification is reliable.

What went well

• Human review quality was excellent. waynesun09's 7 review sweeps caught 21 findings including correctness bugs, concurrency issues, and documentation gaps. All were addressed.
• Concurrency cancellation worked. 4 of 19 dispatch attempts were correctly cancelled when superseded by a newer push.
• The PR ultimately shipped correctly. Despite the fix agent failures, the human resolved all findings and the code merged with 86% patch coverage.

No new proposals filed

All improvement opportunities identified map to existing open issues in the agents repo or fullsend repo. The evidence above has been documented for reference.