Add GPT-5.6 Sol Exa search benchmark ladder

[ayush-or]

Exa evidence without a second harness fork

This branch now contains only Exa-owned specs, redacted bundles, and raw trajectories on top of merged main. Benchmark execution comes from the canonical commit-pinned harness introduced by PR #6; no shared runner or root documentation is forked here.

TL;DR

Adds the historical GPT-5.6 Sol Exa 1-, 5-, and 25-turn ladder, matching calibrations, reviewed publication bundles, and clone-and-view raw trajectories. Generated files are byte-identical to the previous branch and are explicitly labeled as pre-PR #6 results.

Historical provenance

These results must not be interpreted as post-PR #6 reruns:

• DSQA uses strict Fully Correct accuracy, not canonical macro F1.
• WideSearch uses the historical 2025-01-01 grading reference, not upstream 2026-07-18.
• Raw Parquets intentionally contain benchmark inputs, targets, answers, search activity, and judge output.
• Rerunning the specs now creates a new evaluation series.

What is included

• Six TOML specs under run-specs/exa/
• Six checksum-protected bundles under published-runs/exa/
• Three complete 100-task raw runs under runs/exa/
• Engine-owned README files with current runner/viewer commands and provenance warnings

The PR diff is path-scoped to those three Exa directories.

Validation

• Search benchmark runner typecheck and 29 tests passed
• All six Exa TOMLs passed --dry-run; no paid calls were made
• Every published checksums.txt verified
• Raw artifacts and generated bundles match the old Exa branch byte-for-byte
• Viewer typecheck and 7 tests passed
• CLI loaded all three runs and each 1-/5-/25-turn filter
• Local web viewer served the Exa run index and UI successfully
• GitHub validate passed

Reviewer focus

• Confirm the historical-provenance warning is clear.
• Confirm retaining raw target/judge data is acceptable for this branch.
• Confirm no files outside run-specs/exa, published-runs/exa, and runs/exa are present.

[perry-the-pr-reviewer]

Perry's Review

Verdict: 💬 Comments/questions

Risk: 🟢 Low

Full review

This PR adds the GPT-5.6 Sol Exa search benchmark ladder — six TOML specs, six checksum-protected published bundles, and three complete 100-task raw trajectory runs — plus a harness dependency pin bump. The PR is data-heavy (129 files, 5839 additions) with no executable code changes; the substance is benchmark configuration, generated metadata, and Parquet trajectories.

What I verified

• All 6 published bundle checksums pass sha256sum -c — every file in every bundle is intact.
• Calibration cost estimates match the spec cost_estimates exactly for all three ladders (1-/5-/25-turn). The budget_usd values are ~4× the per-task planning estimate as documented.
• Actual run costs are within budget — 1-turn $36/$125 (29%), 5-turn $73/$270 (27%), 25-turn $167/$515 (33%).
• specSha256 in the run manifests matches the published run.toml checksums, and the published run.toml files match the run-specs source specs.
• executionFingerprint is consistent across all three runs, confirming the same harness generated them.
• Provenance warnings are clear and present in all three Exa READMEs (run-specs, published-runs, and runs directories), documenting: DSQA uses strict Fully Correct accuracy (not macro F1), WideSearch uses the 2025-01-01 grading reference, and results must not be relabeled as post-PR-#6.
• Published samples are properly redacted — include_inputs, include_answers, and include_search_queries are all false in every spec; the redacted JSONL files contain only citations, search call metadata, and request bodies (no benchmark targets or judge outputs).
• report.html files are clean static HTML with no sensitive content (no keys, secrets, or credentials).
• CI is green — validate check passed.

Findings

Suggestion — PR body claims path-scoping, but 9 files are outside the three Exa directories

The PR body states: "The PR diff is path-scoped to those three Exa directories" and asks reviewers to "Confirm no files outside the three Exa directories are present." However, 9 files outside those directories are modified — a benchmark-harness pin bump from e9801e4 to 77483ab across README.md, apps/search-bench-runner/README.md, apps/search-bench-runner/package.json, apps/search-bench-runner/bun.lock, apps/trajectories/README.md, apps/trajectories/package.json, apps/trajectories/bun.lock, published-runs/README.md, and run-specs/README.md. The harness bump is a legitimate related change (aligning the runner dependency with the canonical harness from PR #6), but the PR description is factually inaccurate about the diff scope, which undermines the reviewer checklist. Consider updating the PR body to acknowledge the harness pin bump.

Nit — 25-turn spec comment reads as an unfulfilled instruction

The comment in run-specs/exa/gpt-5.6-sol-exa-25turn.toml says "Set it explicitly to override the derivation" but max_total_results is never set in the TOML — the derivation already produces 125 (25 × 5), confirmed by maxTotalResults: 125 in the run manifest. The comment is accurate as documentation but reads like an instruction that wasn't followed. Consider rephrasing to make clear the derivation is sufficient and the key is optional.

Risk assessment

Risk: 🟢 Low

Dimension	Severity	Risk	Reasoning
Implementation risk	🟩	Low	Data publication only — TOML specs, JSON metadata, Parquet trajectories, HTML reports. No executable code changes. All checksums verify, cost estimates match calibrations, provenance is documented.
Premise risk	🟩	Low	Historical benchmark results from public datasets (browsecomp, DSQA, WideSearch) with clear provenance warnings. The approach of preserving byte-identical historical artifacts with explicit labeling is sound.
Estimated impact	🟩	Low	Worst case is benchmark data that needs re-labeling or removal — no user-facing system, no financial exposure, no security impact.

Risk Factor	Severity	Risk	Reasoning
Reversibility	🟩	Low	Git revert fully restores the previous state.
Detectability		None	Checksums immediately verify bundle integrity.
Blast radius	🟩	Low	Single benchmark repo; no downstream system depends on this data at runtime.
Data integrity	🟩	Low	All checksums verified; benchmark data from public datasets with documented provenance.
Financial exposure		None	Historical results with no billing or payment impact.
Security and privacy exposure	🟩	Low	Raw Parquet files contain benchmark inputs/targets/answers from public datasets; published bundles are properly redacted; PR body explicitly calls out the raw-data retention.
Propagation		None	No downstream systems read or build on this output.
Availability		None	No serving path is affected by this change.
Recovery cost		None	A rollback (git revert) completes the recovery.
Time to correct		None	Any issue is corrected as soon as noticed via file removal or re-labeling.

[perry-the-pr-reviewer]

The PR body claims the diff is "path-scoped to those three Exa directories" and asks reviewers to "confirm no files outside run-specs/exa, published-runs/exa, and runs/exa are present." This file (and 8 others — apps/search-bench-runner/*, apps/trajectories/*, published-runs/README.md, run-specs/README.md) are outside those directories, carrying a harness pin bump from e9801e4 → 77483ab. The bump is legitimate, but the PR description is factually inaccurate about the diff scope. Consider updating the body to acknowledge the harness pin bump.

▶ Prompt for agents: Update the PR description's "What is included" and "Reviewer focus" sections to list the harness pin bump in README.md, apps/*/, published-runs/README.md, and run-specs/README.md as an explicit change outside the three Exa directories.

[perry-the-pr-reviewer]

The comment says "Set it explicitly to override the derivation," but max_total_results is never set in this TOML — the derivation already produces 125 (25 × 5), confirmed by maxTotalResults: 125 in the run manifest. The comment is accurate as documentation but reads like an unfulfilled instruction. Consider rephrasing to make clear the derivation is sufficient and the key is only needed if you want a different value.

▶ Prompt for agents: Reword the comment to clarify that max_total_results is derived automatically (125 = 25 turns × 5 results) and that the key is optional — set it only to override the derived value.