Add GPT-5.6 Sol Parallel search benchmark ladder

[ayush-or]

Parallel evidence without a second harness fork

This branch now contains only Parallel-owned specs, redacted bundles, and raw trajectories on top of merged main. Benchmark execution comes from the canonical commit-pinned harness introduced by PR #6; no shared runner or root documentation is forked here.

TL;DR

Adds the historical GPT-5.6 Sol Parallel 1-, 5-, and 25-turn ladder, matching calibrations, reviewed publication bundles, and clone-and-view raw trajectories. Generated files are byte-identical to the previous branch and are explicitly labeled as pre-PR #6 results.

Historical provenance

These results must not be interpreted as post-PR #6 reruns:

• DSQA uses strict Fully Correct accuracy, not canonical macro F1.
• WideSearch uses the historical 2025-01-01 grading reference, not upstream 2026-07-18.
• Raw Parquets intentionally contain benchmark inputs, targets, answers, search activity, and judge output.
• Rerunning the specs now creates a new evaluation series.

What is included

• Six TOML specs under run-specs/parallel/
• Six checksum-protected bundles under published-runs/parallel/
• Three complete 100-task raw runs under runs/parallel/
• Engine-owned README files with current runner/viewer commands and provenance warnings

The PR diff is path-scoped to those three Parallel directories.

Validation

• Search benchmark runner typecheck and 29 tests passed
• All six Parallel TOMLs passed --dry-run; no paid calls were made
• Every published checksums.txt verified
• Raw artifacts and generated bundles match the old Parallel branch byte-for-byte
• Viewer typecheck and 7 tests passed
• CLI loaded all three runs and each 1-/5-/25-turn filter
• Local web viewer served the Parallel run index and UI successfully
• GitHub validate passed

Reviewer focus

• Confirm the historical-provenance warning is clear.
• Confirm retaining raw target/judge data is acceptable for this branch.
• Confirm no files outside run-specs/parallel, published-runs/parallel, and runs/parallel are present.

[perry-the-pr-reviewer]

Perry's Review

Verdict: 💬 Comments / questions

Risk: 🟢 Low

Note: The maintainer app lacks pull_requests:write on OpenRouterTeam, so this review is posted as COMMENT rather than APPROVE. The findings below are all suggestions/nits — no blockers were found. A human reviewer should approve if they agree.

Full review

Summary

This PR adds the historical GPT-5.6 Sol Parallel search benchmark ladder — six TOML specs, six published (redacted) run bundles, and three complete 100-task raw trajectory runs — to the search-benchmarks repo. All changes are data/spec artifacts; no executable code is modified. The harness commit pin is also updated (e9801e4 → 77483ab) across the root README, app package.json/bun.lock, and parent-directory READMEs.

Verification performed

• Checksums: All 36 files across the 6 published bundles verified against checksums.txt. All 63 raw Parquet files across the 3 raw runs verified against their manifest.json chunk entries.
• Spec consistency: source_spec_sha256 / specSha256 in every manifest matches the corresponding run.toml AND the run-specs/parallel/ source spec — all three SHA values agree for all 9 runs.
• Execution fingerprint: Identical across all 9 runs, confirming the same harness was used throughout.
• Redaction: Published samples.redacted.jsonl files contain only citations, search-call metadata, request config, and scores — no benchmark questions, expected answers, model responses, or search queries. Every spec sets include_inputs/answers/search_queries = false.
• Cost consistency: Suite costs in summary.json sum to totalCost for the 100-task runs.
• No secrets/PII: Scanned all published text for API keys, tokens, and credentials — none found.
• CI: validate check passes (typecheck + tests + dry-run for both runner and viewer).

Findings

Suggestion — max_total_results comment reads as an instruction but the field isn't set. The 25-turn spec's comment says "Set it explicitly to override the derivation" but the field is absent — the value (125) is derived. Inline comment below with a suggested reword.

Suggestion — runs directory is gitignored but this PR force-adds 69 files under it. The root .gitignore ignores all of runs/, which silently prevents git add from picking up new files under the tracked runs/parallel subtree without -f. Adding a negation pattern for the tracked subtree would make the intent explicit and prevent future contributors from hitting a silent no-op when adding new artifacts.

Nit — PR body "path-scoped" claim is inaccurate. The body says "The PR diff is path-scoped to those three Parallel directories" but 9 files outside those directories are modified (harness commit-pin updates in the root README, published-runs and run-specs parent READMEs, and the two apps' configs). The changes are legitimate and the "What is included" section does mention "Engine-owned README files", but the "path-scoped" claim contradicts the actual diff.

Risk assessment

Risk: 🟢 Low

Risk assessment:

Dimension	Severity	Risk	Reasoning
Implementation risk	🟩	Low	No executable code changes — all data/spec artifacts with verified checksums and consistent manifests.
Premise risk	🟩	Low	Provenance framing is clear and consistent across READMEs, manifests, and specs; historical-pre-PR-6 warnings are prominent.
Estimated impact	🟩	Low	Worst case is someone misinterprets historical results as current, but provenance warnings in every README and the PR body mitigate this.

Risk Factor	Severity	Risk	Reasoning
Reversibility	🟩	Low	Removing the artifacts is a standard revert.
Detectability	🟩	Low	All artifacts are checksummed; any tampering is caught by checksums.txt verification.
Blast radius	🟩	Low	A single repo's benchmark data; no user-facing or production impact.
Data integrity	🟩	Low	All checksums verified; no persisted production state is touched.
Financial exposure		None	No billing, payment, or accounting systems affected.
Security and privacy exposure	🟩	Low	Published bundles properly redacted; raw runs are repo-internal with access-boundary warnings.
Propagation	🟩	Low	No downstream systems consume this data beyond the local trajectory viewer.
Availability		None	No serving paths affected.
Recovery cost	🟩	Low	A revert fully restores the previous state.
Time to correct	🟩	Low	Any issue is corrected by a follow-up commit.

[perry-the-pr-reviewer]

This comment says "Set it explicitly to override the derivation," but max_total_results isn't set in this spec — the value (125) is derived from max_agent_turns × per-search results. The phrasing reads as an instruction to set the field, which could confuse readers into thinking it was forgotten.

Consider rewording to clarify the derivation is intentional:

Suggested change

	# explicitly to override the derivation.
	# max_total_results defaults to turns × per-search results (125 here); the
	# server default of 50 would otherwise bind around turn ten. Set it
	# explicitly only to override this derivation.