Add GPT-5.6 Sol Parallel search benchmark ladder

README.md

@@ -1,6 +1,6 @@
 # OpenRouter Search Benchmarks
 
-> **Runner / harness:** Use the [search benchmark runner](apps/search-bench-runner/README.md) for TOML specs, cost approval, resumable runs, and publication. Model execution comes from the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/e9801e4ddfd070f30d188ed26ebcda62b3234625).
+> **Runner / harness:** Use the [search benchmark runner](apps/search-bench-runner/README.md) for TOML specs, cost approval, resumable runs, and publication. Model execution comes from the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/77483ab4d585baeaaacd4dab9f4f60cc7823e65b).
 
 Standalone TypeScript tooling for running and inspecting OpenRouter search
 benchmarks. The harness supports BrowseComp, DeepSearchQA, and WideSearch over

apps/search-bench-runner/README.md

@@ -1,6 +1,6 @@
 # OpenRouter Search Benchmark Runner
 
-> **Runner / harness:** This package owns TOML specs, cost approval, resumable runs, and publication. Model execution comes from the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/e9801e4ddfd070f30d188ed26ebcda62b3234625).
+> **Runner / harness:** This package owns TOML specs, cost approval, resumable runs, and publication. Model execution comes from the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/77483ab4d585baeaaacd4dab9f4f60cc7823e65b).
 
 Search campaign orchestration for three benchmark suites:
 

apps/search-bench-runner/package.json

@@ -11,7 +11,7 @@
     "typecheck": "tsgo --noEmit"
   },
   "dependencies": {
-    "@openrouter/bench-harness": "git+https://github.com/OpenRouterTeam/benchmark-harness.git#e9801e4ddfd070f30d188ed26ebcda62b3234625",
+    "@openrouter/bench-harness": "git+https://github.com/OpenRouterTeam/benchmark-harness.git#77483ab4d585baeaaacd4dab9f4f60cc7823e65b",
     "effect": "3.20.0",
     "smol-toml": "1.6.1",
     "zod": "4.4.3"

apps/trajectories/README.md

@@ -1,6 +1,6 @@
 # Trajectory Reader
 
-> **Runner / harness:** Runs are created by the [search benchmark runner](../search-bench-runner/README.md) using the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/e9801e4ddfd070f30d188ed26ebcda62b3234625).
+> **Runner / harness:** Runs are created by the [search benchmark runner](../search-bench-runner/README.md) using the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/77483ab4d585baeaaacd4dab9f4f60cc7823e65b).
 
 Local tooling for inspecting raw bench-harness Parquet trajectories. It accepts
 either one `.parquet` file or a complete `runs/ts/<run-id>` directory and

apps/trajectories/package.json

@@ -10,7 +10,7 @@
     "web": "bun run web.ts"
   },
   "dependencies": {
-    "@openrouter/bench-harness": "git+https://github.com/OpenRouterTeam/benchmark-harness.git#e9801e4ddfd070f30d188ed26ebcda62b3234625"
+    "@openrouter/bench-harness": "git+https://github.com/OpenRouterTeam/benchmark-harness.git#77483ab4d585baeaaacd4dab9f4f60cc7823e65b"
   },
   "devDependencies": {
     "@types/node": "24.10.0",

published-runs/README.md

@@ -1,6 +1,6 @@
 # Published Runs
 
-> **Runner / harness:** Bundles are produced by the [search benchmark runner](../apps/search-bench-runner/README.md) from results generated by the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/e9801e4ddfd070f30d188ed26ebcda62b3234625).
+> **Runner / harness:** Bundles are produced by the [search benchmark runner](../apps/search-bench-runner/README.md) from results generated by the commit-pinned [OpenRouter benchmark harness](https://github.com/OpenRouterTeam/benchmark-harness/tree/77483ab4d585baeaaacd4dab9f4f60cc7823e65b).
 
 This directory contains intentionally tracked, redacted benchmark bundles
 grouped by search engine under `<engine>/<run-id>/`. Each bundle is

published-runs/parallel/README.md

@@ -0,0 +1,5 @@
+# Parallel Published Runs
+
+> **Historical provenance:** These redacted bundles were generated before the canonical harness dependency merged in PR #6. Their DSQA headline is strict Fully Correct accuracy, not macro F1, and their WideSearch grading reference date is `2025-01-01`.
+
+The generated bundle directories below are preserved byte-for-byte with their original checksums. Do not relabel them as post-PR #6 results or regenerate them during branch maintenance.

published-runs/parallel/gpt-5.6-sol-parallel-1turn-calibration/README.md

@@ -0,0 +1,16 @@
+# GPT-5.6 Sol with Parallel search, 1-turn calibration
+
+One task per suite to measure cost before the 100-task run.
+
+- Status: `complete`
+- Model: `openai/gpt-5.6-sol`
+- Full configuration: [`run.toml`](run.toml)
+- Machine-readable summary: [`summary.json`](summary.json)
+- Redacted trajectories: [`samples.redacted.jsonl`](samples.redacted.jsonl)
+- Self-contained report: [`report.html`](report.html)
+
+The published trajectory export excludes benchmark targets, grader details,
+raw inputs, model answers, search queries, session IDs, and generation IDs by
+default. Optional content must be explicitly enabled in the run spec and reviewed
+before sharing. Raw Parquet artifacts are identified by checksum in
+`manifest.json` and remain outside Git.

published-runs/parallel/gpt-5.6-sol-parallel-1turn-calibration/checksums.txt

@@ -0,0 +1,6 @@
+e0e90c5aca9eb1f5f609fd1a06581201cc1baafa9facf456aa5f70dc7fd68686  README.md
+7816f75f05e5ffe2138067001f1a2497ec2fe0cecfd77e6c0999c0668eacdfdc  manifest.json
+796a0db8959c93536aa05d7d88cf5d53e5d3f334e5cecc94d51d32f22621c1ce  report.html
+8b506c86070e73605c6c6f6a353feb36a0b3873f8f442691989f1e85bb42b11e  run.toml
+d91bf907191337889ac2a0dcf3b6ab3f38f20086f9dc974b94dfbe1c4f625b29  samples.redacted.jsonl
+ff9b9707cfe4520ce7266cf4ba9e5431651119048b76ba6623b16a85952886ef  summary.json

published-runs/parallel/gpt-5.6-sol-parallel-1turn-calibration/manifest.json

@@ -0,0 +1,122 @@
+{
+  "version": 1,
+  "run_id": "gpt-5.6-sol-parallel-1turn-calibration",
+  "status": "complete",
+  "created_at": "2026-08-03T01:45:04.324Z",
+  "updated_at": "2026-08-03T01:47:59.990Z",
+  "source_spec_sha256": "8b506c86070e73605c6c6f6a353feb36a0b3873f8f442691989f1e85bb42b11e",
+  "execution_fingerprint": "b79d48558bd7a8057e47c50d20bf03899b14a3cf7f43a387cf955c933ab025cf",
+  "harness_changes": [],
+  "repository_commit": "fb1f5c771ae757d958b0abf4c6ffdc9b9059df7a",
+  "repository_dirty": true,
+  "estimated_cost_usd": null,
+  "approved_cost_usd": 2,
+  "effective_concurrency": 15,
+  "resolved_configs": {
+    "browsecomp": {
+      "benchmarkId": "search_browsecomp",
+      "model": "openai/gpt-5.6-sol",
+      "temperature": 0,
+      "providerOnly": [
+        "OpenAI"
+      ],
+      "allowFallbacks": false,
+      "lane": {
+        "webSearch": "server-tool",
+        "engine": "parallel",
+        "maxAgentTurns": 1
+      },
+      "maxRetries": 10
+    },
+    "dsqa": {
+      "benchmarkId": "search_dsqa",
+      "model": "openai/gpt-5.6-sol",
+      "temperature": 0,
+      "providerOnly": [
+        "OpenAI"
+      ],
+      "allowFallbacks": false,
+      "lane": {
+        "webSearch": "server-tool",
+        "engine": "parallel",
+        "maxAgentTurns": 1
+      },
+      "maxRetries": 10
+    },
+    "widesearch": {
+      "benchmarkId": "search_widesearch",
+      "model": "openai/gpt-5.6-sol",
+      "temperature": 0,
+      "providerOnly": [
+        "OpenAI"
+      ],
+      "allowFallbacks": false,
+      "lane": {
+        "webSearch": "server-tool",
+        "engine": "parallel",
+        "maxAgentTurns": 1
+      },
+      "maxRetries": 10
+    }
+  },
+  "dataset_contracts": {
+    "browsecomp": {
+      "benchmarkId": "search_browsecomp",
+      "source": "https://openaipublic.blob.core.windows.net/simple-evals/browse_comp_test_set.csv",
+      "sha256": "7b24471cd5b3eb2a46830a14802b5c029ea62f488ff75a0f88af7923d1454abf",
+      "rows": 1266,
+      "promptSha256": "2250aa21b4a98647dbcfed62a30261eb914016c81cb33e7694c15859205de78f",
+      "judgeModel": "openai/gpt-4.1"
+    },
+    "dsqa": {
+      "benchmarkId": "search_dsqa",
+      "source": "https://huggingface.co/datasets/google/deepsearchqa/resolve/b2623f8653065c2672de6d941fc5434cd652376c/DSQA-full.csv",
+      "sha256": "25d48dcf7efa872e5467032e8b8eedf38d301f59a252d0da95cda584baa78396",
+      "rows": 900,
+      "promptSha256": "2250aa21b4a98647dbcfed62a30261eb914016c81cb33e7694c15859205de78f",
+      "judgeModel": "google/gemini-2.5-flash"
+    },
+    "widesearch": {
+      "benchmarkId": "search_widesearch",
+      "source": "https://huggingface.co/datasets/ByteDance-Seed/WideSearch/resolve/6531a7e5b497d44c8912407e0cb3dc95bd98cc09/widesearch.jsonl",
+      "revision": "ByteDance-Seed/WideSearch@6531a7e5b497d44c8912407e0cb3dc95bd98cc09",
+      "sha256": "bba28ec51dce28fa617f82617d88fcd6bdd4cd4d7f0a4d70db07d7fa8a90bdf4",
+      "rows": 200,
+      "promptSha256": "14061a8a9476ecd7f5b5a4ca3de70248bd9a46f6ff963992a813538382ef23e8",
+      "judgeModel": "openai/gpt-4.1",
+      "gradingReferenceDate": "2025-01-01T00:00:00.000Z"
+    }
+  },
+  "raw_artifacts": [
+    {
+      "suite": "browsecomp",
+      "range": [
+        0,
+        1
+      ],
+      "filename": "000000-000001.parquet",
+      "sha256": "ff7c4b8d241fe4b1160e497c736d0186f04c9a703c17c8874f3a4bf934036413",
+      "bytes": 23243
+    },
+    {
+      "suite": "dsqa",
+      "range": [
+        0,
+        1
+      ],
+      "filename": "000000-000001.parquet",
+      "sha256": "18cc916be7f82d68ca10e2e9282bf13d61ed7b0fcff13e434f84cc47d7d44424",
+      "bytes": 25355
+    },
+    {
+      "suite": "widesearch",
+      "range": [
+        0,
+        1
+      ],
+      "filename": "000000-000001.parquet",
+      "sha256": "37e7996b470e64af0343ce0689d0f50597f51818c10a4eed4b0d2b31fbf15111",
+      "bytes": 38963
+    }
+  ]
+}

published-runs/parallel/gpt-5.6-sol-parallel-1turn-calibration/report.html

@@ -0,0 +1,26 @@
+<!doctype html>
+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+<title>GPT-5.6 Sol with Parallel search, 1-turn calibration</title>
+<style>
+:root{color-scheme:light;font-family:Inter,ui-sans-serif,system-ui,sans-serif;background:#f7f7fb;color:#17171d}body{margin:0}.wrap{max-width:1080px;margin:auto;padding:56px 24px 80px}.top{display:flex;justify-content:space-between;gap:20px;border-bottom:1px solid #dedee8;padding-bottom:18px}.brand{font-weight:750}.status{font:600 12px ui-monospace,SFMono-Regular,monospace;text-transform:uppercase;color:#d24f2a}h1{font-size:clamp(36px,7vw,72px);letter-spacing:-.055em;line-height:.96;margin:64px 0 18px;max-width:850px}.lede{font-size:18px;color:#5d5d6b;max-width:720px;line-height:1.55}.meta{font:12px ui-monospace,SFMono-Regular,monospace;color:#77778a;margin:28px 0 42px}.grid{display:grid;grid-template-columns:repeat(3,1fr);gap:16px}.card{background:white;border:1px solid #e4e4ec;border-radius:14px;padding:24px}.eyebrow{text-transform:uppercase;letter-spacing:.14em;font-size:10px;color:#d24f2a;font-weight:700}.card h2{text-transform:capitalize;margin:8px 0 30px}.score{font-size:48px;font-weight:750;letter-spacing:-.05em}dl{display:grid;grid-template-columns:1fr auto;gap:8px;margin:24px 0 0;font-size:12px}dt{color:#77778a}dd{margin:0;font-variant-numeric:tabular-nums}.footer{margin-top:32px;padding-top:20px;border-top:1px solid #dedee8;font-size:12px;color:#77778a}@media(max-width:760px){.grid{grid-template-columns:1fr}.top{flex-direction:column}h1{margin-top:42px}}
+</style></head><body><main class="wrap"><header class="top"><div class="brand">OpenRouter Search Benchmarks</div><div class="status">complete</div></header>
+<h1>GPT-5.6 Sol with Parallel search, 1-turn calibration</h1><p class="lede">One task per suite to measure cost before the 100-task run.</p>
+<div class="meta">openai/gpt-5.6-sol · run gpt-5.6-sol-parallel-1turn-calibration · $0.2536 provider-reported</div>
+<section class="grid"><article class="card">
+        <div class="eyebrow">accuracy</div>
+        <h2>browsecomp</h2>
+        <div class="score">0.0%</div>
+        <dl><dt>Tasks</dt><dd>1/1</dd><dt>Cost</dt><dd>$0.0538</dd><dt>Tokens</dt><dd>3,994</dd></dl>
+      </article>
+<article class="card">
+        <div class="eyebrow">accuracy</div>
+        <h2>dsqa</h2>
+        <div class="score">0.0%</div>
+        <dl><dt>Tasks</dt><dd>1/1</dd><dt>Cost</dt><dd>$0.0585</dd><dt>Tokens</dt><dd>3,951</dd></dl>
+      </article>
+<article class="card">
+        <div class="eyebrow">f1_by_item</div>
+        <h2>widesearch</h2>
+        <div class="score">15.5%</div>
+        <dl><dt>Tasks</dt><dd>1/1</dd><dt>Cost</dt><dd>$0.1412</dd><dt>Tokens</dt><dd>9,308</dd></dl>
+      </article></section><footer class="footer">Generated 2026-08-03T01:47:59.990Z · targets and judge trajectories excluded from this published bundle</footer></main></body></html>

published-runs/parallel/gpt-5.6-sol-parallel-1turn-calibration/run.toml

@@ -0,0 +1,26 @@
+version = 1
+title = "GPT-5.6 Sol with Parallel search, 1-turn calibration"
+description = "One task per suite to measure cost before the 100-task run."
+model = "openai/gpt-5.6-sol"
+suites = ["browsecomp", "dsqa", "widesearch"]
+start = 0
+limit = 1
+epochs = 1
+concurrency = 15
+chunk_size = 1
+
+[inference]
+temperature = 0
+max_retries = 10
+provider_only = ["OpenAI"]
+allow_fallbacks = false
+
+[search]
+surface = "server-tool"
+engine = "parallel"
+max_agent_turns = 1
+
+[publish]
+include_inputs = false
+include_answers = false
+include_search_queries = false

published-runs/parallel/gpt-5.6-sol-parallel-1turn-calibration/samples.redacted.jsonl

@@ -0,0 +1,3 @@
+{"suite":"browsecomp","sampleId":"browsecomp-0","epoch":0,"score":"I","citations":[{"url":"https://www.theguardian.com/","title":"www.theguardian.com"},{"url":"https://assembly.malala.org/","title":"assembly.malala.org"},{"url":"https://www.degruyterbrill.com/","title":"www.degruyterbrill.com"},{"url":"https://apa.si.edu/","title":"apa.si.edu"},{"url":"https://thenewinquiry.com/","title":"thenewinquiry.com"}],"searchCalls":[{"id":"st_tmp_h3ent6bf8pj","status":"completed","action":"search","executed":true,"sourceCount":5}],"requestBody":{"model":"openai/gpt-5.6-sol","maxOutputTokens":128000,"temperature":0,"provider":{"only":["OpenAI"],"allowFallbacks":false},"tools":[{"type":"openrouter:web_search","parameters":{"engine":"parallel","maxUses":1,"excludedDomains":["huggingface.co","huggingface.org","aifasthub.com","hf-mirror.com","hf.co"]}}],"maxToolCalls":1}}
+{"suite":"dsqa","sampleId":"dsqa-0","epoch":0,"score":"I","citations":[{"url":"https://migrationobservatory.ox.ac.uk/","title":"migrationobservatory.ox.ac.uk"},{"url":"https://migrationobservatory.ox.ac.uk/","title":"migrationobservatory.ox.ac.uk"},{"url":"https://migrationobservatory.ox.ac.uk/","title":"migrationobservatory.ox.ac.uk"},{"url":"https://data-explorer.oecd.org/","title":"data-explorer.oecd.org"},{"url":"https://www.oecd.org/","title":"www.oecd.org"}],"searchCalls":[{"id":"st_tmp_etfqlc92b3","status":"completed","action":"search","executed":true,"sourceCount":5}],"requestBody":{"model":"openai/gpt-5.6-sol","maxOutputTokens":128000,"temperature":0,"provider":{"only":["OpenAI"],"allowFallbacks":false},"tools":[{"type":"openrouter:web_search","parameters":{"engine":"parallel","maxUses":1,"excludedDomains":["huggingface.co","huggingface.org","aifasthub.com","hf-mirror.com","hf.co"]}}],"maxToolCalls":1}}
+{"suite":"widesearch","sampleId":"ws_en_001","epoch":0,"score":"I","citations":[{"url":"https://www.qs.com/","title":"www.qs.com"},{"url":"https://magazine.qs.com/","title":"magazine.qs.com"},{"url":"https://www.qs.com/","title":"www.qs.com"},{"url":"https://www.hindustantimes.com/","title":"www.hindustantimes.com"},{"url":"https://humanities.web.ox.ac.uk/","title":"humanities.web.ox.ac.uk"}],"searchCalls":[{"id":"st_tmp_r3ra4tbgr6i","status":"completed","action":"search","executed":true,"sourceCount":5},{"id":"st_tmp_ipiicin44nm","status":"completed","action":"search","executed":false,"sourceCount":0},{"id":"st_tmp_j79csjq27r","status":"completed","action":"search","executed":false,"sourceCount":0},{"id":"st_tmp_t5g2syw2gpd","status":"completed","action":"search","executed":false,"sourceCount":0},{"id":"st_tmp_3nshb8tsom7","status":"completed","action":"search","executed":false,"sourceCount":0}],"requestBody":{"model":"openai/gpt-5.6-sol","maxOutputTokens":128000,"temperature":0,"provider":{"only":["OpenAI"],"allowFallbacks":false},"tools":[{"type":"openrouter:web_search","parameters":{"engine":"parallel","maxUses":1,"excludedDomains":["huggingface.co","huggingface.org","aifasthub.com","hf-mirror.com","hf.co"]}}],"maxToolCalls":1}}