Use the canonical benchmark harness dependency

[ayush-or]

The search repository owns a drifting fork of the harness

search-benchmarks currently carries a second implementation of @openrouter/bench-harness. Fixes must be ported twice, and downstream-only changes can silently fork prompts, graders, metrics, or persistence. PR #5 exposed this directly: the official DSQA repair landed upstream and also required a separate local rewrite.

TL;DR

Delete the local packages/bench-harness fork, pin both search applications directly to merged benchmark-harness#11 at immutable commit e9801e4, and move repository-specific runner policy into apps/search-bench-runner. TOML specs, dry-run, cost approval, deterministic resumable chunks, publication, redaction, and viewer behavior remain available; only the runner working directory changes.

This PR targets main and supersedes the now-closed #5. After merge, the perplexity, parallel, and exa branches should be restacked onto the new main.

What Changed?

One canonical benchmark implementation

• Removes packages/bench-harness entirely.
• Pins @openrouter/bench-harness in both app manifests and lockfiles to merged commit e9801e4ddfd070f30d188ed26ebcda62b3234625.
• Removes subtree update, integrity, metadata, nested repository, and duplicated upstream CI files.
• Benchmark behavior changes now land in OpenRouterTeam/benchmark-harness; consumers only bump an immutable SHA.

Search benchmark runner boundary

• Moves TOML parsing, paid-run approval, budgets, chunk planning, run manifests, model-ceiling discovery, publication, redaction, and reports to apps/search-bench-runner.
• Uses only public @openrouter/bench-harness exports.
• Adds a runner-owned result-store adapter so deterministic filenames such as 000000-000010.parquet survive without an upstream patch.
• Includes the runner source and lockfile in resumable-run fingerprints, so an upstream SHA bump invalidates stale execution state.

Setup

The runner flags and relative spec paths are unchanged; only the working directory moves:

cd apps/search-bench-runner
bun install
bun run bench -- \
  --spec ../../run-specs/example-partner-search.toml \
  --run-id example \
  --dry-run

Each search-benchmarks-owned README links the runner and pinned canonical harness at the top. Root CI validates runner typecheck/tests/dry-run and viewer compatibility. Upstream owns harness format/lint/typecheck/tests/build.

Compatibility and behavior

• Existing trajectory imports remain unchanged because all consumed package subpaths are public upstream exports.
• The checked-in Parquet fixture remains readable at format version 1.
• The upstream DSQA dataset resolves all 900 rows.
• Existing incomplete runs detect the changed runner/dependency lock fingerprint and require explicit --allow-harness-change to resume.
• WideSearch now follows upstream's recorded grading reference date (2026-07-18) rather than the fork's 2025-01-01; this is visible in run provenance and is an intentional consequence of upstream ownership.

Validation

Completed locally with Bun 1.3.14:

• Runner typecheck and 29 tests passed
• Deterministic result-store path and Parquet round trip passed
• Trajectory viewer typecheck, 7 tests, and demo.parquet CLI load passed
• Full DSQA dataset size resolved to 900
• Three-suite example dry-run passed with no paid calls
• Perplexity five-turn, ten-task DSQA dry-run passed with no paid calls
• Frozen installs resolve @openrouter/bench-harness to e9801e4 in both apps
• Host workflow YAML and diff whitespace checks passed

The merged upstream commit independently passed its format, lint, typecheck, 1,173 tests, build, CodeQL, and CI before this dependency pin.

Sequencing

1. Merge this PR into main.
2. Restack perplexity, parallel, and exa onto the new main, preserving only engine-specific specs and reviewed result bundles.

Reviewer Focus

• Confirm no benchmark implementation remains in this repository.
• Confirm both manifests and locks resolve the same immutable upstream commit.
• Confirm all paid-run safety gates remain in apps/search-bench-runner.
• Confirm deterministic chunk names and manifest provenance remain stable.
• Confirm the command-location and WideSearch reference-date changes are acceptable.

[ayush-or]

Paid local integration validation

Validated the merged Git dependency end to end with an untracked DSQA-only Perplexity TOML:

model:             openai/gpt-5.6-sol
tasks:             5
search turns:      25
cumulative cap:    125 results
concurrency:       2
judged:            5
skipped:            0
grader parse errors: 0
macro F1:          0.880000
macro precision:   0.900000
macro recall:      0.866667
Fully Correct:     0.800000 (4/5)
Partially Correct: 0.200000
actual cost:       $1.184759
approved ceiling:  $3.00

One generation ended without response.completed; the configured transient retry recovered and the run completed. All five trajectories contained nonempty correctness-detail maps. The partial sample received precision 0.5, recall 0.3333, and F1 0.4, confirming partial-credit grading through the Git-installed harness, deterministic Parquet store, publication, and trajectory reader.

The local TOML and generated result bundle are intentionally untracked and are not part of this PR.