Add cross-version benchmarks CI workflow

[mattleibow]

What

Adds a PR-triggered Benchmarks workflow that runs the SkiaSharp micro-benchmarks across multiple SkiaSharp versions on Linux, Windows, and macOS, then merges the results into a single comparison report posted to the job summary.

This grew out of validating the native blur fix (SK_AVOID_SLOW_RASTER_PIPELINE_BLURS): there was no way to measure such changes with real numbers across versions and platforms. Now there is.

Why a build matrix instead of one process

BenchmarkDotNet cannot host two different versions of the same assembly in a single process (type-identity clash — there is no built-in WithNuGet). So each (os, version) combination is benchmarked in its own job and the JSON exports are merged afterwards.

How it works

flowchart LR
  setup[setup: resolve versions] --> pub[published matrix: os x version]
  setup --> cur[current matrix: linux + macos, native build]
  pub --> rep[report: merge -> job summary]
  cur --> rep

Loading

• published — SkiaSharp.Benchmarks.Compare, an isolated harness that links the same benchmark sources but restores a published SkiaSharp NuGet version from nuget.org. It deliberately opts out of the repo build infrastructure (empty Directory.Build.props/.targets) and carries its own NuGet.config so it resolves the exact released version requested. Fast and reliable — always reports real numbers on all three OSes. Default versions: 4.150.0-preview.2.1 (baseline) and 3.119.4.
• current — benchmarks the working tree's native code. Rather than building the multi-targeted in-repo binding graph (which drags in the mobile native-asset projects and needs the Android/iOS workloads the runners don't have — NETSDK1147), it reuses the same Compare harness: it restores the baseline published managed package, then replaces that package's libSkiaSharp in the NuGet global cache with the one freshly built from this PR. BenchmarkDotNet builds its child project against that cache at run time, so the benchmark exercises this PR's native code with a known, stable managed API. The job logs the SHA before/after the swap to prove the working-tree binary is the one being measured. Best-effort (continue-on-error); scoped to the OSes whose native library reliably builds from source on a hosted runner (Linux + macOS).
• report — scripts/benchmarks/merge-benchmarks.py merges every run into one Markdown table (mean µs + ratio-vs-baseline) written to $GITHUB_STEP_SUMMARY and uploaded as an artifact.

Reading the current column ⚠️

The from-source externals-* native is not built with the same optimization/official-build flags as the shipped NuGet native, so its absolute timings are systematically different (typically slower) than the optimized published packages. Do not read a current-vs-published ratio as the effect of a PR's code. To measure a native change, compare two current runs built the same way (the PR branch vs its base). The published columns are the apples-to-apples comparison across released versions.

The Windows hosted image currently cannot build the native library from source (missing Windows SDK 10.0.19041 and Spectre-mitigated MSVC libs on the VS preview image), so Windows is excluded from current by default and still appears in the published comparison. It can be opted back in via the current_oses dispatch input once the toolchain is available.

Other changes

• BenchmarkConfig (shared) adds JsonExporter.FullCompressed — that export cannot be selected from the command line in this BenchmarkDotNet version, so it is configured in code to guarantee every run produces the JSON the merge step consumes.
• BlurImageFilterBenchmark exercises the 8888 raster blur path affected by the native flag (small-sigma slow path vs large-sigma control).
• SurfaceCanvasBenchmark switched to SKPath (CS0618 suppressed) so the shared sources compile against older releases that predate SKPathBuilder.

Triggers

• pull_request touching benchmarks/**, scripts/benchmarks/**, or the workflow file.
• workflow_dispatch with inputs: versions, filter, job (short/default), build_current, current_oses, extra_feed (e.g. point at a PR preview package source).

Validation — real CI numbers

Proven on real GitHub-hosted runners (run 28379961911): all six published cells (3 OSes × 2 versions) plus both current cells (Linux x64, macOS arm64) succeeded and merged into one report. The current cells logged the native SHA swap, e.g. macOS built 726ab1f0… replacing the published 4523f8db…, confirming the working-tree binary was measured. Example (mean µs, baseline 4.150.0-preview.2.1):

OS	Benchmark	baseline	3.119.4	current
windows-x64	BlurImage 1024/σ1	1,793	2,134	—
linux-x64	BlurImage 1024/σ1	4,199	3,955	4,502
osx-arm64	BlurImage 1024/σ1	414	472	643

(The macOS current being slower than published despite identical m150 code is exactly the build-flag caveat above.)

Note: this is infrastructure only — no library code or public API changes.

[github-actions]

📦 Try the packages from this PR

Warning

Do not run these scripts without first reviewing the code in this PR.

Step 1 — Download the packages

bash / macOS / Linux:

curl -fsSL https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.sh | bash -s -- 4276

PowerShell / Windows:

iex "& { $(irm https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.ps1) } 4276"

Step 2 — Add the local NuGet source

dotnet nuget add source ~/.skiasharp/hives/pr-4276/packages --name skiasharp-pr-4276

More options

Option	Description
--successful-only / -SuccessfulOnly	Only use successful builds
--force / -Force	Overwrite previously downloaded packages
--list / -List	List available artifacts without downloading
--build-id ID / -BuildId ID	Download from a specific build

Or download manually from Azure Pipelines — look for the nuget artifact on the build for this PR.

Remove the source when you're done:

dotnet nuget remove source skiasharp-pr-4276

[github-actions]

📖 Documentation Preview

The documentation for this PR has been deployed and is available at:

🔗 View Staging Site
🔗 View Staging Docs
🔗 View Staging Gallery (Blazor)
🔗 View Staging Gallery (Uno Platform)
🔗 View Staging SkiaFiddle

This preview will be updated automatically when you push new commits to this PR.

---

This comment is automatically updated by the documentation staging workflow.

[Copilot]

Pull request overview

Adds a PR-triggered GitHub Actions workflow to run SkiaSharp micro-benchmarks across multiple released versions (and optionally the PR’s native build), then merges all BenchmarkDotNet JSON exports into a single Markdown comparison report posted to the job summary.

Changes:

• Add .github/workflows/benchmarks.yml with a setup → published matrix → optional current (native swap-in) → report merge pipeline.
• Add benchmarks/SkiaSharp.Benchmarks.Compare harness that benchmarks published SkiaSharp NuGet versions while reusing the same benchmark source files.
• Add scripts/benchmarks/merge-benchmarks.py and shared BenchmarkConfig to ensure *-report-full-compressed.json is emitted and mergeable.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/infra/caching/repo-deps.json	Include scripts/benchmarks/** in dependency/caching inputs.
scripts/benchmarks/merge-benchmarks.py	New merger that produces a consolidated Markdown comparison report from multiple BDN JSON artifacts.
benchmarks/SkiaSharp.Benchmarks/Program.cs	Run benchmarks with shared BenchmarkConfig to guarantee required JSON export.
benchmarks/SkiaSharp.Benchmarks/Benchmarks/SurfaceCanvasBenchmark.cs	Switch star-path creation to SKPath + suppress CS0618 for cross-version compilation.
benchmarks/SkiaSharp.Benchmarks/Benchmarks/BlurImageFilterBenchmark.cs	New benchmark targeting CPU raster blur paths relevant to native blur flag changes.
benchmarks/SkiaSharp.Benchmarks/BenchmarkConfig.cs	Shared BenchmarkDotNet config adding JsonExporter.FullCompressed.
benchmarks/SkiaSharp.Benchmarks.Compare/SkiaSharp.Benchmarks.Compare.csproj	New isolated harness that restores a specified published SkiaSharp version and links shared benchmark sources.
benchmarks/SkiaSharp.Benchmarks.Compare/Program.cs	Entry point to run the compare harness with shared config.
benchmarks/SkiaSharp.Benchmarks.Compare/NuGet.config	Ensure compare harness restores from nuget.org (with optional extra feed via workflow).
benchmarks/SkiaSharp.Benchmarks.Compare/Directory.Build.props	Opt out of repo-level build infrastructure to allow resolving exact published package versions.
benchmarks/SkiaSharp.Benchmarks.Compare/Directory.Build.targets	Empty targets file for symmetry/clarity (prevents walking up to repo build infrastructure).
.github/workflows/benchmarks.yml	New CI workflow implementing published/current benchmark runs and summary report merge.

[mattleibow]

Validated — this is a valid but low-severity robustness point.

The division in question is actually at line 139 (ratios.append(f"{v}: {cols[v] / base:.2f}x"), where base = cols[baseline]); the comment is anchored a few lines below at 142.

Reachability of base == 0:

• Normal runs: not reachable. base traces back to float(mean) from a BenchmarkDotNet Statistics.Mean, and a benchmark that actually ran always reports a positive mean — never exactly 0. load_records already skips Mean is None, but not Mean == 0.
• Only reachable via a contrived corrupt-but-structurally-valid BDN JSON whose baseline benchmark has Mean: 0. Truncated/partial files fail json.load (already caught), so this edge case is narrow.

Given this is report-only tooling on the continue-on-error path (not the product build), the impact is limited to failing the report job in that contrived case. A 2-line guard (skip non-positive baseline means before computing ratios) would fully close it.

Note for triage: this PR is likely to be closed as superseded by #4469 (the track-benchmarks system already on main). If the Compare harness / merge script is salvaged into #4469, the base <= 0 guard is best added there rather than spending a commit here.