ci(codspeed): move the simulation bench gate to a dedicated self-hosted runner - #75
ci(codspeed): move the simulation bench gate to a dedicated self-hosted runner#75sedghi wants to merge 3 commits into
Conversation
…d runner Move the codspeed-bench job off GitHub's shared pool (ubuntu-24.04) onto a fixed self-hosted runner labeled 'codspeed-bench'. Simulation instruction counts derive from the runner CPU's cache model, and the shared pool randomly assigns Intel vs AMD hardware, producing spurious cross-run 'Different runtime environments' deltas. A dedicated fixed box keeps every run on identical hardware so the gate is stable. Adds docs/ci/self-hosted-runner.md with the box requirements and cutover order.
📝 WalkthroughWalkthroughThe PR moves simulation benchmarks to the shared ChangesSelf-hosted CodSpeed runner migration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant NashuaRunner
participant LockWrapper
participant LernaBenchmarks
GitHubActions->>NashuaRunner: Start simulation benchmark job
NashuaRunner->>LockWrapper: Run with-nashua-lock.sh
LockWrapper->>LockWrapper: Acquire shared flock lock
LockWrapper->>LernaBenchmarks: Execute parallel benchmarks
LernaBenchmarks-->>LockWrapper: Return benchmark result
LockWrapper->>LockWrapper: Release lock
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ore is a throughput win
Merging this PR will degrade performance by 19.39%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decode CT-512x512-near-lossless.JLS (.81 near-lossless) — cold |
19.8 ms | 37.9 ms | -47.78% |
| ❌ | Simulation | decode CT-512x512-near-lossless.JLS (.81 near-lossless) — warm |
19.8 ms | 37.9 ms | -47.76% |
| ❌ | Simulation | HTJ2K Lossless (.201) |
141.4 ms | 189 ms | -25.16% |
| ❌ | WallTime | 8-bit passthrough, 512x512 x100 |
9 µs | 10.2 µs | -11.92% |
| ❌ | WallTime | 16-bit signed, 512x512 x100 |
11.4 µs | 13 µs | -11.67% |
| ❌ | WallTime | 16-bit unsigned, 512x512 x100 |
11.7 µs | 13.1 µs | -10.28% |
| ❌ | WallTime | 32-bit float, 512x512 x100 |
13.1 µs | 14.5 µs | -9.76% |
| ❌ | WallTime | instantiate+destroy J2KEncoder x50 |
148.8 µs | 161.7 µs | -7.99% |
| ❌ | WallTime | instantiate+destroy JPEGEncoder x50 |
139.5 µs | 151.6 µs | -7.99% |
| ⚡ | WallTime | instantiate+destroy JPEGDecoder x50 |
152.2 µs | 143.1 µs | +6.39% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ci/codspeed-self-hosted (b4e5387) with main (a88a461)2
Footnotes
-
13 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
main(9c086c9) during the generation of this report, so a88a461 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
| # (valgrind, node 22, a pinned/isolated CPU) and the cutover steps. | ||
| # IMPORTANT: after this lands, the baseline on main must be re-seeded by | ||
| # one main run on THIS runner before PR comparisons are meaningful. | ||
| runs-on: [self-hosted, codspeed-bench] |
There was a problem hiding this comment.
This should use an organization level OHIF/cornerstonejs runner if the two organizations are owned by the same enterprise, or a shared cornerstonejs runner. Having a shared runner will reduce overlaps between builds making this more stable.
Also, the settings to require approval for running this should be in place before we release, so that users on forks can't run without admin approval first.
- flock: add tools/ci/with-nashua-lock.sh (same lock path as cornerstone3D and OHIF), wrapping the bench command inside the CodSpeed action's `run:`; plus timeout-minutes: 180 to bound the added lock wait. - node: pin both codspeed jobs to 22.23.1 — a range reuses whatever 22.x is in the box's persistent tool cache, and V8 patch releases move the numbers. - yarn: provision per-job via Corepack; nashua has none and setup-node installs node + npm only. - docs: document the shared-box mutex, and correct the valgrind requirement — CodSpeed installs its own patched build, pre-installed by hand and held here.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/pr-checks.yml:
- Line 484: Update the detect-changes TOOLCHAIN_PATHS configuration to include
tools/ci/with-nashua-lock.sh (or the tools/ci/ directory), so changes to the
lock wrapper set any=true and trigger the benchmark command using it.
In `@docs/ci/self-hosted-runner.md`:
- Around line 24-27: Update the supported OS list in the self-hosted runner
documentation to remove Debian 11, leaving Ubuntu 22.04/24.04 and Debian 12 as
the documented releases. Keep the surrounding architecture and CodSpeed
compatibility guidance unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 6589db6b-8e7e-46cb-8c5e-ee4d0b38d965
📒 Files selected for processing (3)
.github/workflows/pr-checks.ymldocs/ci/self-hosted-runner.mdtools/ci/with-nashua-lock.sh
| with: | ||
| mode: simulation | ||
| run: yarn lerna run bench --parallel --stream ${{ steps.scope.outputs.flags }} | ||
| run: bash tools/ci/with-nashua-lock.sh yarn lerna run bench --parallel --stream ${{ steps.scope.outputs.flags }} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Trigger the benchmark gate when the lock wrapper changes.
detect-changes does not include tools/ci/with-nashua-lock.sh in TOOLCHAIN_PATHS. A PR that changes only this command sets any=false, so this benchmark command does not run before merge. Add this file, or tools/ci/, to the toolchain paths.
Proposed fix
TOOLCHAIN_PATHS=(
".github/workflows/"
"package.json"
"yarn.lock"
+ "tools/ci/with-nashua-lock.sh"
"vitest.workspace.mjs"
)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/pr-checks.yml at line 484, Update the detect-changes
TOOLCHAIN_PATHS configuration to include tools/ci/with-nashua-lock.sh (or the
tools/ci/ directory), so changes to the lock wrapper set any=true and trigger
the benchmark command using it.
| - **OS**: **Ubuntu 22.04 / 24.04 or Debian 11 / 12**, x86_64 or aarch64 — not | ||
| general "Linux x64" latitude. CodSpeed publishes its patched valgrind `.deb` | ||
| only for those releases, and the CodSpeed runner bails with "Unsupported system" | ||
| on anything else. nashua is Ubuntu 24.04 x86_64. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
curl -fsSL \
https://raw.githubusercontent.com/CodSpeedHQ/runner/v4.18.2/src/executor/valgrind/setup.rs |
sed -n '11,39p'Repository: cornerstonejs/codecs
Length of output: 1316
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
doc = Path("docs/ci/self-hosted-runner.md").read_text()
runner_ref = "CodSpeedHQ/runner@v4.18.2"
snippet_line = 24
# Extract the supported OS list and the runner ref from the provided context.
supported = "Ubuntu 22.04 / 24.04 or Debian 11 / 12"
supported_set = { "Ubuntu 22.04", "Ubuntu 24.04", "Debian 11", "Debian 12" }
accepted_set = { "Ubuntu 22.04", "Ubuntu 24.04", "Debian 12" }
extra = supported_set - accepted_set
missing = accepted_set - supported_set
print(f"runner_ref={runner_ref}")
print(f"currently_listed_ubuntu_24_04={'Ubuntu 24.04' in supported_set}")
print(f"currently_listed_debian_11={'Debian 11' in supported_set}")
print(f"runner_accepts_debian_11={'Debian 11' in accepted_set}")
print(f"os_list_has_extra_os_untouched_by_runner={'Debian 11' in extra}")
print(f"os_list_extra={'Debian 11'}")
print(f"runner_missing_from_os_list={'Ubuntu 24.04' in missing}")
PY
# Print the exact documented line with line numbers.
sed -n '20,29p' docs/ci/self-hosted-runner.md | nl -ba -v20Repository: cornerstonejs/codecs
Length of output: 438
Remove Debian 11 from the supported OS list.
CodSpeedHQ/runner@v4.18.2 accepts Debian 12 only; Debian 11 reaches Unsupported system. Keep the documented list aligned with the runner support.
Proposed fix
- **OS**: **Ubuntu 22.04 / 24.04 or Debian 11 / 12**, x86_64 or aarch64 — not
+ **OS**: **Ubuntu 22.04 / 24.04 or Debian 12**, x86_64 or aarch64 — not📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - **OS**: **Ubuntu 22.04 / 24.04 or Debian 11 / 12**, x86_64 or aarch64 — not | |
| general "Linux x64" latitude. CodSpeed publishes its patched valgrind `.deb` | |
| only for those releases, and the CodSpeed runner bails with "Unsupported system" | |
| on anything else. nashua is Ubuntu 24.04 x86_64. | |
| - **OS**: **Ubuntu 22.04 / 24.04 or Debian 12**, x86_64 or aarch64 — not | |
| general "Linux x64" latitude. CodSpeed publishes its patched valgrind `.deb` | |
| only for those releases, and the CodSpeed runner bails with "Unsupported system" | |
| on anything else. nashua is Ubuntu 24.04 x86_64. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/ci/self-hosted-runner.md` around lines 24 - 27, Update the supported OS
list in the self-hosted runner documentation to remove Debian 11, leaving Ubuntu
22.04/24.04 and Debian 12 as the documented releases. Keep the surrounding
architecture and CodSpeed compatibility guidance unchanged.
3 participants
Why
CodSpeed simulation mode derives its modeled CPU cache from the physical runner CPU. GitHub's shared runners randomly assign Intel Xeon 8370C vs AMD EPYC 7763, so identical source shifts instruction counts run-to-run and CodSpeed flags "Different runtime environments detected" — the noise behind the phantom regressions we've been triaging. Pinning the OS wasn't enough; the CPU model is the remaining variable.
Change
codspeed-bench(the blocking simulation gate) now runs onruns-on: [self-hosted, codspeed-bench]instead ofubuntu-24.04. One fixed box → every baseline and PR run on identical hardware → Simulation is stable.docs/ci/self-hosted-runner.md: box requirements (valgrind, node 22, isolated/pinned CPU), how to register the runner, and the cutover order.build/test/dist-size/browser-smokestay on GitHub-hosted runners. (codspeed-walltimealready targetscodspeed-macrobehind a repo variable — unchanged.)Cutover order matters:
codspeed-benchlabel FIRST (see the doc). If this merges before a runner with that label exists, thecodspeed-benchjob queues indefinitely.mainpush re-seeds the CodSpeed baseline on the fixed hardware. The first PR comparisons after cutover show a one-time environment shift (old shared-runner baseline vs new fixed head) — expected, self-resolves once the baseline is re-seeded.Rollback: revert
runs-ontoubuntu-24.04.Summary by CodeRabbit
Chores
Documentation