[Zbig9000]

Overview

Follow-up to #410 in three parts.

Jetson Orin Nano (qvac-dev-linux-arm64) lane, and the Strix Halo moves to qvac-dev-strix-1

• lib.sh builds only the cmake targets of the selected families (plus whisper-cli), so a host whose SKIP_FAMILIES removes the MOSS families never compiles moss-cli.
• The cmake generator falls back to Unix Makefiles where ninja is missing (qvac-dev-strix-1 has none).
• EVICT_PAGE_CACHE=1 drops the model files from the page cache before every family, through a new evict subcommand of prepare-derived.py (posix_fadvise DONTNEED); the Jetson's CUDA allocations fail while the page cache is full.
• CMAKE_PREFIX_PATH in config.env is passed to the ggml and engine configures and is part of the build signature: the Jetson has no Vulkan SDK and no root, so SPIRV-Headers and Vulkan-Headers live under the bench home.
• qvac-dev-strix-1 replaces qvac-dev-strix-0 (the same hardware) as the one Strix Halo of the weekly set, under the plain name "Strix Halo"; the campaign's Strix Halo rows go with it.
• config.example.env and the skill's machine table document the Jetson and the Strix Halo.

Speech Bench hub with task pages and per-machine dashboards, in the layout of the approved Speech Bench report

• A hub page lists, per task (TTS and voice cloning, ASR, music), every machine with its models, backends and other engines, linking to that machine's task page, then the quality flags, the engines and how the bench runs. build-landing.py renders it from hub.json.
• The task pages follow the approved report's per-machine pages: one section per model with a horizontal speed chart per backend and input in engine colors, a details panel per chart (exact values, precision per row, the competitor ÷ QVAC time ratio with its qualifier, notes and sources) and what was not measured there, and why. task_views.py renders them from each machine's runs.json and coverage.json, with dashboard/task.css; build-landing.py embeds all of them in the hub as views reached at the hub's link plus #<machine>-<task> (for example #dgx-spark-tts), so they need no artifact of their own and are visible to everyone who can open the hub.
• One Dashboard artifact per machine runs the same page (dashboard/index.html, app.js, app.css, adapted from the approved report's dashboard script): an overview (status, a summary of the week, quality flags, which models ran on which backend, gaps, engines, rules), a tab per task with one section per model (a speed chart per backend and input; weekly charts hold QVAC's variants, campaign charts hold QVAC and transcribe.cpp, audio.cpp, LocalAI parakeet.cpp or acestep.cpp in engine colors, with details, time ratios and sources), a week-over-week line per backend, and a runs and coverage tab.
• build-dashboard-data.py writes per host the six files a dashboard reads (runs, history, coverage, engines, machine, commentary) and hub.json. Campaign rows keep their own build and date and never share a chart with weekly rows; a competitor without a ratio (OOM in the campaign) is a failed row, and a competitor row records its precision as not recorded instead of QVAC's quantization. Two campaign notes in comparisons.json that were cut at 400 characters are complete again. engines.json lists the engines, dashboards.json the links.
• The skill's publish step and the Monday refresh upload each machine's changed files to its dashboard and republish the hub.

fetch-registry.js discards a stale .part left by a stopped download, and downloads beside it when that file belongs to another user (the shared store is sticky).

Additional information

• test-desktop-weekly.sh covers the selected-family target list, the generator fallback, the evict subcommand, the cmake prefixes in the build signature, the partial-download handling, the data builder, the hub and the task pages (27 checks pass; it also syntax-checks the dashboard script).
• Rehearsal runs with these tools are on the dashboards: DGX Spark 40/40, RTX 5090 60/60, Strix Halo 40/40, Mac mini 34/34, Jetson 34/39. The five Jetson Vulkan cells finish their measurement and then segfault at process exit on the Tegra driver (exit 139); the fix is QVAC-27853 fix(vulkan): destroy devices and instance before driver unload qvac-ext-ggml#148, after which the engine needs a ggml-speech registry cut.

🤖 Generated with Claude Code

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude Code wrote the change, the tests and this description.

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[GustavoA1604]

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 170 (if row.get("task") == "TTS cloning":)

This labels cosyvoice3-clone as including model load and reference encoding, although its native benchmark excludes both. The dashboard and task pages therefore misrepresent the measured latency. Apply TIMING_CLONING only to time-wrapped cloning, and add a native-cloning test.

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 636 (by_host = group_by_host(summaries))

Retaining historical summaries as documented leaves both qvac-dev-strix-0 and qvac-dev-strix-1 in the hub, producing an extra obsolete machine and three task pages despite the intended single Strix Halo. Filter the retired host from current views or explicitly consolidate its history. Add a test containing both host names.

[Zbig9000]

Both review findings are addressed in a4e6d98:

• timing_of applies the "fresh process incl. load and reference encoding" scope only to time-wrapped cloning families; the native CosyVoice3 bench keeps "inference only", with a test for both kinds (test_only_time_wrapped_cloning_includes_load_and_reference_encoding).
• load_summaries skips RETIRED_HOSTS (qvac-dev-strix-0), so the hub shows one Strix Halo even when the old summaries are still stored; test test_retired_hosts_are_left_out_of_the_summaries holds both host names. README and the skill say so.

🤖 Addressed by Claude Code

[Zbig9000]

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 170 (if row.get("task") == "TTS cloning":)

This labels cosyvoice3-clone as including model load and reference encoding, although its native benchmark excludes both. The dashboard and task pages therefore misrepresent the measured latency. Apply TIMING_CLONING only to time-wrapped cloning, and add a native-cloning test.

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 636 (by_host = group_by_host(summaries))

Retaining historical summaries as documented leaves both qvac-dev-strix-0 and qvac-dev-strix-1 in the hub, producing an extra obsolete machine and three task pages despite the intended single Strix Halo. Filter the retired host from current views or explicitly consolidate its history. Add a test containing both host names.

Both findings are addressed in a4e6d98 (branch head is now 46f10f0, which keeps the fixes).

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 170 (if row.get(“task”) == “TTS cloning”:)
This labels cosyvoice3-clone as including model load and reference encoding, although its native benchmark excludes both. The dashboard and task pages therefore misrepresent the measured latency. Apply TIMING_CLONING only to time-wrapped cloning, and add a native-cloning test.

Fixed: timing_of applies the “fresh process incl. load and reference encoding” scope only when the cloning family is time-wrapped; native cloning (cosyvoice3-clone) keeps the “inference only” scope of its bench. Test test_only_time_wrapped_cloning_includes_load_and_reference_encoding covers both families. No published page needed a rebuild yet: the Mac and Linux hosts have no CosyVoice weekly rows until Sunday.

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 636 (by_host = group_by_host(summaries))
Retaining historical summaries as documented leaves both qvac-dev-strix-0 and qvac-dev-strix-1 in the hub, producing an extra obsolete machine and three task pages despite the intended single Strix Halo. Filter the retired host from current views or explicitly consolidate its history. Add a test containing both host names.

Fixed: RETIRED_HOSTS = ("qvac-dev-strix-0",) and current_summary_paths drop the retired host’s summaries at load time, so the hub shows one Strix Halo (strix-1) and three task pages for it; the strix-0 files stay in the history folder for the record. Test test_retired_hosts_are_left_out_of_the_summaries loads both host names and asserts only strix-1 survives. README and the skill document the constant.

[Zbig9000]

/review

[GustavoA1604]

[medium] scripts/benchmarks/desktop-weekly/lib.sh — line 483 (if family_runs_on_backend "$1" "$2"; then)

Metal-only families are skipped on CPU, but summarize.collect_rows() still creates every backend × family pair. On a Mac running cpu,metal, those skipped cells become not-run failures and mark an otherwise successful run partial. Apply the same backend filter when collecting summary rows, and add a mixed-backend regression test.

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 267 ("group": f"{host}|{row['backend']}|{model}|{task}|{scenario}|weekly")

Include timing_class in this grouping key. On Mac Studio Metal, whisper-small includes startup and model loading, while mlx-whisper times inference after loading. Their matching keys put them into one chart and produce a misleading comparison ratio. The ABCBA pair has the same issue. Add tests confirming these timing scopes stay separate.

[Zbig9000]

[medium] scripts/benchmarks/desktop-weekly/lib.sh — line 483 (if family_runs_on_backend "$1" "$2"; then)

Metal-only families are skipped on CPU, but summarize.collect_rows() still creates every backend × family pair. On a Mac running cpu,metal, those skipped cells become not-run failures and mark an otherwise successful run partial. Apply the same backend filter when collecting summary rows, and add a mixed-backend regression test.

[medium] scripts/benchmarks/desktop-weekly/build-dashboard-data.py — line 267 ("group": f"{host}|{row['backend']}|{model}|{task}|{scenario}|weekly")

Include timing_class in this grouping key. On Mac Studio Metal, whisper-small includes startup and model loading, while mlx-whisper times inference after loading. Their matching keys put them into one chart and produce a misleading comparison ratio. The ABCBA pair has the same issue. Add tests confirming these timing scopes stay separate.

Both addressed in d6b1dd7.

lib.sh:483 / summarize rows. collect_rows() now applies the same backend filter as the runner (family_runs_on_backend), so a Metal-only family is no longer an expected cell on CPU. Regression test test_a_metal_only_family_is_not_expected_on_the_other_backends runs a cpu,metal matrix and checks the counts come out 3 expected, 2 ok, 1 failed with only the real CPU gap in the problem list.

build-dashboard-data.py:267 / grouping key. The weekly group key now carries timing_class, so whisper-small (fresh process, whisper-cli) and mlx-whisper (in-process, inference only) land in separate groups and the dashboard draws no ratio between them; same for the ABCBA pair. Test test_rows_of_different_timing_scopes_never_share_a_chart covers both pairs. The README row for mlx-whisper already stated they stay in their own charts; now the code does it too.

[Zbig9000]

/review