metal: route skinny f16 matmuls through mlx gemv_wide (~2.9x on M4) - #2547
Open
czoli1976 wants to merge 2 commits into
Open
metal: route skinny f16 matmuls through mlx gemv_wide (~2.9x on M4)#2547czoli1976 wants to merge 2 commits into
czoli1976 wants to merge 2 commits into
Conversation
Skinny f16 matmuls (M in 2..15, the shape a small batch or speculative decode produces) go to a GEMM tile that pads M to 64 and leaves most of the bandwidth unused. Port mlx's gemv_wide kernel and its launch config, with correctness tests and a bake-off bench against the current MLX and Ggml routes. Nothing dispatches to it yet: mlx keeps it off below architecture generation 15 and the routing decision needs numbers from an M3-or-later GPU. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f16 products of a few rows against a large transposed weight - what a small batch or a speculative decode step produces - went to a GEMM tile that pads M to 64 and leaves most of the bandwidth unused. Port mlx's gemv_wide kernel and route M in 2..15 to it, keeping mlx's own gate: f16, K a multiple of 4, and architecture generation 15 or later. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
On the architecture gate: I've posted the M1 Pro measurements upstream as ml-explore/mlx#3967 in case they want to revisit the generation-15 cutoff. This PR keeps mlx's gate either way. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
f16 products of a few rows against a large transposed weight —
x[M,K] @ w[N,K]ᵀwith M in 2..15, which is what a small batch or a speculative decode step
produces — currently go to a GEMM tile that pads M to 64 and leaves most of the
bandwidth unused.
MlxGemmonly takes the gemv path at M == 1.This ports mlx's
gemv_widekernel (ml-explore/mlx#3888) intomlx_gemv.metal,same vendoring idiom as the rest of that file, and routes those shapes to it,
keeping mlx's own gate: f16, K a multiple of 4, N > 1, and architecture
generation ≥ 15.
Numbers
bench_skinny_gemm(added here), K=2048, ms/dispatch, f16,x @ w.T.M4, 8-core GPU (
applegpu_g16g, macOS 26.6) — before vs after, through thereal
MlxGemmdispatch:M == 1 and M ≥ 16 are untouched. At N=32000 the same band gains 1.34×.
The kernel also beats routing these shapes to
GgmlGemm, which was the otheroption: after this change Ggml is 1.10× (M=2) to 1.80× (M=8) slower than the MLX
path at N=2048.
On the architecture gate
mlx keeps
gemv_wideoff below generation 15 on the grounds that pre-M3 partsare limited by load issue rate rather than bandwidth. That gate is kept here, but
for what it is worth the kernel is a clear win on an M1 Pro too (
applegpu_g13s,generation 13): 0.123 → 0.043 ms at M=2, 0.064 at M=4, 0.083 at M=8, i.e.
1.5–2.9×. Happy to widen the gate if you would rather take that; I have no M2 to
check the rest of the pre-M3 range, which is why I left mlx's rule alone.
f32 is deliberately excluded, matching mlx, which instantiates f16/bf16 only:
measured on the M4 it is 0.50–0.67× at M ≥ 4, i.e. a regression.
Validation
cargo test -p tract-metal --releaseon the M4 (where the gate is live): 81passed, 1 failed — that one failure is
test_mfa_attention_causal_const_is_noop,pre-existing on main and unrelated (#2546). Same on an M1 Pro, where the gate
leaves the path inactive. New tests:
gemv_wide_*cover f16 M=2..15, f32, tailrows and the gate itself against a host reference, and
mlx_gemm_skinny_f16_matches_referencedrives M=2..15 through the real dispatch.fmt and clippy clean.
mlx (ml-explore/mlx) is MIT, Copyright (c) 2023-2025 Apple Inc.; attributed in
the source header.
🍍