Skip to content

metal: route skinny f16 matmuls through mlx gemv_wide (~2.9x on M4) - #2547

Open
czoli1976 wants to merge 2 commits into
sonos:mainfrom
czoli1976:perf/metal-skinny-gemm
Open

metal: route skinny f16 matmuls through mlx gemv_wide (~2.9x on M4)#2547
czoli1976 wants to merge 2 commits into
sonos:mainfrom
czoli1976:perf/metal-skinny-gemm

Conversation

@czoli1976

Copy link
Copy Markdown
Contributor

f16 products of a few rows against a large transposed weight — x[M,K] @ w[N,K]ᵀ
with M in 2..15, which is what a small batch or a speculative decode step
produces — currently go to a GEMM tile that pads M to 64 and leaves most of the
bandwidth unused. MlxGemm only takes the gemv path at M == 1.

This ports mlx's gemv_wide kernel (ml-explore/mlx#3888) into mlx_gemv.metal,
same vendoring idiom as the rest of that file, and routes those shapes to it,
keeping mlx's own gate: f16, K a multiple of 4, N > 1, and architecture
generation ≥ 15.

Numbers

bench_skinny_gemm (added here), K=2048, ms/dispatch, f16, x @ w.T.

M4, 8-core GPU (applegpu_g16g, macOS 26.6) — before vs after, through the
real MlxGemm dispatch:

M before after
2 0.218 0.073 2.99×
4 0.220 0.078 2.83×
8 0.218 0.077 2.83×

M == 1 and M ≥ 16 are untouched. At N=32000 the same band gains 1.34×.

The kernel also beats routing these shapes to GgmlGemm, which was the other
option: after this change Ggml is 1.10× (M=2) to 1.80× (M=8) slower than the MLX
path at N=2048.

On the architecture gate

mlx keeps gemv_wide off below generation 15 on the grounds that pre-M3 parts
are limited by load issue rate rather than bandwidth. That gate is kept here, but
for what it is worth the kernel is a clear win on an M1 Pro too (applegpu_g13s,
generation 13): 0.123 → 0.043 ms at M=2, 0.064 at M=4, 0.083 at M=8, i.e.
1.5–2.9×. Happy to widen the gate if you would rather take that; I have no M2 to
check the rest of the pre-M3 range, which is why I left mlx's rule alone.

f32 is deliberately excluded, matching mlx, which instantiates f16/bf16 only:
measured on the M4 it is 0.50–0.67× at M ≥ 4, i.e. a regression.

Validation

cargo test -p tract-metal --release on the M4 (where the gate is live): 81
passed, 1 failed — that one failure is test_mfa_attention_causal_const_is_noop,
pre-existing on main and unrelated (#2546). Same on an M1 Pro, where the gate
leaves the path inactive. New tests: gemv_wide_* cover f16 M=2..15, f32, tail
rows and the gate itself against a host reference, and
mlx_gemm_skinny_f16_matches_reference drives M=2..15 through the real dispatch.
fmt and clippy clean.

mlx (ml-explore/mlx) is MIT, Copyright (c) 2023-2025 Apple Inc.; attributed in
the source header.

🍍

czoli1976 and others added 2 commits August 1, 2026 15:31
Skinny f16 matmuls (M in 2..15, the shape a small batch or speculative
decode produces) go to a GEMM tile that pads M to 64 and leaves most of
the bandwidth unused. Port mlx's gemv_wide kernel and its launch config,
with correctness tests and a bake-off bench against the current MLX and
Ggml routes. Nothing dispatches to it yet: mlx keeps it off below
architecture generation 15 and the routing decision needs numbers from
an M3-or-later GPU.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f16 products of a few rows against a large transposed weight - what a
small batch or a speculative decode step produces - went to a GEMM tile
that pads M to 64 and leaves most of the bandwidth unused. Port mlx's
gemv_wide kernel and route M in 2..15 to it, keeping mlx's own gate:
f16, K a multiple of 4, and architecture generation 15 or later.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@czoli1976

Copy link
Copy Markdown
Contributor Author

On the architecture gate: I've posted the M1 Pro measurements upstream as ml-explore/mlx#3967 in case they want to revisit the generation-15 cutoff. This PR keeps mlx's gate either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant