gemv_wide_config returns nullopt below architecture generation 15, on the
grounds that pre-M3 parts are limited by load issue rate rather than bandwidth
and do not profit from the amortized stream. On an M1 Pro (applegpu_g13s,
generation 13) I measure the opposite for f16: the kernel is a clear win across
its whole M range.
Measurements
x[M,K] @ w[N,K]ᵀ, f16, K=2048, ms per dispatch, best of 5×20 with the inputs
and output preallocated on device. Apple M1 Pro, 14-core GPU, macOS 26.5.2.
N=2048:
| M |
steel GEMM |
gemv_wide |
|
| 2 |
0.1246 |
0.0423 |
2.94× |
| 4 |
0.1264 |
0.0667 |
1.90× |
| 8 |
0.1225 |
0.0825 |
1.49× |
N=32000:
| M |
steel GEMM |
gemv_wide |
|
| 2 |
1.291 |
0.768 |
1.68× |
| 4 |
1.294 |
0.917 |
1.41× |
| 8 |
1.295 |
1.181 |
1.10× |
For reference, the same bench on an M4 (applegpu_g16g, generation 16, where the
gate is already open) gives 2.78×/2.66×/2.73× at N=2048 — so the M1 Pro gain at
M=2 is in the same range as the M4's, and only tapers with M.
f32 behaves as your instantiation choice implies and is not interesting here
(0.5–0.7× at M ≥ 4 on the M4).
Caveat on methodology
These numbers do not come from mlx. I flattened the kernel and
gemv_wide_config into another Metal runtime (tract, sonos/tract#2547) that
vendors your steel GEMM and gemv as source, and benched the two dispatch paths
against each other there. So this isolates the kernel and its launch config, but
not mlx's own dispatch overheads, layout handling or the shapes your routing
sees in practice — the gate may well be right inside mlx for reasons this does
not capture.
Also, one device is not the pre-M3 range: I have no M1 base, M1 Max/Ultra or M2
to check, and 'g13s' is a Pro part. It is plausible the gate is right for the
smaller pre-M3 GPUs and conservative for the larger ones.
Posting it as a data point in case the generation-15 cutoff is worth revisiting
or widening to something like "generation ≥ 15, or ≥ 13 with enough cores". Happy
to run any specific shapes on the M1 Pro or M4 if that would help.
gemv_wide_configreturnsnulloptbelow architecture generation 15, on thegrounds that pre-M3 parts are limited by load issue rate rather than bandwidth
and do not profit from the amortized stream. On an M1 Pro (
applegpu_g13s,generation 13) I measure the opposite for f16: the kernel is a clear win across
its whole M range.
Measurements
x[M,K] @ w[N,K]ᵀ, f16, K=2048, ms per dispatch, best of 5×20 with the inputsand output preallocated on device. Apple M1 Pro, 14-core GPU, macOS 26.5.2.
N=2048:
N=32000:
For reference, the same bench on an M4 (
applegpu_g16g, generation 16, where thegate is already open) gives 2.78×/2.66×/2.73× at N=2048 — so the M1 Pro gain at
M=2 is in the same range as the M4's, and only tapers with M.
f32 behaves as your instantiation choice implies and is not interesting here
(0.5–0.7× at M ≥ 4 on the M4).
Caveat on methodology
These numbers do not come from mlx. I flattened the kernel and
gemv_wide_configinto another Metal runtime (tract, sonos/tract#2547) thatvendors your steel GEMM and gemv as source, and benched the two dispatch paths
against each other there. So this isolates the kernel and its launch config, but
not mlx's own dispatch overheads, layout handling or the shapes your routing
sees in practice — the gate may well be right inside mlx for reasons this does
not capture.
Also, one device is not the pre-M3 range: I have no M1 base, M1 Max/Ultra or M2
to check, and 'g13s' is a Pro part. It is plausible the gate is right for the
smaller pre-M3 GPUs and conservative for the larger ones.
Posting it as a data point in case the generation-15 cutoff is worth revisiting
or widening to something like "generation ≥ 15, or ≥ 13 with enough cores". Happy
to run any specific shapes on the M1 Pro or M4 if that would help.