Is your feature request related to a problem? Please describe.
Batch-invariant MCore MoE inference currently runs token permutation and MXFP8 quantization as separate GPU operations. The ordinary permutation is needed to produce the token/local-expert inverse map used by deterministic unpermute, so the existing fused permute_and_quantize_mxfp8 path is disabled in batch-invariant mode.
This adds launch overhead and writes the full BF16 permuted activation before reading it back for quantization. It affects both squared-ReLU and SwiGLU MXFP8 inference.
Tag @NVIDIA/mcore-oncall.
Describe the solution you'd like
Extend permute_and_quantize_mxfp8 so its Triton kernel can optionally emit the batch-invariant inverse map while producing zeroed MXFP8 alignment padding. Then dispatch batch-invariant MXFP8 MoE preprocessing through that fused path, leaving activation handling unchanged.
The API should retain its existing four-value return by default and append the inverse map only when requested.
Describe alternatives you've considered
Keep permute_tokens followed by MXFP8Tensor.from_bf16. This is correct but retains an avoidable BF16 intermediate and a separate full-buffer quantization launch. A second kernel just for the inverse map would also preserve correctness, but would give back part of the launch reduction.
Additional context
RTX 5090 microbenchmarks for batch-invariant preprocessing (permute + zero padding + inverse map + MXFP8 quantization) show the fused path at 2.56x-2.98x the throughput of the separate path across representative MoE shapes:
| tokens |
hidden |
top-k |
local experts |
separate |
fused |
speedup |
| 128 |
2688 |
8 |
128 |
0.121 ms |
0.045 ms |
2.70x |
| 512 |
4096 |
8 |
64 |
0.121 ms |
0.047 ms |
2.56x |
| 2048 |
4096 |
8 |
16 |
0.260 ms |
0.087 ms |
2.98x |
Is your feature request related to a problem? Please describe.
Batch-invariant MCore MoE inference currently runs token permutation and MXFP8 quantization as separate GPU operations. The ordinary permutation is needed to produce the token/local-expert inverse map used by deterministic unpermute, so the existing fused
permute_and_quantize_mxfp8path is disabled in batch-invariant mode.This adds launch overhead and writes the full BF16 permuted activation before reading it back for quantization. It affects both squared-ReLU and SwiGLU MXFP8 inference.
Tag @NVIDIA/mcore-oncall.
Describe the solution you'd like
Extend
permute_and_quantize_mxfp8so its Triton kernel can optionally emit the batch-invariant inverse map while producing zeroed MXFP8 alignment padding. Then dispatch batch-invariant MXFP8 MoE preprocessing through that fused path, leaving activation handling unchanged.The API should retain its existing four-value return by default and append the inverse map only when requested.
Describe alternatives you've considered
Keep
permute_tokensfollowed byMXFP8Tensor.from_bf16. This is correct but retains an avoidable BF16 intermediate and a separate full-buffer quantization launch. A second kernel just for the inverse map would also preserve correctness, but would give back part of the launch reduction.Additional context
RTX 5090 microbenchmarks for batch-invariant preprocessing (
permute + zero padding + inverse map + MXFP8 quantization) show the fused path at 2.56x-2.98x the throughput of the separate path across representative MoE shapes: