Skip to content

Fuse batch-invariant MoE permutation with MXFP8 quantization #7659

Description

@CharlesXu-HQ

Is your feature request related to a problem? Please describe.

Batch-invariant MCore MoE inference currently runs token permutation and MXFP8 quantization as separate GPU operations. The ordinary permutation is needed to produce the token/local-expert inverse map used by deterministic unpermute, so the existing fused permute_and_quantize_mxfp8 path is disabled in batch-invariant mode.

This adds launch overhead and writes the full BF16 permuted activation before reading it back for quantization. It affects both squared-ReLU and SwiGLU MXFP8 inference.

Tag @NVIDIA/mcore-oncall.

Describe the solution you'd like

Extend permute_and_quantize_mxfp8 so its Triton kernel can optionally emit the batch-invariant inverse map while producing zeroed MXFP8 alignment padding. Then dispatch batch-invariant MXFP8 MoE preprocessing through that fused path, leaving activation handling unchanged.

The API should retain its existing four-value return by default and append the inverse map only when requested.

Describe alternatives you've considered

Keep permute_tokens followed by MXFP8Tensor.from_bf16. This is correct but retains an avoidable BF16 intermediate and a separate full-buffer quantization launch. A second kernel just for the inverse map would also preserve correctness, but would give back part of the launch reduction.

Additional context

RTX 5090 microbenchmarks for batch-invariant preprocessing (permute + zero padding + inverse map + MXFP8 quantization) show the fused path at 2.56x-2.98x the throughput of the separate path across representative MoE shapes:

tokens hidden top-k local experts separate fused speedup
128 2688 8 128 0.121 ms 0.045 ms 2.70x
512 4096 8 64 0.121 ms 0.047 ms 2.56x
2048 4096 8 16 0.260 ms 0.087 ms 2.98x

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions