Summary
Decode workers fail during initialization with AssertionError: NVFP4 dispatch scale tensor has stride(-1)=14680064 when SGLANG_MOE_NVFP4_DISPATCH=1 is set in the decode environment.
Error
From inkwell-copper-cn06_decode_w0.out:2007 (and all 8 decode workers):
AssertionError: NVFP4 dispatch scale tensor has stride(-1)=14680064, dtype=torch.float8_e4m3fn; expected 1 for .view(float8_e4m3fn). Try SGLANG_MOE_NVFP4_DISPATCH=0 or check DeepEP version.
The error occurs during model loading initialization in DeepEP's MoE dispatch path.
Config
Recipe: gb200-fp4/1k1k/mid-curve.yaml
Model: DeepSeek-R1 FP4 (modelopt_fp4 quantization)
Topology: 4 prefill nodes + 8 decode nodes, tp=32, ep=32
Relevant flags on decode workers:
moe-a2a-backend: deepepmoe-runner-backend: flashinfer_cutedslSGLANG_MOE_NVFP4_DISPATCH: "1" (set in decode_environment)
Job
Job ID: 24576077428 (sweep_4666)
Decode workers on: inkwell-copper-cn06, cn11, cn12, cn13, cn15, cn16, cn17, cn18
Suggested Fix
Remove or set SGLANG_MOE_NVFP4_DISPATCH to "0" in the decode_environment section of the recipe:
decode_environment:
SGLANG_MOE_NVFP4_DISPATCH: "0" # was \"1\"
Alternatively, if NVFP4 dispatch is required for this topology, the recipe must ensure the DeepEP version supports the tensor layout produced by modelopt_fp4 quantization with ep-size=32 and ep-num-redundant-experts=32.
Summary
Decode workers fail during initialization with
AssertionError: NVFP4 dispatch scale tensor has stride(-1)=14680064whenSGLANG_MOE_NVFP4_DISPATCH=1is set in the decode environment.Error
From
inkwell-copper-cn06_decode_w0.out:2007(and all 8 decode workers):The error occurs during model loading initialization in DeepEP's MoE dispatch path.
Config
Recipe:
gb200-fp4/1k1k/mid-curve.yamlModel: DeepSeek-R1 FP4 (modelopt_fp4 quantization)
Topology: 4 prefill nodes + 8 decode nodes, tp=32, ep=32
Relevant flags on decode workers:
moe-a2a-backend: deepepmoe-runner-backend: flashinfer_cutedslSGLANG_MOE_NVFP4_DISPATCH: "1"(set in decode_environment)Job
Job ID: 24576077428 (sweep_4666)
Decode workers on: inkwell-copper-cn06, cn11, cn12, cn13, cn15, cn16, cn17, cn18
Suggested Fix
Remove or set
SGLANG_MOE_NVFP4_DISPATCHto"0"in the decode_environment section of the recipe:Alternatively, if NVFP4 dispatch is required for this topology, the recipe must ensure the DeepEP version supports the tensor layout produced by modelopt_fp4 quantization with ep-size=32 and ep-num-redundant-experts=32.