MUSA: enable CAR-RMSNorm by default under contract

[Elon-yf]

Default-enable MUSA CAR-RMSNorm with per-model batch routing

Summary

The original PR #140 added CAR-RMSNorm support but did not enable it by
default. This update enables it by default for supported routes. A normal
vllm serve command needs no CAR-specific override.

To disable only CAR-RMSNorm while keeping compilation and CUDAGraph enabled,
pass -cc.pass_config.fuse_allreduce_rms=False. The nested field syntax merges with the
existing -cc.mode and -cc.cudagraph_mode arguments.

Scenario	Configuration	Result
Supported model/TP/precision/range	pass_config.fuse_allreduce_rms unspecified (auto/default)	Eligible rows use CAR-RMSNorm; native rows fall back to native
Any configuration	-cc.pass_config.fuse_allreduce_rms=False	CAR-RMSNorm disabled; use native
Unsupported model/TP/precision/range	pass_config.fuse_allreduce_rms unspecified (auto/default)	Use native

Problem

CAR-RMSNorm support depends on the model family, tensor-parallel size, hidden
size, explicit BF16 or FP8 format, active rows, and compile range. A broad global
switch cannot safely decide every row. Unsupported configurations and compile
ranges that cross a native boundary must remain on the native path.

Implementation

• Enable CAR-RMSNorm by default at the platform layer.
• Use one routing predicate for model properties, active rows, and the compile
  range.
• Keep unsupported configurations and ranges crossing a native boundary on
  the native path.
• Use the standard vLLM compilation config as the sole user-visible OFF
  control: -cc.pass_config.fuse_allreduce_rms=False.

Supported routing matrix

TP2 contract

Model	Hidden	Precision	Native rows	Compile endpoints	Row routing
Qwen3.5/3.6-27B	5120	BF16	[16]	[3,4,15,16,63]	Eligible [[1,3],[4,4],[5,15],[17,63],[64,8192]]; Native [[16,16]]
Qwen3.5/3.6-27B	5120	FP8 weights, BF16 activations	[4,16]	[3,4,15,16,63]	Eligible [[1,3],[5,15],[17,63],[64,8192]]; Native [[4,4],[16,16]]

TP4 contract

Model	Hidden	Precision	Native rows	Compile endpoints	Row routing
Qwen3.5/3.6-35B-A3B	2048	BF16	[16,64]	[15,16,63,64]	Eligible [[1,15]]; Native [[16,16],[17,63],[64,64],[65,8192]]
Qwen3.5/3.6-35B-A3B	2048	FP8 weights, BF16 activations	[64]	[15,16,63,64]	Eligible [[1,15],[16,16],[17,63],[65,8192]]; Native [[64,64]]

Other / unsupported

Model	Hidden	Precision	Native rows	Compile endpoints	Row routing
Unsupported	-	-	[]	[]	Native [[1,8192]]

Compile endpoints split the graph into stable intervals. If a requested range
crosses a native row, the provider uses the native path for that range; the
endpoint partition keeps adjacent fused intervals separate.

Qwen3.5 serve and benchmark

Run the benchmark once per reported BS (1, 4, 16, or 64); set
--num-prompts and --max-concurrency to that same BS, with
--request-rate inf. The command below shows the 4096/1024 form; use the same
BS mapping for the 2500/1500 rows by changing the two random-length arguments.

The serve command below is the default-ON arm:

vllm serve <MODEL_PATH> \
  --tensor-parallel-size 4 \
  --host 127.0.0.1 --port 39720 \
  --served-model-name <NAME> \
  --max-model-len 5184 --max-num-seqs 64 \
  --gpu-memory-utilization 0.8 \
  --no-enable-chunked-prefill --no-enable-prefix-caching \
  -ac.backend FLASH_ATTN \
  -cc.cudagraph_mode=FULL_DECODE_ONLY \
  --trust-remote-code

For the matched OFF arm, append this argument to the same serve command:

-cc.pass_config.fuse_allreduce_rms=False

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:39720 \
  --model <NAME> --tokenizer <MODEL_PATH> \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 4096 --random-output-len 1024 \
  --random-range-ratio 0 \
  --num-prompts <BS> --request-rate inf --max-concurrency <BS> \
  --endpoint /v1/chat/completions \
  --temperature 0 --seed 0 --ignore-eos \
  --save-detailed --save-result \
  --result-dir <RESULT_DIR> --result-filename result.json \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99

Validation

The existing FULL_DECODE_ONLY report is copied without re-aggregation:

• 8 model/shape combinations.
• 32 batch cells, with BS 1, 4, 16, and 64 for every model/shape.
• All four report metrics: output throughput, TTFT, TPOT, and E2E.
• 7 cells identify pooled n=10 repeated data;
  25 cells retain the report's unavailable sample count.
• The report observed zero fused operator rewrites, so the ON/OFF differences
  are not attributed to fused execution.

FULL_DECODE_ONLY measures the decode graph path; mixed execution is outside
this report.

Complete results

Qwen3.5-35B-A3B-BF16 · 4096/1024 · TP4 · BF16

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	107.659	107.677	+0.016%	138.889	132.939	-4.284%	9.162	9.166	+0.047%	9511.144	9509.553	-0.017%
4	235.084	235.799	+0.304%	314.709	305.605	-2.893%	16.701	16.658	-0.257%	17400.034	17347.035	-0.305%
16	772.098	773.469	+0.177%	983.281	979.171	-0.418%	19.658	19.626	-0.165%	21093.438	21056.213	-0.176%
64	2071.835	2081.693	+0.476%	3955.759	3907.095	-1.230%	26.443	26.347	-0.363%	31006.594	30859.936	-0.473%

Qwen3.5-35B-A3B-BF16 · 2500/1500 · TP4 · BF16

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	108.351	108.525	+0.161%	117.775	114.069	-3.147%	9.157	9.144	-0.134%	13843.379	13821.272	-0.160%
4	241.122	242.627	+0.624%	249.507	242.773	-2.699%	16.428	16.329	-0.603%	24874.633	24720.359	-0.620%
16	822.195	823.547	+0.164%	594.325	590.756	-0.601%	19.050	19.020	-0.157%	29149.608	29101.621	-0.165%
64	2545.598	2550.594	+0.196%	2064.172	2028.081	-1.748%	23.644	23.619	-0.107%	37506.679	37432.655	-0.197%

Qwen3.5-27B-BF16 · 4096/1024 · TP2 · BF16

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	35.389	34.993	-1.119%	523.734	525.983	+0.429%	27.773	28.090	+1.143%	28935.471	29262.525	+1.130%
4	74.167	74.187	+0.027%	1401.545	1398.062	-0.249%	52.543	52.532	-0.021%	55153.487	55138.267	-0.028%
16	272.059	272.071	+0.004%	4757.672	4766.073	+0.177%	53.858	53.848	-0.019%	59854.785	59852.130	-0.004%
64	748.831	750.589	+0.235%	18933.805	18920.050	-0.073%	65.529	65.348	-0.276%	85969.845	85770.715	-0.232%

Qwen3.5-27B-BF16 · 2500/1500 · TP2 · BF16

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	35.865	35.420	-1.242%	330.049	331.331	+0.388%	27.680	28.030	+1.265%	41822.397	42348.384	+1.258%
4	76.591	76.751	+0.209%	1127.543	1119.179	-0.742%	51.490	51.387	-0.200%	78311.387	78148.967	-0.207%
16	301.952	301.991	+0.013%	3218.372	3210.642	-0.240%	50.800	50.798	-0.004%	79367.236	79357.052	-0.013%
64	1057.008	1058.454	+0.137%	11583.176	11512.706	-0.608%	52.527	52.491	-0.069%	90320.781	90197.366	-0.137%

Qwen3.5-27B-FP8 · 4096/1024 · TP2 · FP8

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	48.961	50.371	+2.880%	459.468	458.781	-0.150%	19.994	19.423	-2.856%	20913.597	20328.585	-2.797%
4	90.605	90.770	+0.182%	1163.026	1153.730	-0.799%	42.997	42.927	-0.163%	45148.964	45068.124	-0.179%
16	333.660	333.264	-0.119%	3936.323	3939.922	+0.091%	43.858	43.912	+0.123%	48803.123	48861.877	+0.120%
64	915.251	914.882	-0.040%	15847.792	15853.100	+0.033%	53.276	53.301	+0.047%	70349.157	70379.614	+0.043%

Qwen3.5-27B-FP8 · 2500/1500 · TP2 · FP8

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	49.642	51.266	+3.271%	285.110	285.212	+0.036%	19.967	19.329	-3.195%	30215.264	29258.683	-3.166%
4	94.286	94.045	-0.256%	882.141	884.404	+0.257%	41.850	41.957	+0.256%	63614.751	63777.853	+0.256%
16	371.998	372.202	+0.055%	2505.177	2505.294	+0.005%	41.305	41.282	-0.056%	64421.031	64386.840	-0.053%
64	1329.534	1331.112	+0.119%	9014.335	9001.902	-0.138%	41.883	41.835	-0.115%	71796.492	71712.146	-0.117%

Qwen3.5-35B-A3B-FP8 · 4096/1024 · TP4 · FP8

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	85.902	86.839	+1.091%	132.523	135.304	+2.099%	11.522	11.394	-1.111%	11919.723	11791.255	-1.078%
4	176.691	176.456	-0.133%	349.138	350.302	+0.333%	22.288	22.318	+0.135%	23150.218	23181.260	+0.134%
16	674.272	672.817	-0.216%	1032.731	1035.399	+0.258%	22.586	22.635	+0.217%	24138.303	24191.030	+0.218%
64	2209.678	2214.483	+0.217%	4090.591	4036.377	-1.325%	24.331	24.316	-0.062%	28981.454	28912.119	-0.239%

Qwen3.5-35B-A3B-FP8 · 2500/1500 · TP4 · FP8

Batch	Output tok/s OFF	Output tok/s ON	Δ	TTFT ms OFF	TTFT ms ON	Δ	TPOT ms/token OFF	TPOT ms/token ON	Δ	E2E ms OFF	E2E ms ON	Δ
1	86.399	87.360	+1.112%	126.968	125.794	-0.925%	11.497	11.370	-1.105%	17360.884	17169.900	-1.100%
4	180.829	180.439	-0.216%	271.039	270.480	-0.206%	21.946	21.994	+0.219%	33168.815	33240.196	+0.215%
16	721.262	718.422	-0.394%	621.708	622.701	+0.160%	21.750	21.836	+0.395%	33224.416	33355.148	+0.393%
64	2790.471	2791.989	+0.054%	2174.998	2150.839	-1.111%	21.350	21.354	+0.019%	34179.295	34159.903	-0.057%

Standalone Qwen3.6-27B case

This case is reported separately from the Qwen3.5 matrix.

Qwen3.6-27B serve and benchmark

The serve command below is the default-ON arm:

vllm serve <MODEL_PATH> \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.80 \
  --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \
    17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 64 \
  --no-enable-chunked-prefill --no-enable-prefix-caching --no-enable-dbo \
  --async-scheduling \
  -ac.backend FLASH_ATTN -cc.mode=3 \
  -cc.cudagraph_mode=FULL_DECODE_ONLY \
  --trust-remote-code

For the matched OFF arm, append this argument to the same serve command:

-cc.pass_config.fuse_allreduce_rms=False

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:39720 \
  --model <NAME> --tokenizer <MODEL_PATH> \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 2500 --random-output-len 1500 \
  --random-range-ratio 0 \
  --num-prompts 20 --request-rate 4 --max-concurrency 4 \
  --endpoint /v1/chat/completions \
  --temperature 0 --seed 0 --ignore-eos \
  --save-detailed --save-result \
  --result-dir <RESULT_DIR> --result-filename result.json \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99

Metric	SGLang-MUSA	vLLM OFF	OFF gain	vLLM ON	ON gain
Input throughput (tok/s)	213.110	219.570	+3.03%	218.354	+2.46%
Output throughput (tok/s)	125.640	131.742	+4.86%	131.012	+4.28%
Mean TTFT (ms)	1114.950	904.651	+18.86%	893.138	+19.89%
Median TTFT (ms)	1293.040	1082.950	+16.25%	1069.033	+17.32%
P90 TTFT (ms)	1345.530	1086.036	+19.29%	1071.941	+20.33%
P99 TTFT (ms)	1347.960	1090.544	+19.10%	1073.799	+20.34%
Mean TPOT (ms/token)	31.110	29.754	+4.36%	29.931	+3.79%
Median TPOT (ms/token)	30.980	29.658	+4.27%	29.834	+3.70%
P90 TPOT (ms/token)	31.630	30.094	+4.86%	30.266	+4.31%
P99 TPOT (ms/token)	32.310	30.102	+6.83%	30.273	+6.31%

Gain is relative to SGLang. Positive values are better; throughput uses
higher-is-better, while TTFT and TPOT use lower-is-better.

Limitations

• TP8 is outside the current CAR-RMSNorm contract and remains on native CAR +
  RMSNorm; ordinary custom all-reduce transport is unchanged.
• Sample coverage differs by cell and is shown directly in the result table.
• Cells whose sample count was not reported must not be interpreted as pooled
  n=10 measurements.
• Zero observed rewrites means these results compare configuration behavior;
  they do not establish fused-operator performance benefit.