[Model] Enable Qwen3.8-Flash-Next on MUSA

[Joey-gvwal]

Summary

This PR enables BF16 Qwen3.8-Flash-Next serving on vllm-musa
v0.28.0-dev.

The implementation backports the upstream Qwen4Exp model family and adapts its
hybrid Qwen Sparse Attention (QSA), Gated DeltaNet recurrent state, packed KV
cache, FlashAttention, and MoE paths to MUSA.

Final validation used a clean source build and successfully completed TP8 +
expert-parallel startup, all 131 checkpoint shards, CUDA Graph capture,
/health, /v1/models, text completion, and chat completion.

Code changes

1. Backport the Qwen4Exp model family

• Add Qwen4ExpForConditionalGeneration configuration, registration, model,
  multimodal processor, MoE, QSA, and Gated DeltaNet implementations.
• Add Qwen4Exp speculative-decode and model-configuration plumbing.
• Add the runtime interfaces and loader compatibility required by the new
  architecture on the vLLM 0.28 base.

2. Adapt QSA to MUSA

• Route QSA through the existing MUSA FlashAttention v3 backend.
• Bind MUSA's [K, V] cache layout directly instead of assuming NVIDIA's
  concatenated last-dimension layout.
• Replace the unavailable Triton tl.gather request lookup with a vectorized
  MUSA-compatible cumsum + searchsorted work-map implementation.
• Explicitly unroll temporal/height/width MRoPE loads because the MUSA Triton
  frontend does not support the upstream tuple constexpr indexing form.
• Use MUSA torch.topk for sparse-block selection instead of the CUDA-only
  _C::persistent_topk extension. QSA score calculation, cache handling,
  MRoPE, and sparse attention remain on Triton/MUSA kernels.

3. Fix packed hybrid KV and recurrent-state layouts

• Add the KV-cache layout metadata required by QSA and hybrid recurrent
  attention.
• Add packed-cache views that honor each layer's byte offset and physical block
  stride.
• Fix the packed Mamba/Gated DeltaNet state path so every layer receives its
  own per-block state view.

The recurrent-state fix is required for semantic correctness. Before the fix,
prefill produced the correct first token, but all recurrent layers viewed state
from byte offset zero and overwrote one another; decode tokens were corrupted.
Using (layer_offset, block_stride) to construct per-layer strided views
restores coherent decode.

4. Add compatibility and tests

• Adapt Qwen3-Next inherited fused-path flags and projection return shapes.
• Add Mamba short-convolution/speculative-state and state-copy compatibility.
• Add a focused multi-request QSA work-metadata test.
• Update the generated vLLM-MUSA patch series from 133 to 134 patches.

Serving command

cd /workspace/vllm-musa

export PYTHONPATH=/workspace/vllm-musa:/workspace/vllm-musa/third_party/vllm
export MUSA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export PYTHONUNBUFFERED=1
unset TORCHDYNAMO_DISABLE VLLM_ATTENTION_BACKEND

vllm serve /home/dist/models/Qwen/Qwen3.8-Flash-Next \
  --host 0.0.0.0 \
  --port 28083 \
  --served-model-name Qwen3.8-Flash-Next \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --trust-remote-code \
  --max-model-len 4096 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --no-enable-prefix-caching \
  --attention-backend FLASH_ATTN \
  --mamba-ssm-cache-dtype float32 \
  --gpu-memory-utilization 0.90 \
  -cc.mode=NONE \
  -cc.cudagraph_mode=FULL_DECODE_ONLY \
  --cudagraph-capture-sizes 1

Test results

Test	Result	Evidence
Patch-series replay	PASS	134 applied, 0 already-applied, 0 conflict
Source build/install	PASS	vLLM 0.28.0 and vLLM-MUSA 0.1.28 built and installed
QSA work metadata, CPU	PASS	1 passed in 0.21s
QSA work metadata, MUSA, 3 requests	PASS	Correct request/work mapping returned on device
Model architecture resolution	PASS	Resolved architecture: Qwen4ExpForConditionalGeneration
Tensor/expert parallel initialization	PASS	TP8; EP rank 0/8; 64/512 local/global experts
Checkpoint load	PASS	131/131; approximately 43.12 GiB per rank
Attention backend	PASS	MUSA FlashAttention v3 selected
KV cache	PASS	347,388 tokens; 84.81× concurrency at model length 4096
CUDA Graph capture	PASS	FULL decode capture size 1 completed
/health	PASS	HTTP 200
/v1/models	PASS	Reports Qwen3.8-Flash-Next
/v1/completions	PASS	HTTP 200 and coherent Beijing completion
/v1/chat/completions	PASS	HTTP 200 and coherent Beijing answer

Startup proof

Resolved architecture: Qwen4ExpForConditionalGeneration
[EP Rank 0/8] Expert parallelism is enabled ... 64/512
Loading safetensors checkpoint shards: 100% Completed | 131/131
Model loading took 43.12 GiB memory
GPU KV cache size: 347,388 tokens
Capturing CUDA graphs (FULL): 100% | 1/1
Application startup complete.
GET /health HTTP/1.1" 200 OK
POST /v1/completions HTTP/1.1" 200 OK
POST /v1/chat/completions HTTP/1.1" 200 OK

Model-list check

curl -s http://127.0.0.1:28083/v1/models

The response reports:

id: Qwen3.8-Flash-Next
root: /home/dist/models/Qwen/Qwen3.8-Flash-Next
max_model_len: 4096

Completion check

curl -s http://127.0.0.1:28083/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next",
    "prompt": "The capital of China is",
    "max_tokens": 32,
    "temperature": 0
  }'

Result:

Beijing.
The capital of China is Beijing.
The capital of China is Beijing.

Chat check

curl -s http://127.0.0.1:28083/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of China? Answer with one city."
      }
    ],
    "max_tokens": 64,
    "temperature": 0
  }'

The returned content is coherent and explicitly identifies Beijing as the
capital of China.

[yeahdongcn]

docker pull registry.mthreads.com/mcconline/inference/vllm/vllm-openai:qwen38-flash-next