[Tianlei Wu (tianleiwu)]

Description

Adds support for inclusionAI/Ling-3.0-tiny (BailingMoeV3ForCausalLM, model_type: "bailing_hybrid") in ONNX Runtime GenAI, covering its hybrid Kimi Delta Attention (KDA) + Multi-Head Latent Attention (MLA) + Sparse MoE architecture across BF16, 128x128 block-FP8 (inclusionAI/Ling-3.0-tiny-fp8), and quantized INT4/INT8 + calibrated INT8 per-channel KV cache recipes. Also adds automatic fallback registration for the V2 CUDA Plugin EP in src/ep/cuda/session_options.cpp.

Summary of Changes

Model Builder (src/python/py/models/)

File	Change
src/python/py/models/builders/bailing.py	Implement BailingMoeV3Model and streaming safetensors loader _BailingWeights: KDA layers (VarlenCausalConvWithState / CausalConvWithState, GatedDeltaNet with per-key-dim decay and gate_activation='sigmoid', GatedRMSNorm(activation='sigmoid')), MLA layers (PagedAttention / GroupQueryAttention with head_size=192 and offline de-interleaved RoPE weights), fused MoEGroupRouter (noaux_tc group-limited sigmoid routing) + shared expert, 128x128 block-FP8 (MatMulBlockQuantizedFp8Weight + QMoE(quant_type='fp8', block_size=128, swiglu_fusion=2)), default CUDA graph capture (enable_cuda_graph='1'), and automatic PCRE possessive-quantifier normalization in tokenizer.json.
src/python/py/models/builders/base.py	Support 2D block-scaled FP8 in make_matmul_block_quantized_fp8_weight, fuse gate_proj + up_proj for MatMulBlockQuantizedFp8Weight in make_mlp_proj_fused, support activation attribute in make_gated_rms_norm, and support gate_lower_bound / separate decay_shape & beta_shape in make_gated_delta_net*.
src/python/py/models/builder.py, src/python/py/models/builders/__init__.py	Register BailingMoeV3Model (BailingMoeV3ForCausalLM -> bailing_hybrid), allow bailing_hybrid for native FP8 checkpoint export.

C++ Runtime (src/)

File	Change
src/models/model_type.h	Add "bailing" and "bailing_hybrid" to ModelType::LLM.
src/ep/cuda/session_options.cpp	Add EnsureCudaPluginEpRegistered to support ORT_CUDA_PLUGIN_EP_PATH and automatic fallback registration when libonnxruntime_providers_cuda.so is built as a V2 CUDA Plugin EP.

Testing

• Built against ONNX Runtime (microsoft/onnxruntime#33817) and verified end-to-end generation, 2k/8k/32k latency & VRAM benchmarks, and MMLU-Pro accuracy across BF16, Block-FP8, INT8+INT8-KV, and INT4+INT8-KV exports of inclusionAI/Ling-3.0-tiny.

• Review-fix validation: 32 offline Python builder/loader tests and 17 native CUDA option/device-selection tests passed; the Linux native library build and formatting checks passed. Windows runtime execution and full Ling generation were not rerun.

Checklist

• Tests added/updated
• No breaking changes
• Documentation updated (if applicable)

[Copilot]

🟡 Changes recommended

Unresolved critical issues bypass the caller’s remote-code opt-out and cause supported exports to fail, omit adapters, or silently produce incorrect weight formats.

9 open issues

• Fallback overrides explicit remote-code opt-out · New
• Run QDQ quantization after dequantizing FP8 weights · New
• Apply recurrent-state dtype overrides before graph emission · New
• Apply or reject adapters in the Bailing weight loader · New
• Resolve Hugging Face repo IDs before loading weights · New
• Support sigmoid gating in the TRT-RTX standard expansion · New
• Preserve per-channel scale layouts with explicit block metadata · New
• Register explicit CUDA plugin paths on Windows · New
• Reject compact capture with unsupported per-key decay · New

What changed in this PR

Adds Ling-3.0-tiny support to ONNX Runtime GenAI’s model builder and runtime, alongside CUDA plugin execution-provider fallback registration.

Changes:

• Introduces a BailingMoeV3 builder for hybrid KDA/MLA attention, sparse MoE, checkpoint loading, and tokenizer compatibility.
• Extends shared graph emitters for block-FP8 weights and additional gated-attention attributes.
• Registers Bailing model types and adds explicit-path and adjacent-library CUDA plugin discovery.

File	Description
src/​python/​py/​models/​builders/​base.py	Extends FP8 scale handling and gated-attention emitters.
src/​python/​py/​models/​builders/​bailing.py	Adds the Bailing loader, graph builder, routing, and tokenizer processing.
src/​python/​py/​models/​builders/​__init__.py	Exposes BailingMoeV3Model.
src/​python/​py/​models/​builder.py	Registers the architecture, adjusts checkpoint validation and config loading, and escapes CLI help text.
src/​models/​model_type.h	Recognizes Bailing variants as language models.
src/​ep/​cuda/​session_options.cpp	Adds CUDA plugin registration and provider-bridge fallback handling.

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

Addressed the configuration-alignment feedback in cb850ed:

• Full attention follows the HF incomplete-tail scheduling rule; missing layer_group_size defaults to 1, and per-layer lists or invalid integers fail explicitly. Current SGLang requires integer-group divisibility, so the incomplete-tail cases use HF as the reference.
• Routing normalization (default false), routed scaling (default 1.0), and integer group-score top-k are configurable. Unnormalized routing restores selected sigmoid mass rather than relying on the MoE softmax alone.
• KDA honors the safe-gate flag and validates its bound only in safe mode. RoPE honors interleaved/noninterleaved configuration without changing checkpoint projection rows.

Validation: 84 targeted tests passed, including actual CPU/A100 CUDA router and KDA numerical comparisons plus RoPE references; CUDA numerical sessions disable CPU fallback. Configured lint passed. These are component-level checks, not full Ling generation parity.

ORT #33817 is now merged. Recipe PR microsoft/olive-recipes#689 pins this GenAI revision, but full Ling exports/generation, accuracy against SGLang/HF, and H200/FP8 latency and throughput results are still required before approval. New-head CI is running; I am not claiming it is green yet.

[Baiju Meswani (baijumeswani)]

What this PR does

This adds Ling-3.0-tiny support to the model builder and runtime. It covers the model's mix of KDA linear attention, MLA full attention, and shared/routed experts, with BF16, block-FP8, and integer weight export paths. It also adds CUDA plugin discovery when the usual provider-loading path does not work.

What looks good

The implementation reuses the existing Model/Qwen builder infrastructure rather than creating a separate exporter. The follow-up fixes add useful checks, including rejecting unsupported adapters and compact state capture instead of silently producing an incomplete export. The tests cover several important details: checkpoint resolution, QDQ after dequantization, recurrent-state types, routing math, KDA math, and RoPE layout. The shared FP8 and normalization changes also preserve the existing builder regressions I ran.

What remains before approval

I would hold approval until the two correctness concerns are fixed: late CUDA plugin discovery can select a different GPU from the one used for the stream, and an explicit FP16 export can silently produce BF16 tensors. The inline comments describe the cases and suggested tests.

The builder should also follow the existing loader and configuration boundaries: keep checkpoint unpacking in loaders, simplify the large methods, follow the project's naming/static-method conventions, use the existing configuration hook, and document the new precision option. Please document the minimum compatible ORT build and make sure a CI job actually runs the new routing/KDA kernel tests rather than skipping them. The dependency microsoft/onnxruntime#33817 merged on November 8, 2026, so that dependency is no longer waiting to merge.

Validation

Reviewed commit cb850ed. Targeted Bailing/QMoE tests and shared FP8, expansion, and precision regressions produced 467 passed, 35 skipped. Key routing/KDA kernel cases skipped with the installed ORT 1.30.0. I reproduced the FP16/BF16 mismatch locally. The CUDA device/stream concern comes from tracing the code; I did not reproduce it on a multi-GPU system. Full-checkpoint CUDA generation and Windows plugin execution were not run as part of this review.