Skip to content

Add configurable GQA and MLA sliding-window draft layers - #767

Draft
maocheng23 wants to merge 2 commits into
sgl-project:mainfrom
maocheng23:agent/general-swa-attention
Draft

Add configurable GQA and MLA sliding-window draft layers#767
maocheng23 wants to merge 2 commits into
sgl-project:mainfrom
maocheng23:agent/general-swa-attention

Conversation

@maocheng23

Copy link
Copy Markdown
Collaborator

Dependency

This is a stacked draft on #758. The first commit is the generic MLA support from that PR; after #758 merges, this PR will contain only the SWA configuration delta.

Summary

  • make context layout and projection layout independent for every DFlash-family draft layer
  • keep dflash_config.attention_mode as the backward-compatible uniform GQA/MHA/MLA shorthand
  • add dflash_config.attention_modes for per-layer GQA/MHA/MLA selection
  • support SWA-GQA, SWA-MLA, full-context GQA/MLA, and mixed stacks without a model-specific architecture
  • validate ambiguous, invalid, and wrong-length configurations early
  • document the paired layer_types / attention_modes contract

layer_types continues to select full_attention or sliding_attention; the projection list is paired with it by layer index. The GQA and MLA parameterizations follow the standard layouts used by TorchSpec, while SpecForge's existing DFlash mask and objective interfaces remain unchanged.

Example

{
  "num_hidden_layers": 4,
  "layer_types": [
    "sliding_attention",
    "sliding_attention",
    "full_attention",
    "full_attention"
  ],
  "use_sliding_window": true,
  "sliding_window": 2048,
  "dflash_config": {
    "attention_modes": ["gqa", "mla", "gqa", "mla"]
  }
}

Validation

  • all pre-commit hooks passed
  • CPU targeted regression: 34 passed, 2 skipped, 19 subtests passed
  • H200 mixed SWA-GQA/SWA-MLA FlexAttention forward/backward: 6 passed, 7 subtests passed
  • broader modeling/algorithm/runtime/training regression: 93 passed, 5 skipped, 54 subtests passed
  • tested with Transformers 5.8.1 and PyTorch 2.11.0+cu130

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant