refactor: extract packed capture_rows from the offline SGLang backend - #780
Draft
maocheng23 wants to merge 1 commit into
Draft
refactor: extract packed capture_rows from the offline SGLang backend#780maocheng23 wants to merge 1 commit into
maocheng23 wants to merge 1 commit into
Conversation
Expose capture_rows(input_ids) on OfflineSGLangCaptureBackend and OfflineSGLangCapture so callers can capture variable-length rows in one packed prefill without building padded tensors. capture_eagle3 delegates to it and keeps its exact request construction and outputs. DSpark capture-layer setup now falls back from the native set_dspark_layers_to_capture hook to the dense DFlash hook (with a log line naming the resolved hook) so dense targets such as Qwen3 can serve DSpark capture on stock SGLang 0.5.14 models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Stack 1/5 replacing draft #766 (colocated online training), split into reviewable steps. This PR is the capture-engine primitive the colocated runtime builds on: a packed variable-length capture API, plus DSpark capture-hook compatibility for dense targets.
Modifications
capture_rows(input_ids: list[list[int]])fromcapture_eagle3onOfflineSGLangCaptureBackendand expose it onOfflineSGLangCapture. One packed prefill captures variable-length rows without building padded tensors.capture_eagle3delegates to it and keeps its exact request construction, outputs, and pool-clearing behavior — offline feature preparation is byte-identical (unlike feat: add scalable colocated online training #766, no attention-mask trimming is bundled in).set_dspark_layers_to_capturehook to the dense DFlash hook, logging the resolved hook. Dense targets such as Qwen3 expose the same auxiliary-hidden-state layout only throughset_dflash_layers_to_captureon stock SGLang 0.5.14, and previously failed outright. (feat: add scalable colocated online training #766's additional EAGLE3-hook fallback tier is dropped: no supported build needs it and its layout equivalence was unverified.)Related Issues
Splits #766. Stack: #1 (this) ← #2 teardown-abort ← #3 rank0-tracker ← #4 colocated-core ← #5 hybrid-shard.
Accuracy Test
tests/test_runtime/test_sglang_0514_compat.pygains hook-preference, fallback, and missing-hook coverage (runs in CI with sglang installed).Checklist
black --checkandisort --check-only).