Add GLM-5.2 recipe for RTX PRO 6000 / RTX 6000D (SM120) 8-GPU boxes - #57
Open
AliceChenyy wants to merge 4 commits into
Open
Add GLM-5.2 recipe for RTX PRO 6000 / RTX 6000D (SM120) 8-GPU boxes#57AliceChenyy wants to merge 4 commits into
AliceChenyy wants to merge 4 commits into
Conversation
First GLM-5.2 cookbook entry, focused on the SM120 workstation platform: the NVFP4 checkpoint (433 GB) fits a single 8-GPU PCIe box. Validated with the official lmsysorg/sglang:v0.5.15 image on 8x RTX 6000D; includes required flags with rationale, ops pitfalls (NCCL ACS hang, zombie schedulers pinning GPU memory), and measured GSM8K + latency numbers. MTP and TP4xPP2 recipes are noted as pending sgl-project/sglang#30994. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both variants validated end to end on 8x RTX 6000D with a v0.5.15 build of sglang#30994 (GSM8K 0.90 each, greedy output identical to TP8): - TP8 + MTP speculative decoding: median TPOT 11.4ms vs 24.4ms at concurrency 1 (low-latency / interactive use) - TP4 x PP2: 2.8x lower TTFT and 28% more output tok/s than TP8 at concurrency 16 (throughput / batch use); documents the index_topk_pattern override needed because the PP2 stage boundary lands on a skip-topk layer Benchmarks section split into per-config tables with selection guidance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Added validated MTP and TP4xPP2 recipes (commit 656a582). Both were run end to end on 8x RTX 6000D with a v0.5.15 build of sglang#30994 — GSM8K 0.90 each, greedy output identical to TP8. Benchmarks section now has per-config tables (TP8 / TP8+MTP / TP4xPP2) plus selection guidance. The two variant sections state the #30994 requirement explicitly; happy to hold this PR until it lands if preferred. |
…chmarks Major update based on the latest SM120 validation: - Switch from dense bf16 KV (workaround) to FlashInfer sparse MLA with FP8 KV (#26928) as the recommended path - Replace old three-config bench data (dense, #30994-based) with four-config real-weight benchmarks on 8x RTX 6000D: TP8 / TP4xPP2 / TP8+MTP / TP4xPP2+MTP - Add pure decode section (ISL 1 / OSL 256) isolating TPOT from prefill - TP4xPP2+MTP is the best config at every batch size in pure decode: bs1 TPOT 10.7ms (2.2x over TP8), bs32 throughput 611 tok/s (2.0x) - Update all launch commands with fp8_e4m3 KV, mem-fraction tuning, disable-radix-cache, and disable-custom-all-reduce tips - Document #26928 and #30775 dependencies clearly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AliceChenyy
marked this pull request as ready for review
July 15, 2026 10:17
68 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first GLM-5.2 cookbook entry, focused on the first workstation/edge SM120 platform recipe (NVIDIA RTX PRO 6000 Blackwell 8x96GB / RTX 6000D 8x84GB): the NVFP4 checkpoint (433 GB) fits a single 8-GPU PCIe box, making this the most accessible way to serve a 753B-class model on-prem.
What's included
cookbook/llm/GLM/GLM-5.2.mdx, validated with the officiallmsysorg/sglang:v0.5.15image on 8x RTX 6000D:--disable-shared-experts-fusionfor sglang#29562; dense FlashInfer MLA + bf16 KV because SM120 has no DSA sparse kernels)NCCL_P2P_DISABLE=1), zombie schedulers pinning GPU memory after SIGKILL (docker run --init)docs.jsonScope notes
topk_indiceson spec-verify / non-overlap decode paths). With a patched build we measured TP4xPP2 ahead of TP8 by 3-6x at ISL 8K on RTX PRO 6000, so these are worth adding once it lands — noted in the doc.(Originally prepared against sgl-cookbook before noticing it is archived; ported to this repo's mdx + docs.json structure.)
🤖 Generated with Claude Code