Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion docs/basic_usage/Ascend/ascend_npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,42 @@ python -m pip install -e . --no-deps
torch/sglang. If a later step reports a missing lightweight dependency, install
just that package, also with `--no-deps`.

### Install Mooncake (Ascend)

Online runs move features through Mooncake; the launcher checks for the
`mooncake_master` binary and the `mooncake` Python package at startup. The
fastest path is the official NPU wheel:

```bash
pip install mooncake-transfer-engine-npu
```

If the wheel does not cover your platform, build from source instead (see the
[vLLM-Ascend Mooncake guide](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_colocated_mooncake_multi_instance.html)
for the full walkthrough):

```bash
git clone -b v0.3.9 --depth 1 https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
git submodule update --init --recursive
apt-get install -y mpich libmpich-dev
bash dependencies.sh -y
mkdir build && cd build
cmake .. -DUSE_ASCEND_DIRECT=ON # mandatory on Ascend
make -j && make install
```

Either way, verify both pieces before continuing:

```bash
mooncake_master --help >/dev/null && echo mooncake_master ok
python -c "from mooncake.store import MooncakeDistributedStore; print('mooncake.store ok')"
```

If the capture server later dies with "no `allocate_and_mount_segment`", the
Mooncake build is too old for the companion patch below — upgrade or rebuild
it first.

### Apply the SGLang capture patches (online runs only)

Online capture needs two patches on top of the installed SGLang, applied **in
Expand All @@ -35,7 +71,7 @@ bash scripts/apply_sglang_spec_capture_patch.sh
# Ascend companion patch: skip the wildcard segment mount that Ascend
# Mooncake rejects, and mount the feature segment with location="cpu"
SGLANG_DIR=$(python -c "import sglang, os; print(os.path.dirname(os.path.dirname(sglang.__file__)))")
cd "$SGLANG_DIR" && git apply /path/to/SpecForge/patches/sglang/v0.5.14/spec-capture-ascend-mount.patch
cd "$SGLANG_DIR" && git apply -p2 /path/to/SpecForge/patches/sglang/v0.5.14/spec-capture-ascend-mount.patch
```

Skip both for offline training, which reads features from disk. The companion
Expand Down
16 changes: 3 additions & 13 deletions patches/sglang/v0.5.14/spec-capture-ascend-mount.patch
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
diff --git a/python/sglang/srt/spec_capture_sink.py b/python/sglang/srt/spec_capture_sink.py
index 4955e11..902f0b1 100644
index d54b920..f831b03 100644
--- a/python/sglang/srt/spec_capture_sink.py
+++ b/python/sglang/srt/spec_capture_sink.py
@@ -81,18 +81,31 @@ class SpecCaptureSink:
@@ -87,18 +87,28 @@ class SpecCaptureSink:
from mooncake.store import MooncakeDistributedStore, ReplicateConfig

store = MooncakeDistributedStore()
Expand Down Expand Up @@ -38,7 +38,7 @@ index 4955e11..902f0b1 100644
rdma_devices=os.environ.get("MOONCAKE_RDMA_DEVICES", ""),
master_server_addr=os.environ.get(
"MOONCAKE_MASTER_SERVER_ADDR", "localhost:50051"
@@ -100,12 +113,34 @@ class SpecCaptureSink:
@@ -106,6 +116,25 @@ class SpecCaptureSink:
)
if rc is not None and int(rc) != 0:
raise RuntimeError(f"spec-capture mooncake setup failed (status {rc})")
Expand All @@ -64,13 +64,3 @@ index 4955e11..902f0b1 100644
# Hard-pin every object: SpecForge (not Mooncake's LRU) is the
# lifetime authority — a committed feature must never be evicted
# before the trainer consumes it.
cfg = ReplicateConfig()
cfg.replica_num = 1
- cfg.with_hard_pin = True
+ # Older Mooncake builds expose no with_hard_pin field; objects
+ # then follow the store's default pin behavior until remove().
+ if hasattr(cfg, "with_hard_pin"):
+ cfg.with_hard_pin = True
self._put_config = cfg
self._store = store
logger.info("spec-capture mooncake sink connected")
Loading