Skip to content

MUSA: switch Ubuntu 22.04 image to system Python 3.12 - #119

Draft
yeahdongcn wants to merge 4 commits into
v0.24.0-devfrom
codex/musa-60032-python312-image
Draft

MUSA: switch Ubuntu 22.04 image to system Python 3.12#119
yeahdongcn wants to merge 4 commits into
v0.24.0-devfrom
codex/musa-60032-python312-image

Conversation

@yeahdongcn

@yeahdongcn yeahdongcn commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Switch the Ubuntu 22.04 vLLM-MUSA image to a system CPython 3.12 runtime.
  • Bootstrap Python 3.12 from Deadsnakes (with configurable mirror/key support),
    without uv or a virtual environment.
  • Align package metadata, Docker documentation, and build-image defaults with
    Python 3.12.
  • Require binary MUSA wheels and fail closed instead of accepting an incompatible
    cp310 wheel in a cp312 image.
  • Add optional rustup mirror arguments for builds that enable vllm-rs.

Wheel compatibility

The live Moore Threads index was checked before implementation. The required
cp312-capable set includes native wheels for torch, torch_musa, torchvision,
torchaudio, deep_ep, triton, apache-tvm-ffi, and
torch-c-dlpack-ext. tilelang_musa is cp38-abi3; MATE, FlashAttention 3,
FlashMLA, DeepGEMM, and SageAttention are pure-Python wheels.

Validation

  • Worktree: codex/musa-60032-python312-image
  • Tested SHA: 2f58508434663abcca0db808afa360b7cf287860
  • Base: v0.24.0-dev at e96a974b91063198f84f432b5fcfac991f95639e
  • Local image: vllm-musa:python312-2f58508-final
  • Image ID: sha256:850d5cb5b78025b77089517158f34667b2fb71f9ce432414a66f96800d89a587
  • Local gate: Python 3.12.13, sys.prefix == sys.base_prefix == /usr, no uv,
    no python3.12-venv, required MUSA imports pass.
  • Static gate: python3 -m pytest -q tests/test_inplace_porting_contract.py
    (12 passed), bash -n docker/build_image.sh, and git diff --check pass.
  • Hardware gate: Qwen3-8B-FP8 semantic request and serving A/B pass on one
    MTT S5000 card; backend logs show MUSA FLASH_ATTN and
    FULL_AND_PIECEWISE compilation/capture.
  • A/B contract: input/output 512/128, BS 1/4/16/64, two warmups and three
    measured runs, same physical S5000 card, Python frontend on both sides.

Review caveats

  • The smoke candidate was built with BUILD_VLLM_RS=0 and BUILD_MOONCAKE=0
    because the optional source downloads were unavailable in the build network;
    the A/B explicitly used VLLM_USE_RUST_FRONTEND=0 on both images.
  • The cp312 image uses generic musa5.2.0 wheels; the registry control uses
    musa5.2.0s5000 wheels. The performance result is therefore a whole-image
    comparison, not a causal Python-only comparison.
  • The S5000 lease was non-isolated and the control ran before the candidate.
  • The measured throughput delta was within roughly +/-1.6% across BS values;
    no stable Python 3.12 uplift was observed. One candidate BS=64 run had a
    one-off capture/scheduling outlier, retained in the local evidence bundle.

The bulky local smoke/A-B logs and normalized table are retained under
generated/MUSA-60032/20260723-python312-image-ab/ in the development
workspace and are not added to this source PR.

Review follow-ups

  • third_party/PINS retains a build-box/package-floor comment mentioning
    Python 3.10; it has no runtime effect but can be refreshed during review.
  • The default Deadsnakes path deliberately bypasses inherited proxy variables;
    retry behavior for transient Launchpad failures is a possible hardening item.

@yeahdongcn

Copy link
Copy Markdown
Collaborator Author

Follow-up: registry-control S5000 A/B

I compared this PR's Python 3.12 image with the new Python 3.10 registry
control requested in the review:

  • control: registry.mthreads.com/mcconline/inference/vllm:v0.24.0-ph1-5.2.0-torch2.9.1.post1-20260724
    • registry digest at run time: sha256:6bf7f99f4e3aef394b39112ddb1401fb88508e702178653f2e4c278c647cb970
    • image ID: sha256:be33c3bc90aa4b6440d417597b9e3dc3065d8eefe152ff25b2b9c0f4898d01cb
  • candidate: this PR at 2f58508434663abcca0db808afa360b7cf287860
    • image ID: sha256:850d5cb5b78025b77089517158f34667b2fb71f9ce432414a66f96800d89a587

Contract

  • One physical S5000 card, TP=1, on a broker-reserved 8 x S5000 node
  • Qwen3-8B-FP8; configured input/output 512/128, observed 520/128 after chat templating
  • Concurrency 1/4/16/64
  • Each server lifetime and BS: 2 warmups + 3 measured runs, 64 requests per measured run
  • Python frontend, FLASH_ATTN, normal FULL_AND_PIECEWISE compiled/captured path, async scheduling

Primary result: uncontaminated A1 -> B1 pair

The timestamps prove this first pair completed before any unrelated GPU
process appeared on the host. Each row pools 192 requests per image.

BS Py3.10 output tok/s Py3.12 output tok/s Delta Py3.10 p99 TTFT ms Py3.12 p99 TTFT ms Py3.10 p99 TPOT ms Py3.12 p99 TPOT ms Py3.10 p99 E2E ms Py3.12 p99 E2E ms
1 90.50 90.80 +0.33% 48.86 48.46 11.17 11.09 1457.62 1446.04
4 331.16 333.03 +0.57% 109.94 108.90 11.77 11.70 1572.81 1565.74
16 1102.03 1095.15 -0.62% 308.76 305.67 14.03 14.05 1904.04 1906.37
64 2350.90 2326.79 -1.03% 1306.07 1286.40 25.89 25.79 3555.07 3519.22

There is no material throughput or p99 regression in the clean pair. There is
also no resolvable uplift to claim.

Isolation caveat

The broker returned an isolated=true whole-node lease, but direct unmanaged
Docker launches are not fenced by that lease. An unrelated GPU0 process began
after B1 completed, and a later SGLang job on GPU4 overlapped A2. I therefore
treat the later B2/A2 pair and the four-session pooled table as secondary
evidence only. A second isolated 8-GPU host was acquired, but two unrelated
profiler containers started on GPU0 before image load; fail-closed preflight
rejected that retry without running vLLM. No unrelated container/process was
stopped.

The later pair was directionally consistent (all throughput deltas remained
within 1.78% and pooled p99 E2E favored the candidate by 0.25% to 2.20%), but it
is not labeled interference-free.

Functional gate and decision

All 3,072 measured requests in the four-session run completed with zero
failures. The candidate also passed a separate normal compiled-path semantic
smoke: finish_reason=stop, content Beijing.

Decision: continue review of this draft PR. The current evidence supports
functional parity and performance non-regression, not a Python-caused speedup.
This remains a whole-image comparison because the Python 3.10 control carries
S5000-specific musa5.2.0s5000 wheels while the Python 3.12 candidate uses the
available generic musa5.2.0 cp312 wheels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant