From 9fda38254318f50d3ca2e1c929907d37134fa3a1 Mon Sep 17 00:00:00 2001 From: Alex M <459478+alez007@users.noreply.github.com> Date: Mon, 6 Jul 2026 08:34:22 +0000 Subject: [PATCH 1/4] feat: extend bench to llama_server/CPU and add --no-preflight for fair A/B bench/run.sh now takes --loader (vllm|llama_server) and --device (gpu|cpu) to A/B any wrapped stack against its vanilla baseline, not just vLLM-on-GPU; shared shell helpers moved into bench/lib.sh and a rawllama_entrypoint.py mirrors llama_server_infer.py's launch for the llama.cpp baseline phase. Preflight hardware auto-sizing can now be disabled via --no-preflight / MSHIP_PREFLIGHT=false (passed through to replicas alongside MSHIP_METRICS), so bench can run modelship on loader defaults + explicit config only, removing the prior "not byte-identical" caveat against the raw baseline. --- AGENTS.md | 1 + bench/README.md | 105 ++-- bench/configs/llama-cpu.yaml | 20 + bench/configs/llama-gpu.yaml | 21 + bench/configs/vllm-cpu.yaml | 19 + bench/configs/{bench.yaml => vllm-gpu.yaml} | 0 bench/lib.sh | 439 ++++++++++++++++ bench/rawllama_entrypoint.py | 97 ++++ bench/run.sh | 525 +++----------------- docs/model-configuration.md | 1 + modelship/deploy/actor_options.py | 7 +- modelship/preflight/base.py | 8 + modelship/utils/cli.py | 11 + tests/test_mship_deploy.py | 16 + tests/test_preflight.py | 14 + 15 files changed, 800 insertions(+), 484 deletions(-) create mode 100644 bench/configs/llama-cpu.yaml create mode 100644 bench/configs/llama-gpu.yaml create mode 100644 bench/configs/vllm-cpu.yaml rename bench/configs/{bench.yaml => vllm-gpu.yaml} (100%) create mode 100644 bench/lib.sh create mode 100644 bench/rawllama_entrypoint.py diff --git a/AGENTS.md b/AGENTS.md index 317205d..fcbc8bb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -105,6 +105,7 @@ Commit messages matter: use Conventional Commits prefixes so the changelog gener - **GGUF is not supported on the `vllm` loader.** 0.24 moved GGUF out of tree; the only external `vllm-gguf-plugin` (`0.0.2`) has a stale `override_quantization_method` signature incompatible with 0.24's quantization API (it breaks *all* quantized models, not just GGUF), so it is deliberately not installed. `resolve_all_model_sources` rejects a `.gguf` on the vllm loader at driver preflight and points to `llama_server`. For GGUF use `loader: llama_server`; feed the vllm loader safetensors or an AWQ/GPTQ/FP8 quant. - `llama_server` loader (GGUF) launches a `llama-server` subprocess — found via `MSHIP_LLAMA_SERVER_BIN`, pinned in the Docker images at `/opt/llama.cpp/llama-server.sh` — and proxies its native OpenAI API instead of parsing output in-process. `num_gpus` must be `0` or a whole integer — fractional is rejected at config time (llama.cpp has no VRAM-fraction knob). `num_gpus >= 1` honors `n_gpu_layers`; `stable_diffusion_cpp` remains forced to `num_gpus: 0` in `mship_deploy.py:build_deployment_options`. `--parallel` slots give real request concurrency instead of serializing behind a single lock. Tool-call/reasoning parsing is llama-server's own, auto-detected per chat template: named-function `tool_choice` forcing is unsupported globally (silently falls back to `auto`), and `tool_choice: required` is grammar-enforced for harmony-style templates but a silent no-op for hermes-style ones (e.g. Qwen3); bare `response_format: {"type": "json_object"}` (no `schema` key) is also unenforced despite llama-server's own docs claiming support — `type: json_schema` requests (what modelship sends whenever a schema is given) are unaffected. No persistent on-disk prompt cache. See `docs/model-configuration.md`'s llama_server section for the full field table and examples. - Metrics are on by default on port **8079** (not 8000). Disable with `--no-metrics` or `MSHIP_METRICS=false`. +- Preflight hardware auto-sizing is on by default. Disable with `--no-preflight` or `MSHIP_PREFLIGHT=false` to run models on loader/library defaults plus explicit `models.yaml` config only — useful for benchmarking across hardware. - Log level `TRACE` (below `DEBUG`) is a custom level and logs full request/response payloads. - Docker images are multi-arch (amd64 + arm64). The CPU image uses the unified `Dockerfile` with `--build-arg MSHIP_VARIANT=cpu` and has a different tag suffix (`:latest-cpu`). diff --git a/bench/README.md b/bench/README.md index c18ffe8..22f5a81 100644 --- a/bench/README.md +++ b/bench/README.md @@ -1,26 +1,46 @@ -# bench — modelship vs raw vLLM A/B harness +# bench — modelship vs vanilla-loader A/B harness -Two-phase benchmark that runs `vllm bench serve` against the **modelship** vLLM -loader, then against **raw `vllm serve`** from the *same Docker image* with the -same engine kwargs, and diffs throughput, latency, and memory. +Two-phase benchmark that runs `vllm bench serve` against a **modelship** +loader, then against the **vanilla server it wraps** (raw `vllm serve`, or a +bare `llama-server` subprocess) from the *same Docker image* with the same +engine config, and diffs throughput, latency, and memory. + +This measures modelship's own wrapping overhead — Ray Serve, the gateway, the +loader's proxy layer — not one inference stack against another. `--loader` +picks which wrapped stack to measure (`vllm` or `llama_server`); `--device` +picks GPU vs CPU for that stack. The four combinations: + +| `--loader` | `--device` | config | baseline | +| --- | --- | --- | --- | +| `vllm` | `gpu` (default) | `configs/vllm-gpu.yaml` | raw `vllm serve` on GPU | +| `vllm` | `cpu` | `configs/vllm-cpu.yaml` | raw `vllm serve` on the CPU backend | +| `llama_server` | `gpu` | `configs/llama-gpu.yaml` | vanilla `llama-server`, fully GPU-offloaded | +| `llama_server` | `cpu` | `configs/llama-cpu.yaml` | vanilla `llama-server`, CPU-only | ## Prerequisites -- A GPU host with the NVIDIA container runtime (`--gpus all` must work). -- `docker`, `curl`, and `nvidia-smi` on the host. -- A built modelship image (default tag `modelship:dev`; override with `--image`). +- `docker` and `curl` on the host. +- For `--device gpu`: the NVIDIA container runtime (`--gpus all` must work) and `nvidia-smi`. +- A built modelship image. Default tag is `modelship:dev` (GPU) / `modelship:dev-cpu` + (CPU) — override with `--image`. GPU and CPU are separate image variants + (`--build-arg MSHIP_VARIANT=gpu|cpu`, see the root `Dockerfile`); the `cpu` + variant is required for `--device cpu` (it ships the `vllm==...+cpu` wheel). ## Run ```bash -bench/run.sh # defaults: 100 prompts, conc 8, in/out 128/512, 20 warmups, 3 repeats -bench/run.sh --image modelship:dev --concurrency 32 --num-prompts 500 +bench/run.sh # vllm on GPU (default): 100 prompts, conc 8, in/out 128/512, 20 warmups, 3 repeats +bench/run.sh --loader vllm --device cpu +bench/run.sh --loader llama_server --device gpu +bench/run.sh --loader llama_server --device cpu +bench/run.sh --loader vllm --image modelship:dev --concurrency 32 --num-prompts 500 ``` `--num-warmups N` (default 20) sends warmup requests that are discarded before -timing so cold-start (CUDA graph capture / compilation) doesn't skew the result. -`--repeats N` (default 3) runs the sweep N times per stack; the summary reports -the **median** so a single noisy run can't dominate. +timing so cold-start (CUDA graph capture / compilation / first-request JIT) +doesn't skew the result. `--repeats N` (default 3) runs the sweep N times per +stack; the summary reports the **median** so a single noisy run can't dominate. +`--config PATH` overrides the config file picked by `--loader`/`--device`. Tunable env vars (forwarded to the modelship phase): @@ -28,15 +48,17 @@ Tunable env vars (forwarded to the modelship phase): - `MSHIP_GATEWAY_MAX_ONGOING` (default 1024) — gateway per-replica concurrency cap. - `MSHIP_CACHE_DIR` — model cache to reuse across phases (default `./models-cache`). -The model and the per-model `max_ongoing_requests` cap come from -[`configs/bench.yaml`](configs/bench.yaml). +The model and the per-model `max_ongoing_requests` cap come from the config +file (`configs/vllm-gpu.yaml`, `configs/vllm-cpu.yaml`, `configs/llama-gpu.yaml`, +or `configs/llama-cpu.yaml`). ## Output Each run writes a timestamped dir under `bench/results/` (gitignored) containing `result_.json` (one per repeat, per phase), `mem.tsv`, `prom.txt`, `components.txt`, container logs, and a `summary.md` whose tables show the -**median across repeats**. +**median across repeats**. The two phase subdirectories are always named +`modelship` and `baseline` regardless of `--loader`/`--device`. `summary.md` also breaks the modelship container's memory down per Ray process (from `components.txt`, scraped from the reporter agent on port 8079): a @@ -46,10 +68,12 @@ control-plane processes (`gcs_server`, `raylet`, `agent`, `ProxyActor`, mapped into every worker — not plasma) reported separately so they aren't charged to any one actor. This attributes the host-RAM overhead — model-serving actor vs fixed Ray control plane. The snapshot is the **peak-private scrape sampled during -the sweep** (not the idle post-sweep state); modelship-only, since raw vLLM has no -Ray. Note this table **undercounts** the true total: the Ray reporter sees only -Ray worker PIDs, so vLLM's `EngineCore` subprocess is missing — the reconciliation -below quantifies the gap. Trust cgroup `anon` for the absolute number. +the sweep** (not the idle post-sweep state); modelship-only, since the baseline +stack has no Ray. Note this table **undercounts** the true total: the Ray +reporter sees only Ray worker PIDs, so a loader's own inference subprocess +(vLLM's `EngineCore`, or the `llama-server` child) is missing — the +reconciliation below quantifies the gap. Trust cgroup `anon` for the absolute +number. Two cross-checks back this up: @@ -57,19 +81,19 @@ Two cross-checks back this up: stacks, the kernel's own accounting: `anon` (real process RSS), `shmem` (tmpfs/plasma — Ray's object store, charged to the cgroup but to no process), and `file` (reclaimable page cache). The memory table reports the peak of each, - so the modelship-vs-raw RSS gap is attributed to real memory vs plasma vs cache. + so the modelship-vs-baseline RSS gap is attributed to real memory vs plasma vs cache. - **reporter-vs-cgroup reconciliation** — the per-component section compares the Ray reporter's Σ private/shared (a second-hand Prometheus gauge that can be stale) against cgroup `anon`/`shmem` (ground truth). A `⚠️ diverges` flag means - the reporter numbers are suspect and shouldn't be quoted. (vLLM exposes a - Prometheus `/metrics` too, but only engine stats — no per-process memory — which - is why the cgroup numbers are the only cross-stack memory signal.) + the reporter numbers are suspect and shouldn't be quoted. (vLLM and llama-server + each expose their own `/metrics` too, but only engine stats — no per-process + memory — which is why the cgroup numbers are the only cross-stack memory signal.) ## Results -Example run — 1×GPU, Qwen2.5-7B-AWQ, 100 prompts @ concurrency 8, median of 3: +Example run — 1×GPU, Qwen2.5-7B-AWQ, `--loader vllm --device gpu`, 100 prompts @ concurrency 8, median of 3: -| metric | modelship | raw vLLM | delta | +| metric | modelship | raw vllm | delta | | --- | ---: | ---: | ---: | | throughput (req/s) | 1.185 | 1.188 | −0.3% | | output (tok/s) | 606.7 | 608.2 | −0.3% | @@ -92,18 +116,25 @@ Notes: by reclaimable page cache and swings multiple GB between runs depending on which cgroup faulted the weights. VRAM overhead is modest (an extra CUDA context), not KV cache. -- Numbers are illustrative; they vary with model, hardware, and load. +- Numbers are illustrative; they vary with model, hardware, load, loader, and device. ## How the two phases stay comparable -- Both phases use the same image (same vLLM wheel) and the same `bench.yaml`. -- The raw phase ([`rawvllm_entrypoint.py`](rawvllm_entrypoint.py)) parses - `bench.yaml` through modelship's own pydantic schema and translates the - vLLM engine kwargs into `vllm serve` flags. -- Keep `gpu_memory_utilization` equal to `num_gpus` in `bench.yaml`: for - fractional `num_gpus` modelship overrides `gpu_memory_utilization` to - `num_gpus`, and the raw phase reads the field verbatim — so a mismatch would - make the comparison unfair. - -> Caveat: modelship applies hardware-aware preflight defaults the raw phase does -> not, so this is a close A/B, not a byte-identical one. +- Both phases use the same image (same vLLM wheel / same `llama-server` binary) + and the same config file. +- The baseline phase parses the config through modelship's own pydantic schema + and translates the engine config into either `vllm serve` flags + ([`rawvllm_entrypoint.py`](rawvllm_entrypoint.py)) or a `llama-server` launch + command ([`rawllama_entrypoint.py`](rawllama_entrypoint.py), mirroring + `modelship/infer/llama_server/llama_server_infer.py`'s `_launch`). +- vLLM: keep `gpu_memory_utilization` equal to what modelship would pick for + that `num_gpus` (0.9 GPU / 0.4 CPU) — for fractional `num_gpus` modelship + overrides `gpu_memory_utilization` to `num_gpus`, and the raw phase reads the + field verbatim, so a mismatch would make the comparison unfair. +- llama_server: set `n_gpu_layers` explicitly in `configs/llama-gpu.yaml` + (rather than the loader's `-1` auto-fit default) — the raw phase has no + preflight to pick a matching value on its own, so an explicit, identical + value keeps both phases offloading the same number of layers. + +> Caveat: modelship applies hardware-aware preflight defaults the baseline +> phase does not, so this is a close A/B, not a byte-identical one. diff --git a/bench/configs/llama-cpu.yaml b/bench/configs/llama-cpu.yaml new file mode 100644 index 0000000..14c23e6 --- /dev/null +++ b/bench/configs/llama-cpu.yaml @@ -0,0 +1,20 @@ +models: + - name: "llm" + model: "lmstudio-community/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf" + usecase: "generate" + loader: "llama_server" + num_gpus: 0 + num_cpus: 8 + # Lift Ray Serve's per-replica concurrency cap (default 100) so the model + # deployment isn't the bottleneck during the sweep. + max_ongoing_requests: 256 + llama_server_config: + # Concurrent llama-server request slots — must cover --concurrency (8 by + # default) so requests overlap instead of queuing behind a full set of + # slots, which would understate throughput for both phases equally but + # muddy the TTFT/ITL comparison with queue-wait time. + parallel: 8 + # n_gpu_layers is irrelevant here: the loader forces `-ngl 0` for any + # num_gpus: 0 deploy regardless of this field (see llama_server_infer.py + # and rawllama_entrypoint.py, which mirrors that branch), so both phases + # run CPU-only either way. diff --git a/bench/configs/llama-gpu.yaml b/bench/configs/llama-gpu.yaml new file mode 100644 index 0000000..d956034 --- /dev/null +++ b/bench/configs/llama-gpu.yaml @@ -0,0 +1,21 @@ +models: + - name: "llm" + model: "lmstudio-community/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf" + usecase: "generate" + loader: "llama_server" + num_gpus: 1 + num_cpus: 2 + # Lift Ray Serve's per-replica concurrency cap (default 100) so the model + # deployment isn't the bottleneck during the sweep. + max_ongoing_requests: 256 + llama_server_config: + # Concurrent llama-server request slots — must cover --concurrency (8 by + # default) so requests overlap instead of queuing behind a full set of + # slots, which would understate throughput for both phases equally but + # muddy the TTFT/ITL comparison with queue-wait time. + parallel: 8 + # Explicit rather than the loader's -1 (auto-fit) default: the raw phase + # reads this field verbatim (no preflight), so leaving it unset would let + # the two phases pick different offload counts. 99 exceeds Qwen2.5-7B's + # 28 layers, i.e. full GPU offload for both phases. + n_gpu_layers: 99 diff --git a/bench/configs/vllm-cpu.yaml b/bench/configs/vllm-cpu.yaml new file mode 100644 index 0000000..40dc929 --- /dev/null +++ b/bench/configs/vllm-cpu.yaml @@ -0,0 +1,19 @@ +models: + - name: "llm" + model: "Qwen/Qwen2.5-7B-Instruct-AWQ" + usecase: "generate" + loader: "vllm" + num_gpus: 0 + num_cpus: 8 + # Lift Ray Serve's per-replica concurrency cap (default 100) so the model + # deployment isn't the bottleneck during the sweep. + max_ongoing_requests: 256 + vllm_engine_kwargs: + tensor_parallel_size: 1 + max_model_len: 8192 + # vLLM's CPU backend repurposes this as the fraction of HOST RAM to + # reserve for the KV cache (not VRAM) — set explicitly (rather than + # relying on modelship's num_gpus==0 default of 0.4) since the raw + # phase reads this field verbatim and must match. See + # config/examples/vllm-cpu.yaml. + gpu_memory_utilization: 0.4 diff --git a/bench/configs/bench.yaml b/bench/configs/vllm-gpu.yaml similarity index 100% rename from bench/configs/bench.yaml rename to bench/configs/vllm-gpu.yaml diff --git a/bench/lib.sh b/bench/lib.sh new file mode 100644 index 0000000..c63f04d --- /dev/null +++ b/bench/lib.sh @@ -0,0 +1,439 @@ +# Shared helpers for the bench/run-*.sh A/B scripts. Sourced, not executed — +# assumes the caller has already set REPO_ROOT, BENCH_DIR, RESULTS_DIR, +# CACHE_DIR, and the sampler/cleanup PID vars it declares below. + +# Extract the first scalar matching a key regex from a bench config yaml. +# Tolerates double-quoted, single-quoted, and unquoted values: strips +# everything up to the first colon, trims surrounding whitespace, then +# removes a matched pair of surrounding quotes (only when both ends use the +# same quote char). +yaml_scalar() { + local pattern="$1" file="$2" + grep -m1 -E "$pattern" "$file" \ + | sed -E "s/^[^:]*:[[:space:]]*//; s/[[:space:]]*\$//; s/^(['\"])(.*)\1\$/\2/" +} + +cleanup() { + for pid in "${MEM_SAMPLER_PID:-}" "${COMPONENT_SAMPLER_PID:-}"; do + if [[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null; then + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + fi + done + for c in "$MODELSHIP_CONTAINER" "$BASELINE_CONTAINER"; do + if [[ -n "$c" ]] && docker inspect "$c" >/dev/null 2>&1; then + docker logs "$c" >"$RESULTS_DIR/${c}.container.log" 2>&1 || true + docker rm -f "$c" >/dev/null 2>&1 || true + fi + done +} + +wait_ready() { + local name="$1" + local deadline=$(( $(date +%s) + READY_TIMEOUT )) + while (( $(date +%s) < deadline )); do + # /v1/models reachable AND lists the served model id + if curl -fsS http://localhost:8000/v1/models 2>/dev/null \ + | grep -q "\"id\":\"$SERVED_NAME\""; then + return 0 + fi + if ! docker ps --filter "name=^${name}$" --format '{{.Names}}' | grep -q "$name"; then + echo "container $name died" >&2 + docker logs --tail 80 "$name" >&2 || true + return 1 + fi + sleep 2 + done + echo "timeout waiting for $name to be ready (served=$SERVED_NAME)" >&2 + docker logs --tail 80 "$name" >&2 || true + return 1 +} + +start_mem_sampler() { + local stack="$1" + local container="$2" + local out="$RESULTS_DIR/$stack/mem.tsv" + : > "$out" + ( + while :; do + local ts vram cmem cgstat amib fmib smib + ts=$(date +%s) + # || true: under pipefail+set -e a failing nvidia-smi/docker stats + # would otherwise abort this backgrounded subshell and silently stop + # sampling. Empty values fall back to 0 in the printf below. On a + # CPU-only host nvidia-smi is simply absent, so vram stays 0 — the + # sampler itself needs no device-aware branching. + vram=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d ' ') || true + # Mem usage like "1.234GiB / 64GiB" — take first field, normalize to + # MiB. Handle binary (GiB/MiB/KiB) and decimal (GB/MB/KB) units, + # any case, with or without a space before the unit; unknown units + # fall back to assuming the value is already in MiB. + cmem=$(docker stats --no-stream --format '{{.MemUsage}}' "$container" 2>/dev/null \ + | awk -F'/' '{print $1}' \ + | awk '{ + s=$0; gsub(/[[:space:]]/,"",s); # e.g. "1.234GiB" + num=s; unit=s; + sub(/[A-Za-z]+$/,"",num); # numeric part + sub(/^[0-9.]+/,"",unit); # unit part + U=toupper(unit); + base=(U ~ /I/)?1024:1000; # *iB binary, *B decimal + p=substr(U,1,1); + if (p=="T") mib=num*base*base*base*base/1048576; + else if (p=="G") mib=num*base*base*base/1048576; + else if (p=="M") mib=num*base*base/1048576; + else if (p=="K") mib=num*base/1048576; + else if (U=="B") mib=num/1048576; + else mib=num; # unknown/unitless: assume MiB + printf "%.1f", mib + }') || true + # cgroup memory.stat breakdown (bytes→MiB), sampled for *both* stacks + # so the peak-RSS gap can be attributed: anon = real process RSS, + # shmem = tmpfs/plasma (Ray's object store — charged to the cgroup but + # to no single process, so the per-component table never sees it), + # file = reclaimable page cache. cgroup v2 path; if absent (v1/missing) + # the awk END still emits zeros. + cgstat=$(docker exec "$container" cat /sys/fs/cgroup/memory.stat 2>/dev/null) || true + read -r amib fmib smib < <(printf '%s\n' "$cgstat" | awk ' + $1=="anon"{a=$2} $1=="file"{f=$2} $1=="shmem"{s=$2} + END {printf "%.1f %.1f %.1f", a/1048576, f/1048576, s/1048576}') || true + printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ + "${ts:-0}" "${vram:-0}" "${cmem:-0}" "${amib:-0}" "${fmib:-0}" "${smib:-0}" >> "$out" + sleep 1 + done + ) & + MEM_SAMPLER_PID=$! +} + +stop_mem_sampler() { + if [[ -n "${MEM_SAMPLER_PID:-}" ]] && kill -0 "$MEM_SAMPLER_PID" 2>/dev/null; then + kill "$MEM_SAMPLER_PID" 2>/dev/null || true + wait "$MEM_SAMPLER_PID" 2>/dev/null || true + fi + MEM_SAMPLER_PID="" +} + +# Sample the Ray reporter's per-component memory (port 8079) *during* the sweep +# and keep the scrape with the highest total private memory, so the breakdown +# reflects peak load instead of the idle post-sweep state. The reporter refreshes +# its gauges on its own interval; polling at 2s catches every refresh over a +# multi-minute phase. modelship-only — the baseline stack has no Ray reporter. +# Loader-agnostic: the Ray Serve control plane emits these regardless of which +# loader (vllm / llama_server) the deployment wraps. Router / request histograms +# are cumulative and still scraped once at the end. +start_component_sampler() { + local out="$1" + : > "$out" + ( + local best=-1 comp score + while :; do + # || true: a failed scrape under pipefail+set -e must not kill the loop. + comp=$(curl -fsS http://localhost:8079/metrics 2>/dev/null \ + | awk '/^ray_component_(uss_mb|rss_mb|mem_shared_bytes)[{ ]/') || true + if [[ -n "$comp" ]]; then + # Score = total private (USS) across components. $NF is the metric + # value — robust to spaces inside Component label values (e.g. + # "ray::ServeReplica:modelship api:modelship api"). + score=$(printf '%s\n' "$comp" \ + | awk '/^ray_component_uss_mb[{ ]/ {s+=$NF} END {printf "%.0f", s+0}') + if [[ -n "$score" ]] && (( score > best )); then + best=$score + printf '%s\n' "$comp" > "$out" + fi + fi + sleep 2 + done + ) & + COMPONENT_SAMPLER_PID=$! +} + +stop_component_sampler() { + if [[ -n "${COMPONENT_SAMPLER_PID:-}" ]] && kill -0 "$COMPONENT_SAMPLER_PID" 2>/dev/null; then + kill "$COMPONENT_SAMPLER_PID" 2>/dev/null || true + wait "$COMPONENT_SAMPLER_PID" 2>/dev/null || true + fi + COMPONENT_SAMPLER_PID="" +} + +# No-op (returns immediately) on a host with no nvidia-smi — there is no VRAM +# to wait on between phases on a CPU-only run. +vram_gate() { + if ! command -v nvidia-smi >/dev/null 2>&1; then + return 0 + fi + local deadline=$(( $(date +%s) + 60 )) + while (( $(date +%s) < deadline )); do + local used + # tr -dc digits → "" when nvidia-smi errors or prints non-numeric + # output; || true keeps pipefail+set -e from aborting the run. Guard + # the arithmetic so an empty operand isn't a syntax error. + used=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -dc '0-9') || true + if [[ -n "$used" ]] && (( used < 500 )); then return 0; fi + sleep 1 + done + echo "warn: VRAM not freed within 60s" >&2 +} + +# Run REPEATS timed sweeps against an already-ready stack, saving each to its +# own result_.json. The summary takes the median across them. +run_stack() { + local stack="$1" + local out_dir="$RESULTS_DIR/$stack" + mkdir -p "$out_dir" + chmod 777 "$out_dir" + for i in $(seq 1 "$REPEATS"); do + echo " sweep $i/$REPEATS ($stack)..." + run_sweep "$stack" "result_${i}.json" + done +} + +run_sweep() { + local stack="$1" + local fname="$2" + local out_dir="$RESULTS_DIR/$stack" + docker run --rm --network host -v "$out_dir:/out:rw" "$IMAGE" \ + bash -lc "cd /modelship && uv run --active --no-sync vllm bench serve \ + --backend openai-chat \ + --base-url http://localhost:8000 \ + --endpoint /v1/chat/completions \ + --model $SERVED_NAME \ + --tokenizer $MODEL_ID \ + --dataset-name random \ + --random-input-len $INPUT_LEN \ + --random-output-len $OUTPUT_LEN \ + --num-prompts $NUM_PROMPTS \ + --max-concurrency $CONCURRENCY \ + --num-warmups $NUM_WARMUPS \ + --ignore-eos \ + --save-result \ + --result-dir /out \ + --result-filename $fname" +} + +scrape_prom() { + local out="$1" + # Router / request histograms only — these are cumulative counters, so a + # single end-of-sweep scrape is correct. Per-component *memory* is a gauge + # that varies with load and is captured under load by start_component_sampler, + # not here. || true: an empty scrape must not abort the run under pipefail. + curl -fsS http://localhost:8079/metrics 2>/dev/null \ + | awk '/^ray_modelship_(request|generation)_duration_seconds_(sum|count)/ \ + || /^ray_serve_request_router_fulfillment_time_ms_(sum|count)/' \ + > "$out" || true +} + +# Renders the shared summary.md body: latency/throughput table (median across +# repeats), memory table (peak across the sweep), modelship per-component +# memory breakdown + cgroup cross-check, and the router-fulfillment Prometheus +# figure. $1/$2 are the stack directory names under $RESULTS_DIR (modelship +# phase first), used only as labels in the printed tables. +write_summary() { + local modelship_stack="$1" baseline_stack="$2" baseline_label="$3" + echo "| metric | modelship | $baseline_label | overhead |" + echo "| --- | ---: | ---: | ---: |" + python3 - "$RESULTS_DIR" "$modelship_stack" "$baseline_stack" <<'PY' +import json, sys, statistics +from pathlib import Path +root = Path(sys.argv[1]) +def load(stack): + # One result_.json per repeat; return them all so we can take medians. + runs = [json.loads(p.read_text()) for p in sorted((root / stack).glob("result_*.json"))] + if not runs: + sys.exit(f"no result_*.json found for {stack}") + return runs +def med(runs, key): + vals = [r[key] for r in runs if r.get(key) is not None] + return statistics.median(vals) if vals else None +m = load(sys.argv[2]); r = load(sys.argv[3]) +keys = [ + ("request_throughput", "req/s", 3), + ("output_throughput", "output tok/s", 2), + ("mean_ttft_ms", "TTFT mean (ms)", 1), + ("p50_ttft_ms", "TTFT p50 (ms)", 1), + ("p95_ttft_ms", "TTFT p95 (ms)", 1), + ("mean_itl_ms", "ITL mean (ms)", 2), + ("p95_itl_ms", "ITL p95 (ms)", 2), + ("mean_e2el_ms", "E2E mean (ms)", 1), + ("p50_e2el_ms", "E2E p50 (ms)", 1), + ("p95_e2el_ms", "E2E p95 (ms)", 1), +] +for key, label, prec in keys: + mv = med(m, key); rv = med(r, key) + if mv is None or rv is None: + continue + if rv == 0: + ratio = "—" + else: + ratio = f"{(mv - rv) / rv * 100:+.1f}%" + print(f"| {label} | {mv:.{prec}f} | {rv:.{prec}f} | {ratio} |") +PY + echo + echo "## memory (peak across all sweeps)" + python3 - "$RESULTS_DIR" "$modelship_stack" "$baseline_stack" "$baseline_label" <<'PY' +import sys +from pathlib import Path +root = Path(sys.argv[1]) +# mem.tsv columns: ts, vram, container_rss, anon, file(cache), shmem — all MiB +# except ts. Older runs only have the first 3; missing columns peak at 0. +COLS = ["vram", "rss", "anon", "file", "shmem"] +def peak(stack): + f = root / stack / "mem.tsv" + if not f.exists(): + return None + peaks = {c: 0.0 for c in COLS} + for line in f.read_text().splitlines(): + parts = line.split("\t") + for i, c in enumerate(COLS, start=1): + if i < len(parts): + try: + peaks[c] = max(peaks[c], float(parts[i])) + except ValueError: + pass + return peaks +m = peak(sys.argv[2]); r = peak(sys.argv[3]) +label = sys.argv[4] +print(f"| metric | modelship | {label} | overhead |") +print("| --- | ---: | ---: | ---: |") +def row(name, key, unit="MiB"): + if m is None or r is None: + return + mv, rv = m[key], r[key] + delta = mv - rv + pct = f"{(delta / rv * 100):+.1f}%" if rv else "—" + print(f"| {name} | {mv:.0f} {unit} | {rv:.0f} {unit} | {delta:+.0f} {unit} ({pct}) |") +row("peak VRAM (GPU0)", "vram") +row("peak container RSS", "rss") +row(" ├─ anon (process RSS)", "anon") +row(" ├─ shmem (tmpfs/plasma)", "shmem") +row(" └─ file (page cache)", "file") +print() +print("_**anon** is the real RAM overhead. **file** (page cache) is reclaimable " + "and non-deterministic — it depends on which cgroup first faulted the weights " + "and can swing GB between runs, so the container-RSS delta over- or " + "under-states the true cost. Each row is an independent peak (different " + "instants), so the sub-rows need not sum to peak RSS._") +PY + echo + echo "## modelship per-component memory (Ray reporter, peak under load)" + echo + # Breaks the modelship container's RSS down by Ray process so we can see + # whether the overhead lives in the model-serving actor (ray::*Deployment* — + # we'd serve differently than the baseline) or the control plane (gcs_server / + # raylet / agent / ProxyActor / ServeController — fixed Ray cost). USS is + # private memory; shared is shared *libraries* (torch/CUDA, mapped into every + # worker — NOT plasma) reported separately so it isn't charged to any one + # actor. Snapshot is the peak-private scrape sampled *during* the sweep + # (start_component_sampler), not the idle post-sweep state. The reconciliation + # below shows this table undercounts the true total (misses non-Ray children). + if [[ -s "$RESULTS_DIR/$modelship_stack/components.txt" ]]; then + python3 - "$RESULTS_DIR" "$modelship_stack" <<'PY' +import sys, re +from pathlib import Path +root = Path(sys.argv[1]) +pat = re.compile(r'^(ray_component_(?:uss_mb|rss_mb|mem_shared_bytes))\{([^}]*)\}\s+([0-9eE+.\-]+)') +key = {"ray_component_uss_mb": "uss", "ray_component_rss_mb": "rss", "ray_component_mem_shared_bytes": "shared"} +comp: dict[str, dict[str, float]] = {} +for line in (root / sys.argv[2] / "components.txt").read_text().splitlines(): + m = pat.match(line) + if not m: + continue + metric, labels, val = m.group(1), m.group(2), float(m.group(3)) + name = dict(re.findall(r'(\w+)="([^"]*)"', labels)).get("Component", "?") + d = comp.setdefault(name, {}) + # Ray emits rss/uss in MB (bytes/1e6); shared is raw bytes — normalize to MB. + v = val / 1e6 if metric == "ray_component_mem_shared_bytes" else val + d[key[metric]] = d.get(key[metric], 0.0) + v +# Private = USS when the agent could read it, else RSS - shared as a floor. +def private(d): + return d["uss"] if "uss" in d else max(d.get("rss", 0.0) - d.get("shared", 0.0), 0.0) +rows = sorted(comp.items(), key=lambda kv: private(kv[1]), reverse=True) +print("| component | private (MB) | rss (MB) | shared (MB) |") +print("| --- | ---: | ---: | ---: |") +tot_priv = tot_rss = tot_shared = 0.0 +for name, d in rows: + p, r, s = private(d), d.get("rss", 0.0), d.get("shared", 0.0) + tot_priv += p; tot_rss += r; tot_shared += s + print(f"| `{name}` | {p:.0f} | {r:.0f} | {s:.0f} |") +print(f"| **total** | **{tot_priv:.0f}** | **{tot_rss:.0f}** | **{tot_shared:.0f}** |") +if not any("uss" in d for _, d in rows): + print() + print("_USS unavailable (reporter couldn't read smaps); private column is " + "rss − shared, an upper bound._") + +# Cross-check the Ray reporter (Prometheus) against the kernel's own accounting +# (cgroup memory.stat, peak under load). The reporter samples /proc smaps on its +# own interval and can be stale or miss workers; cgroup is ground truth. anon ≈ +# Σ private, shmem ≈ Σ shared. Peaks are sampled independently so expect rough, +# not exact, agreement — a large gap means the per-component table is suspect. +def cgroup_peak(col): # mem.tsv: ts,vram,rss,anon,file,shmem (MiB) + f = root / sys.argv[2] / "mem.tsv" + if not f.exists(): + return None + peak = 0.0 + for line in f.read_text().splitlines(): + parts = line.split("\t") + if len(parts) > col: + try: + peak = max(peak, float(parts[col])) + except ValueError: + pass + return peak +anon = cgroup_peak(3) # MiB ≈ MB for this sanity check +if anon: + print() + gap = (tot_priv - anon) / anon * 100 + flag = " ⚠️ diverges" if abs(gap) > 25 else "" + print("_Reporter cross-check vs cgroup `memory.stat` (kernel ground truth, peak):_") + print(f"- private: reporter Σ USS **{tot_priv:.0f} MB** vs cgroup anon " + f"**{anon:.0f} MB** ({gap:+.0f}%){flag}") + if gap < -25: + print(" - reporter undercounts — it sees only Ray worker PIDs, so memory in " + "non-Ray child processes (notably the loader's own inference subprocess/engine) " + "is missing. Trust the cgroup figure; treat the per-component split as " + "relative attribution, not an absolute total.") + # NOTE: Ray's mem_shared_bytes is shared *libraries* (torch/CUDA, PSS-shared), + # not plasma — so it has no clean cgroup counterpart and is deliberately not + # reconciled. Actual tmpfs/plasma is cgroup `shmem` (see the memory table); it + # is tiny here, confirming the streaming path barely touches the object store. +PY + else + echo "_no component metrics scraped (reporter agent down or 8079 unreachable)_" + fi + echo + echo "## modelship internal (Prometheus)" + echo + # NOTE: modelship_request_duration_seconds and modelship_generation_duration_seconds + # are observed when the streaming generator is *created*, not after it drains + # (model_deployment.py:230, api.py:347), so for streaming responses they capture + # setup/TTFT only — not end-to-end or full-generation time. We therefore do NOT + # derive "gateway overhead" from them (it's meaningless and was wildly wrong). + # Only the router fulfillment time below reflects real request handling. + if [[ -s "$RESULTS_DIR/$modelship_stack/prom.txt" ]]; then + python3 - "$RESULTS_DIR/$modelship_stack/prom.txt" <<'PY' +import sys, re +sums = {}; counts = {} +pat = re.compile( + r'(ray_serve_request_router_fulfillment_time_ms)' + r'_(sum|count)\S*\s+([0-9eE+\-.]+)' +) +for line in open(sys.argv[1]): + m = pat.match(line) + if not m: continue + name, kind, val = m.group(1), m.group(2), float(m.group(3)) + (sums if kind == "sum" else counts).setdefault(name, 0.0) + if kind == "sum": sums[name] += val + else: counts[name] += val +n = "ray_serve_request_router_fulfillment_time_ms" +cnt = counts.get(n, 0.0) +if cnt: + print(f"- mean router fulfillment (routing + queue wait): **{sums.get(n, 0.0) / cnt:.1f} ms** " + f"over {cnt:.0f} routed requests") +else: + print("- no router metrics scraped") +print() +print("_E2E / engine durations omitted: their histograms are recorded before " + "streaming completes and are not meaningful for streaming responses._") +PY + else + echo "_no metrics scraped_" + fi +} diff --git a/bench/rawllama_entrypoint.py b/bench/rawllama_entrypoint.py new file mode 100644 index 0000000..95947a5 --- /dev/null +++ b/bench/rawllama_entrypoint.py @@ -0,0 +1,97 @@ +"""Run `llama-server` directly using the same models.yaml modelship reads. + +Mounted-only entrypoint — bypasses ray + modelship pipeline so a benchmark can +A/B the modelship llama_server loader against vanilla llama-server with an +identical launch command and the identical llama-server binary that ships in +the image (MSHIP_LLAMA_SERVER_BIN). Mirrors the flag-building logic in +modelship/infer/llama_server/llama_server_infer.py's `_launch` — see that +module for the source of truth this must stay in sync with. +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +import yaml + +from modelship.infer.infer_config import LlamaServerConfig, ModelLoader, ModelshipConfig, ModelUsecase +from modelship.infer.model_resolver import resolve_model_source + +CONFIG_PATH = Path(os.environ.get("MSHIP_CONFIG", "/modelship/config/models.yaml")) + + +def main() -> int: + binary = os.environ.get("MSHIP_LLAMA_SERVER_BIN") + if not binary or not os.path.isfile(binary): + print(f"MSHIP_LLAMA_SERVER_BIN must point at a llama-server executable; got {binary!r}", file=sys.stderr) + return 2 + + raw = yaml.safe_load(CONFIG_PATH.read_text()) + cfg = ModelshipConfig.model_validate(raw) + llama_models = [m for m in cfg.models if m.loader == ModelLoader.llama_server] + if len(llama_models) != 1: + print( + f"bench expects exactly one llama_server model in {CONFIG_PATH}, got {len(llama_models)}", file=sys.stderr + ) + return 2 + + m = llama_models[0] + k = m.llama_server_config or LlamaServerConfig() + + model_path = resolve_model_source(m.model) + print(f"rawllama resolved model -> {model_path}", flush=True) + + args = [ + binary, + "--host", + "0.0.0.0", + "--port", + "8000", + "-m", + model_path, + "-c", + str(k.n_ctx * k.parallel), + "-b", + str(k.n_batch), + "--parallel", + str(k.parallel), + "--jinja", + "--reasoning-format", + "auto", + "--no-webui", + # Reported as the "id" in /v1/models — lets the harness's wait_ready + # (which greps for the served name) work identically to the modelship + # phase. Vanilla llama-server has no auth by default; skipping + # --api-key here matches an out-of-the-box `llama-server` invocation. + "--alias", + m.name, + ] + # Same branch as LlamaServerInfer._launch: Ray only sets CUDA_VISIBLE_DEVICES + # for actors that reserve GPUs, so a num_gpus=0 deploy may still see every + # GPU — force no offload rather than trusting the container's device + # visibility. Mirrored here so the raw phase can't accidentally offload + # when the modelship phase wouldn't. + if m.num_gpus > 0: + args += ["-ngl", str(k.n_gpu_layers)] + else: + args += ["-ngl", "0"] + if k.threads is not None: + args += ["--threads", str(k.threads)] + if k.chat_template: + flag = "--chat-template-file" if os.path.isfile(k.chat_template) else "--chat-template" + args += [flag, k.chat_template] + if k.mmproj: + mmproj_path = resolve_model_source(k.mmproj) + args += ["--mmproj", mmproj_path] + if m.usecase == ModelUsecase.embed: + args += ["--embedding"] + args += list(k.extra_args) + + print("rawllama exec:", " ".join(args), flush=True) + os.execvp(args[0], args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/run.sh b/bench/run.sh index 81d6be6..3ce2702 100755 --- a/bench/run.sh +++ b/bench/run.sh @@ -1,17 +1,25 @@ #!/usr/bin/env bash -# A/B benchmark: modelship vLLM loader vs raw vLLM, same image, same config. -# Usage: bench/run.sh [--image TAG] [--num-prompts N] [--concurrency N] [--input-len N] [--output-len N] -# [--num-warmups N] [--repeats N] +# A/B benchmark: a modelship loader vs the vanilla inference server it wraps, +# same image, same engine config. Answers "how much does modelship's wrapping +# cost?" — not a vllm-vs-llama.cpp comparison, so --loader picks which wrapped +# stack to measure, and --device picks CPU vs GPU for that stack. +# +# Usage: bench/run.sh [--loader vllm|llama_server] [--device gpu|cpu] [--image TAG] +# [--config PATH] [--num-prompts N] [--concurrency N] +# [--input-len N] [--output-len N] [--num-warmups N] [--repeats N] set -euo pipefail -IMAGE="${IMAGE:-modelship:dev}" +LOADER="vllm" +DEVICE="gpu" +IMAGE="" +CONFIG="" NUM_PROMPTS=100 CONCURRENCY=8 INPUT_LEN=128 OUTPUT_LEN=512 # Warmup requests sent (and discarded) before timing begins. Warms CUDA graph -# capture / lazy compilation so the first few real requests don't eat a -# multi-second cold-start tail that skews mean/p99 TTFT and throughput. +# capture / lazy compilation (GPU) or first-request JIT paths so the first few +# real requests don't eat a cold-start tail that skews mean/p99 TTFT and throughput. NUM_WARMUPS=20 # Timed sweeps per stack. We report the median across repeats so a single cold # or noisy run can't dominate; tail metrics on one ~100-prompt run are unstable. @@ -20,7 +28,10 @@ READY_TIMEOUT=900 while [[ $# -gt 0 ]]; do case "$1" in + --loader) LOADER="$2"; shift 2 ;; + --device) DEVICE="$2"; shift 2 ;; --image) IMAGE="$2"; shift 2 ;; + --config) CONFIG="$2"; shift 2 ;; --num-prompts) NUM_PROMPTS="$2"; shift 2 ;; --concurrency) CONCURRENCY="$2"; shift 2 ;; --input-len) INPUT_LEN="$2"; shift 2 ;; @@ -31,8 +42,39 @@ while [[ $# -gt 0 ]]; do esac done +case "$LOADER" in + vllm|llama_server) ;; + *) echo "--loader must be vllm or llama_server, got: $LOADER" >&2; exit 2 ;; +esac +case "$DEVICE" in + gpu|cpu) ;; + *) echo "--device must be gpu or cpu, got: $DEVICE" >&2; exit 2 ;; +esac + REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" BENCH_DIR="$REPO_ROOT/bench" +# shellcheck source=bench/lib.sh +source "$BENCH_DIR/lib.sh" + +# modelship's own CLAUDE.md/Dockerfile convention: gpu/cpu are separate image +# variants (mutually exclusive extras, different vllm wheel index), and the +# published image tags a cpu build with a "-cpu" suffix. +if [[ -z "$IMAGE" ]]; then + IMAGE="modelship:dev" + [[ "$DEVICE" == "cpu" ]] && IMAGE="modelship:dev-cpu" +fi + +CONFIG_PREFIX="vllm" +BASELINE_ENTRYPOINT="rawvllm_entrypoint.py" +BASELINE_LABEL="raw vllm" +if [[ "$LOADER" == "llama_server" ]]; then + CONFIG_PREFIX="llama" + BASELINE_ENTRYPOINT="rawllama_entrypoint.py" + BASELINE_LABEL="vanilla llama-server" +fi +[[ -n "$CONFIG" ]] || CONFIG="$BENCH_DIR/configs/${CONFIG_PREFIX}-${DEVICE}.yaml" +[[ -f "$CONFIG" ]] || { echo "config not found: $CONFIG" >&2; exit 2; } + TS="$(date -u +%Y%m%dT%H%M%SZ)" RESULTS_DIR="$BENCH_DIR/results/$TS" mkdir -p "$RESULTS_DIR" @@ -40,492 +82,85 @@ mkdir -p "$RESULTS_DIR" CACHE_DIR="${MSHIP_CACHE_DIR:-$REPO_ROOT/models-cache}" mkdir -p "$CACHE_DIR" -# Extract the first scalar matching a key regex from bench.yaml. Tolerates -# double-quoted, single-quoted, and unquoted values: strips everything up to -# the first colon, trims surrounding whitespace, then removes a matched pair of -# surrounding quotes (only when both ends use the same quote char). -yaml_scalar() { - grep -m1 -E "$1" "$BENCH_DIR/configs/bench.yaml" \ - | sed -E "s/^[^:]*:[[:space:]]*//; s/[[:space:]]*\$//; s/^(['\"])(.*)\1\$/\2/" -} -SERVED_NAME="$(yaml_scalar '^[[:space:]]*-[[:space:]]*name:')" -MODEL_ID="$(yaml_scalar '^[[:space:]]*model:')" -[[ -n "$MODEL_ID" && -n "$SERVED_NAME" ]] || { echo "failed to parse bench.yaml" >&2; exit 2; } +SERVED_NAME="$(yaml_scalar '^[[:space:]]*-[[:space:]]*name:' "$CONFIG")" +MODEL_ID="$(yaml_scalar '^[[:space:]]*model:' "$CONFIG")" +[[ -n "$MODEL_ID" && -n "$SERVED_NAME" ]] || { echo "failed to parse $CONFIG" >&2; exit 2; } -cleanup() { - for pid in "${MEM_SAMPLER_PID:-}" "${COMPONENT_SAMPLER_PID:-}"; do - if [[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null; then - kill "$pid" 2>/dev/null || true - wait "$pid" 2>/dev/null || true - fi - done - for c in bench-modelship bench-rawvllm; do - if docker inspect "$c" >/dev/null 2>&1; then - docker logs "$c" >"$RESULTS_DIR/${c}.container.log" 2>&1 || true - docker rm -f "$c" >/dev/null 2>&1 || true - fi - done -} +MODELSHIP_CONTAINER=bench-modelship +BASELINE_CONTAINER=bench-baseline trap cleanup EXIT # Defensive: remove any pre-existing bench containers from a prior aborted run. -docker rm -f bench-modelship bench-rawvllm >/dev/null 2>&1 || true - -wait_ready() { - local name="$1" - local deadline=$(( $(date +%s) + READY_TIMEOUT )) - while (( $(date +%s) < deadline )); do - # /v1/models reachable AND lists the served model id - if curl -fsS http://localhost:8000/v1/models 2>/dev/null \ - | grep -q "\"id\":\"$SERVED_NAME\""; then - return 0 - fi - if ! docker ps --filter "name=^${name}$" --format '{{.Names}}' | grep -q "$name"; then - echo "container $name died" >&2 - docker logs --tail 80 "$name" >&2 || true - return 1 - fi - sleep 2 - done - echo "timeout waiting for $name to be ready (served=$SERVED_NAME)" >&2 - docker logs --tail 80 "$name" >&2 || true - return 1 -} - -start_mem_sampler() { - local stack="$1" - local container="$2" - local out="$RESULTS_DIR/$stack/mem.tsv" - : > "$out" - ( - while :; do - local ts vram cmem cgstat amib fmib smib - ts=$(date +%s) - # || true: under pipefail+set -e a failing nvidia-smi/docker stats - # would otherwise abort this backgrounded subshell and silently stop - # sampling. Empty values fall back to 0 in the printf below. - vram=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d ' ') || true - # Mem usage like "1.234GiB / 64GiB" — take first field, normalize to - # MiB. Handle binary (GiB/MiB/KiB) and decimal (GB/MB/KB) units, - # any case, with or without a space before the unit; unknown units - # fall back to assuming the value is already in MiB. - cmem=$(docker stats --no-stream --format '{{.MemUsage}}' "$container" 2>/dev/null \ - | awk -F'/' '{print $1}' \ - | awk '{ - s=$0; gsub(/[[:space:]]/,"",s); # e.g. "1.234GiB" - num=s; unit=s; - sub(/[A-Za-z]+$/,"",num); # numeric part - sub(/^[0-9.]+/,"",unit); # unit part - U=toupper(unit); - base=(U ~ /I/)?1024:1000; # *iB binary, *B decimal - p=substr(U,1,1); - if (p=="T") mib=num*base*base*base*base/1048576; - else if (p=="G") mib=num*base*base*base/1048576; - else if (p=="M") mib=num*base*base/1048576; - else if (p=="K") mib=num*base/1048576; - else if (U=="B") mib=num/1048576; - else mib=num; # unknown/unitless: assume MiB - printf "%.1f", mib - }') || true - # cgroup memory.stat breakdown (bytes→MiB), sampled for *both* stacks - # so the peak-RSS gap can be attributed: anon = real process RSS, - # shmem = tmpfs/plasma (Ray's object store — charged to the cgroup but - # to no single process, so the per-component table never sees it), - # file = reclaimable page cache. cgroup v2 path; if absent (v1/missing) - # the awk END still emits zeros. - cgstat=$(docker exec "$container" cat /sys/fs/cgroup/memory.stat 2>/dev/null) || true - read -r amib fmib smib < <(printf '%s\n' "$cgstat" | awk ' - $1=="anon"{a=$2} $1=="file"{f=$2} $1=="shmem"{s=$2} - END {printf "%.1f %.1f %.1f", a/1048576, f/1048576, s/1048576}') || true - printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ - "${ts:-0}" "${vram:-0}" "${cmem:-0}" "${amib:-0}" "${fmib:-0}" "${smib:-0}" >> "$out" - sleep 1 - done - ) & - MEM_SAMPLER_PID=$! -} - -stop_mem_sampler() { - if [[ -n "${MEM_SAMPLER_PID:-}" ]] && kill -0 "$MEM_SAMPLER_PID" 2>/dev/null; then - kill "$MEM_SAMPLER_PID" 2>/dev/null || true - wait "$MEM_SAMPLER_PID" 2>/dev/null || true - fi - MEM_SAMPLER_PID="" -} +docker rm -f "$MODELSHIP_CONTAINER" "$BASELINE_CONTAINER" >/dev/null 2>&1 || true -# Sample the Ray reporter's per-component memory (port 8079) *during* the sweep -# and keep the scrape with the highest total private memory, so the breakdown -# reflects peak load instead of the idle post-sweep state. The reporter refreshes -# its gauges on its own interval; polling at 2s catches every refresh over a -# multi-minute phase. modelship-only — raw vLLM has no Ray reporter. Router / -# request histograms are cumulative and still scraped once at the end. -start_component_sampler() { - local out="$1" - : > "$out" - ( - local best=-1 comp score - while :; do - # || true: a failed scrape under pipefail+set -e must not kill the loop. - comp=$(curl -fsS http://localhost:8079/metrics 2>/dev/null \ - | awk '/^ray_component_(uss_mb|rss_mb|mem_shared_bytes)[{ ]/') || true - if [[ -n "$comp" ]]; then - # Score = total private (USS) across components. $NF is the metric - # value — robust to spaces inside Component label values (e.g. - # "ray::ServeReplica:modelship api:modelship api"). - score=$(printf '%s\n' "$comp" \ - | awk '/^ray_component_uss_mb[{ ]/ {s+=$NF} END {printf "%.0f", s+0}') - if [[ -n "$score" ]] && (( score > best )); then - best=$score - printf '%s\n' "$comp" > "$out" - fi - fi - sleep 2 - done - ) & - COMPONENT_SAMPLER_PID=$! -} - -stop_component_sampler() { - if [[ -n "${COMPONENT_SAMPLER_PID:-}" ]] && kill -0 "$COMPONENT_SAMPLER_PID" 2>/dev/null; then - kill "$COMPONENT_SAMPLER_PID" 2>/dev/null || true - wait "$COMPONENT_SAMPLER_PID" 2>/dev/null || true - fi - COMPONENT_SAMPLER_PID="" -} - -vram_gate() { - local deadline=$(( $(date +%s) + 60 )) - while (( $(date +%s) < deadline )); do - local used - # tr -dc digits → "" when nvidia-smi is missing, errors, or prints - # non-numeric output; || true keeps pipefail+set -e from aborting the - # run. Guard the arithmetic so an empty operand isn't a syntax error. - used=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -dc '0-9') || true - if [[ -n "$used" ]] && (( used < 500 )); then return 0; fi - sleep 1 - done - echo "warn: VRAM not freed within 60s" >&2 -} - -run_sweep() { - local stack="$1" - local fname="$2" - local out_dir="$RESULTS_DIR/$stack" - docker run --rm --network host -v "$out_dir:/out:rw" "$IMAGE" \ - bash -lc "cd /modelship && uv run --active --no-sync vllm bench serve \ - --backend openai-chat \ - --base-url http://localhost:8000 \ - --endpoint /v1/chat/completions \ - --model $SERVED_NAME \ - --tokenizer $MODEL_ID \ - --dataset-name random \ - --random-input-len $INPUT_LEN \ - --random-output-len $OUTPUT_LEN \ - --num-prompts $NUM_PROMPTS \ - --max-concurrency $CONCURRENCY \ - --num-warmups $NUM_WARMUPS \ - --ignore-eos \ - --save-result \ - --result-dir /out \ - --result-filename $fname" -} - -# Run REPEATS timed sweeps against an already-ready stack, saving each to its -# own result_.json. The summary takes the median across them. -run_stack() { - local stack="$1" - local out_dir="$RESULTS_DIR/$stack" - mkdir -p "$out_dir" - chmod 777 "$out_dir" - for i in $(seq 1 "$REPEATS"); do - echo " sweep $i/$REPEATS ($stack)..." - run_sweep "$stack" "result_${i}.json" - done -} +DOCKER_GPU_ARGS=() +[[ "$DEVICE" == "gpu" ]] && DOCKER_GPU_ARGS=(--gpus all) start_modelship() { # Mount local source over the prebuilt image so the bench exercises the # working tree. mship_deploy.py is the entry point invoked by the image's # default CMD (scripts/start.sh). - docker run -d --gpus all --ipc=host --network host \ + docker run -d "${DOCKER_GPU_ARGS[@]}" --ipc=host --network host \ -e MSHIP_METRICS=true \ -e MSHIP_GATEWAY_REPLICAS="${MSHIP_GATEWAY_REPLICAS:-1}" \ -e MSHIP_GATEWAY_MAX_ONGOING="${MSHIP_GATEWAY_MAX_ONGOING:-1024}" \ - -v "$BENCH_DIR/configs/bench.yaml:/modelship/config/models.yaml:ro" \ + -v "$CONFIG:/modelship/config/models.yaml:ro" \ -v "$REPO_ROOT/mship_deploy.py:/modelship/mship_deploy.py:ro" \ -v "$REPO_ROOT/modelship:/modelship/modelship:ro" \ -v "$CACHE_DIR:/.cache:rw" \ - --name bench-modelship "$IMAGE" >/dev/null + --name "$MODELSHIP_CONTAINER" "$IMAGE" >/dev/null } -start_rawvllm() { - docker run -d --gpus all --ipc=host --network host \ +start_baseline() { + docker run -d "${DOCKER_GPU_ARGS[@]}" --ipc=host --network host \ -e PYTHONPATH=/modelship \ - -v "$BENCH_DIR/configs/bench.yaml:/modelship/config/models.yaml:ro" \ - -v "$BENCH_DIR/rawvllm_entrypoint.py:/modelship/bench/rawvllm_entrypoint.py:ro" \ + -v "$CONFIG:/modelship/config/models.yaml:ro" \ + -v "$BENCH_DIR/$BASELINE_ENTRYPOINT:/modelship/bench/$BASELINE_ENTRYPOINT:ro" \ -v "$CACHE_DIR:/.cache:rw" \ -w /modelship \ --entrypoint /.venv/bin/python \ - --name bench-rawvllm "$IMAGE" \ - /modelship/bench/rawvllm_entrypoint.py >/dev/null -} - -scrape_prom() { - local out="$1" - # Router / request histograms only — these are cumulative counters, so a - # single end-of-sweep scrape is correct. Per-component *memory* is a gauge - # that varies with load and is captured under load by start_component_sampler, - # not here. || true: an empty scrape must not abort the run under pipefail. - curl -fsS http://localhost:8079/metrics 2>/dev/null \ - | awk '/^ray_modelship_(request|generation)_duration_seconds_(sum|count)/ \ - || /^ray_serve_request_router_fulfillment_time_ms_(sum|count)/' \ - > "$out" || true + --name "$BASELINE_CONTAINER" "$IMAGE" \ + "/modelship/bench/$BASELINE_ENTRYPOINT" >/dev/null } -echo "=== bench $TS — image=$IMAGE prompts=$NUM_PROMPTS conc=$CONCURRENCY in=$INPUT_LEN out=$OUTPUT_LEN warmups=$NUM_WARMUPS repeats=$REPEATS ===" +echo "=== bench $TS — loader=$LOADER device=$DEVICE image=$IMAGE config=$(basename "$CONFIG") prompts=$NUM_PROMPTS conc=$CONCURRENCY in=$INPUT_LEN out=$OUTPUT_LEN warmups=$NUM_WARMUPS repeats=$REPEATS ===" # Phase A — modelship echo "[A] starting modelship..." start_modelship -wait_ready bench-modelship +wait_ready "$MODELSHIP_CONTAINER" echo "[A] running $REPEATS sweep(s)..." mkdir -p "$RESULTS_DIR/modelship" -start_mem_sampler modelship bench-modelship +start_mem_sampler modelship "$MODELSHIP_CONTAINER" start_component_sampler "$RESULTS_DIR/modelship/components.txt" run_stack modelship stop_mem_sampler stop_component_sampler scrape_prom "$RESULTS_DIR/modelship/prom.txt" -docker rm -f bench-modelship >/dev/null +docker rm -f "$MODELSHIP_CONTAINER" >/dev/null vram_gate -# Phase B — rawvllm -echo "[B] starting rawvllm..." -start_rawvllm -wait_ready bench-rawvllm +# Phase B — baseline (vanilla vllm or vanilla llama-server, same image/config) +echo "[B] starting baseline ($BASELINE_LABEL)..." +start_baseline +wait_ready "$BASELINE_CONTAINER" echo "[B] running $REPEATS sweep(s)..." -mkdir -p "$RESULTS_DIR/rawvllm" -start_mem_sampler rawvllm bench-rawvllm -run_stack rawvllm +mkdir -p "$RESULTS_DIR/baseline" +start_mem_sampler baseline "$BASELINE_CONTAINER" +run_stack baseline stop_mem_sampler -docker rm -f bench-rawvllm >/dev/null +docker rm -f "$BASELINE_CONTAINER" >/dev/null # Summary SUMMARY="$RESULTS_DIR/summary.md" { - echo "# bench $TS" + echo "# bench $TS — $LOADER / $DEVICE" echo - echo "image: \`$IMAGE\` prompts: $NUM_PROMPTS concurrency: $CONCURRENCY input/output: $INPUT_LEN/$OUTPUT_LEN warmups: $NUM_WARMUPS repeats: $REPEATS" + echo "image: \`$IMAGE\` config: \`$(basename "$CONFIG")\` prompts: $NUM_PROMPTS concurrency: $CONCURRENCY input/output: $INPUT_LEN/$OUTPUT_LEN warmups: $NUM_WARMUPS repeats: $REPEATS" echo echo "Values are the median across \`repeats\` sweeps." echo - echo "| metric | modelship | rawvllm | overhead |" - echo "| --- | ---: | ---: | ---: |" - python3 - "$RESULTS_DIR" <<'PY' -import json, sys, statistics -from pathlib import Path -root = Path(sys.argv[1]) -def load(stack): - # One result_.json per repeat; return them all so we can take medians. - runs = [json.loads(p.read_text()) for p in sorted((root / stack).glob("result_*.json"))] - if not runs: - sys.exit(f"no result_*.json found for {stack}") - return runs -def med(runs, key): - vals = [r[key] for r in runs if r.get(key) is not None] - return statistics.median(vals) if vals else None -m = load("modelship"); r = load("rawvllm") -keys = [ - ("request_throughput", "req/s", 3), - ("output_throughput", "output tok/s", 2), - ("mean_ttft_ms", "TTFT mean (ms)", 1), - ("p50_ttft_ms", "TTFT p50 (ms)", 1), - ("p95_ttft_ms", "TTFT p95 (ms)", 1), - ("mean_itl_ms", "ITL mean (ms)", 2), - ("p95_itl_ms", "ITL p95 (ms)", 2), - ("mean_e2el_ms", "E2E mean (ms)", 1), - ("p50_e2el_ms", "E2E p50 (ms)", 1), - ("p95_e2el_ms", "E2E p95 (ms)", 1), -] -for key, label, prec in keys: - mv = med(m, key); rv = med(r, key) - if mv is None or rv is None: - continue - if rv == 0: - ratio = "—" - else: - ratio = f"{(mv - rv) / rv * 100:+.1f}%" - print(f"| {label} | {mv:.{prec}f} | {rv:.{prec}f} | {ratio} |") -PY - echo - echo "## memory (peak across all sweeps)" - python3 - "$RESULTS_DIR" <<'PY' -import sys -from pathlib import Path -root = Path(sys.argv[1]) -# mem.tsv columns: ts, vram, container_rss, anon, file(cache), shmem — all MiB -# except ts. Older runs only have the first 3; missing columns peak at 0. -COLS = ["vram", "rss", "anon", "file", "shmem"] -def peak(stack): - f = root / stack / "mem.tsv" - if not f.exists(): - return None - peaks = {c: 0.0 for c in COLS} - for line in f.read_text().splitlines(): - parts = line.split("\t") - for i, c in enumerate(COLS, start=1): - if i < len(parts): - try: - peaks[c] = max(peaks[c], float(parts[i])) - except ValueError: - pass - return peaks -m = peak("modelship"); r = peak("rawvllm") -print("| metric | modelship | rawvllm | overhead |") -print("| --- | ---: | ---: | ---: |") -def row(label, key, unit="MiB"): - if m is None or r is None: - return - mv, rv = m[key], r[key] - delta = mv - rv - pct = f"{(delta / rv * 100):+.1f}%" if rv else "—" - print(f"| {label} | {mv:.0f} {unit} | {rv:.0f} {unit} | {delta:+.0f} {unit} ({pct}) |") -row("peak VRAM (GPU0)", "vram") -row("peak container RSS", "rss") -row(" ├─ anon (process RSS)", "anon") -row(" ├─ shmem (tmpfs/plasma)", "shmem") -row(" └─ file (page cache)", "file") -print() -print("_**anon** is the real RAM overhead. **file** (page cache) is reclaimable " - "and non-deterministic — it depends on which cgroup first faulted the weights " - "and can swing GB between runs, so the container-RSS delta over- or " - "under-states the true cost. Each row is an independent peak (different " - "instants), so the sub-rows need not sum to peak RSS._") -PY - echo - echo "## modelship per-component memory (Ray reporter, peak under load)" - echo - # Breaks the modelship container's RSS down by Ray process so we can see - # whether the overhead lives in the model-serving actor (ray::*Deployment* — - # we'd serve differently than raw vLLM) or the control plane (gcs_server / - # raylet / agent / ProxyActor / ServeController — fixed Ray cost). USS is - # private memory; shared is shared *libraries* (torch/CUDA, mapped into every - # worker — NOT plasma) reported separately so it isn't charged to any one - # actor. Snapshot is the peak-private scrape sampled *during* the sweep - # (start_component_sampler), not the idle post-sweep state. The reconciliation - # below shows this table undercounts the true total (misses non-Ray children). - if [[ -s "$RESULTS_DIR/modelship/components.txt" ]]; then - python3 - "$RESULTS_DIR" <<'PY' -import sys, re -from pathlib import Path -root = Path(sys.argv[1]) -pat = re.compile(r'^(ray_component_(?:uss_mb|rss_mb|mem_shared_bytes))\{([^}]*)\}\s+([0-9eE+.\-]+)') -key = {"ray_component_uss_mb": "uss", "ray_component_rss_mb": "rss", "ray_component_mem_shared_bytes": "shared"} -comp: dict[str, dict[str, float]] = {} -for line in (root / "modelship" / "components.txt").read_text().splitlines(): - m = pat.match(line) - if not m: - continue - metric, labels, val = m.group(1), m.group(2), float(m.group(3)) - name = dict(re.findall(r'(\w+)="([^"]*)"', labels)).get("Component", "?") - d = comp.setdefault(name, {}) - # Ray emits rss/uss in MB (bytes/1e6); shared is raw bytes — normalize to MB. - v = val / 1e6 if metric == "ray_component_mem_shared_bytes" else val - d[key[metric]] = d.get(key[metric], 0.0) + v -# Private = USS when the agent could read it, else RSS - shared as a floor. -def private(d): - return d["uss"] if "uss" in d else max(d.get("rss", 0.0) - d.get("shared", 0.0), 0.0) -rows = sorted(comp.items(), key=lambda kv: private(kv[1]), reverse=True) -print("| component | private (MB) | rss (MB) | shared (MB) |") -print("| --- | ---: | ---: | ---: |") -tot_priv = tot_rss = tot_shared = 0.0 -for name, d in rows: - p, r, s = private(d), d.get("rss", 0.0), d.get("shared", 0.0) - tot_priv += p; tot_rss += r; tot_shared += s - print(f"| `{name}` | {p:.0f} | {r:.0f} | {s:.0f} |") -print(f"| **total** | **{tot_priv:.0f}** | **{tot_rss:.0f}** | **{tot_shared:.0f}** |") -if not any("uss" in d for _, d in rows): - print() - print("_USS unavailable (reporter couldn't read smaps); private column is " - "rss − shared, an upper bound._") - -# Cross-check the Ray reporter (Prometheus) against the kernel's own accounting -# (cgroup memory.stat, peak under load). The reporter samples /proc smaps on its -# own interval and can be stale or miss workers; cgroup is ground truth. anon ≈ -# Σ private, shmem ≈ Σ shared. Peaks are sampled independently so expect rough, -# not exact, agreement — a large gap means the per-component table is suspect. -def cgroup_peak(col): # mem.tsv: ts,vram,rss,anon,file,shmem (MiB) - f = root / "modelship" / "mem.tsv" - if not f.exists(): - return None - peak = 0.0 - for line in f.read_text().splitlines(): - parts = line.split("\t") - if len(parts) > col: - try: - peak = max(peak, float(parts[col])) - except ValueError: - pass - return peak -anon = cgroup_peak(3) # MiB ≈ MB for this sanity check -if anon: - print() - gap = (tot_priv - anon) / anon * 100 - flag = " ⚠️ diverges" if abs(gap) > 25 else "" - print("_Reporter cross-check vs cgroup `memory.stat` (kernel ground truth, peak):_") - print(f"- private: reporter Σ USS **{tot_priv:.0f} MB** vs cgroup anon " - f"**{anon:.0f} MB** ({gap:+.0f}%){flag}") - if gap < -25: - print(" - reporter undercounts — it sees only Ray worker PIDs, so memory in " - "non-Ray child processes (notably vLLM's `EngineCore` subprocess) is " - "missing. Trust the cgroup figure; treat the per-component split as " - "relative attribution, not an absolute total.") - # NOTE: Ray's mem_shared_bytes is shared *libraries* (torch/CUDA, PSS-shared), - # not plasma — so it has no clean cgroup counterpart and is deliberately not - # reconciled. Actual tmpfs/plasma is cgroup `shmem` (see the memory table); it - # is tiny here, confirming the streaming path barely touches the object store. -PY - else - echo "_no component metrics scraped (reporter agent down or 8079 unreachable)_" - fi - echo - echo "## modelship internal (Prometheus)" - echo - # NOTE: modelship_request_duration_seconds and modelship_generation_duration_seconds - # are observed when the streaming generator is *created*, not after it drains - # (model_deployment.py:230, api.py:347), so for streaming responses they capture - # setup/TTFT only — not end-to-end or full-generation time. We therefore do NOT - # derive "gateway overhead" from them (it's meaningless and was wildly wrong). - # Only the router fulfillment time below reflects real request handling. - if [[ -s "$RESULTS_DIR/modelship/prom.txt" ]]; then - python3 - "$RESULTS_DIR/modelship/prom.txt" <<'PY' -import sys, re -sums = {}; counts = {} -pat = re.compile( - r'(ray_serve_request_router_fulfillment_time_ms)' - r'_(sum|count)\S*\s+([0-9eE+\-.]+)' -) -for line in open(sys.argv[1]): - m = pat.match(line) - if not m: continue - name, kind, val = m.group(1), m.group(2), float(m.group(3)) - (sums if kind == "sum" else counts).setdefault(name, 0.0) - if kind == "sum": sums[name] += val - else: counts[name] += val -n = "ray_serve_request_router_fulfillment_time_ms" -cnt = counts.get(n, 0.0) -if cnt: - print(f"- mean router fulfillment (routing + queue wait): **{sums.get(n, 0.0) / cnt:.1f} ms** " - f"over {cnt:.0f} routed requests") -else: - print("- no router metrics scraped") -print() -print("_E2E / engine durations omitted: their histograms are recorded before " - "streaming completes and are not meaningful for streaming responses._") -PY - else - echo "_no metrics scraped_" - fi + write_summary modelship baseline "$BASELINE_LABEL" } | tee "$SUMMARY" echo diff --git a/docs/model-configuration.md b/docs/model-configuration.md index 445755c..4414fd3 100644 --- a/docs/model-configuration.md +++ b/docs/model-configuration.md @@ -21,6 +21,7 @@ Models are configured in a YAML file (default: `config/models.yaml`). Each entry | `--log-target` | `MSHIP_LOG_TARGET` | `console` | Log target: `console` or syslog URI (e.g. `syslog://host:514`, `syslog+tcp://host:514`) | | `--otel-endpoint` | `OTEL_EXPORTER_OTLP_ENDPOINT` | — | OpenTelemetry OTLP endpoint (e.g. `http://collector:4317`) | | `--no-metrics` | `MSHIP_METRICS` | enabled | Disable Prometheus metrics | +| `--no-preflight` | `MSHIP_PREFLIGHT` | enabled | Disable preflight hardware auto-sizing; models run on loader/library defaults plus explicit config. Useful for benchmarking | | `--api-keys` | `MSHIP_API_KEYS` | — | Comma-separated API keys | | `--max-request-body-bytes` | `MSHIP_MAX_REQUEST_BODY_BYTES` | `52428800` | Max request body size in bytes | diff --git a/modelship/deploy/actor_options.py b/modelship/deploy/actor_options.py index 47f9757..35996ff 100644 --- a/modelship/deploy/actor_options.py +++ b/modelship/deploy/actor_options.py @@ -17,14 +17,17 @@ logger = get_logger("startup") # Forwarded from the driver to each replica's runtime_env: logging vars, the gateway -# name (metrics.py stamps every metric with it), and MSHIP_METRICS so --no-metrics on -# the driver also disables metrics in the replicas (else they'd default to on). +# name (metrics.py stamps every metric with it), MSHIP_METRICS so --no-metrics on +# the driver also disables metrics in the replicas (else they'd default to on), and +# MSHIP_PREFLIGHT so --no-preflight on the driver also disables it in the replicas +# (preflight runs inside each loader's actor __init__, not on the driver). _PASSTHROUGH_ENV_VARS = ( "MSHIP_LOG_LEVEL", "MSHIP_LOG_FORMAT", "MSHIP_LOG_TARGET", "MSHIP_GATEWAY_NAME", "MSHIP_METRICS", + "MSHIP_PREFLIGHT", ) diff --git a/modelship/preflight/base.py b/modelship/preflight/base.py index 66bb444..85fbe57 100644 --- a/modelship/preflight/base.py +++ b/modelship/preflight/base.py @@ -1,6 +1,7 @@ from __future__ import annotations import contextlib +import os from dataclasses import dataclass, field from typing import Any, Protocol @@ -324,6 +325,13 @@ def run_preflight(config: ModelshipModelConfig, hw: HardwareProfile) -> dict[str etc.). For `loader='custom'`, dispatches to the plugin's `ModelPlugin.preflight()` classmethod via a registered adapter. Never raises — preflight failures must not block a deploy.""" + if os.environ.get("MSHIP_PREFLIGHT", "true").lower() == "false": + logger.info( + "preflight disabled via MSHIP_PREFLIGHT=false for '%s'; using loader defaults + user config", + config.name, + ) + return {} + # Register-on-first-call so importing this module doesn't pull in # backend-specific deps (vllm, transformers) when they're not installed. _ensure_registered() diff --git a/modelship/utils/cli.py b/modelship/utils/cli.py index e744bb8..0cc0016 100644 --- a/modelship/utils/cli.py +++ b/modelship/utils/cli.py @@ -64,6 +64,15 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: help="OpenTelemetry OTLP endpoint e.g. http://collector:4317 (env: OTEL_EXPORTER_OTLP_ENDPOINT)", ) parser.add_argument("--no-metrics", action="store_true", default=None, help="Disable metrics (env: MSHIP_METRICS)") + parser.add_argument( + "--no-preflight", + action="store_true", + default=None, + help=( + "Disable preflight hardware-based auto-sizing; models run on loader/library " + "defaults plus explicit config (env: MSHIP_PREFLIGHT). Useful for benchmarking." + ), + ) parser.add_argument( "--prune-ray-sessions", choices=["true", "false"], @@ -117,6 +126,8 @@ def apply_args_to_env(args: argparse.Namespace) -> None: os.environ["MSHIP_USE_EXISTING_RAY_CLUSTER"] = "true" if args.no_metrics is True: os.environ["MSHIP_METRICS"] = "false" + if args.no_preflight is True: + os.environ["MSHIP_PREFLIGHT"] = "false" if args.prune_ray_sessions is not None: os.environ["MSHIP_PRUNE_RAY_SESSIONS"] = args.prune_ray_sessions if args.max_request_body_bytes is not None: diff --git a/tests/test_mship_deploy.py b/tests/test_mship_deploy.py index 3a87617..22acaae 100644 --- a/tests/test_mship_deploy.py +++ b/tests/test_mship_deploy.py @@ -117,6 +117,18 @@ def test_prune_ray_sessions_absent_leaves_env_untouched(self): apply_args_to_env(parse_args([])) assert "MSHIP_PRUNE_RAY_SESSIONS" not in os.environ + def test_no_preflight_sets_env(self): + with patch.dict(os.environ, {}, clear=False): + os.environ.pop("MSHIP_PREFLIGHT", None) + apply_args_to_env(parse_args(["--no-preflight"])) + assert os.environ["MSHIP_PREFLIGHT"] == "false" + + def test_no_preflight_absent_leaves_env_untouched(self): + with patch.dict(os.environ, {}, clear=False): + os.environ.pop("MSHIP_PREFLIGHT", None) + apply_args_to_env(parse_args([])) + assert "MSHIP_PREFLIGHT" not in os.environ + class TestRandSuffix: def test_default_length(self): @@ -202,6 +214,7 @@ def test_passthrough_env_vars_forwarded_to_replicas(self, monkeypatch): # via runtime_env, else the replica defaults to metrics-on (inconsistent). monkeypatch.setenv("MSHIP_METRICS", "false") monkeypatch.setenv("MSHIP_GATEWAY_NAME", "edge") + monkeypatch.setenv("MSHIP_PREFLIGHT", "false") config = ModelshipModelConfig( name="test-model", model="some-model", @@ -212,10 +225,12 @@ def test_passthrough_env_vars_forwarded_to_replicas(self, monkeypatch): env_vars = build_deployment_options(config)["ray_actor_options"]["runtime_env"]["env_vars"] assert env_vars["MSHIP_METRICS"] == "false" assert env_vars["MSHIP_GATEWAY_NAME"] == "edge" + assert env_vars["MSHIP_PREFLIGHT"] == "false" def test_unset_passthrough_env_vars_not_forwarded(self, monkeypatch): # Unset on the driver → not forwarded, so the replica keeps its own default. monkeypatch.delenv("MSHIP_METRICS", raising=False) + monkeypatch.delenv("MSHIP_PREFLIGHT", raising=False) config = ModelshipModelConfig( name="test-model", model="some-model", @@ -225,6 +240,7 @@ def test_unset_passthrough_env_vars_not_forwarded(self, monkeypatch): ) env_vars = build_deployment_options(config)["ray_actor_options"]["runtime_env"]["env_vars"] assert "MSHIP_METRICS" not in env_vars + assert "MSHIP_PREFLIGHT" not in env_vars def test_log_level_in_passthrough_and_deployment_env(self): # The gateway-replica bug: MSHIP_LOG_LEVEL must flow through the shared diff --git a/tests/test_preflight.py b/tests/test_preflight.py index d4016a1..4a8bcfd 100644 --- a/tests/test_preflight.py +++ b/tests/test_preflight.py @@ -114,6 +114,20 @@ def test_swallows_estimator_exceptions(self): result = run_preflight(cfg, HardwareProfile(gpus=[GPUInfo(0, 24 * 1024**3, "test")])) assert result == {} + def test_disabled_via_env_returns_empty_even_with_recommendation(self, monkeypatch): + monkeypatch.setenv("MSHIP_PREFLIGHT", "false") + cfg = _make_config() + with patch.object(VllmPreflight, "recommend", return_value={"max_model_len": 4096}): + result = run_preflight(cfg, HardwareProfile(gpus=[GPUInfo(0, 24 * 1024**3, "test")])) + assert result == {} + + def test_enabled_by_default_when_env_unset(self, monkeypatch): + monkeypatch.delenv("MSHIP_PREFLIGHT", raising=False) + cfg = _make_config() + with patch.object(VllmPreflight, "recommend", return_value={"max_model_len": 4096}): + result = run_preflight(cfg, HardwareProfile(gpus=[GPUInfo(0, 24 * 1024**3, "test")])) + assert result == {"max_model_len": 4096} + class TestVllmPreflight: def test_no_gpus_returns_empty(self): From 38fdd999b83bc3f1d99cf1418d584f104886d9ca Mon Sep 17 00:00:00 2001 From: Alex M <459478+alez007@users.noreply.github.com> Date: Tue, 7 Jul 2026 08:42:13 +0000 Subject: [PATCH 2/4] docs: remove Home Assistant/Wyoming integration doc The HA add-on no longer bundles the Wyoming bridge (moved to a vanilla modelship server setup), so the doc's setup instructions are stale. --- README.md | 3 +-- docs/home-assistant.md | 37 ------------------------------------- 2 files changed, 1 insertion(+), 39 deletions(-) delete mode 100644 docs/home-assistant.md diff --git a/README.md b/README.md index 45fa332..33af08e 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Most self-hosted inference tools focus on running a single model. Modelship is f - **GPU memory control** — allocate exact GPU fractions per model (e.g. 70% for the LLM, 5% for TTS) so everything fits on your hardware - **Mix and match backends** — use vLLM for high-throughput GPU inference, vLLM or llama.cpp for CPU-only workloads, Diffusers for images, and plugins for custom backends — in the same deployment - **Agentic-ready** — reasoning (`` → `reasoning_content`), universal tool/function calling, and the `/v1/responses` API work across the vLLM and llama.cpp (`llama_server`) loaders — the OpenAI-spec surface agents already target -- **Drop-in OpenAI replacement** — any OpenAI SDK client works out of the box, making it easy to integrate with existing apps and tools like [Home Assistant](docs/home-assistant.md) +- **Drop-in OpenAI replacement** — any OpenAI SDK client works out of the box, making it easy to integrate with existing apps and tools ## Architecture @@ -179,7 +179,6 @@ For a full guide on writing your own plugin, see [Plugin Development](docs/plugi - [Model Configuration](docs/model-configuration.md) — full `models.yaml` reference, GPU pinning, environment variables - [Architecture](docs/architecture.md) — system design, request lifecycle, plugin loading - [Plugin Development](docs/plugins.md) — writing custom TTS/STT backends -- [Home Assistant Integration](docs/home-assistant.md) — Wyoming protocol setup for voice automation - [Monitoring & Logging](docs/monitoring.md) — Prometheus metrics, Grafana dashboard, structured logging, health checks - [Troubleshooting](docs/troubleshooting.md) — common first-run errors and fixes diff --git a/docs/home-assistant.md b/docs/home-assistant.md deleted file mode 100644 index f379ddd..0000000 --- a/docs/home-assistant.md +++ /dev/null @@ -1,37 +0,0 @@ -# Home Assistant Integration - -Modelship exposes an OpenAI-compatible API, but Home Assistant's built-in voice assistant pipeline does not speak the OpenAI protocol directly. Instead, it uses the **Wyoming protocol** — a lightweight, purpose-built protocol for local voice services (STT, TTS, wake word). - -To bridge the two, Modelship uses [wyoming-openai](https://github.com/roryeckel/wyoming_openai): a small Docker image that sits between Home Assistant and Modelship, translating Wyoming requests into OpenAI API calls. - -## Setup - -Run `wyoming-openai` alongside Modelship, pointing it at Modelship's API endpoint: - -```bash -docker run -it -p 10300:10300 \ - -e WYOMING_URI="tcp://0.0.0.0:10300" \ - -e WYOMING_LOG_LEVEL="INFO" \ - -e WYOMING_LANGUAGES="en" \ - -e STT_OPENAI_URL="http://:8000/v1" \ - -e STT_MODELS="whisper" \ - -e STT_STREAMING_MODELS="whisper" \ - -e STT_BACKEND="OPENAI" \ - -e TTS_OPENAI_URL="http://:8000/v1" \ - -e TTS_MODELS="kokoro" \ - -e TTS_STREAMING_MODELS="kokoro" \ - -e TTS_VOICES="af_heart" \ - -e TTS_SPEED="1.0" \ - -e TTS_BACKEND="OPENAI" \ - ghcr.io/roryeckel/wyoming_openai:latest -``` - -Replace `` with the address of your Modelship instance: - -- `localhost` or `0.0.0.0` — if running on the same machine -- A Docker container name — if both containers share a Docker network -- A domain or IP — if Modelship is exposed over the network or internet - -Then in Home Assistant, add a **Wyoming** integration pointing to `:10300`. - -The `STT_MODELS` and `TTS_MODELS` values must match the `name` fields in your `models.yaml`. From 6897c177d5930aa87adb7ef08310f8967dba00c7 Mon Sep 17 00:00:00 2001 From: Alex M <459478+alez007@users.noreply.github.com> Date: Tue, 7 Jul 2026 08:42:25 +0000 Subject: [PATCH 3/4] fix: harden bench A/B harness for fair modelship-vs-raw comparisons Several sources of unfairness were biasing the modelship-vs-raw comparison: GPU JIT-compile toolchain gaps in the image, unwarmed/ uneven page cache between phases, llama-server inheriting every GPU on a multi-GPU host instead of just its reservation, connection resets under concurrency on the llama_server CPU baseline, and GGUF configs having no HF tokenizer id for the bench client. Add launch-parity and result-parity gates that fail the run before summarizing if the two arms didn't actually run identical engine args or dropped requests. --- Dockerfile | 21 ++- bench/README.md | 39 +++-- bench/configs/llama-cpu.yaml | 14 ++ bench/configs/llama-gpu.yaml | 14 ++ bench/configs/vllm-cpu.yaml | 8 +- bench/configs/vllm-gpu.yaml | 3 + bench/lib.sh | 292 ++++++++++++++++++++++++++++++++++- bench/rawllama_entrypoint.py | 13 ++ bench/rawvllm_entrypoint.py | 7 +- bench/run.sh | 37 +++++ 10 files changed, 419 insertions(+), 29 deletions(-) diff --git a/Dockerfile b/Dockerfile index 8d92466..ff1e78f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -76,16 +76,21 @@ RUN apt-get update -y && \ curl \ espeak-ng \ gcc \ + g++ \ gnupg \ gosu \ libc6-dev \ - libgomp1 && \ + libgomp1 \ + libnuma1 \ + ninja-build && \ rm -rf /var/lib/apt/lists/* -# Register the NVIDIA CUDA apt repo and install cuda-cudart (GPU variant only). -# gcc + libc6-dev stay because torch/triton JIT-compile kernels at model-load -# time and shell out to $CC; without them, vllm crashes in _inductor with -# "Failed to find C compiler". +# Register the NVIDIA CUDA apt repo and install cuda-cudart, cuda-nvcc, and +# cuda-cuobjdump (GPU variant only). gcc/g++ + libc6-dev and ninja-build stay +# because torch/triton and flashinfer JIT-compile kernels at model-load time +# and shell out to $CC/nvcc; without them, vllm crashes in _inductor (needs +# g++ for its CPU codegen backend, e.g. the vllm CPU loader's torch.compile +# path) or flashinfer on newer architectures (such as Blackwell). RUN if [ "$MSHIP_VARIANT" = "gpu" ]; then \ CUDA_VERSION_DASH=$(echo $CUDA_VERSION | cut -d. -f1,2 | tr '.' '-') && \ curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \ @@ -93,7 +98,11 @@ RUN if [ "$MSHIP_VARIANT" = "gpu" ]; then \ echo "deb [signed-by=/usr/share/keyrings/cuda-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /" \ > /etc/apt/sources.list.d/cuda.list && \ apt-get update -y && \ - apt-get install -y --no-install-recommends cuda-cudart-${CUDA_VERSION_DASH} && \ + apt-get install -y --no-install-recommends \ + cuda-cudart-${CUDA_VERSION_DASH} \ + cuda-nvcc-${CUDA_VERSION_DASH} \ + cuda-cuobjdump-${CUDA_VERSION_DASH} \ + libcurand-dev-${CUDA_VERSION_DASH} && \ apt-get purge -y --auto-remove gnupg && \ rm -f /etc/apt/sources.list.d/cuda.list /usr/share/keyrings/cuda-keyring.gpg && \ rm -rf /var/lib/apt/lists/*; \ diff --git a/bench/README.md b/bench/README.md index 22f5a81..ba5507c 100644 --- a/bench/README.md +++ b/bench/README.md @@ -93,24 +93,29 @@ Two cross-checks back this up: Example run — 1×GPU, Qwen2.5-7B-AWQ, `--loader vllm --device gpu`, 100 prompts @ concurrency 8, median of 3: -| metric | modelship | raw vllm | delta | +| metric | modelship | raw vllm | overhead | | --- | ---: | ---: | ---: | -| throughput (req/s) | 1.185 | 1.188 | −0.3% | -| output (tok/s) | 606.7 | 608.2 | −0.3% | -| TTFT mean (ms) | 71.3 | 63.4 | +8 ms | -| ITL mean (ms) | 12.99 | 12.53 | +0.5 ms | -| TPOT mean (ms) | 12.57 | 12.55 | ~0 | -| peak VRAM (MiB) | 14671 | 14070 | +601 | -| peak host RAM, anon (MiB) | 4598 | 3733 | **+865** | +| throughput (req/s) | 1.199 | 1.203 | −0.4% | +| output (tok/s) | 613.7 | 616.1 | −0.4% | +| TTFT mean (ms) | 62.5 | 54.9 | +13.9% | +| TTFT p95 (ms) | 89.5 | 65.0 | +37.8% | +| ITL mean (ms) | 12.96 | 12.39 | +4.6% | +| TPOT mean (ms) | 12.44 | 12.42 | +0.2% | +| peak VRAM (MiB) | 14533 | 14020 | +513 MiB | +| peak host RAM, anon (MiB) | 4957 | 3781 | **+1176 MiB** | Notes: - **Throughput and decode (TPOT) are at parity** — same vLLM wheel and GPU, so the engine's hot path is identical. modelship adds no per-token overhead. - **TTFT/ITL** carry modelship's expected cost: the extra hop through the Ray - Serve proxy/router adds a small *fixed* first-token latency (here ~8 ms) and a + Serve proxy/router adds a small *fixed* first-token latency (here ~7.6 ms) and a tiny per-chunk cost. Negligible for a 512-token response. -- **Host RAM (~+0.9 GB)** is the real resource cost — the Ray + Serve control +- **TTFT's tail is fatter than its mean**: p95 overhead (+37.8%) runs well above + the mean/p50 gap (+13.9%/+1.6%) — occasional scheduling jitter through the Ray + Serve proxy/router under concurrent load, not a fixed per-request cost. Still + small in absolute terms (~25 ms) against a multi-second E2E latency. +- **Host RAM (+1176 MiB)** is the real resource cost — the Ray + Serve control plane (including the prewarmed idle-worker pool) plus the replica. This is `anon` (real process memory); **don't** use the container-RSS delta, which is dominated by reclaimable page cache and swings multiple GB between runs depending on which @@ -122,11 +127,14 @@ Notes: - Both phases use the same image (same vLLM wheel / same `llama-server` binary) and the same config file. +- The modelship phase runs with **`MSHIP_PREFLIGHT=false`** (passed via env). This is the linchpin: it disables hardware-aware automatic preflight tuning, ensuring that unset fields fall back to loader/pydantic defaults. Consequently, both phases run identical engine parameters out-of-the-box. - The baseline phase parses the config through modelship's own pydantic schema and translates the engine config into either `vllm serve` flags ([`rawvllm_entrypoint.py`](rawvllm_entrypoint.py)) or a `llama-server` launch command ([`rawllama_entrypoint.py`](rawllama_entrypoint.py), mirroring `modelship/infer/llama_server/llama_server_infer.py`'s `_launch`). +- **Launch parity check**: After both phases run, the harness extracts each phase's effective launch command from its container logs, normalizes legitimately-different tokens (such as ports, hostnames, and api keys), and fails (exits non-zero) if there are any remaining differences. This guarantees that both phases run identical engine parameters. +- **Tokenizer extraction**: GGUF configs (which use GGUF model paths) cannot be used directly as Hugging Face repository IDs by the bench client. To handle this, the harness looks for a `# bench-tokenizer: ` comment inside the yaml config file (inert to modelship) and parses it using `yaml_scalar` to use as the tokenizer for the bench client. You can also override it using the `--tokenizer` CLI flag. - vLLM: keep `gpu_memory_utilization` equal to what modelship would pick for that `num_gpus` (0.9 GPU / 0.4 CPU) — for fractional `num_gpus` modelship overrides `gpu_memory_utilization` to `num_gpus`, and the raw phase reads the @@ -135,6 +143,11 @@ Notes: (rather than the loader's `-1` auto-fit default) — the raw phase has no preflight to pick a matching value on its own, so an explicit, identical value keeps both phases offloading the same number of layers. - -> Caveat: modelship applies hardware-aware preflight defaults the baseline -> phase does not, so this is a close A/B, not a byte-identical one. +- llama_server on a multi-GPU host: `rawllama_entrypoint.py` sets + `CUDA_VISIBLE_DEVICES` to exactly `num_gpus` device(s) before exec'ing + `llama-server`, mirroring the GPU reservation Ray gives the modelship + actor. Without this the raw phase — a bare subprocess with no Ray actor — + inherits every GPU the container's `--gpus` flag exposed, and llama.cpp + auto-splits the model across all of them (no `--tensor-split`/`--main-gpu` + is passed), handing the baseline more aggregate VRAM/bandwidth than the + single-GPU modelship deploy and invalidating the comparison. diff --git a/bench/configs/llama-cpu.yaml b/bench/configs/llama-cpu.yaml index 14c23e6..a695379 100644 --- a/bench/configs/llama-cpu.yaml +++ b/bench/configs/llama-cpu.yaml @@ -1,6 +1,7 @@ models: - name: "llm" model: "lmstudio-community/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf" + # bench-tokenizer: Qwen/Qwen2.5-7B-Instruct usecase: "generate" loader: "llama_server" num_gpus: 0 @@ -8,6 +9,9 @@ models: # Lift Ray Serve's per-replica concurrency cap (default 100) so the model # deployment isn't the bottleneck during the sweep. max_ongoing_requests: 256 + # Note: the bench runs modelship with MSHIP_PREFLIGHT=false, so both phases + # use loader defaults for unset fields; explicit values here are read + # identically by both. llama_server_config: # Concurrent llama-server request slots — must cover --concurrency (8 by # default) so requests overlap instead of queuing behind a full set of @@ -18,3 +22,13 @@ models: # num_gpus: 0 deploy regardless of this field (see llama_server_infer.py # and rawllama_entrypoint.py, which mirrors that branch), so both phases # run CPU-only either way. + # Size llama-server's HTTP worker pool to comfortably cover --concurrency + # (default 8). The baseline arm points the bench client's many concurrent + # short-lived connections *straight* at llama-server's cpp-httplib, which + # oversubscribes the default pool and resets connections (observed as + # ServerDisconnectedError → failed requests); the modelship arm hides this + # behind a single pooled httpx client, so without this the baseline drops + # requests the modelship phase doesn't and the two arms compare unequal + # populations. Applied to BOTH phases (both append extra_args verbatim), + # so launch parity is preserved. + extra_args: ["--threads-http", "16"] diff --git a/bench/configs/llama-gpu.yaml b/bench/configs/llama-gpu.yaml index d956034..6983266 100644 --- a/bench/configs/llama-gpu.yaml +++ b/bench/configs/llama-gpu.yaml @@ -1,6 +1,7 @@ models: - name: "llm" model: "lmstudio-community/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf" + # bench-tokenizer: Qwen/Qwen2.5-7B-Instruct usecase: "generate" loader: "llama_server" num_gpus: 1 @@ -8,6 +9,9 @@ models: # Lift Ray Serve's per-replica concurrency cap (default 100) so the model # deployment isn't the bottleneck during the sweep. max_ongoing_requests: 256 + # Note: the bench runs modelship with MSHIP_PREFLIGHT=false, so both phases + # use loader defaults for unset fields; explicit values here are read + # identically by both. llama_server_config: # Concurrent llama-server request slots — must cover --concurrency (8 by # default) so requests overlap instead of queuing behind a full set of @@ -19,3 +23,13 @@ models: # the two phases pick different offload counts. 99 exceeds Qwen2.5-7B's # 28 layers, i.e. full GPU offload for both phases. n_gpu_layers: 99 + # Size llama-server's HTTP worker pool to comfortably cover --concurrency + # (default 8). The baseline arm points the bench client's many concurrent + # short-lived connections *straight* at llama-server's cpp-httplib, which + # oversubscribes the default pool and resets connections (observed as + # ServerDisconnectedError → failed requests); the modelship arm hides this + # behind a single pooled httpx client, so without this the baseline drops + # requests the modelship phase doesn't and the two arms compare unequal + # populations. Applied to BOTH phases (both append extra_args verbatim), + # so launch parity is preserved. + extra_args: ["--threads-http", "16"] diff --git a/bench/configs/vllm-cpu.yaml b/bench/configs/vllm-cpu.yaml index 40dc929..53a2769 100644 --- a/bench/configs/vllm-cpu.yaml +++ b/bench/configs/vllm-cpu.yaml @@ -1,6 +1,9 @@ models: - name: "llm" - model: "Qwen/Qwen2.5-7B-Instruct-AWQ" + # Unquantized (not -AWQ): this vllm+cpu wheel doesn't ship the CPU AWQ/GPTQ + # kernels (cpu_gemm_wna16 / int4_scaled_mm_cpu aren't in torch.ops._C), so + # any AWQ model crashes at engine warmup on the vllm CPU backend. + model: "Qwen/Qwen2.5-7B-Instruct" usecase: "generate" loader: "vllm" num_gpus: 0 @@ -8,6 +11,9 @@ models: # Lift Ray Serve's per-replica concurrency cap (default 100) so the model # deployment isn't the bottleneck during the sweep. max_ongoing_requests: 256 + # Note: the bench runs modelship with MSHIP_PREFLIGHT=false, so both phases + # use loader defaults for unset fields; explicit values here are read + # identically by both. vllm_engine_kwargs: tensor_parallel_size: 1 max_model_len: 8192 diff --git a/bench/configs/vllm-gpu.yaml b/bench/configs/vllm-gpu.yaml index 5048fc2..07117f0 100644 --- a/bench/configs/vllm-gpu.yaml +++ b/bench/configs/vllm-gpu.yaml @@ -7,6 +7,9 @@ models: # Lift Ray Serve's per-replica concurrency cap (default 100) so the model # deployment isn't the bottleneck during the sweep. max_ongoing_requests: 256 + # Note: the bench runs modelship with MSHIP_PREFLIGHT=false, so both phases + # use loader defaults for unset fields; explicit values here are read + # identically by both. vllm_engine_kwargs: tensor_parallel_size: 1 max_model_len: 8192 diff --git a/bench/lib.sh b/bench/lib.sh index c63f04d..7f5c75b 100644 --- a/bench/lib.sh +++ b/bench/lib.sh @@ -10,7 +10,8 @@ yaml_scalar() { local pattern="$1" file="$2" grep -m1 -E "$pattern" "$file" \ - | sed -E "s/^[^:]*:[[:space:]]*//; s/[[:space:]]*\$//; s/^(['\"])(.*)\1\$/\2/" + | sed -E "s/^[^:]*:[[:space:]]*//; s/[[:space:]]*\$//; s/^(['\"])(.*)\1\$/\2/" \ + || true } cleanup() { @@ -22,7 +23,7 @@ cleanup() { done for c in "$MODELSHIP_CONTAINER" "$BASELINE_CONTAINER"; do if [[ -n "$c" ]] && docker inspect "$c" >/dev/null 2>&1; then - docker logs "$c" >"$RESULTS_DIR/${c}.container.log" 2>&1 || true + docker logs "$c" >"$RESULTS_DIR/${c}.log" 2>&1 || true docker rm -f "$c" >/dev/null 2>&1 || true fi done @@ -33,9 +34,11 @@ wait_ready() { local deadline=$(( $(date +%s) + READY_TIMEOUT )) while (( $(date +%s) < deadline )); do # /v1/models reachable AND lists the served model id - if curl -fsS http://localhost:8000/v1/models 2>/dev/null \ - | grep -q "\"id\":\"$SERVED_NAME\""; then - return 0 + local response + if response=$(curl -fsS http://localhost:8000/v1/models 2>/dev/null); then + if python3 -c "import sys, json; data = json.loads(sys.argv[1]); print('match' if any(m.get('id') == sys.argv[2] for m in data.get('data', [])) else '')" "$response" "$SERVED_NAME" | grep -q "match"; then + return 0 + fi fi if ! docker ps --filter "name=^${name}$" --format '{{.Names}}' | grep -q "$name"; then echo "container $name died" >&2 @@ -173,14 +176,48 @@ vram_gate() { echo "warn: VRAM not freed within 60s" >&2 } +# Read the model weights into the host page cache so both phases enter their +# timed sweeps equally warm. drop_host_caches can't drop without privileges (it +# warns and no-ops in most environments), and run.sh always runs modelship first +# / baseline second against the same GGUF — so the baseline would otherwise +# inherit a page cache the modelship phase had to cold-fault in. Called before +# each phase's sweeps: cheap (a no-op re-read once cached) and symmetric. +# Globs CACHE_DIR for *.gguf; on a cold host before the model is downloaded it +# simply finds nothing and returns, which is fine (that phase faults it in during +# its own model load, and the next phase is pre-warmed here). +warm_model_cache() { + echo " pre-warming model cache..." + local found=0 + while IFS= read -r -d '' f; do + found=1 + cat "$f" > /dev/null 2>&1 || true + done < <(find "$CACHE_DIR" -type f -name '*.gguf' -print0 2>/dev/null) + if (( found )); then + echo " model cache warmed." + else + echo " no .gguf found under cache dir yet — skipping warm." + fi +} + +drop_host_caches() { + echo " dropping host page caches..." + if { sync && echo 3 > /proc/sys/vm/drop_caches; } >/dev/null 2>&1; then + echo " caches dropped successfully." + elif sudo -n sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches' >/dev/null 2>&1; then + echo " caches dropped successfully via sudo." + else + echo " warn: failed to drop host caches (no write access and no passwordless sudo, or unsupported in this environment). Page-cache states may differ." >&2 + fi +} + # Run REPEATS timed sweeps against an already-ready stack, saving each to its # own result_.json. The summary takes the median across them. run_stack() { local stack="$1" local out_dir="$RESULTS_DIR/$stack" mkdir -p "$out_dir" - chmod 777 "$out_dir" for i in $(seq 1 "$REPEATS"); do + drop_host_caches echo " sweep $i/$REPEATS ($stack)..." run_sweep "$stack" "result_${i}.json" done @@ -190,13 +227,32 @@ run_sweep() { local stack="$1" local fname="$2" local out_dir="$RESULTS_DIR/$stack" - docker run --rm --network host -v "$out_dir:/out:rw" "$IMAGE" \ + + local extra_client_args=() + # E3. Disjoint cores for client vs server on `--device cpu` + if [[ "$DEVICE" == "cpu" ]]; then + local num_cores + num_cores=$(nproc) + if (( num_cores > 2 )); then + local c_start=$(( num_cores - 2 )) + local c_end=$(( num_cores - 1 )) + extra_client_args+=(--cpuset-cpus "${c_start}-${c_end}") + echo " pinning client container to cpuset ${c_start}-${c_end} (of ${num_cores} cores)" + else + extra_client_args+=(--cpuset-cpus "0") + echo " pinning client container to cpuset 0" + fi + fi + + docker run --rm --network host --user "$(id -u):$(id -g)" \ + "${extra_client_args[@]}" \ + -v "$out_dir:/out:rw" "$IMAGE" \ bash -lc "cd /modelship && uv run --active --no-sync vllm bench serve \ --backend openai-chat \ --base-url http://localhost:8000 \ --endpoint /v1/chat/completions \ --model $SERVED_NAME \ - --tokenizer $MODEL_ID \ + --tokenizer $TOKENIZER \ --dataset-name random \ --random-input-len $INPUT_LEN \ --random-output-len $OUTPUT_LEN \ @@ -204,6 +260,8 @@ run_sweep() { --max-concurrency $CONCURRENCY \ --num-warmups $NUM_WARMUPS \ --ignore-eos \ + --percentile-metrics ttft,tpot,itl,e2el \ + --metric-percentiles 50,95,99 \ --save-result \ --result-dir /out \ --result-filename $fname" @@ -221,6 +279,199 @@ scrape_prom() { > "$out" || true } +assert_launch_parity() { + echo "=== verifying launch-args parity ===" + python3 - "$RESULTS_DIR" "$LOADER" <<'PY' +import sys, re, ast, shlex, os +from pathlib import Path + +root = Path(sys.argv[1]) +loader = sys.argv[2] + +modelship_log = root / "bench-modelship.log" +baseline_log = root / "bench-baseline.log" + +if not modelship_log.exists() or not baseline_log.exists(): + sys.exit(f"Logs missing. modelship log exists: {modelship_log.exists()}, baseline log exists: {baseline_log.exists()}") + +m_content = modelship_log.read_text() +b_content = baseline_log.read_text() + +def normalize_path(p): + if not p: + return "" + if p.startswith("/") or "/" in p: + return os.path.basename(p) + return p + +if loader == "llama_server": + m_match = re.search(r"llama-server launch args for '.*':\s*(\[.*\])", m_content) + if not m_match: + sys.exit("Could not find 'llama-server launch args for' in modelship log") + m_args = ast.literal_eval(m_match.group(1)) + + b_match = re.search(r"rawllama exec:\s*(.*)", b_content) + if not b_match: + sys.exit("Could not find 'rawllama exec:' in baseline log") + b_args = shlex.split(b_match.group(1)) + + def normalize_llama_args(args): + res = list(args[1:]) + normalized = [] + i = 0 + while i < len(res): + arg = res[i] + if arg in ["--host", "--port", "--api-key"]: + i += 2 + elif arg in ["--alias"]: + i += 2 + elif arg in ["-m", "--mmproj", "--chat-template-file", "--chat-template"]: + if i + 1 < len(res): + normalized.append((arg, normalize_path(res[i+1]))) + i += 2 + else: + normalized.append((arg, "")) + i += 1 + else: + normalized.append((arg, "")) + i += 1 + return sorted(normalized) + + m_norm = normalize_llama_args(m_args) + b_norm = normalize_llama_args(b_args) + + with open(root / "launch-parity.txt", "w") as f: + f.write(f"Modelship normalized: {m_norm}\n") + f.write(f"Baseline normalized: {b_norm}\n") + + if m_norm != b_norm: + print("LAUNCH PARITY FAILED for llama_server!", file=sys.stderr) + print(f"Modelship: {m_norm}", file=sys.stderr) + print(f"Baseline: {b_norm}", file=sys.stderr) + sys.exit(1) + else: + print("LAUNCH PARITY PASSED for llama_server.") + +elif loader == "vllm": + m_match = re.search(r"initialising vllm engine with args:\s*(\{.*\})", m_content) + if not m_match: + sys.exit("Could not find 'initialising vllm engine with args:' in modelship log") + m_dict = ast.literal_eval(m_match.group(1)) + + b_match = re.search(r"rawvllm exec:\s*(.*)", b_content) + if not b_match: + sys.exit("Could not find 'rawvllm exec:' in baseline log") + b_args = shlex.split(b_match.group(1)) + + def parse_vllm_flags(args): + parsed = {} + flag_start = 0 + for idx, arg in enumerate(args): + if arg.startswith('--'): + flag_start = idx + break + + i = flag_start + while i < len(args): + arg = args[i] + if arg.startswith('--'): + name = arg[2:].replace('-', '_') + if name in ['enforce_eager', 'trust_remote_code', 'enable_auto_tool_choice']: + parsed[name] = True + elif i + 1 < len(args): + val = args[i+1] + if val.isdigit(): + parsed[name] = int(val) + else: + try: + parsed[name] = float(val) + except ValueError: + parsed[name] = val + i += 1 + i += 1 + return parsed + + b_dict = parse_vllm_flags(b_args) + + fields = [ + 'gpu_memory_utilization', + 'tensor_parallel_size', + 'pipeline_parallel_size', + 'dtype', + 'quantization', + 'kv_cache_dtype', + 'enforce_eager', + 'trust_remote_code', + 'max_model_len', + ] + + m_norm = {} + b_norm = {} + + for fld in fields: + mv = m_dict.get(fld) + bv = b_dict.get(fld) + if mv in [None, False]: + mv = None + if bv in [None, False]: + bv = None + if isinstance(mv, float) and isinstance(bv, float): + if abs(mv - bv) < 1e-5: + bv = mv + m_norm[fld] = mv + b_norm[fld] = bv + + with open(root / "launch-parity.txt", "w") as f: + f.write(f"Modelship normalized: {m_norm}\n") + f.write(f"Baseline normalized: {b_norm}\n") + + if m_norm != b_norm: + print("LAUNCH PARITY FAILED for vllm!", file=sys.stderr) + print(f"Modelship: {m_norm}", file=sys.stderr) + print(f"Baseline: {b_norm}", file=sys.stderr) + sys.exit(1) + else: + print("LAUNCH PARITY PASSED for vllm.") +PY +} + +# Gate the run on result-population parity. The latency/throughput medians in +# summary.md are computed only over each arm's *successful* requests, with no +# check that both arms completed the same population — so if one arm silently +# drops its slowest requests as failures (e.g. the baseline's direct-to- +# cpp-httplib connections resetting under load), its tail metrics look better +# purely by survivorship. Fail loudly rather than publish a biased comparison. +assert_result_parity() { + echo "=== verifying result-population parity ===" + python3 - "$RESULTS_DIR" <<'PY' +import json, sys +from pathlib import Path + +root = Path(sys.argv[1]) +bad = False +for stack in ("modelship", "baseline"): + for p in sorted((root / stack).glob("result_*.json")): + d = json.loads(p.read_text()) + completed = d.get("completed", 0) + failed = d.get("failed", 0) + total = d.get("num_prompts", completed + failed) + if failed: + print(f" {stack}/{p.name}: {failed} FAILED / {completed} completed of {total}", file=sys.stderr) + bad = True + +if bad: + print( + "RESULT PARITY FAILED: an arm dropped requests. The latency/throughput " + "medians compare unequal populations (survivorship bias) and are NOT " + "trustworthy. Re-run; if the baseline keeps failing, raise " + "--threads-http via llama_server_config.extra_args in the bench config.", + file=sys.stderr, + ) + sys.exit(1) +print("RESULT PARITY PASSED: both arms completed every request.") +PY +} + # Renders the shared summary.md body: latency/throughput table (median across # repeats), memory table (peak across the sweep), modelship per-component # memory breakdown + cgroup cross-check, and the router-fulfillment Prometheus @@ -245,11 +496,18 @@ def med(runs, key): return statistics.median(vals) if vals else None m = load(sys.argv[2]); r = load(sys.argv[3]) keys = [ + # Population first: latency/throughput below are over *successful* requests + # only, so these two rows are the caveat for reading the rest of the table. + # assert_result_parity fails the run when `failed` is non-zero on either arm, + # so in a passing run both these are N and 0 respectively. + ("completed", "completed", 0), + ("failed", "failed", 0), ("request_throughput", "req/s", 3), ("output_throughput", "output tok/s", 2), ("mean_ttft_ms", "TTFT mean (ms)", 1), ("p50_ttft_ms", "TTFT p50 (ms)", 1), ("p95_ttft_ms", "TTFT p95 (ms)", 1), + ("mean_tpot_ms", "TPOT mean (ms)", 1), ("mean_itl_ms", "ITL mean (ms)", 2), ("p95_itl_ms", "ITL p95 (ms)", 2), ("mean_e2el_ms", "E2E mean (ms)", 1), @@ -265,7 +523,25 @@ for key, label, prec in keys: else: ratio = f"{(mv - rv) / rv * 100:+.1f}%" print(f"| {label} | {mv:.{prec}f} | {rv:.{prec}f} | {ratio} |") + +# Token-count parity: prove both arms ran equivalent prompts. total_input_tokens +# is the client-side (shared --tokenizer) accounting; dividing by completed makes +# it robust to any success-count gap. modelship drains its llama-server subprocess +# logs at TRACE (suppressed in the bench container), so this reconciliation is the +# only independent check that the two arms tokenized the same work — the launch +# args being identical doesn't guarantee the prompt bodies were. +def per_prompt_in(runs): + vals = [rr["total_input_tokens"] / rr["completed"] for rr in runs if rr.get("completed")] + return statistics.median(vals) if vals else None +mi = per_prompt_in(m); ri = per_prompt_in(r) +if mi is not None and ri is not None: + delta = mi - ri + flag = "⚠️ prompts differ" if abs(delta) > 1.0 else "✓" + print() + print(f"_input tokens/prompt (client tokenizer): modelship **{mi:.1f}** vs " + f"baseline **{ri:.1f}** (Δ {delta:+.1f}) {flag}_") PY + echo echo "## memory (peak across all sweeps)" python3 - "$RESULTS_DIR" "$modelship_stack" "$baseline_stack" "$baseline_label" <<'PY' diff --git a/bench/rawllama_entrypoint.py b/bench/rawllama_entrypoint.py index 95947a5..3bf26cf 100644 --- a/bench/rawllama_entrypoint.py +++ b/bench/rawllama_entrypoint.py @@ -75,6 +75,19 @@ def main() -> int: # when the modelship phase wouldn't. if m.num_gpus > 0: args += ["-ngl", str(k.n_gpu_layers)] + # Ray's GPU reservation restricts the modelship actor's + # CUDA_VISIBLE_DEVICES to exactly num_gpus device(s); this bare + # subprocess bypasses Ray entirely and would otherwise inherit every + # GPU the container's --gpus flag exposed. Without this, + # llama-server auto-splits the model across every visible device + # (no --tensor-split/--main-gpu is passed), giving the baseline more + # aggregate VRAM/bandwidth than the modelship phase on a multi-GPU + # host and invalidating the A/B. + # num_gpus is typed float on ModelConfig (shared with the vllm loader's + # fractional sharing), but the llama_server loader's own validator + # (validate_llama_server_num_gpus) rejects non-whole values — so the + # int() cast here is safe and just satisfies range()'s signature. + os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(str(i) for i in range(int(m.num_gpus))) else: args += ["-ngl", "0"] if k.threads is not None: diff --git a/bench/rawvllm_entrypoint.py b/bench/rawvllm_entrypoint.py index 53eeed0..4e4e3c9 100644 --- a/bench/rawvllm_entrypoint.py +++ b/bench/rawvllm_entrypoint.py @@ -32,7 +32,12 @@ def main() -> int: args += ["--tensor-parallel-size", str(k.tensor_parallel_size)] args += ["--pipeline-parallel-size", str(k.pipeline_parallel_size)] args += ["--dtype", k.dtype] - args += ["--gpu-memory-utilization", str(k.gpu_memory_utilization)] + from modelship.infer.infer_config import default_gpu_memory_utilization + + gmu = k.gpu_memory_utilization + if gmu is None: + gmu = default_gpu_memory_utilization(m) + args += ["--gpu-memory-utilization", str(gmu)] if k.max_model_len is not None: args += ["--max-model-len", str(k.max_model_len)] if k.tokenizer: diff --git a/bench/run.sh b/bench/run.sh index 3ce2702..43dbf03 100755 --- a/bench/run.sh +++ b/bench/run.sh @@ -13,6 +13,7 @@ LOADER="vllm" DEVICE="gpu" IMAGE="" CONFIG="" +TOKENIZER="" NUM_PROMPTS=100 CONCURRENCY=8 INPUT_LEN=128 @@ -32,6 +33,7 @@ while [[ $# -gt 0 ]]; do --device) DEVICE="$2"; shift 2 ;; --image) IMAGE="$2"; shift 2 ;; --config) CONFIG="$2"; shift 2 ;; + --tokenizer) TOKENIZER="$2"; shift 2 ;; --num-prompts) NUM_PROMPTS="$2"; shift 2 ;; --concurrency) CONCURRENCY="$2"; shift 2 ;; --input-len) INPUT_LEN="$2"; shift 2 ;; @@ -86,6 +88,19 @@ SERVED_NAME="$(yaml_scalar '^[[:space:]]*-[[:space:]]*name:' "$CONFIG")" MODEL_ID="$(yaml_scalar '^[[:space:]]*model:' "$CONFIG")" [[ -n "$MODEL_ID" && -n "$SERVED_NAME" ]] || { echo "failed to parse $CONFIG" >&2; exit 2; } +if [[ -z "$TOKENIZER" ]]; then + TOKENIZER="$(yaml_scalar '^[[:space:]]*#[[:space:]]*bench-tokenizer:' "$CONFIG")" +fi +if [[ -z "$TOKENIZER" ]]; then + TOKENIZER="$MODEL_ID" +fi + +NUM_CPUS="$(yaml_scalar '^[[:space:]]*num_cpus:' "$CONFIG")" +BASELINE_ENV_ARGS=() +if [[ -n "${NUM_CPUS:-}" ]]; then + BASELINE_ENV_ARGS+=(-e "OMP_NUM_THREADS=$NUM_CPUS") +fi + MODELSHIP_CONTAINER=bench-modelship BASELINE_CONTAINER=bench-baseline trap cleanup EXIT @@ -100,8 +115,12 @@ start_modelship() { # Mount local source over the prebuilt image so the bench exercises the # working tree. mship_deploy.py is the entry point invoked by the image's # default CMD (scripts/start.sh). + # + # MSHIP_PREFLIGHT=false ensures modelship loader falls back to standard defaults + # matching the baseline phase exactly. docker run -d "${DOCKER_GPU_ARGS[@]}" --ipc=host --network host \ -e MSHIP_METRICS=true \ + -e MSHIP_PREFLIGHT=false \ -e MSHIP_GATEWAY_REPLICAS="${MSHIP_GATEWAY_REPLICAS:-1}" \ -e MSHIP_GATEWAY_MAX_ONGOING="${MSHIP_GATEWAY_MAX_ONGOING:-1024}" \ -v "$CONFIG:/modelship/config/models.yaml:ro" \ @@ -112,9 +131,13 @@ start_modelship() { } start_baseline() { + # Mount the local modelship code into the baseline container so that + # any pydantic schema or other changes in the working tree are shared. docker run -d "${DOCKER_GPU_ARGS[@]}" --ipc=host --network host \ -e PYTHONPATH=/modelship \ + "${BASELINE_ENV_ARGS[@]}" \ -v "$CONFIG:/modelship/config/models.yaml:ro" \ + -v "$REPO_ROOT/modelship:/modelship/modelship:ro" \ -v "$BENCH_DIR/$BASELINE_ENTRYPOINT:/modelship/bench/$BASELINE_ENTRYPOINT:ro" \ -v "$CACHE_DIR:/.cache:rw" \ -w /modelship \ @@ -130,6 +153,7 @@ echo "[A] starting modelship..." start_modelship wait_ready "$MODELSHIP_CONTAINER" echo "[A] running $REPEATS sweep(s)..." +warm_model_cache mkdir -p "$RESULTS_DIR/modelship" start_mem_sampler modelship "$MODELSHIP_CONTAINER" start_component_sampler "$RESULTS_DIR/modelship/components.txt" @@ -137,6 +161,7 @@ run_stack modelship stop_mem_sampler stop_component_sampler scrape_prom "$RESULTS_DIR/modelship/prom.txt" +docker logs "$MODELSHIP_CONTAINER" > "$RESULTS_DIR/${MODELSHIP_CONTAINER}.log" 2>&1 || true docker rm -f "$MODELSHIP_CONTAINER" >/dev/null vram_gate @@ -145,12 +170,18 @@ echo "[B] starting baseline ($BASELINE_LABEL)..." start_baseline wait_ready "$BASELINE_CONTAINER" echo "[B] running $REPEATS sweep(s)..." +warm_model_cache mkdir -p "$RESULTS_DIR/baseline" start_mem_sampler baseline "$BASELINE_CONTAINER" run_stack baseline stop_mem_sampler +docker logs "$BASELINE_CONTAINER" > "$RESULTS_DIR/${BASELINE_CONTAINER}.log" 2>&1 || true docker rm -f "$BASELINE_CONTAINER" >/dev/null +# Launch parity (config correctness): fail before summarizing if the two arms +# weren't launched with identical engine args — nothing downstream is meaningful. +assert_launch_parity + # Summary SUMMARY="$RESULTS_DIR/summary.md" { @@ -165,3 +196,9 @@ SUMMARY="$RESULTS_DIR/summary.md" echo echo "results: $RESULTS_DIR" + +# Result-population gate LAST: the summary above is written first (so the +# completed/failed rows and token-parity line are always available for +# inspection), then we fail the run with a non-zero exit if either arm dropped +# requests — so a survivorship-biased comparison is never treated as a pass. +assert_result_parity From f4506f5a35ddd1a64c7a6889b898d9bb91741cdb Mon Sep 17 00:00:00 2001 From: Alex M <459478+alez007@users.noreply.github.com> Date: Tue, 7 Jul 2026 10:55:12 +0000 Subject: [PATCH 4/4] feat: pin greedy load client and make bench result-parity gate relative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The llama_server A/B occasionally saw modelship "truncate" a request: with --ignore-eos the model can babble a malformed past EOS that llama-server's own grammar parser rejects mid-stream, which modelship faithfully relays as an in-band SSE error. This is engine behaviour present in both arms; under sampling it landed on one arm by luck. - Pin --temperature 0 in the vllm-bench load client so both arms decode an identical deterministic token stream — the A/B is reproducible and any shared engine-level in-band error appears symmetrically instead of randomly. - Rewrite assert_result_parity to compare drop/truncation counts RELATIVE between arms: modelship dropping more hard-fails; the baseline dropping more is a FINDING (run passes); equal is parity. The old absolute per-arm gate could mis-attribute an upstream engine artifact to modelship's wrapping. - Document a llama_server / GPU results table in bench/README.md mirroring the vllm one, and drop the peak-host-RAM row from both tables. - Update the config NOTE comments to match the relative gate. --- bench/README.md | 59 +++++++++++-- bench/configs/llama-cpu.yaml | 22 ++--- bench/configs/llama-gpu.yaml | 22 ++--- bench/lib.sh | 155 +++++++++++++++++++++++++++++++---- 4 files changed, 216 insertions(+), 42 deletions(-) diff --git a/bench/README.md b/bench/README.md index ba5507c..7b24a8a 100644 --- a/bench/README.md +++ b/bench/README.md @@ -91,6 +91,8 @@ Two cross-checks back this up: ## Results +### vllm / GPU + Example run — 1×GPU, Qwen2.5-7B-AWQ, `--loader vllm --device gpu`, 100 prompts @ concurrency 8, median of 3: | metric | modelship | raw vllm | overhead | @@ -102,7 +104,6 @@ Example run — 1×GPU, Qwen2.5-7B-AWQ, `--loader vllm --device gpu`, 100 prompt | ITL mean (ms) | 12.96 | 12.39 | +4.6% | | TPOT mean (ms) | 12.44 | 12.42 | +0.2% | | peak VRAM (MiB) | 14533 | 14020 | +513 MiB | -| peak host RAM, anon (MiB) | 4957 | 3781 | **+1176 MiB** | Notes: @@ -115,12 +116,58 @@ Notes: the mean/p50 gap (+13.9%/+1.6%) — occasional scheduling jitter through the Ray Serve proxy/router under concurrent load, not a fixed per-request cost. Still small in absolute terms (~25 ms) against a multi-second E2E latency. -- **Host RAM (+1176 MiB)** is the real resource cost — the Ray + Serve control - plane (including the prewarmed idle-worker pool) plus the replica. This is `anon` - (real process memory); **don't** use the container-RSS delta, which is dominated +- **VRAM overhead is modest** (an extra CUDA context, not KV cache). The real + host-RAM cost is the Ray + Serve control plane (including the prewarmed + idle-worker pool) plus the replica — but read it from `anon` in the + per-component table below, **not** the container-RSS delta, which is dominated by reclaimable page cache and swings multiple GB between runs depending on which - cgroup faulted the weights. VRAM overhead is modest (an extra CUDA context), not - KV cache. + cgroup faulted the weights. +- Numbers are illustrative; they vary with model, hardware, load, loader, and device. + +### llama_server / GPU + +Example run — 1×GPU, Qwen2.5-7B-Instruct Q4_K_M GGUF (fully offloaded, `n_gpu_layers: 99`), +`--loader llama_server --device gpu`, 100 prompts @ concurrency 8, greedy (`--temperature 0`), median of 3: + +| metric | modelship | vanilla llama-server | overhead | +| --- | ---: | ---: | ---: | +| completed / 100 | **100** | 93 | — | +| failed | **0** | 7 | — | +| throughput (req/s) | 0.495 | 0.501 | −1.2% | +| output (tok/s) | 253.5 | 256.7 | −1.2% | +| TTFT mean (ms) | 341.2 | 331.6 | +2.9% | +| TTFT p95 (ms) | 491.3 | 443.9 | +10.7% | +| ITL mean (ms) | 30.29 | 29.99 | +1.0% | +| TPOT mean (ms) | 30.2 | 29.9 | +1.0% | +| peak VRAM (MiB) | 5505 | 5372 | +133 MiB | + +Notes: + +- **Decode is at parity** — same `llama-server` binary and GPU, so the engine's + hot path is identical. TPOT/ITL sit within ~1%; modelship adds no per-token cost. +- **modelship completes every request; the raw baseline drops ~7%.** Vanilla + `llama-server`'s failures are all `ServerDisconnectedError` — the bench client + (aiohttp) hits `llama-server`'s cpp-httplib keep-alive close behaviour directly, + a race that Ray Serve's uvicorn front door structurally absorbs. It is **not** + tunable away via `llama-server` flags (`--threads-http` sizes the worker pool, + not the keep-alive lifecycle). So the baseline's small throughput/TTFT edge is + partly **survivorship** — it decoded fewer requests. The survivorship-immune + per-token metrics (TPOT/ITL) are the honest read, and they're at parity. +- **The load client runs greedy (`--temperature 0`).** Both arms then decode an + identical deterministic token stream, so the A/B is reproducible and any + *shared* engine-level in-band error appears symmetrically instead of landing on + one arm by sampling luck. (Under sampling, `--ignore-eos` occasionally makes the + model babble a malformed `` past EOS that `llama-server`'s **own** + grammar parser rejects mid-stream — modelship faithfully relays that as an + in-band SSE error, which greedy decoding eliminates.) +- **The result-parity gate is relative between arms**: modelship dropping or + truncating *more* than the baseline hard-fails the run; the baseline dropping + more (as here) is reported as a **FINDING** and the run passes. This run is a + finding in modelship's favour — 0 drops vs 7. +- **VRAM overhead is modest** (+133 MiB, an extra CUDA context). As with vllm, + read host-RAM cost from `anon` in the per-component table, not the container-RSS + delta (page-cache-dominated and non-deterministic — here the baseline's `file` + cache is actually ~4.4 GiB *higher*). - Numbers are illustrative; they vary with model, hardware, load, loader, and device. ## How the two phases stay comparable diff --git a/bench/configs/llama-cpu.yaml b/bench/configs/llama-cpu.yaml index a695379..b5eba2c 100644 --- a/bench/configs/llama-cpu.yaml +++ b/bench/configs/llama-cpu.yaml @@ -22,13 +22,15 @@ models: # num_gpus: 0 deploy regardless of this field (see llama_server_infer.py # and rawllama_entrypoint.py, which mirrors that branch), so both phases # run CPU-only either way. - # Size llama-server's HTTP worker pool to comfortably cover --concurrency - # (default 8). The baseline arm points the bench client's many concurrent - # short-lived connections *straight* at llama-server's cpp-httplib, which - # oversubscribes the default pool and resets connections (observed as - # ServerDisconnectedError → failed requests); the modelship arm hides this - # behind a single pooled httpx client, so without this the baseline drops - # requests the modelship phase doesn't and the two arms compare unequal - # populations. Applied to BOTH phases (both append extra_args verbatim), - # so launch parity is preserved. - extra_args: ["--threads-http", "16"] + # NOTE: the baseline arm drops ~4-7% of requests with ServerDisconnectedError + # because the bench client hits llama-server's cpp-httplib keep-alive close + # behaviour directly, while the modelship arm faces Ray Serve's uvicorn front + # door and never does. This is NOT tunable away via llama-server flags + # (--threads-http sizes the worker pool, not the keep-alive lifecycle, and + # was verified ineffective), so there is no extra_args mitigation here. The + # asymmetry is real and in modelship's favour: assert_result_parity compares + # drop/truncation counts BETWEEN the two arms — the baseline dropping more is + # reported as a FINDING (run passes), and only modelship dropping MORE than + # the baseline hard-fails. (The load client runs greedy, --temperature 0, so + # both arms process an identical token stream and any shared engine-level + # in-band error shows up in both rather than landing randomly on one.) diff --git a/bench/configs/llama-gpu.yaml b/bench/configs/llama-gpu.yaml index 6983266..edae3aa 100644 --- a/bench/configs/llama-gpu.yaml +++ b/bench/configs/llama-gpu.yaml @@ -23,13 +23,15 @@ models: # the two phases pick different offload counts. 99 exceeds Qwen2.5-7B's # 28 layers, i.e. full GPU offload for both phases. n_gpu_layers: 99 - # Size llama-server's HTTP worker pool to comfortably cover --concurrency - # (default 8). The baseline arm points the bench client's many concurrent - # short-lived connections *straight* at llama-server's cpp-httplib, which - # oversubscribes the default pool and resets connections (observed as - # ServerDisconnectedError → failed requests); the modelship arm hides this - # behind a single pooled httpx client, so without this the baseline drops - # requests the modelship phase doesn't and the two arms compare unequal - # populations. Applied to BOTH phases (both append extra_args verbatim), - # so launch parity is preserved. - extra_args: ["--threads-http", "16"] + # NOTE: the baseline arm drops ~4-7% of requests with ServerDisconnectedError + # because the bench client hits llama-server's cpp-httplib keep-alive close + # behaviour directly, while the modelship arm faces Ray Serve's uvicorn front + # door and never does. This is NOT tunable away via llama-server flags + # (--threads-http sizes the worker pool, not the keep-alive lifecycle, and + # was verified ineffective), so there is no extra_args mitigation here. The + # asymmetry is real and in modelship's favour: assert_result_parity compares + # drop/truncation counts BETWEEN the two arms — the baseline dropping more is + # reported as a FINDING (run passes), and only modelship dropping MORE than + # the baseline hard-fails. (The load client runs greedy, --temperature 0, so + # both arms process an identical token stream and any shared engine-level + # in-band error shows up in both rather than landing randomly on one.) diff --git a/bench/lib.sh b/bench/lib.sh index 7f5c75b..8668d99 100644 --- a/bench/lib.sh +++ b/bench/lib.sh @@ -185,13 +185,21 @@ vram_gate() { # Globs CACHE_DIR for *.gguf; on a cold host before the model is downloaded it # simply finds nothing and returns, which is fine (that phase faults it in during # its own model load, and the next phase is pre-warmed here). +# +# -L (dereference symlinks) is load-bearing: huggingface_hub stores the weights +# as a hash-named blob under blobs/ and exposes it via a snapshots/*.gguf +# *symlink*. Without -L, `-type f` skips the symlink and `-name '*.gguf'` misses +# the extensionless blob, so the glob matches nothing and the warm silently +# no-ops even with the model fully cached. With -L the .gguf symlink resolves to +# its blob and passes -type f. (2>/dev/null swallows the "No such file" find +# prints on any broken symlink.) warm_model_cache() { echo " pre-warming model cache..." local found=0 while IFS= read -r -d '' f; do found=1 cat "$f" > /dev/null 2>&1 || true - done < <(find "$CACHE_DIR" -type f -name '*.gguf' -print0 2>/dev/null) + done < <(find -L "$CACHE_DIR" -type f -name '*.gguf' -print0 2>/dev/null) if (( found )); then echo " model cache warmed." else @@ -244,6 +252,14 @@ run_sweep() { fi fi + # --temperature 0 pins greedy decoding: vllm bench serve no longer forces it + # (it warns and defers to the server default), so without this the two arms + # sample independently and the comparison is nondeterministic run-to-run. With + # --ignore-eos forcing generation past EOS, sampling also occasionally makes a + # request emit a stray `` + malformed tool-call syntax that + # llama-server's own grammar parser rejects mid-stream (an in-band 200-with- + # error, propagated identically by both arms) — greedy makes that deterministic + # and symmetric instead of landing randomly on one arm and reding the run. docker run --rm --network host --user "$(id -u):$(id -g)" \ "${extra_client_args[@]}" \ -v "$out_dir:/out:rw" "$IMAGE" \ @@ -260,9 +276,11 @@ run_sweep() { --max-concurrency $CONCURRENCY \ --num-warmups $NUM_WARMUPS \ --ignore-eos \ + --temperature 0 \ --percentile-metrics ttft,tpot,itl,e2el \ --metric-percentiles 50,95,99 \ --save-result \ + --save-detailed \ --result-dir /out \ --result-filename $fname" } @@ -442,33 +460,138 @@ PY # cpp-httplib connections resetting under load), its tail metrics look better # purely by survivorship. Fail loudly rather than publish a biased comparison. assert_result_parity() { - echo "=== verifying result-population parity ===" - python3 - "$RESULTS_DIR" <<'PY' + echo "=== verifying result-population parity (header + in-band SSE errors) ===" + python3 - "$RESULTS_DIR" "$OUTPUT_LEN" <<'PY' import json, sys from pathlib import Path +# Two ways a request can fail, and the load client (vllm bench serve) only +# reliably catches one of them: +# +# 1. Header-level failure — the connection errors before/at the response +# (e.g. baseline's cpp-httplib keep-alive resets → ServerDisconnectedError). +# The client sets success=False and counts it in `failed`. Visible. +# +# 2. In-band failure — a streaming response whose HTTP 200 headers are already +# flushed, then the body carries an OpenAI-style `data: {"error": ...}` +# chunk followed by `[DONE]`. This is standard OpenAI streaming semantics +# (you cannot downgrade a status once bytes are sent), which modelship +# faithfully reproduces — and note the *error itself* often originates in +# llama-server (e.g. its grammar parser rejecting malformed tool-call output +# the model emits when --ignore-eos forces it past EOS), not in modelship's +# wrapping. vllm bench serve's hand-rolled SSE parser only reads +# `choices`/`usage` and silently skips the error chunk, so it counts the +# request as `completed` with a *truncated* token stream. Invisible to +# `failed`. +# +# Because the sweep runs with --ignore-eos, every healthy request emits exactly +# --random-output-len tokens. So a per-request output length below that (from +# --save-detailed's `output_lens`) is a hidden in-band failure — the only signal +# that survives an in-band error. We count it too, otherwise an arm that silently +# truncated N requests would still show completed==num_prompts and pass a +# survivorship-biased comparison the header check can't catch. +# +# Severity is RELATIVE between the two arms, because the bench's question is "does +# modelship cost anything *versus the raw server it wraps*?" — not "is either arm +# perfectly reliable". Both arms drive the *same* llama-server binary with the +# same greedy (--temperature 0) workload, so an in-band error that is really the +# engine's own (grammar rejection, etc.) shows up in both and is not a wrapping +# cost. Therefore: +# * modelship drops/truncates MORE than baseline → HARD FAIL (exit 1): a real +# cost of the wrapper, and its medians compare unequal populations. +# * baseline drops/truncates MORE than modelship → FINDING (exit 0): a point in +# modelship's favour (its uvicorn front door absorbs the cpp-httplib keep-alive +# resets the raw server exposes). The baseline medians are then over its +# surviving population — the completed/failed rows in summary.md flag that. +# * equal (incl. both zero) → PASS: any drops are shared workload/engine +# behaviour, not attributable to the wrapper. root = Path(sys.argv[1]) -bad = False -for stack in ("modelship", "baseline"): +expected = int(sys.argv[2]) + +def scan(d): + """Return (header_failed, hidden_inband, worst_partial_tok) for one result.""" + completed = d.get("completed", 0) + failed = d.get("failed", 0) + output_lens = d.get("output_lens") + if output_lens: # exact per-request path (needs --save-detailed) + # Header failures already appended output_len 0, so subtract them to + # isolate the hidden (200-with-error) failures miscounted as completed. + short = sum(1 for ol in output_lens if ol < expected) + hidden = max(0, short - failed) + worst = min((ol for ol in output_lens if 0 < ol < expected), default=0) + return failed, hidden, worst + if completed and expected: # aggregate fallback if arrays were stripped + got = d.get("total_output_tokens", 0) + full = completed * expected + hidden = max(0, round((full - got) / expected)) if got < full else 0 + return failed, hidden, 0 + return failed, 0, 0 + +def summarize(stack): + """Print per-sweep detail for one arm and return its (header, hidden) totals.""" + header_tot = hidden_tot = 0 for p in sorted((root / stack).glob("result_*.json")): d = json.loads(p.read_text()) completed = d.get("completed", 0) - failed = d.get("failed", 0) - total = d.get("num_prompts", completed + failed) + total = d.get("num_prompts", completed + d.get("failed", 0)) + failed, hidden, worst = scan(d) + header_tot += failed + hidden_tot += hidden + msgs = [] if failed: - print(f" {stack}/{p.name}: {failed} FAILED / {completed} completed of {total}", file=sys.stderr) - bad = True - -if bad: + msgs.append(f"{failed} header FAILED / {completed} completed of {total}" + + (" (cpp-httplib keep-alive resets)" if stack == "baseline" else "")) + if hidden: + partial = f", shortest partial {worst} tok" if worst else "" + msgs.append(f"{hidden} HIDDEN in-band failure(s) — HTTP 200 but truncated " + f"below output_len={expected}{partial}, miscounted as completed") + for m in msgs: + print(f" {stack}/{p.name}: {m}") + return header_tot, hidden_tot + +m_header, m_hidden = summarize("modelship") +b_header, b_hidden = summarize("baseline") +m_drops = m_header + m_hidden +b_drops = b_header + b_hidden +print(f" totals: modelship {m_drops} dropped/truncated ({m_header} header + {m_hidden} in-band); " + f"baseline {b_drops} ({b_header} header + {b_hidden} in-band)") + +if m_drops > b_drops: + sys.stdout.flush() # keep the per-sweep detail (stdout) ahead of the verdict (stderr) print( - "RESULT PARITY FAILED: an arm dropped requests. The latency/throughput " - "medians compare unequal populations (survivorship bias) and are NOT " - "trustworthy. Re-run; if the baseline keeps failing, raise " - "--threads-http via llama_server_config.extra_args in the bench config.", + f"\nRESULT PARITY FAILED: the modelship arm dropped or truncated MORE requests " + f"than the raw baseline ({m_drops} vs {b_drops}). That excess is a cost of the " + f"wrapper under test, and its latency/throughput medians compare unequal " + f"populations (survivorship bias), so they are NOT trustworthy. Investigate " + f"before trusting this run.", file=sys.stderr, ) sys.exit(1) -print("RESULT PARITY PASSED: both arms completed every request.") +if b_drops > m_drops: + print() + print("FINDING — baseline robustness gap (NOT a failure; run still passes):") + print( + f"The raw llama-server baseline dropped/truncated more requests than modelship " + f"({b_drops} vs {m_drops}) under this load — largely the bench client hitting " + f"cpp-httplib's keep-alive resets directly, which modelship's uvicorn front door " + f"absorbs. This is a point in modelship's favour. Caveat: the baseline " + f"latency/throughput medians in summary.md are computed over its surviving " + f"requests only — read them as an upper bound on the baseline's advantage, not a " + f"like-for-like population (see the completed/failed rows)." + + (f" (modelship itself truncated {m_drops} request(s) — an in-band error it " + f"shares with the baseline's engine, not a drop the baseline avoided.)" + if m_drops else "") + ) + sys.exit(0) +if m_drops: # equal and non-zero + print( + f"\nRESULT PARITY PASSED: both arms dropped/truncated the same number of requests " + f"({m_drops}) — shared workload/engine behaviour (e.g. llama-server rejecting " + f"--ignore-eos-forced malformed tool calls), not a cost of the wrapper. The " + f"populations match, so the medians are comparable." + ) + sys.exit(0) +print("RESULT PARITY PASSED: both arms completed every request in full (no header or in-band errors).") PY }