diff --git a/docs/model-configuration.md b/docs/model-configuration.md index aba4412..cfe3f9d 100644 --- a/docs/model-configuration.md +++ b/docs/model-configuration.md @@ -9,7 +9,6 @@ Models are configured in a YAML file (default: `config/models.yaml`). Each entry | Argument | Env Var | Default | Description | |---|---|---|---| | `--config` | — | `config/models.yaml` | Path to models config file | -| `--model-stack` | `MSHIP_MODEL_STACK` | — | Auto-generate a config from a [profile](#profiles-mship_model_stack) (`chat`/`assistant`/`studio`/`everything`) sized to detected hardware | | `--gateway-name` | `MSHIP_GATEWAY_NAME` | `modelship api` | Name for the API gateway app | | `--use-existing-ray-cluster` | `MSHIP_USE_EXISTING_RAY_CLUSTER` | `false` | Connect to a Ray cluster you manage (must run on a cluster node) instead of starting one. Implies deploy-and-exit (no teardown) | | `--prune-ray-sessions` | `MSHIP_PRUNE_RAY_SESSIONS` | `true` | When starting its own Ray head, delete stale `session_*` dirs left under the Ray temp root (default `/tmp/ray`) by previous, no-longer-running heads — Ray never cleans these up, so they fill the disk across restarts. A live head's session is always kept. Set `false` to keep them (e.g. for debugging). No effect with `--use-existing-ray-cluster` | @@ -64,47 +63,6 @@ python mship_deploy.py --config config/llm.yaml --gateway-name "llm-api" python mship_deploy.py --config config/tts.yaml --gateway-name "tts-api" ``` -## Profiles (`MSHIP_MODEL_STACK`) - -Don't want to pick models, write YAML, or hand-allocate GPU/CPU? Set a **profile** and modelship generates a config sized to your detected hardware: - -```bash -MSHIP_MODEL_STACK=studio uv run mship_deploy.py -# equivalently: -uv run mship_deploy.py --model-stack studio -``` - -A profile is a set of capabilities to serve. Every model in the catalog is **ungated**, so the one-click path needs no `HF_TOKEN`. - -| Profile | Capabilities | -|---|---| -| `chat` | generate + embed | -| `assistant` | generate + transcription + tts | -| `studio` | generate + image + embed | -| `everything` | generate + image + embed + transcription + tts | - -### Models per tier - -The `generate` and `image` models scale with a **tier** (small/medium/large) picked from your hardware — GPU VRAM if a GPU is present, otherwise RAM and core count. Bigger box → bigger tier. The other capabilities are the same on every box. - -| Capability | small | medium | large | -|---|---|---|---| -| generate (GPU, vLLM/AWQ) | Qwen2.5-7B | Qwen2.5-14B | Qwen2.5-32B | -| generate (CPU, llama.cpp/GGUF) | Llama-3.2-3B | Qwen2.5-7B | Qwen2.5-14B | -| image (GPU, diffusers) | SD-Turbo | SDXL-Turbo | playground-v2.5 | -| image (CPU, stable-diffusion.cpp) | SD-Turbo | SDXL-Turbo | SDXL-base | -| embed (CPU, llama.cpp) | nomic-embed-text-v1.5 | ← | ← | -| tts (CPU, `kokoroonnx`) | Kokoro-82M | ← | ← | -| transcription (CPU, `whispercpp`) | whisper `base` | whisper `small` | ← | - -Selection is **all-or-nothing**: the highest tier whose *complete* stack fits is chosen. A capability is never dropped to squeeze something in — if even the smallest tier won't fit, the deploy refuses with a clear message instead of serving a partial stack. Resource requests (`num_cpus`/`num_gpus`) are filled in automatically; on a shared GPU the LLM gets the larger slice. - -### File behavior and precedence - -- The generated `config/models_stack_.yaml` is **regenerated from scratch on every start** while the profile is set — switch profiles just by changing the value, with no stale file to delete. -- Resolution order: an explicit `--config` always wins; otherwise `MSHIP_MODEL_STACK`/`--model-stack`; otherwise the default `config/models.yaml`. -- The file is normal, editable YAML, but **edits are overwritten** on the next profile-driven start. To keep changes, copy it to `config/models.yaml` (or pass `--config `) and unset `MSHIP_MODEL_STACK`. - ## Fields | Field | Type | Description | @@ -509,7 +467,6 @@ deployment. | Variable | Description | Default | |---|---|---| | `HF_TOKEN` | HuggingFace access token | — | -| `MSHIP_MODEL_STACK` | [Profile](#profiles-mship_model_stack) to auto-generate a hardware-sized config from (`chat`/`assistant`/`studio`/`everything`) | — | | `MSHIP_CACHE_DIR` | Model cache directory (HuggingFace + plugins) | `/.cache` | | `MSHIP_STATE_STORE` | State-store connection URI for the effective config + deploy coordinator (see [State store](#state-store-mship_state_store)) | `memory://` | | `MSHIP_STATE_DIR` | Default directory for a `file://` state store with no path | `/state` | diff --git a/modelship/deploy/config.py b/modelship/deploy/config.py index a1eec79..e98a124 100644 --- a/modelship/deploy/config.py +++ b/modelship/deploy/config.py @@ -17,55 +17,20 @@ def resolve_config_path(arg_path: str | None, config_dir: Path | None = None) -> Precedence: 1. An explicit ``--config`` path always wins (most specific signal); it must exist. - 2. ``MSHIP_MODEL_STACK=`` (or ``--model-stack``) → regenerate - ``models_stack_.yaml`` from scratch on every start, sized to the - detected hardware, and deploy that. Regenerating fresh each time lets the - user switch profiles by just changing the value — no stale file to delete by - hand. Refuses with a clean exit (no partial deploy) if the profile can't fit. - 3. Otherwise the default ``config/models.yaml`` must exist. + 2. Otherwise the default ``config/models.yaml`` must exist. """ config_dir = config_dir or Path(__file__).resolve().parent.parent.parent / "config" - stack = os.environ.get("MSHIP_MODEL_STACK") if arg_path: if not os.path.exists(arg_path): raise FileNotFoundError(f"--config {arg_path} not found.") return arg_path - if stack: - from modelship.deploy.profiles.catalog import PROFILES - from modelship.deploy.profiles.generator import generate_models_yaml - from modelship.deploy.profiles.selector import ProfileDoesNotFitError - - # Validate against the known profiles BEFORE building a path or touching the - # filesystem — `stack` is operator-supplied (env var / CLI), and feeding it - # into the filename unchecked would allow path traversal on the unlink below. - if stack not in PROFILES: - raise SystemExit(f"MSHIP_MODEL_STACK={stack!r}: unknown profile; choose one of {sorted(PROFILES)}.") - - path = config_dir / f"models_stack_{stack}.yaml" - logger.info("MSHIP_MODEL_STACK=%s: generating %s for the detected hardware...", stack, path) - try: - # Remove any prior generation first so a refusal never leaves a stale - # file behind that a later run could mistake for hand-authored config. - path.unlink(missing_ok=True) - generate_models_yaml(stack, str(path)) - except (ProfileDoesNotFitError, ValueError) as e: - raise SystemExit(f"MSHIP_MODEL_STACK={stack}: {e}") from e - except OSError as e: - # Read-only / permission-denied config dir, etc. — fail cleanly instead - # of dumping a traceback. - raise SystemExit(f"MSHIP_MODEL_STACK={stack}: cannot write {path}: {e}") from e - return str(path) - default = config_dir / "models.yaml" if default.exists(): return str(default) - raise FileNotFoundError( - f"{default} not found. Set MSHIP_MODEL_STACK= (or pass --model-stack) to " - f"auto-generate one, or copy an example config from config/examples/ to config/models.yaml." - ) + raise FileNotFoundError(f"{default} not found. Copy an example config from config/examples/ to config/models.yaml.") def load_yaml_config(arg_path: str | None) -> ModelshipConfig: @@ -78,9 +43,7 @@ def load_raw_models(arg_path: str | None) -> list[dict]: The effective-config store keeps raw dicts (not validated configs, which don't round-trip through num_gpus/tp normalization), so the deploy path merges at the - raw-dict level and validates only the merged result. Shares resolve_config_path - with load_yaml_config so MSHIP_MODEL_STACK generation runs at most once per - deploy (callers should use one or the other, not both).""" + raw-dict level and validates only the merged result.""" with open(resolve_config_path(arg_path)) as f: data = yaml.safe_load(f) or {} if not isinstance(data, dict): diff --git a/modelship/deploy/profiles/__init__.py b/modelship/deploy/profiles/__init__.py deleted file mode 100644 index d4ed39e..0000000 --- a/modelship/deploy/profiles/__init__.py +++ /dev/null @@ -1,12 +0,0 @@ -"""One-click model "profiles" (a.k.a. model stacks). - -A user picks a capability set via `MSHIP_MODEL_STACK` (e.g. `chat`, `assistant`, -`studio`, `everything`) and the generator writes a concrete, editable -`config/models.yaml` sized to the detected hardware — removing the need to know -models, allocate resources, or hand-write config. See `budget.py` for how -hardware budgets are read. -""" - -from modelship.deploy.profiles.budget import DeployBudget, read_deploy_budget - -__all__ = ["DeployBudget", "read_deploy_budget"] diff --git a/modelship/deploy/profiles/budget.py b/modelship/deploy/profiles/budget.py deleted file mode 100644 index 085bb04..0000000 --- a/modelship/deploy/profiles/budget.py +++ /dev/null @@ -1,103 +0,0 @@ -"""The resource budget the profile generator sizes models against. - -Split-sourced, per the detection model we locked: - -- **CPU / GPU counts** come from Ray's ledger (`ray.cluster_resources()`), which - already reflects `RAY_HEAD_CPU_NUM` / `RAY_HEAD_GPU_NUM` (or Ray's auto-detected - value, including the container CFS-quota). These are what a generated - deployment may *request* of the scheduler, so they bound `num_cpus`/`num_gpus` - and gate which capabilities we offer. -- **RAM** comes from `detect_ram_bytes()` (cgroup-aware physical RAM) and - **per-GPU VRAM** from `detect_gpus()` — both standalone utilities rather than - the whole `discover_hardware()`, since we don't need its CPU-count or the - HardwareProfile wrapper here. Ray doesn't ledger usable RAM/VRAM bytes (its - `memory` resource under-reports, reserving object-store space), so these come - from physical/container detection. - -This runs on the driver (outside any actor) — the single-node / homogeneous-pool -assumption applies (see the profiles plan in memory). Heterogeneous multi-node -clusters write `models.yaml` by hand. -""" - -from __future__ import annotations - -from dataclasses import dataclass - -from modelship.logging import get_logger -from modelship.preflight import detect_available_ram_bytes, detect_gpus, detect_ram_bytes - -logger = get_logger("deploy.profiles.budget") - - -@dataclass(frozen=True) -class DeployBudget: - """What the generator may allocate on this box. - - `cpu_units` / `gpu_count` are Ray-ledger counts (schedulable). `ram_bytes` is - the container-aware *total* physical RAM (the box's stable size — used for the - human-facing header). `available_ram_bytes` is the container-aware *free* RAM at - deploy time — what the selector actually sizes against. `vram_bytes_per_gpu` is - the conservative per-GPU VRAM across the pool (0 when no usable GPU).""" - - cpu_units: float - gpu_count: int - ram_bytes: int - vram_bytes_per_gpu: int - available_ram_bytes: int = 0 - - @property - def has_gpu(self) -> bool: - """True when Ray will schedule GPUs *and* we measured their VRAM — the - signal to pick the GPU bundle over the CPU bundle.""" - return self.gpu_count > 0 and self.vram_bytes_per_gpu > 0 - - -def read_deploy_budget() -> DeployBudget: - """Read the deploy budget from Ray's ledger + physical detection. - - Must be called after `ray.init()` (the deploy driver is already connected). - """ - import ray - - ledger = ray.cluster_resources() - cpu_units = float(ledger.get("CPU", 0.0)) - gpu_count = int(ledger.get("GPU", 0)) - - ram_bytes = detect_ram_bytes() - available_ram_bytes = detect_available_ram_bytes() - - # Per-GPU VRAM for tiering: take the smallest across the pool so a - # homogeneous fleet sizes to its real per-card budget and a (mis)matched one - # degrades conservatively rather than over-promising. - vram_per_gpu = min((g.available_bytes for g in detect_gpus()), default=0) - - if gpu_count == 0: - # Ray won't schedule GPUs here (no GPU, or fenced to 0) — force the CPU - # path even if the driver physically sees cards. - vram_per_gpu = 0 - elif vram_per_gpu == 0: - # Ray ledgers GPUs but the driver couldn't read their VRAM (no CUDA - # context / pynvml). Degrade to the CPU bundle rather than guess a tier. - logger.warning( - "profiles: Ray ledger reports %d GPU(s) but no VRAM was detected on the driver; " - "falling back to the CPU model bundle.", - gpu_count, - ) - - budget = DeployBudget( - cpu_units=cpu_units, - gpu_count=gpu_count, - ram_bytes=ram_bytes, - available_ram_bytes=available_ram_bytes, - vram_bytes_per_gpu=vram_per_gpu, - ) - logger.info( - "profiles: deploy budget — cpu_units=%.1f gpu_count=%d ram=%.1f GiB (%.1f free) vram/gpu=%.1f GiB (has_gpu=%s)", - budget.cpu_units, - budget.gpu_count, - budget.ram_bytes / 1024**3, - budget.available_ram_bytes / 1024**3, - budget.vram_bytes_per_gpu / 1024**3, - budget.has_gpu, - ) - return budget diff --git a/modelship/deploy/profiles/catalog.py b/modelship/deploy/profiles/catalog.py deleted file mode 100644 index ca439f5..0000000 --- a/modelship/deploy/profiles/catalog.py +++ /dev/null @@ -1,302 +0,0 @@ -"""The curated model catalog behind `MSHIP_MODEL_STACK`. - -Shape (kept deliberately small and hand-verifiable): -- **profiles** = a capability set (which `ModelUsecase`s to serve). -- For each usecase there is a **candidate pool** per accelerator. The selector runs - a weighted knapsack over these pools (one pick per capability) — there is no - "ladder vs satellite" split; embed/tts/transcription are just smaller pools. - -Each spec carries: -- `footprint_bytes` — coarse VRAM (vllm/diffusers) or RAM (CPU loaders); the GPU - figure still gates VRAM bin-packing in the selector/generator. -- `req_min` / `req_rec` — each a `ModeReq(cpu cores, ram bytes, weight)`: the - resources the model needs running at minimum vs. comfortably, plus that mode's - quality `weight`. The knapsack fits the cpu/ram scalars against the host's free - cpu/RAM and maximises total weight. Weights are hand-set so a smaller model at - `req_rec` can outscore a bigger model at `req_min` (the tuning lever) — keep that - ordering in mind when editing them. - -Every `model:` here was verified live against HF: it resolves AND is ungated, so -the one-click path needs no HF token. Gated models (FLUX, SD3.5) are deliberately -excluded; a user can swap them in by editing the generated yaml + setting HF_TOKEN. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any - -from modelship.deploy.profiles.tiers import Accelerator -from modelship.infer.infer_config import ModelLoader, ModelUsecase - -_GiB = 1024**3 - - -@dataclass(frozen=True) -class ModeReq: - """One resource mode (minimum or recommended) of a model: the host resources it - needs plus that mode's quality `weight`. cpu = cores; ram in bytes. VRAM is NOT - here — on a GPU box vllm/diffusers expand to whatever the card allows, so the - VRAM check stays the coarse `footprint_bytes` gate, not a min/rec scalar.""" - - cpu: float - ram_bytes: int - weight: float - - -def _req(cpu: float, ram_gib: float, weight: float) -> ModeReq: - return ModeReq(cpu=cpu, ram_bytes=int(ram_gib * _GiB), weight=weight) - - -@dataclass(frozen=True) -class ModelSpec: - """One catalog entry — enough to build a `models.yaml` model minus the resource - allocation, which the generator fills from the budget. - - `loader_config` is the inner loader-config dict (e.g. sd.cpp's - `sample_steps`/`cfg_scale`); the generator wraps it under the loader's field - name. `footprint_bytes` is coarse — it gates VRAM bin-packing, not runtime - sizing (the per-loader preflight does that at deploy time).""" - - model: str - loader: ModelLoader - usecase: ModelUsecase - footprint_bytes: int - req_min: ModeReq - req_rec: ModeReq - plugin: str | None = None - plugin_config: dict[str, Any] | None = None - loader_config: dict[str, Any] | None = None - - @property - def draws_from_vram(self) -> bool: - """vllm/diffusers consume VRAM on a GPU box; every other loader (the - CPU-pinned ones) consumes system RAM.""" - return self.loader in (ModelLoader.vllm, ModelLoader.diffusers) - - -# --- Profiles: capability sets ------------------------------------------------- - -PROFILES: dict[str, tuple[ModelUsecase, ...]] = { - "chat": (ModelUsecase.generate, ModelUsecase.embed), - "assistant": (ModelUsecase.generate, ModelUsecase.transcription, ModelUsecase.tts), - "studio": (ModelUsecase.generate, ModelUsecase.image, ModelUsecase.embed), - "everything": ( - ModelUsecase.generate, - ModelUsecase.image, - ModelUsecase.embed, - ModelUsecase.transcription, - ModelUsecase.tts, - ), -} - - -# --- Generate pool ------------------------------------------------------------ - -_GENERATE_CPU = ( - ModelSpec( - model="bartowski/Qwen2.5-1.5B-Instruct-GGUF:*Q4_K_M.gguf", - loader=ModelLoader.llama_server, - usecase=ModelUsecase.generate, - footprint_bytes=int(1.5 * _GiB), - req_min=_req(1, 1.8, 18), - req_rec=_req(2, 2.5, 35), - ), - ModelSpec( - model="bartowski/Llama-3.2-3B-Instruct-GGUF:*Q4_K_M.gguf", - loader=ModelLoader.llama_server, - usecase=ModelUsecase.generate, - footprint_bytes=int(3.0 * _GiB), - req_min=_req(2, 3.0, 30), - req_rec=_req(4, 4.5, 50), - ), - ModelSpec( - model="bartowski/Qwen2.5-7B-Instruct-GGUF:*Q4_K_M.gguf", - loader=ModelLoader.llama_server, - usecase=ModelUsecase.generate, - footprint_bytes=int(6.0 * _GiB), - req_min=_req(4, 5.0, 45), - req_rec=_req(6, 7.0, 80), - ), - ModelSpec( - model="bartowski/Qwen2.5-14B-Instruct-GGUF:*Q4_K_M.gguf", - loader=ModelLoader.llama_server, - usecase=ModelUsecase.generate, - footprint_bytes=int(11.0 * _GiB), - req_min=_req(6, 9.5, 70), - req_rec=_req(8, 12.0, 120), - ), -) - -_GENERATE_GPU = ( - ModelSpec( - model="Qwen/Qwen2.5-7B-Instruct-AWQ", - loader=ModelLoader.vllm, - usecase=ModelUsecase.generate, - footprint_bytes=int(6.0 * _GiB), - req_min=_req(1, 2.0, 45), - req_rec=_req(2, 4.0, 80), - ), - ModelSpec( - model="Qwen/Qwen2.5-14B-Instruct-AWQ", - loader=ModelLoader.vllm, - usecase=ModelUsecase.generate, - footprint_bytes=int(11.0 * _GiB), - req_min=_req(1, 3.0, 70), - req_rec=_req(2, 5.0, 120), - ), - ModelSpec( - model="Qwen/Qwen2.5-32B-Instruct-AWQ", - loader=ModelLoader.vllm, - usecase=ModelUsecase.generate, - footprint_bytes=int(20.0 * _GiB), - req_min=_req(2, 4.0, 100), - req_rec=_req(2, 6.0, 170), - ), -) - - -# --- Image pool --------------------------------------------------------------- -# Turbo checkpoints are distilled for ~4 steps with no CFG; SDXL-base wants the -# full ~30 steps. Step counts matter most on CPU (fewer = faster). - -_IMAGE_CPU = ( - ModelSpec( - model="stabilityai/sd-turbo:sd_turbo.safetensors", - loader=ModelLoader.stable_diffusion_cpp, - usecase=ModelUsecase.image, - footprint_bytes=int(2.5 * _GiB), - req_min=_req(2, 2.5, 25), - req_rec=_req(4, 4.0, 40), - loader_config={"sample_steps": 4, "cfg_scale": 1.0, "wtype": "q8_0"}, - ), - ModelSpec( - model="stabilityai/sdxl-turbo:sd_xl_turbo_1.0_fp16.safetensors", - loader=ModelLoader.stable_diffusion_cpp, - usecase=ModelUsecase.image, - footprint_bytes=int(6.0 * _GiB), - req_min=_req(4, 6.0, 50), - req_rec=_req(6, 8.0, 80), - loader_config={"sample_steps": 4, "cfg_scale": 1.0, "wtype": "q8_0"}, - ), - ModelSpec( - model="stabilityai/stable-diffusion-xl-base-1.0:sd_xl_base_1.0.safetensors", - loader=ModelLoader.stable_diffusion_cpp, - usecase=ModelUsecase.image, - footprint_bytes=int(7.0 * _GiB), - req_min=_req(4, 7.0, 70), - req_rec=_req(8, 10.0, 110), - loader_config={"sample_steps": 30, "cfg_scale": 7.0, "wtype": "q8_0"}, - ), -) - -_IMAGE_GPU = ( - ModelSpec( - model="stabilityai/sd-turbo", - loader=ModelLoader.diffusers, - usecase=ModelUsecase.image, - footprint_bytes=int(4.0 * _GiB), - req_min=_req(1, 2.0, 25), - req_rec=_req(2, 3.0, 40), - loader_config={"num_inference_steps": 4, "guidance_scale": 0.0}, - ), - ModelSpec( - model="stabilityai/sdxl-turbo", - loader=ModelLoader.diffusers, - usecase=ModelUsecase.image, - footprint_bytes=int(7.0 * _GiB), - req_min=_req(1, 2.0, 50), - req_rec=_req(2, 3.0, 80), - loader_config={"num_inference_steps": 4, "guidance_scale": 0.0}, - ), - ModelSpec( - model="playgroundai/playground-v2.5-1024px-aesthetic", - loader=ModelLoader.diffusers, - usecase=ModelUsecase.image, - footprint_bytes=int(8.0 * _GiB), - req_min=_req(1, 3.0, 75), - req_rec=_req(2, 4.0, 120), - loader_config={"num_inference_steps": 30, "guidance_scale": 3.0}, - ), -) - - -# --- Satellite pools (CPU-pinned, same on every accelerator) ------------------- - -_EMBED = ( - ModelSpec( - model="nomic-ai/nomic-embed-text-v1.5-GGUF:*f16.gguf", - loader=ModelLoader.llama_server, - usecase=ModelUsecase.embed, - footprint_bytes=int(0.6 * _GiB), - req_min=_req(0.5, 0.6, 7), - req_rec=_req(1, 1.0, 10), - ), -) - -_TTS = ( - ModelSpec( - model="hexgrad/Kokoro-82M", - loader=ModelLoader.custom, - usecase=ModelUsecase.tts, - footprint_bytes=int(0.5 * _GiB), - req_min=_req(1, 0.5, 7), - req_rec=_req(2, 1.0, 10), - plugin="kokoroonnx", - plugin_config={"onnx_provider": "CPUExecutionProvider"}, - ), -) - -# whisper.cpp model *names* (pywhispercpp downloads ggml by name). tiny is the floor -# rung for the smallest boxes (~75 MB; swap to `tiny.en` for English-only HA voice); -# base/small are better and outweigh it, so the knapsack only falls to tiny when it -# must. -_TRANSCRIPTION = ( - ModelSpec( - model="tiny", - loader=ModelLoader.custom, - usecase=ModelUsecase.transcription, - footprint_bytes=int(0.2 * _GiB), - req_min=_req(1, 0.3, 10), - req_rec=_req(1, 0.5, 18), - plugin="whispercpp", - plugin_config={"n_threads": 2}, - ), - ModelSpec( - model="base", - loader=ModelLoader.custom, - usecase=ModelUsecase.transcription, - footprint_bytes=int(0.4 * _GiB), - req_min=_req(1, 0.4, 20), - req_rec=_req(2, 0.8, 30), - plugin="whispercpp", - plugin_config={"n_threads": 2}, - ), - ModelSpec( - model="small", - loader=ModelLoader.custom, - usecase=ModelUsecase.transcription, - footprint_bytes=int(0.7 * _GiB), - req_min=_req(2, 0.7, 40), - req_rec=_req(2, 1.2, 60), - plugin="whispercpp", - plugin_config={"n_threads": 2}, - ), -) - - -# Per-usecase candidate pools, indexed by accelerator. Satellites list the same -# tuple under both accelerators (they're always CPU-pinned). -_CANDIDATES: dict[ModelUsecase, dict[Accelerator, tuple[ModelSpec, ...]]] = { - ModelUsecase.generate: {Accelerator.cpu: _GENERATE_CPU, Accelerator.gpu: _GENERATE_GPU}, - ModelUsecase.image: {Accelerator.cpu: _IMAGE_CPU, Accelerator.gpu: _IMAGE_GPU}, - ModelUsecase.embed: {Accelerator.cpu: _EMBED, Accelerator.gpu: _EMBED}, - ModelUsecase.tts: {Accelerator.cpu: _TTS, Accelerator.gpu: _TTS}, - ModelUsecase.transcription: {Accelerator.cpu: _TRANSCRIPTION, Accelerator.gpu: _TRANSCRIPTION}, -} - - -def candidates(usecase: ModelUsecase, accel: Accelerator) -> tuple[ModelSpec, ...]: - """The candidate model pool for `usecase` on this accelerator (ascending - footprint). Empty tuple if the catalog has no models for it.""" - return _CANDIDATES.get(usecase, {}).get(accel, ()) diff --git a/modelship/deploy/profiles/generator.py b/modelship/deploy/profiles/generator.py deleted file mode 100644 index 3704a42..0000000 --- a/modelship/deploy/profiles/generator.py +++ /dev/null @@ -1,180 +0,0 @@ -"""Generate a transparent, editable `models.yaml` from a profile. - -`MSHIP_MODEL_STACK=` → read the hardware budget → select the stack → -allocate per-deployment resources → write a normal, commented `models.yaml`. From -there the existing deploy path runs unchanged; the file is the user's to edit. - -Resource allocation (the request budgeting we locked): -- **CPU cores** (`num_cpus`) are split from Ray's ledger so the sum never exceeds - it (or Ray would never schedule): the generate anchor gets the bulk, satellites - ~1 core each, image ~2. -- **GPU** (`num_gpus`) is shared fractionally when a profile puts both generate - and image on one GPU (`studio`/`everything`): each gets `gpu_count / n_gpu_models`. - `num_gpus` is the *single* sharing knob — the loader derives each process's VRAM - cap from it (`base_infer._get_memory_fraction` feeds it to vLLM's - `gpu_memory_utilization` and diffusers' per-process fraction), so we deliberately - do NOT also write `gpu_memory_utilization` here (it would be overridden anyway). -""" - -from __future__ import annotations - -from pathlib import Path - -import yaml - -from modelship.deploy.profiles.budget import DeployBudget, read_deploy_budget -from modelship.deploy.profiles.catalog import ModelSpec -from modelship.deploy.profiles.selector import select_stack -from modelship.infer.infer_config import ModelLoader, ModelUsecase -from modelship.logging import get_logger - -logger = get_logger("deploy.profiles.generator") - -# Base CPU-core reservation per role before the generate anchor absorbs leftover. -_CORES_GENERATE_BASE = 2.0 -_CORES_IMAGE = 2.0 -_CORES_SATELLITE = 1.0 - -# Maps a built-in loader to the models.yaml field its inner config lives under. -_LOADER_CONFIG_FIELD = { - ModelLoader.llama_server: "llama_server_config", - ModelLoader.stable_diffusion_cpp: "stable_diffusion_cpp_config", - ModelLoader.diffusers: "diffusers_config", - ModelLoader.vllm: "vllm_engine_kwargs", -} - - -def generate_models_yaml(profile: str, path: str) -> None: - """Select a stack for `profile` on the detected hardware and write it to - `path`. Raises `ProfileDoesNotFitError` (from the selector) without writing - anything if the profile can't be delivered in full.""" - budget = read_deploy_budget() - specs = select_stack(profile, budget) # raises before any file is written - entries = _to_entries(specs, budget) - text = _render(profile, budget, entries) - Path(path).parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: - f.write(text) - logger.info("profiles: wrote %d-model '%s' stack to %s", len(entries), profile, path) - - -def _to_entries(specs: list[ModelSpec], budget: DeployBudget) -> list[dict]: - cpus = _cpu_allocation(specs, budget.cpu_units) - gpu_share = _gpu_allocation(specs, budget.gpu_count) - - entries: list[dict] = [] - for spec, num_cpus in zip(specs, cpus, strict=True): - entry: dict = { - "name": spec.usecase.value, - "model": spec.model, - "usecase": spec.usecase.value, - "loader": spec.loader.value, - "num_cpus": num_cpus, - "num_gpus": gpu_share.get(id(spec), 0), - } - if spec.plugin: - entry["plugin"] = spec.plugin - if spec.plugin_config: - entry["plugin_config"] = dict(spec.plugin_config) - - # No gpu_memory_utilization here: for a fractional num_gpus the loader - # derives the VRAM cap from num_gpus itself (base_infer._get_memory_fraction), - # and that override would clobber anything we wrote anyway. - loader_cfg = dict(spec.loader_config) if spec.loader_config else {} - if loader_cfg and spec.loader in _LOADER_CONFIG_FIELD: - entry[_LOADER_CONFIG_FIELD[spec.loader]] = loader_cfg - - entries.append(entry) - return entries - - -def _gpu_allocation(specs: list[ModelSpec], gpu_count: int) -> dict[int, float]: - """Per-deployment `num_gpus` for the VRAM-drawing models, keyed by `id(spec)`. - - A lone GPU model takes the whole allocation (`num_gpus == gpu_count`). - - When generate + image co-locate, allocation depends on the GPU count: - - - **Fewer GPUs than models** (in practice a single GPU): the GPU is shared - **fractionally, by footprint** — the bigger model (the LLM) gets the larger - slice, so a small co-resident image model doesn't wall off VRAM the LLM - could spend on KV cache. Shares are < 1 and sum to `gpu_count`, which the - config validator accepts only as a single-GPU share. (`num_gpus` is the - single VRAM knob: it drives gpu_memory_utilization for vLLM and the - per-process cap for diffusers.) - - **At least as many GPUs as models**: each model gets ≥ 1 **whole** GPU - (any surplus goes to the largest-footprint models first). Fractional - `num_gpus >= 1` is rejected by the validator, so we must never emit e.g. - 1.4 — multi-GPU boxes get integer allocations. - """ - gpu_models = [s for s in specs if s.draws_from_vram] - if not gpu_models or gpu_count <= 0: - return {} - n = len(gpu_models) - if n == 1: - return {id(gpu_models[0]): float(gpu_count)} - - if gpu_count < n: - # Share a single GPU fractionally, weighted by footprint. - total_fp = sum(s.footprint_bytes for s in gpu_models) or 1 - shares = [round(gpu_count * s.footprint_bytes / total_fp, 3) for s in gpu_models[:-1]] - shares.append(round(gpu_count - sum(shares), 3)) - return {id(s): share for s, share in zip(gpu_models, shares, strict=True)} - - # Enough GPUs to give each model its own: whole-integer allocations, surplus - # to the largest footprints first. - alloc = {id(s): 1 for s in gpu_models} - by_footprint = sorted(gpu_models, key=lambda s: s.footprint_bytes, reverse=True) - for i in range(gpu_count - n): - alloc[id(by_footprint[i % n])] += 1 - return {k: float(v) for k, v in alloc.items()} - - -def _cpu_allocation(specs: list[ModelSpec], cpu_units: float) -> list[float]: - """Per-deployment `num_cpus`, summing to <= `cpu_units` so Ray can always - schedule. Generate gets the leftover; if even the bases don't fit, scale - everything down proportionally.""" - bases = [ - _CORES_GENERATE_BASE - if s.usecase == ModelUsecase.generate - else _CORES_IMAGE - if s.usecase == ModelUsecase.image - else _CORES_SATELLITE - for s in specs - ] - total = sum(bases) - if total <= cpu_units: - leftover = cpu_units - total - # Give the whole leftover to a single anchor (the first generate model), - # not every generate — otherwise a multi-LLM profile would over-subscribe. - anchor = next((i for i, s in enumerate(specs) if s.usecase == ModelUsecase.generate), None) - return [round(b + leftover, 2) if i == anchor else b for i, b in enumerate(bases)] - scale = cpu_units / total - allocs = [round(b * scale, 2) for b in bases] - # Rounding each term to the cent can nudge the sum just above the budget, - # which Ray rejects. Trim the excess in 0.01 steps off the largest allocation - # until the total fits (the guard stops a pathological all-zero loop). - overshoot = round(sum(allocs) - cpu_units, 2) - while overshoot > 0 and any(a > 0 for a in allocs): - i = max(range(len(allocs)), key=lambda j: allocs[j]) - allocs[i] = round(allocs[i] - 0.01, 2) - overshoot = round(overshoot - 0.01, 2) - return allocs - - -def _render(profile: str, budget: DeployBudget, entries: list[dict]) -> str: - accel = "GPU" if budget.has_gpu else "CPU" - detected = f"{budget.ram_bytes / 1024**3:.0f} GiB RAM, {budget.cpu_units:.0f} cores" - if budget.has_gpu: - detected += f", {budget.gpu_count}x {budget.vram_bytes_per_gpu / 1024**3:.0f} GiB GPU" - header = ( - f"# Auto-generated by MSHIP_MODEL_STACK={profile} ({accel} stack).\n" - f"# Detected hardware: {detected}.\n" - f"#\n" - f"# REGENERATED FROM SCRATCH on every start while MSHIP_MODEL_STACK={profile} is set\n" - f"# (or --model-stack {profile} is passed) — edits to this file are overwritten.\n" - f"# To customize and keep your changes, copy it to config/models.yaml (or pass\n" - f"# --config ) and unset MSHIP_MODEL_STACK.\n\n" - ) - body = yaml.safe_dump({"models": entries}, sort_keys=False, default_flow_style=False) - return header + body diff --git a/modelship/deploy/profiles/selector.py b/modelship/deploy/profiles/selector.py deleted file mode 100644 index ddf4e15..0000000 --- a/modelship/deploy/profiles/selector.py +++ /dev/null @@ -1,232 +0,0 @@ -"""Select the concrete model stack for a profile on a given budget. - -Policy (locked with the user): **weighted, fail-fast.** A profile is all-or-nothing -on capabilities — we never drop one. For each capability we pick exactly one model -from its candidate pool, at either its *minimum* or *recommended* resource set, and -choose the combination that **maximises total quality weight** while fitting the -box's free cpu / RAM (and per-GPU VRAM). Weights are hand-set in the catalog so a -smaller model at `recommended` can outscore a bigger one at `min` — the tuning -lever. If no combination fits even at minimum, we raise `ProfileDoesNotFitError` -rather than ship a partial stack. - -The search is a Multiple-Choice Multi-Dimensional Knapsack, but the instances are -tiny (≤ ~1k combos for `everything`, dozens for the rest) so we brute-force every -one-pick-per-capability combination — optimal, no DP. -""" - -from __future__ import annotations - -import itertools -from collections.abc import Iterator -from dataclasses import dataclass -from typing import Literal - -from modelship.deploy.profiles.budget import DeployBudget -from modelship.deploy.profiles.catalog import ( - PROFILES, - ModelSpec, - ModeReq, - candidates, -) -from modelship.deploy.profiles.tiers import Accelerator, accelerator_for -from modelship.infer.infer_config import ModelUsecase -from modelship.logging import get_logger - -logger = get_logger("deploy.profiles.selector") - -# Fraction of free RAM a stack may occupy — headroom for the OS, page cache, and -# KV-cache growth. Matches the llama_server preflight's RAM util. -_UTILIZATION = 0.8 - -# Absolute floor: below this we refuse outright (the user's "don't even try"). -_MIN_CPU_UNITS = 2 - -# Capabilities whose models adapt their own footprint at runtime (llama.cpp n_ctx, -# diffusers tiling) and so must deploy LAST — after the fixed satellites are resident -# — so their preflight sizes against the RAM that's actually left. -_DEPLOY_LAST = (ModelUsecase.generate, ModelUsecase.image) - -Mode = Literal["min", "rec"] - - -class ProfileDoesNotFitError(RuntimeError): - """The chosen profile cannot be delivered in full on this hardware.""" - - -@dataclass(frozen=True) -class _Candidate: - """One (model, resource-mode) option the knapsack may pick for a capability. - `req` carries that mode's cpu/ram demand and its quality weight.""" - - spec: ModelSpec - mode: Mode - req: ModeReq - - -def select_stack(profile: str, budget: DeployBudget) -> list[ModelSpec]: - """Return the model specs for `profile` — the highest-weight combination that - fits this box, ordered satellites-first / generate+image-last (deploy order). - - Raises `ProfileDoesNotFitError` if no combination fits even at minimum or the box - is below the absolute core floor; `ValueError` for an unknown profile.""" - if profile not in PROFILES: - raise ValueError(f"unknown profile {profile!r}; choose one of {sorted(PROFILES)}") - caps = PROFILES[profile] - - if budget.cpu_units < _MIN_CPU_UNITS: - raise ProfileDoesNotFitError( - f"profile {profile!r} needs at least {_MIN_CPU_UNITS} CPU cores; Ray reports " - f"{budget.cpu_units:.0f}. Free up cores, raise RAY_HEAD_CPU_NUM, or write " - f"config/models.yaml by hand." - ) - - accel = accelerator_for(budget) - cpu_cap = budget.cpu_units - ram_cap = int((budget.available_ram_bytes or budget.ram_bytes) * _UTILIZATION) - vram_cap = budget.vram_bytes_per_gpu - - # Per-capability candidate lists (each model in min + rec mode), admission-filtered: - # a candidate whose own demand already exceeds a single host cap can never be in - # a feasible combo, so drop it before enumerating. An empty group means we can't - # place that capability at all → fail fast with a specific message. - groups: list[list[_Candidate]] = [] - for uc in caps: - admitted = [ - cand for spec in candidates(uc, accel) for cand in _modes(spec) if _admits(cand, cpu_cap, ram_cap, vram_cap) - ] - if not admitted: - raise ProfileDoesNotFitError(_no_candidate_message(profile, uc, accel, budget)) - groups.append(admitted) - - best_combo: tuple[_Candidate, ...] | None = None - best_key = (float("-inf"), 0) - for combo in itertools.product(*groups): - cpu_sum = sum(c.req.cpu for c in combo) - # Every model burns host RAM (GPU models too — vLLM/diffusers hold weights, - # KV, and CUDA context in system RAM); VRAM is an *extra* gate below, not a - # substitute for the host-RAM budget. - ram_sum = sum(c.req.ram_bytes for c in combo) - if cpu_sum > cpu_cap or ram_sum > ram_cap: - continue - if not _vram_fits([c.spec for c in combo], budget.gpu_count, vram_cap): - continue - weight = sum(c.req.weight for c in combo) - # Maximise weight; tie-break toward more RAM headroom (lower ram_sum). - key = (weight, -ram_sum) - if key > best_key: - best_key, best_combo = key, combo - - if best_combo is None: - raise ProfileDoesNotFitError(_does_not_fit_message(profile, accel, budget, groups)) - - logger.info( - "profiles: %r -> %s stack (%d models, weight=%.0f): %s", - profile, - accel.value, - len(best_combo), - best_key[0], - ", ".join(f"{c.spec.usecase.value}={_short(c.spec.model)}@{c.mode}" for c in best_combo), - ) - - specs = [c.spec for c in best_combo] - # Deploy order == list order: satellites first, the adaptive generate/image last. - specs.sort(key=lambda s: s.usecase in _DEPLOY_LAST) - return specs - - -def _modes(spec: ModelSpec) -> Iterator[_Candidate]: - """The two resource modes (recommended, minimum) of a model as candidates.""" - yield _Candidate(spec, "rec", spec.req_rec) - yield _Candidate(spec, "min", spec.req_min) - - -def _admits(cand: _Candidate, cpu_cap: float, ram_cap: int, vram_cap: int) -> bool: - """True if this single candidate's own demand fits the host caps. cpu + host RAM - apply to every model; GPU models must additionally fit a card's VRAM (their coarse - footprint).""" - if cand.req.cpu > cpu_cap or cand.req.ram_bytes > ram_cap: - return False - if cand.spec.draws_from_vram: - return cand.spec.footprint_bytes <= vram_cap - return True - - -def _vram_fits(specs: list[ModelSpec], gpu_count: int, vram_cap: int) -> bool: - """Whether the VRAM-drawing models fit a single GPU's budget. Mirrors how the - generator places them: with at least as many GPUs as GPU models each gets its - own card (constraint = the largest single model), otherwise they share one card - (constraint = their sum).""" - gpu_specs = [s for s in specs if s.draws_from_vram] - if not gpu_specs: - return True - if gpu_count >= len(gpu_specs): - need = max(s.footprint_bytes for s in gpu_specs) # one model per GPU - else: - need = sum(s.footprint_bytes for s in gpu_specs) # shared on one GPU - return need <= vram_cap - - -def _short(model: str) -> str: - """Trailing path component of a model id, for compact log lines.""" - return model.split("/")[-1] - - -def _no_candidate_message(profile: str, uc: ModelUsecase, accel: Accelerator, budget: DeployBudget) -> str: - """Message for a capability with no admissible model — even its smallest option - is too big for a single host dimension.""" - pool = candidates(uc, accel) - lighter = ", ".join(p for p in PROFILES if p != profile) - if not pool: - # The catalog has no model for this capability on this accelerator (e.g. an - # unsupported/custom usecase). Bail with a clean message rather than letting - # the `min()` calls below blow up on an empty sequence. - return ( - f"profile {profile!r} requires a {uc.value} model but the catalog has none for " - f"{accel.value}. Choose a lighter profile ({lighter}) or write config/models.yaml by hand." - ) - if accel == Accelerator.gpu and any(s.draws_from_vram for s in pool): - smallest = min(s.footprint_bytes for s in pool if s.draws_from_vram) - return ( - f"profile {profile!r} can't place its {uc.value} model: the smallest option needs " - f"~{smallest / 1024**3:.0f} GiB VRAM, but only {budget.vram_bytes_per_gpu / 1024**3:.0f} GiB/GPU is " - f"free. Choose a lighter profile ({lighter}), add VRAM, or write config/models.yaml by hand." - ) - ram_avail = budget.available_ram_bytes or budget.ram_bytes - smallest_ram = min(s.req_min.ram_bytes for s in pool) - smallest_cpu = min(s.req_min.cpu for s in pool) - return ( - f"profile {profile!r} can't place its {uc.value} model: the smallest option needs " - f"~{smallest_ram / 1024**3:.1f} GiB RAM and {smallest_cpu:.0f} cores, but only " - f"{ram_avail / 1024**3 * _UTILIZATION:.1f} GiB usable RAM and {budget.cpu_units:.0f} cores are free. " - f"Choose a lighter profile ({lighter}), free up RAM, or write config/models.yaml by hand." - ) - - -def _does_not_fit_message( - profile: str, accel: Accelerator, budget: DeployBudget, groups: list[list[_Candidate]] -) -> str: - """Message when each capability has options but no combination fits together — - reports the lightest possible combined demand against the caps.""" - lighter = ", ".join(p for p in PROFILES if p != profile) - cpu_need = sum(min(c.req.cpu for c in g) for g in groups) - # Every model draws host RAM (GPU models too), so the lightest combined host-RAM - # need sums each group's smallest req.ram_bytes regardless of accelerator. - ram_need = sum(min(c.req.ram_bytes for c in g) for g in groups) - ram_avail = budget.available_ram_bytes or budget.ram_bytes - if accel == Accelerator.gpu: - gpu_lightest = [min((c.spec.footprint_bytes for c in g if c.spec.draws_from_vram), default=0) for g in groups] - gpu_lightest = [x for x in gpu_lightest if x] - vram_need = max(gpu_lightest, default=0) if budget.gpu_count >= len(gpu_lightest) else sum(gpu_lightest) - return ( - f"profile {profile!r} does not fit: its lightest stack needs ~{vram_need / 1024**3:.0f} GiB VRAM, " - f"~{ram_need / 1024**3:.0f} GiB host RAM and {cpu_need:.0f} cores, but only " - f"{budget.vram_bytes_per_gpu / 1024**3:.0f} GiB/GPU, {ram_avail / 1024**3 * _UTILIZATION:.0f} GiB usable " - f"RAM and {budget.cpu_units:.0f} cores are free. Choose a lighter profile ({lighter}), add VRAM/RAM, or " - f"write config/models.yaml by hand." - ) - return ( - f"profile {profile!r} does not fit: its lightest stack needs ~{ram_need / 1024**3:.0f} GiB RAM and " - f"{cpu_need:.0f} cores, but only {ram_avail / 1024**3 * _UTILIZATION:.0f} GiB usable RAM and " - f"{budget.cpu_units:.0f} cores are free. Choose a lighter profile ({lighter}), free up RAM, or write " - f"config/models.yaml by hand." - ) diff --git a/modelship/deploy/profiles/tiers.py b/modelship/deploy/profiles/tiers.py deleted file mode 100644 index e2ff21c..0000000 --- a/modelship/deploy/profiles/tiers.py +++ /dev/null @@ -1,24 +0,0 @@ -"""Accelerator family for a `DeployBudget`. - -The accelerator split (GPU present vs not) chooses which loader family the catalog -draws from. Model *sizing* within a family is no longer a tier bucket — the -selector runs a weighted knapsack over each capability's candidate pool against the -box's free cpu/RAM (and per-GPU VRAM). See `selector.py`. -""" - -from __future__ import annotations - -from enum import StrEnum - -from modelship.deploy.profiles.budget import DeployBudget - - -class Accelerator(StrEnum): - cpu = "cpu" - gpu = "gpu" - - -def accelerator_for(budget: DeployBudget) -> Accelerator: - """GPU bundle when Ray will schedule GPUs *and* we measured their VRAM, - else the CPU bundle.""" - return Accelerator.gpu if budget.has_gpu else Accelerator.cpu diff --git a/modelship/utils/cli.py b/modelship/utils/cli.py index 3118cd4..e744bb8 100644 --- a/modelship/utils/cli.py +++ b/modelship/utils/cli.py @@ -17,21 +17,12 @@ "otel_endpoint": "OTEL_EXPORTER_OTLP_ENDPOINT", "api_keys": "MSHIP_API_KEYS", "gateway_name": "MSHIP_GATEWAY_NAME", - "model_stack": "MSHIP_MODEL_STACK", } def parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser = argparse.ArgumentParser(description="Modelship — serve LLMs with Ray Serve") parser.add_argument("--config", help="Path to models.yaml config file (default: config/models.yaml)") - parser.add_argument( - "--model-stack", - help=( - "Profile to auto-generate a config from, e.g. chat/assistant/studio/everything " - "(env: MSHIP_MODEL_STACK). Regenerates config/models_stack_.yaml sized to " - "the detected hardware on every start. Ignored when --config is given." - ), - ) parser.add_argument("--cache-dir", help="Model cache directory (env: MSHIP_CACHE_DIR)") parser.add_argument( "--state-dir", diff --git a/tests/test_integration_profiles.py b/tests/test_integration_profiles.py deleted file mode 100644 index 5b506cb..0000000 --- a/tests/test_integration_profiles.py +++ /dev/null @@ -1,214 +0,0 @@ -"""Integration tests for one-click profiles (`--model-stack`). - -Each test starts its OWN Ray head at a chosen `(num_cpus, num_gpus)` stage, then -runs `mship_deploy.py --model-stack `. Selection reads the deploy budget -from `ray.cluster_resources()`, so the cpu/gpu counts we pass to `ray start` drive -which models the knapsack picks. We then assert: - - * `/v1/models` lists exactly the profile's capability set (deployment names are - the usecases — `generate`, `embed`, `image`, `transcription`, `tts`), proving - the all-or-nothing stack actually loaded and serves; and - * the generated `config/models_stack_.yaml` selected the model we expect - for that stage (e.g. the 1.5B at 2 cores, something larger at 8). - -Host RAM can't be fenced per cluster, so model-size assertions stay machine-robust -(the small box gets the smallest rung; a roomier box gets *something larger*), -never pinning an absolute pick that depends on free RAM. - -NOTE: this module fully owns the Ray cluster and port 8000. Do NOT run it in the -same pytest invocation as `test_integration.py` (which holds a session-scoped -cluster on the same port). Select it on its own with `-m profiles`. -""" - -from __future__ import annotations - -import contextlib -import subprocess -import time -from collections.abc import Iterator -from dataclasses import dataclass -from pathlib import Path - -import httpx -import pytest -import yaml - -from modelship.deploy.profiles.catalog import PROFILES -from modelship.infer.infer_config import ModelUsecase -from openai import OpenAI - -OPENAI_API_BASE = "http://localhost:8000/v1" -HEALTH_URL = "http://localhost:8000/health" - -_REPO_ROOT = Path(__file__).resolve().parents[1] -_CONFIG_DIR = _REPO_ROOT / "config" - -# Generous: a CPU profile pulls 1-2 GiB of GGUF weights on first run, then loads. -_READY_TIMEOUT_S = 600 - - -def _expected_usecase_ids(profile: str) -> set[str]: - return {uc.value for uc in PROFILES[profile]} - - -@dataclass -class _Deployment: - client: OpenAI - config_path: Path # the generated models_stack_.yaml - - def model_ids(self) -> set[str]: - return {m.id for m in self.client.models.list().data} - - def selected_model(self, usecase: ModelUsecase) -> str: - """The `model:` string the generator chose for a usecase, read from the - generated yaml (the /v1/models id is just the usecase name).""" - doc = yaml.safe_load(self.config_path.read_text()) - entry = next(m for m in doc["models"] if m["usecase"] == usecase.value) - return entry["model"] - - -def _ray_start(num_cpus: int, num_gpus: int) -> None: - subprocess.run(["ray", "stop", "--force"], check=False) - subprocess.run( - [ - "ray", - "start", - "--head", - f"--num-cpus={num_cpus}", - f"--num-gpus={num_gpus}", - "--dashboard-host=0.0.0.0", - "--disable-usage-stats", - ], - check=True, - ) - - -def _deploy_cmd(profile: str) -> list[str]: - return ["uv", "run", "mship_deploy.py", "--model-stack", profile, "--use-existing-ray-cluster"] - - -def _wait_for_gateway(log_path: Path) -> None: - """Poll /health until the just-deployed gateway serves (the deploy already - completed, so this is a short confirmation, not a long wait).""" - deadline = time.time() + 120 - while time.time() < deadline: - with contextlib.suppress(Exception): - if httpx.get(HEALTH_URL, timeout=5).status_code == 200: - return - time.sleep(2) - tail = log_path.read_text()[-4000:] if log_path.exists() else "" - pytest.fail(f"gateway not serving after a successful deploy.\nLast 4KB:\n{tail}") - - -@contextlib.contextmanager -def _profile_cluster(profile: str, num_cpus: int, num_gpus: int, tmp_path: Path) -> Iterator[_Deployment]: - """Start a Ray head at `(num_cpus, num_gpus)`, deploy `profile` onto it, and - yield the live `_Deployment`. The deploy runs to completion (models loaded) - before we assert, then `ray stop` tears the whole thing down.""" - log_path = tmp_path / f"mship_deploy_{profile}.log" - config_path = _CONFIG_DIR / f"models_stack_{profile}.yaml" - config_path.unlink(missing_ok=True) - - _ray_start(num_cpus, num_gpus) - try: - with open(log_path, "w") as log_file: - result = subprocess.run( - _deploy_cmd(profile), stdout=log_file, stderr=subprocess.STDOUT, text=True, timeout=_READY_TIMEOUT_S - ) - if result.returncode != 0: - tail = log_path.read_text()[-4000:] - pytest.fail(f"profile {profile!r} ({num_cpus}cpu/{num_gpus}gpu) deploy exited {result.returncode}.\n{tail}") - _wait_for_gateway(log_path) - yield _Deployment(OpenAI(base_url=OPENAI_API_BASE, api_key="not-needed"), config_path) - finally: - subprocess.run(["ray", "stop", "--force"], check=False) - config_path.unlink(missing_ok=True) - - -def _run_deploy_expecting_failure(profile: str, num_cpus: int, num_gpus: int, tmp_path: Path) -> str: - """Deploy a profile that should be refused; return the captured log. Asserts the - deploy exited non-zero (a clean refusal, no gateway).""" - log_path = tmp_path / f"mship_deploy_{profile}_refused.log" - _ray_start(num_cpus, num_gpus) - try: - with open(log_path, "w") as log_file: - result = subprocess.run( - _deploy_cmd(profile), stdout=log_file, stderr=subprocess.STDOUT, text=True, timeout=120 - ) - assert result.returncode != 0, "expected mship_deploy to refuse the profile and exit non-zero" - return log_path.read_text() - finally: - subprocess.run(["ray", "stop", "--force"], check=False) - - -# --- CPU stages --------------------------------------------------------------- - - -@pytest.mark.integration -@pytest.mark.profiles -@pytest.mark.llama_server -def test_chat_on_2_cores_deploys_smallest_generate(tmp_path): - # 2 cores: chat (generate + embed) can only afford the 1.5B generate rung - # alongside embed — the smallest box gets the smallest model. - with _profile_cluster("chat", num_cpus=2, num_gpus=0, tmp_path=tmp_path) as dep: - assert dep.model_ids() == _expected_usecase_ids("chat") - assert "1.5B" in dep.selected_model(ModelUsecase.generate) - - -@pytest.mark.integration -@pytest.mark.profiles -@pytest.mark.llama_server -def test_chat_on_8_cores_scales_generate_up(tmp_path): - # 8 cores: the knapsack must pick a larger generate than the 2-core box did - # (at least the 3B), proving cpu headroom scales the selection up. - with _profile_cluster("chat", num_cpus=8, num_gpus=0, tmp_path=tmp_path) as dep: - assert dep.model_ids() == _expected_usecase_ids("chat") - assert "1.5B" not in dep.selected_model(ModelUsecase.generate) - - -@pytest.mark.integration -@pytest.mark.profiles -@pytest.mark.llama_server -def test_assistant_deploys_full_capability_set(tmp_path): - # assistant = generate + transcription + tts. Needs the whispercpp + kokoroonnx - # plugin wheels available (MSHIP_PLUGIN_WHEEL_DIR / installed extras). - with _profile_cluster("assistant", num_cpus=6, num_gpus=0, tmp_path=tmp_path) as dep: - assert dep.model_ids() == _expected_usecase_ids("assistant") - - -@pytest.mark.integration -@pytest.mark.profiles -def test_everything_refused_on_2_cores_writes_no_gateway(tmp_path): - # 5 models can't each get their minimum cores on a 2-core box → clean refusal - # before any model loads (cheap: no downloads). - log = _run_deploy_expecting_failure("everything", num_cpus=2, num_gpus=0, tmp_path=tmp_path) - assert "does not fit" in log - - -# --- GPU stage ---------------------------------------------------------------- - - -def _has_cuda() -> bool: - try: - import torch - - return torch.cuda.is_available() - except Exception: - return False - - -@pytest.mark.integration -@pytest.mark.profiles -@pytest.mark.vllm -@pytest.mark.diffusers -@pytest.mark.skipif(not _has_cuda(), reason="GPU profile stage requires a CUDA device") -def test_studio_on_one_gpu_uses_gpu_loaders(tmp_path): - # A single GPU: studio (generate + image + embed) must route generate→vllm and - # image→diffusers (embed stays CPU), and serve all three. - with _profile_cluster("studio", num_cpus=8, num_gpus=1, tmp_path=tmp_path) as dep: - assert dep.model_ids() == _expected_usecase_ids("studio") - doc = yaml.safe_load(dep.config_path.read_text()) - by_uc = {m["usecase"]: m for m in doc["models"]} - assert by_uc["generate"]["loader"] == "vllm" - assert by_uc["image"]["loader"] == "diffusers" - assert by_uc["embed"]["loader"] == "llama_server" diff --git a/tests/test_profiles_budget.py b/tests/test_profiles_budget.py deleted file mode 100644 index 715312a..0000000 --- a/tests/test_profiles_budget.py +++ /dev/null @@ -1,69 +0,0 @@ -"""Tests for the profile generator's deploy-budget reader. - -CPU/GPU counts come from Ray's ledger; RAM and per-GPU VRAM from physical -detection. See modelship/deploy/profiles/budget.py.""" - -from __future__ import annotations - -from unittest.mock import patch - -from modelship.deploy.profiles.budget import DeployBudget, read_deploy_budget -from modelship.preflight import GPUInfo - -_GiB = 1024**3 - - -def _read(*, ledger: dict, ram: int, gpus: list[GPUInfo]) -> DeployBudget: - with ( - patch("ray.cluster_resources", return_value=ledger), - patch("modelship.deploy.profiles.budget.detect_ram_bytes", return_value=ram), - patch("modelship.deploy.profiles.budget.detect_gpus", return_value=gpus), - ): - return read_deploy_budget() - - -def test_cpu_only_box(): - b = _read(ledger={"CPU": 8.0}, ram=16 * _GiB, gpus=[]) - assert b.cpu_units == 8.0 - assert b.gpu_count == 0 - assert b.ram_bytes == 16 * _GiB - assert b.vram_bytes_per_gpu == 0 - assert b.has_gpu is False - - -def test_gpu_box_reports_vram(): - gpus = [GPUInfo(0, 24 * _GiB, "L4"), GPUInfo(1, 24 * _GiB, "L4")] - b = _read(ledger={"CPU": 16.0, "GPU": 2.0}, ram=64 * _GiB, gpus=gpus) - assert b.gpu_count == 2 - assert b.vram_bytes_per_gpu == 24 * _GiB - assert b.has_gpu is True - - -def test_heterogeneous_gpus_size_to_smallest(): - gpus = [GPUInfo(0, 16 * _GiB, "A4000"), GPUInfo(1, 24 * _GiB, "L4")] - b = _read(ledger={"CPU": 16.0, "GPU": 2.0}, ram=64 * _GiB, gpus=gpus) - assert b.vram_bytes_per_gpu == 16 * _GiB # conservative: the smaller card - - -def test_ledger_fences_gpus_to_zero_forces_cpu_bundle(): - # Driver physically sees a GPU, but RAY_HEAD_GPU_NUM=0 → ledger has no GPU. - gpus = [GPUInfo(0, 24 * _GiB, "L4")] - b = _read(ledger={"CPU": 8.0, "GPU": 0.0}, ram=32 * _GiB, gpus=gpus) - assert b.gpu_count == 0 - assert b.vram_bytes_per_gpu == 0 - assert b.has_gpu is False - - -def test_ledger_has_gpu_but_no_vram_detected_degrades_to_cpu(): - # Ray ledgers a GPU but the driver couldn't read VRAM (no CUDA ctx / pynvml). - b = _read(ledger={"CPU": 16.0, "GPU": 1.0}, ram=64 * _GiB, gpus=[]) - assert b.gpu_count == 1 - assert b.vram_bytes_per_gpu == 0 - assert b.has_gpu is False - - -def test_missing_ledger_keys_default_to_zero(): - b = _read(ledger={}, ram=8 * _GiB, gpus=[]) - assert b.cpu_units == 0.0 - assert b.gpu_count == 0 - assert b.has_gpu is False diff --git a/tests/test_profiles_catalog.py b/tests/test_profiles_catalog.py deleted file mode 100644 index 9144f77..0000000 --- a/tests/test_profiles_catalog.py +++ /dev/null @@ -1,109 +0,0 @@ -"""Tests for the profile catalog. - -Pure data/functions — no cluster, no model downloads. Source strings are verified -live against HF separately; here we assert structure, resource sets, and weights.""" - -from __future__ import annotations - -import pytest - -from modelship.deploy.profiles.catalog import ( - PROFILES, - ModelSpec, - candidates, -) -from modelship.deploy.profiles.tiers import Accelerator -from modelship.infer.infer_config import ModelLoader, ModelUsecase - -_ALL_USECASES = sorted({uc for caps in PROFILES.values() for uc in caps}, key=lambda u: u.value) - - -def _every_spec() -> list[ModelSpec]: - specs: list[ModelSpec] = [] - for uc in _ALL_USECASES: - for accel in (Accelerator.cpu, Accelerator.gpu): - specs.extend(candidates(uc, accel)) - return specs - - -# --- profiles ----------------------------------------------------------------- - - -def test_profiles_are_the_locked_four(): - assert set(PROFILES) == {"chat", "assistant", "studio", "everything"} - - -# --- candidate pools ---------------------------------------------------------- - - -@pytest.mark.parametrize("uc", _ALL_USECASES) -def test_every_required_usecase_has_candidates_on_both_accelerators(uc): - assert candidates(uc, Accelerator.cpu), f"{uc} has no CPU candidates" - assert candidates(uc, Accelerator.gpu), f"{uc} has no GPU candidates" - - -def test_generate_and_image_split_loaders_by_accelerator(): - for s in candidates(ModelUsecase.generate, Accelerator.cpu) + candidates(ModelUsecase.image, Accelerator.cpu): - assert s.draws_from_vram is False - for s in candidates(ModelUsecase.generate, Accelerator.gpu): - assert s.loader == ModelLoader.vllm and s.draws_from_vram is True - for s in candidates(ModelUsecase.image, Accelerator.gpu): - assert s.loader == ModelLoader.diffusers and s.draws_from_vram is True - - -def test_satellites_are_the_same_cpu_pinned_pool_on_both_accelerators(): - for uc in (ModelUsecase.embed, ModelUsecase.tts, ModelUsecase.transcription): - cpu_pool = candidates(uc, Accelerator.cpu) - assert candidates(uc, Accelerator.gpu) == cpu_pool # identical tuple - assert all(s.draws_from_vram is False for s in cpu_pool) - - -def test_transcription_pool_offers_tiny_base_and_small(): - models = {s.model for s in candidates(ModelUsecase.transcription, Accelerator.cpu)} - assert models == {"tiny", "base", "small"} - - -def test_unknown_usecase_returns_empty_pool(): - # ModelUsecase has members not in any profile (e.g. rerank); pool is empty, not an error. - missing = [uc for uc in ModelUsecase if uc not in _ALL_USECASES] - for uc in missing: - assert candidates(uc, Accelerator.cpu) == () - - -# --- resource sets + weights -------------------------------------------------- - - -def test_every_spec_has_coherent_min_and_rec_reqs(): - for s in _every_spec(): - assert s.req_min.cpu <= s.req_rec.cpu, s.model - assert s.req_min.ram_bytes <= s.req_rec.ram_bytes, s.model - # A comfortable run is worth at least as much as the same model starved. - assert s.req_min.weight <= s.req_rec.weight, s.model - assert s.req_min.weight > 0, s.model - - -@pytest.mark.parametrize("accel", [Accelerator.cpu, Accelerator.gpu]) -def test_generate_and_image_pools_ascend_in_footprint_and_weight(accel): - for uc in (ModelUsecase.generate, ModelUsecase.image): - pool = candidates(uc, accel) - assert [s.footprint_bytes for s in pool] == sorted(s.footprint_bytes for s in pool) - # Bigger rungs are worth more at the same (recommended) mode. - assert [s.req_rec.weight for s in pool] == sorted(s.req_rec.weight for s in pool) - - -def test_recommended_of_a_rung_outscores_minimum_of_the_next(): - # The tuning lever: a smaller model running comfortably should beat the next - # rung up running starved, so the selector prefers the comfortable fit. - from itertools import pairwise - - gen = candidates(ModelUsecase.generate, Accelerator.cpu) - for smaller, bigger in pairwise(gen): - assert smaller.req_rec.weight > bigger.req_min.weight - - -def test_image_turbo_rungs_use_few_steps(): - cpu_img = candidates(ModelUsecase.image, Accelerator.cpu) - # SD-Turbo / SDXL-Turbo: ~4 steps; SDXL-base: ~30. - assert cpu_img[0].loader_config["sample_steps"] == 4 - assert cpu_img[1].loader_config["sample_steps"] == 4 - assert cpu_img[2].loader_config["sample_steps"] == 30 diff --git a/tests/test_profiles_generator.py b/tests/test_profiles_generator.py deleted file mode 100644 index 78ef8c7..0000000 --- a/tests/test_profiles_generator.py +++ /dev/null @@ -1,211 +0,0 @@ -"""Tests for the profile -> models.yaml generator. - -Mocks the budget reader (no Ray) and writes to tmp files. Asserts the rendered -yaml is valid, complete, resource-allocated within the ledger, and that the -refuse path writes nothing.""" - -from __future__ import annotations - -from unittest.mock import patch - -import pytest -import yaml - -from modelship.deploy.profiles.budget import DeployBudget -from modelship.deploy.profiles.catalog import ModelSpec, ModeReq -from modelship.deploy.profiles.generator import _cpu_allocation, generate_models_yaml -from modelship.deploy.profiles.selector import ProfileDoesNotFitError -from modelship.infer.infer_config import ModelLoader, ModelUsecase - -_GiB = 1024**3 - -# _cpu_allocation only reads `usecase`, so the resource sets are placeholders here. -_REQ = ModeReq(cpu=1, ram_bytes=0, weight=1) - - -def _spec(name: str, loader: ModelLoader, usecase: ModelUsecase) -> ModelSpec: - return ModelSpec(name, loader, usecase, 0, req_min=_REQ, req_rec=_REQ) - - -def _cpu(ram_gib: float, cores: float = 8.0) -> DeployBudget: - return DeployBudget(cpu_units=cores, gpu_count=0, ram_bytes=int(ram_gib * _GiB), vram_bytes_per_gpu=0) - - -def _gpu(vram_gib: float, cores: float = 16.0, gpus: int = 1) -> DeployBudget: - return DeployBudget(cpu_units=cores, gpu_count=gpus, ram_bytes=64 * _GiB, vram_bytes_per_gpu=int(vram_gib * _GiB)) - - -def _generate(profile: str, budget: DeployBudget, path) -> dict: - p = str(path / "models.yaml") - with patch("modelship.deploy.profiles.generator.read_deploy_budget", return_value=budget): - generate_models_yaml(profile, p) - with open(p) as f: - text = f.read() - return {"text": text, "doc": yaml.safe_load(text)} - - -def test_chat_cpu_yaml_is_valid_and_complete(tmp_path): - out = _generate("chat", _cpu(16), tmp_path) - models = out["doc"]["models"] - assert {m["usecase"] for m in models} == {"generate", "embed"} - # every entry has the fields the deploy path needs - for m in models: - assert m["name"] and m["model"] and m["loader"] - assert "num_cpus" in m and "num_gpus" in m - assert out["text"].startswith("# Auto-generated by MSHIP_MODEL_STACK=chat") - - -def test_cpu_cores_sum_within_ledger(tmp_path): - out = _generate("everything", _cpu(32, cores=8), tmp_path) - total = sum(m["num_cpus"] for m in out["doc"]["models"]) - assert total <= 8.0 + 1e-6 - - -def test_generate_anchor_gets_the_most_cores(tmp_path): - models = _generate("everything", _cpu(32, cores=8), tmp_path)["doc"]["models"] - by_uc = {m["usecase"]: m for m in models} - gen_cpus = by_uc["generate"]["num_cpus"] - assert gen_cpus == max(m["num_cpus"] for m in models) - - -def test_cpu_stack_requests_no_gpu(tmp_path): - models = _generate("studio", _cpu(32), tmp_path)["doc"]["models"] - assert all(m["num_gpus"] == 0 for m in models) - # CPU image uses the stable_diffusion_cpp loader + its config block - img = next(m for m in models if m["usecase"] == "image") - assert img["loader"] == "stable_diffusion_cpp" - assert "sample_steps" in img["stable_diffusion_cpp_config"] - - -def test_gpu_studio_shares_one_gpu_fractionally(tmp_path): - models = _generate("studio", _gpu(24, gpus=1), tmp_path)["doc"]["models"] - gpu_models = [m for m in models if m["num_gpus"] > 0] - # generate (vllm) + image (diffusers) both on the GPU, footprint-weighted share - assert {m["usecase"] for m in gpu_models} == {"generate", "image"} - gen = next(m for m in models if m["usecase"] == "generate") - img = next(m for m in models if m["usecase"] == "image") - # the LLM (bigger footprint) gets the larger slice; shares cover the whole GPU - assert gen["num_gpus"] > img["num_gpus"] - assert abs(gen["num_gpus"] + img["num_gpus"] - 1.0) < 1e-6 - # num_gpus is the single sharing knob — the loader derives the VRAM cap from - # it, so we must NOT also write gpu_memory_utilization (it would be clobbered). - assert "gpu_memory_utilization" not in gen.get("vllm_engine_kwargs", {}) - # satellites stay on CPU - emb = next(m for m in models if m["usecase"] == "embed") - assert emb["num_gpus"] == 0 - - -def test_gpu_studio_small_split_covers_the_whole_gpu(tmp_path): - # Two GPU models sharing one ~16 GiB card split it fractionally by footprint; - # the shares are each < 1 and cover the whole GPU. (Which exact pair the weighted - # selector picks is asserted elsewhere; here we pin the allocation invariant.) - models = _generate("studio", _gpu(15.3), tmp_path)["doc"]["models"] - gen = next(m for m in models if m["usecase"] == "generate") - img = next(m for m in models if m["usecase"] == "image") - assert 0 < gen["num_gpus"] < 1 and 0 < img["num_gpus"] < 1 - assert abs(gen["num_gpus"] + img["num_gpus"] - 1.0) < 1e-6 - - -def test_multi_gpu_studio_allocates_whole_integer_gpus(tmp_path): - # On a 2-GPU box the co-located models must each get a whole-integer num_gpus - # (>= 1) — fractional num_gpus >= 1 is rejected by the config validator. - out = _generate("studio", _gpu(24, gpus=2), tmp_path) - gpu_models = [m for m in out["doc"]["models"] if m["num_gpus"] > 0] - assert {m["usecase"] for m in gpu_models} == {"generate", "image"} - for m in gpu_models: - assert m["num_gpus"] >= 1 and m["num_gpus"] == int(m["num_gpus"]) - # 2 GPUs, 2 models -> one whole GPU each - assert sum(m["num_gpus"] for m in gpu_models) == 2 - # the generated YAML must actually validate (this is where the old 1.4 crashed) - from pydantic_yaml import parse_yaml_raw_as - - from modelship.infer.infer_config import ModelshipConfig - - parse_yaml_raw_as(ModelshipConfig, out["text"]) - - -def test_multi_gpu_surplus_goes_to_largest_footprint(tmp_path): - # 3 GPUs, 2 models: each gets 1, the surplus GPU goes to the larger (the LLM). - models = _generate("studio", _gpu(24, gpus=3), tmp_path)["doc"]["models"] - gen = next(m for m in models if m["usecase"] == "generate") - img = next(m for m in models if m["usecase"] == "image") - assert gen["num_gpus"] == 2 - assert img["num_gpus"] == 1 - - -def test_gpu_chat_takes_whole_gpu(tmp_path): - models = _generate("chat", _gpu(16, gpus=1), tmp_path)["doc"]["models"] - gen = next(m for m in models if m["usecase"] == "generate") - assert gen["num_gpus"] == 1.0 - # single GPU model → no forced gmu cap - assert "gpu_memory_utilization" not in gen.get("vllm_engine_kwargs", {}) - - -def test_refuse_path_writes_nothing(tmp_path): - p = str(tmp_path / "models.yaml") - with ( - patch("modelship.deploy.profiles.generator.read_deploy_budget", return_value=_cpu(8, cores=1)), - pytest.raises(ProfileDoesNotFitError), - ): - generate_models_yaml("chat", p) - import os - - assert not os.path.exists(p) - - -def test_cpu_allocation_never_exceeds_budget_when_scaling_down(): - # Adversarial tiny budget: naive per-term rounding would sum to 0.05 > 0.04. - specs = [ - _spec("g", ModelLoader.llama_server, ModelUsecase.generate), - _spec("i", ModelLoader.stable_diffusion_cpp, ModelUsecase.image), - _spec("e", ModelLoader.llama_server, ModelUsecase.embed), - ] - allocs = _cpu_allocation(specs, 0.04) - assert round(sum(allocs), 2) <= 0.04 - assert all(a >= 0 for a in allocs) - - -def test_cpu_allocation_leftover_goes_to_a_single_anchor(): - # Two generate models: the leftover must land on only one, not both, or the - # sum would exceed the budget. - specs = [ - _spec("g1", ModelLoader.vllm, ModelUsecase.generate), - _spec("g2", ModelLoader.vllm, ModelUsecase.generate), - _spec("e", ModelLoader.llama_server, ModelUsecase.embed), - ] - allocs = _cpu_allocation(specs, 16.0) - assert sum(allocs) <= 16.0 + 1e-9 - gen = sorted(allocs[:2]) - assert gen[0] == 2.0 # the non-anchor generate stays at its base reservation - assert gen[1] > 2.0 # the anchor absorbed the leftover - - -def test_everything_refused_when_cores_below_per_model_minimum(tmp_path): - # everything is five models; even at their minimums their cores sum past 4, so a - # 4-core box can't give each model its floor — the selector refuses rather than - # cram five models into four cores (writing nothing). - p = str(tmp_path / "models.yaml") - with ( - patch("modelship.deploy.profiles.generator.read_deploy_budget", return_value=_cpu(32, cores=4)), - pytest.raises(ProfileDoesNotFitError), - ): - generate_models_yaml("everything", p) - import os - - assert not os.path.exists(p) - - -def test_cpu_cores_sum_within_ledger_on_a_tight_fit(tmp_path): - # everything just fits an 8-core box; the generator's per-role split must still - # sum within the ledger (the scale-down path is unit-tested separately). - total = sum(m["num_cpus"] for m in _generate("everything", _cpu(32, cores=8), tmp_path)["doc"]["models"]) - assert total <= 8.0 + 1e-9 - - -def test_generate_creates_missing_parent_dir(tmp_path): - import os - - p = str(tmp_path / "does" / "not" / "exist" / "models.yaml") - with patch("modelship.deploy.profiles.generator.read_deploy_budget", return_value=_cpu(16)): - generate_models_yaml("chat", p) - assert os.path.exists(p) diff --git a/tests/test_profiles_hook.py b/tests/test_profiles_hook.py deleted file mode 100644 index 8ef7afc..0000000 --- a/tests/test_profiles_hook.py +++ /dev/null @@ -1,146 +0,0 @@ -"""Tests for the MSHIP_MODEL_STACK / --model-stack hook in resolve_config_path. - -Precedence: explicit --config > profile generation > default config/models.yaml. -The profile always regenerates its own ``models_stack_.yaml`` from scratch, -so switching profiles never requires deleting an old file by hand.""" - -from __future__ import annotations - -from unittest.mock import patch - -import pytest - -from modelship.deploy.config import resolve_config_path -from modelship.deploy.profiles.budget import DeployBudget - -_GIB = 1024**3 - - -def _budget(*, cores: float, ram_gib: float) -> DeployBudget: - return DeployBudget(cpu_units=cores, gpu_count=0, ram_bytes=int(ram_gib * _GIB), vram_bytes_per_gpu=0) - - -def _with_budget(budget: DeployBudget): - return ( - patch("ray.cluster_resources", return_value={"CPU": budget.cpu_units}), - patch("modelship.deploy.profiles.budget.detect_ram_bytes", return_value=budget.ram_bytes), - patch("modelship.deploy.profiles.budget.detect_gpus", return_value=[]), - ) - - -def test_explicit_config_wins_over_env(tmp_path, monkeypatch): - path = tmp_path / "mine.yaml" - path.write_text("models: []\n") - monkeypatch.setenv("MSHIP_MODEL_STACK", "everything") - # No budget mocks needed — generation must not be attempted when --config is given. - assert resolve_config_path(str(path), config_dir=tmp_path) == str(path) - assert path.read_text() == "models: []\n" # untouched - - -def test_explicit_config_missing_raises(tmp_path): - with pytest.raises(FileNotFoundError): - resolve_config_path(str(tmp_path / "nope.yaml"), config_dir=tmp_path) - - -def test_profile_generates_per_profile_file(tmp_path, monkeypatch): - monkeypatch.setenv("MSHIP_MODEL_STACK", "chat") - a, b, c = _with_budget(_budget(cores=8, ram_gib=16)) - with a, b, c: - out = resolve_config_path(None, config_dir=tmp_path) - expected = tmp_path / "models_stack_chat.yaml" - assert out == str(expected) - assert expected.exists() - assert "MSHIP_MODEL_STACK=chat" in expected.read_text() - - -def test_profile_regenerates_fresh_each_time(tmp_path, monkeypatch): - # A stale file from a prior run must be replaced, not preserved. - monkeypatch.setenv("MSHIP_MODEL_STACK", "chat") - stale = tmp_path / "models_stack_chat.yaml" - stale.write_text("models: [stale junk]\n") - a, b, c = _with_budget(_budget(cores=8, ram_gib=16)) - with a, b, c: - out = resolve_config_path(None, config_dir=tmp_path) - text = (tmp_path / "models_stack_chat.yaml").read_text() - assert out == str(stale) - assert "stale junk" not in text - assert "MSHIP_MODEL_STACK=chat" in text - - -def test_switching_profiles_uses_distinct_files(tmp_path, monkeypatch): - a, b, c = _with_budget(_budget(cores=8, ram_gib=16)) - with a, b, c: - monkeypatch.setenv("MSHIP_MODEL_STACK", "chat") - chat = resolve_config_path(None, config_dir=tmp_path) - monkeypatch.setenv("MSHIP_MODEL_STACK", "assistant") - assistant = resolve_config_path(None, config_dir=tmp_path) - assert chat.endswith("models_stack_chat.yaml") - assert assistant.endswith("models_stack_assistant.yaml") - assert (tmp_path / "models_stack_chat.yaml").exists() - assert (tmp_path / "models_stack_assistant.yaml").exists() - - -def test_unfittable_profile_exits_clean_no_file(tmp_path, monkeypatch): - monkeypatch.setenv("MSHIP_MODEL_STACK", "chat") - a, b, c = _with_budget(_budget(cores=1, ram_gib=8)) - with a, b, c, pytest.raises(SystemExit) as exc: - resolve_config_path(None, config_dir=tmp_path) - assert "chat" in str(exc.value) - assert not (tmp_path / "models_stack_chat.yaml").exists() # no partial file - - -def test_unfittable_profile_removes_stale_prior_file(tmp_path, monkeypatch): - # On a refusal, a stale file from a prior fitting run must be gone — never deployed. - monkeypatch.setenv("MSHIP_MODEL_STACK", "chat") - stale = tmp_path / "models_stack_chat.yaml" - stale.write_text("models: [old]\n") - a, b, c = _with_budget(_budget(cores=1, ram_gib=8)) - with a, b, c, pytest.raises(SystemExit): - resolve_config_path(None, config_dir=tmp_path) - assert not stale.exists() - - -def test_no_env_no_default_raises_filenotfound(tmp_path, monkeypatch): - monkeypatch.delenv("MSHIP_MODEL_STACK", raising=False) - with pytest.raises(FileNotFoundError): - resolve_config_path(None, config_dir=tmp_path) - - -def test_no_env_uses_existing_default(tmp_path, monkeypatch): - monkeypatch.delenv("MSHIP_MODEL_STACK", raising=False) - default = tmp_path / "models.yaml" - default.write_text("models: []\n") - assert resolve_config_path(None, config_dir=tmp_path) == str(default) - - -def test_unknown_profile_exits_clean(tmp_path, monkeypatch): - monkeypatch.setenv("MSHIP_MODEL_STACK", "bogus") - a, b, c = _with_budget(_budget(cores=8, ram_gib=16)) - with a, b, c, pytest.raises(SystemExit) as exc: - resolve_config_path(None, config_dir=tmp_path) - assert "bogus" in str(exc.value) - - -def test_oserror_during_generation_exits_clean(tmp_path, monkeypatch): - # A read-only / permission-denied config dir must fail cleanly, not traceback. - monkeypatch.setenv("MSHIP_MODEL_STACK", "chat") - with ( - patch( - "modelship.deploy.profiles.generator.generate_models_yaml", - side_effect=PermissionError("read-only file system"), - ), - pytest.raises(SystemExit) as exc, - ): - resolve_config_path(None, config_dir=tmp_path) - assert "cannot write" in str(exc.value) - assert "chat" in str(exc.value) - - -def test_path_traversal_profile_name_rejected_before_any_fs_op(tmp_path, monkeypatch): - # `stack` is operator-supplied and goes into a filename + unlink, so a crafted - # value must be rejected up front — no path built, no file touched. - monkeypatch.setenv("MSHIP_MODEL_STACK", "../../evil") - with pytest.raises(SystemExit) as exc: - resolve_config_path(None, config_dir=tmp_path) - assert "unknown profile" in str(exc.value) - assert list(tmp_path.iterdir()) == [] # nothing created or deleted diff --git a/tests/test_profiles_selector.py b/tests/test_profiles_selector.py deleted file mode 100644 index d5e5998..0000000 --- a/tests/test_profiles_selector.py +++ /dev/null @@ -1,198 +0,0 @@ -"""Tests for the profile selector — weighted knapsack, fail-fast. - -Asserts the full capability set is always delivered (never partial), that sizing -runs against FREE RAM, that the weight lever prefers a comfortable smaller model -over a starved larger one, that generate/image deploy last, and that too-small -hardware errors rather than degrading.""" - -from __future__ import annotations - -import pytest - -from modelship.deploy.profiles.budget import DeployBudget -from modelship.deploy.profiles.catalog import PROFILES -from modelship.deploy.profiles.selector import ( - _DEPLOY_LAST, - _UTILIZATION, - ProfileDoesNotFitError, - select_stack, -) -from modelship.infer.infer_config import ModelLoader, ModelUsecase - -_GiB = 1024**3 - - -def _cpu(ram_gib: float, cores: float = 8.0, avail_gib: float | None = None) -> DeployBudget: - avail = ram_gib if avail_gib is None else avail_gib - return DeployBudget( - cpu_units=cores, - gpu_count=0, - ram_bytes=int(ram_gib * _GiB), - vram_bytes_per_gpu=0, - available_ram_bytes=int(avail * _GiB), - ) - - -def _gpu(vram_gib: float, ram_gib: float = 64.0, gpus: int = 1, cores: float = 16.0) -> DeployBudget: - return DeployBudget( - cpu_units=cores, - gpu_count=gpus, - ram_bytes=int(ram_gib * _GiB), - vram_bytes_per_gpu=int(vram_gib * _GiB), - available_ram_bytes=int(ram_gib * _GiB), - ) - - -def _usecases(specs) -> set[ModelUsecase]: - return {s.usecase for s in specs} - - -def _gen(specs): - return next(s for s in specs if s.usecase == ModelUsecase.generate) - - -# --- full capability set is always delivered ---------------------------------- - - -@pytest.mark.parametrize("profile", list(PROFILES)) -def test_roomy_box_delivers_every_capability(profile): - specs = select_stack(profile, _cpu(64, cores=16)) - assert _usecases(specs) == set(PROFILES[profile]) - - -@pytest.mark.parametrize("profile", list(PROFILES)) -def test_never_partial_when_it_fits(profile): - specs = select_stack(profile, _cpu(32, cores=16)) - assert _usecases(specs) == set(PROFILES[profile]) - - -# --- deploy order: satellites first, generate/image last ---------------------- - - -@pytest.mark.parametrize("profile", list(PROFILES)) -def test_generate_and_image_deploy_last(profile): - specs = select_stack(profile, _cpu(64, cores=16)) - adaptive_seen = False - for s in specs: - if s.usecase in _DEPLOY_LAST: - adaptive_seen = True - else: - # No satellite may appear after a generate/image model. - assert not adaptive_seen, f"{s.usecase} deployed after a generate/image model" - # The very last model is always an adaptive one (every profile has generate). - assert specs[-1].usecase in _DEPLOY_LAST - - -# --- weighted picks: bigger box, comfortable-over-starved lever ---------------- - - -def test_bigger_box_picks_a_larger_generate(): - small = _gen(select_stack("chat", _cpu(8))) - big = _gen(select_stack("chat", _cpu(64, cores=16))) - assert big.footprint_bytes > small.footprint_bytes - - -def test_prefers_comfortable_smaller_over_starved_larger(): - # ~14 GiB free + plenty of cores: the 14B's *minimum* fits (≈10.5 GiB) and so - # does the 7B's *recommended* (≈8 GiB). The weights make 7B@rec outscore - # 14B@min, so the selector takes the comfortable 7B rather than a starved 14B. - specs = select_stack("chat", _cpu(64, cores=16, avail_gib=14)) - assert "7B" in _gen(specs).model - - -# --- sizing runs against FREE RAM, not total ---------------------------------- - - -def test_available_ram_caps_the_stack_below_total(): - # Huge total RAM but only 8 GiB free → must size like an 8 GiB box (3B), not - # like a 64 GiB one. This is the core fix: co-resident models ate the RAM. - roomy = _gen(select_stack("chat", _cpu(64, cores=16))) - busy = _gen(select_stack("chat", _cpu(64, cores=16, avail_gib=8))) - assert "3B" in busy.model - assert busy.footprint_bytes < roomy.footprint_bytes - - -def test_zero_available_falls_back_to_total(): - # available_ram_bytes == 0 (probe read nothing) must behave like the old total - # basis, not refuse everything. - budget = DeployBudget(cpu_units=16.0, gpu_count=0, ram_bytes=64 * _GiB, vram_bytes_per_gpu=0, available_ram_bytes=0) - specs = select_stack("chat", budget) - assert _usecases(specs) == set(PROFILES["chat"]) - - -# --- fail-fast, never partial ------------------------------------------------- - - -def test_everything_on_tiny_box_is_refused_not_partially_deployed(): - with pytest.raises(ProfileDoesNotFitError) as exc: - select_stack("everything", _cpu(6)) - assert "everything" in str(exc.value) - assert "RAM" in str(exc.value) - - -def test_below_core_floor_is_refused(): - with pytest.raises(ProfileDoesNotFitError): - select_stack("chat", _cpu(32, cores=1)) - - -def test_unknown_profile_raises_valueerror(): - with pytest.raises(ValueError): - select_stack("nonexistent", _cpu(32)) - - -def test_capability_with_no_catalog_models_refuses_cleanly(monkeypatch): - # A profile naming a usecase the catalog has no models for must raise a clean - # ProfileDoesNotFitError, not crash on min() over an empty pool. - from modelship.deploy.profiles import selector as sel - - monkeypatch.setitem(sel.PROFILES, "translate-only", (ModelUsecase.translation,)) - with pytest.raises(ProfileDoesNotFitError) as exc: - select_stack("translate-only", _cpu(32)) - assert "translation" in str(exc.value) - - -# --- accelerator routing + VRAM placement ------------------------------------- - - -def test_gpu_box_uses_gpu_loaders_for_generate_image_cpu_for_satellites(): - specs = select_stack("studio", _gpu(24)) - assert _usecases(specs) == set(PROFILES["studio"]) - gen = _gen(specs) - img = next(s for s in specs if s.usecase == ModelUsecase.image) - emb = next(s for s in specs if s.usecase == ModelUsecase.embed) - assert gen.loader == ModelLoader.vllm - assert img.loader == ModelLoader.diffusers - assert emb.loader == ModelLoader.llama_server # satellite stays CPU - - -def test_studio_on_tiny_gpu_is_refused_on_vram(): - # 8 GiB VRAM can't co-host even the smallest LLM + image model on one card. - with pytest.raises(ProfileDoesNotFitError) as exc: - select_stack("studio", _gpu(8)) - assert "VRAM" in str(exc.value) - - -def test_multi_gpu_fit_checks_largest_model_not_sum(): - # 2x 16 GiB: a 14B (11) + an image model (≤8) sum to >16 but each gets its own - # card, so the binding constraint is the largest single model (11 ≤ 16). The - # selector must reach the 14B, not fall back to the 7B as a summed check would. - specs = select_stack("studio", _gpu(16, gpus=2)) - assert "14B" in _gen(specs).model - - -def test_gpu_models_host_ram_counts_against_the_ram_cap(): - # Ample VRAM but a starved host: studio's GPU generate + image each need a few - # GiB of *host* RAM (weights/KV/CUDA ctx), which must count. With only ~4 GiB - # free the stack can't fit even though every model fits VRAM — the old code - # excluded GPU host RAM and would have wrongly accepted it. - starved = DeployBudget( - cpu_units=16.0, gpu_count=2, ram_bytes=64 * _GiB, vram_bytes_per_gpu=24 * _GiB, available_ram_bytes=4 * _GiB - ) - with pytest.raises(ProfileDoesNotFitError): - select_stack("studio", starved) - - -def test_utilization_headroom_is_applied(): - assert _UTILIZATION == 0.8 - with pytest.raises(ProfileDoesNotFitError): - select_stack("chat", _cpu(2.5, cores=8))