Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 0 additions & 43 deletions docs/model-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ Models are configured in a YAML file (default: `config/models.yaml`). Each entry
| Argument | Env Var | Default | Description |
|---|---|---|---|
| `--config` | — | `config/models.yaml` | Path to models config file |
| `--model-stack` | `MSHIP_MODEL_STACK` | — | Auto-generate a config from a [profile](#profiles-mship_model_stack) (`chat`/`assistant`/`studio`/`everything`) sized to detected hardware |
| `--gateway-name` | `MSHIP_GATEWAY_NAME` | `modelship api` | Name for the API gateway app |
| `--use-existing-ray-cluster` | `MSHIP_USE_EXISTING_RAY_CLUSTER` | `false` | Connect to a Ray cluster you manage (must run on a cluster node) instead of starting one. Implies deploy-and-exit (no teardown) |
| `--prune-ray-sessions` | `MSHIP_PRUNE_RAY_SESSIONS` | `true` | When starting its own Ray head, delete stale `session_*` dirs left under the Ray temp root (default `/tmp/ray`) by previous, no-longer-running heads — Ray never cleans these up, so they fill the disk across restarts. A live head's session is always kept. Set `false` to keep them (e.g. for debugging). No effect with `--use-existing-ray-cluster` |
Expand Down Expand Up @@ -64,47 +63,6 @@ python mship_deploy.py --config config/llm.yaml --gateway-name "llm-api"
python mship_deploy.py --config config/tts.yaml --gateway-name "tts-api"
```

## Profiles (`MSHIP_MODEL_STACK`)

Don't want to pick models, write YAML, or hand-allocate GPU/CPU? Set a **profile** and modelship generates a config sized to your detected hardware:

```bash
MSHIP_MODEL_STACK=studio uv run mship_deploy.py
# equivalently:
uv run mship_deploy.py --model-stack studio
```

A profile is a set of capabilities to serve. Every model in the catalog is **ungated**, so the one-click path needs no `HF_TOKEN`.

| Profile | Capabilities |
|---|---|
| `chat` | generate + embed |
| `assistant` | generate + transcription + tts |
| `studio` | generate + image + embed |
| `everything` | generate + image + embed + transcription + tts |

### Models per tier

The `generate` and `image` models scale with a **tier** (small/medium/large) picked from your hardware — GPU VRAM if a GPU is present, otherwise RAM and core count. Bigger box → bigger tier. The other capabilities are the same on every box.

| Capability | small | medium | large |
|---|---|---|---|
| generate (GPU, vLLM/AWQ) | Qwen2.5-7B | Qwen2.5-14B | Qwen2.5-32B |
| generate (CPU, llama.cpp/GGUF) | Llama-3.2-3B | Qwen2.5-7B | Qwen2.5-14B |
| image (GPU, diffusers) | SD-Turbo | SDXL-Turbo | playground-v2.5 |
| image (CPU, stable-diffusion.cpp) | SD-Turbo | SDXL-Turbo | SDXL-base |
| embed (CPU, llama.cpp) | nomic-embed-text-v1.5 | ← | ← |
| tts (CPU, `kokoroonnx`) | Kokoro-82M | ← | ← |
| transcription (CPU, `whispercpp`) | whisper `base` | whisper `small` | ← |

Selection is **all-or-nothing**: the highest tier whose *complete* stack fits is chosen. A capability is never dropped to squeeze something in — if even the smallest tier won't fit, the deploy refuses with a clear message instead of serving a partial stack. Resource requests (`num_cpus`/`num_gpus`) are filled in automatically; on a shared GPU the LLM gets the larger slice.

### File behavior and precedence

- The generated `config/models_stack_<profile>.yaml` is **regenerated from scratch on every start** while the profile is set — switch profiles just by changing the value, with no stale file to delete.
- Resolution order: an explicit `--config` always wins; otherwise `MSHIP_MODEL_STACK`/`--model-stack`; otherwise the default `config/models.yaml`.
- The file is normal, editable YAML, but **edits are overwritten** on the next profile-driven start. To keep changes, copy it to `config/models.yaml` (or pass `--config <file>`) and unset `MSHIP_MODEL_STACK`.

## Fields

| Field | Type | Description |
Expand Down Expand Up @@ -509,7 +467,6 @@ deployment.
| Variable | Description | Default |
|---|---|---|
| `HF_TOKEN` | HuggingFace access token | — |
| `MSHIP_MODEL_STACK` | [Profile](#profiles-mship_model_stack) to auto-generate a hardware-sized config from (`chat`/`assistant`/`studio`/`everything`) | — |
| `MSHIP_CACHE_DIR` | Model cache directory (HuggingFace + plugins) | `/.cache` |
| `MSHIP_STATE_STORE` | State-store connection URI for the effective config + deploy coordinator (see [State store](#state-store-mship_state_store)) | `memory://` |
| `MSHIP_STATE_DIR` | Default directory for a `file://` state store with no path | `<cache-dir>/state` |
Expand Down
43 changes: 3 additions & 40 deletions modelship/deploy/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,55 +17,20 @@ def resolve_config_path(arg_path: str | None, config_dir: Path | None = None) ->

Precedence:
1. An explicit ``--config`` path always wins (most specific signal); it must exist.
2. ``MSHIP_MODEL_STACK=<profile>`` (or ``--model-stack``) → regenerate
``models_stack_<profile>.yaml`` from scratch on every start, sized to the
detected hardware, and deploy that. Regenerating fresh each time lets the
user switch profiles by just changing the value — no stale file to delete by
hand. Refuses with a clean exit (no partial deploy) if the profile can't fit.
3. Otherwise the default ``config/models.yaml`` must exist.
2. Otherwise the default ``config/models.yaml`` must exist.
"""
config_dir = config_dir or Path(__file__).resolve().parent.parent.parent / "config"
stack = os.environ.get("MSHIP_MODEL_STACK")

if arg_path:
if not os.path.exists(arg_path):
raise FileNotFoundError(f"--config {arg_path} not found.")
return arg_path

if stack:
from modelship.deploy.profiles.catalog import PROFILES
from modelship.deploy.profiles.generator import generate_models_yaml
from modelship.deploy.profiles.selector import ProfileDoesNotFitError

# Validate against the known profiles BEFORE building a path or touching the
# filesystem — `stack` is operator-supplied (env var / CLI), and feeding it
# into the filename unchecked would allow path traversal on the unlink below.
if stack not in PROFILES:
raise SystemExit(f"MSHIP_MODEL_STACK={stack!r}: unknown profile; choose one of {sorted(PROFILES)}.")

path = config_dir / f"models_stack_{stack}.yaml"
logger.info("MSHIP_MODEL_STACK=%s: generating %s for the detected hardware...", stack, path)
try:
# Remove any prior generation first so a refusal never leaves a stale
# file behind that a later run could mistake for hand-authored config.
path.unlink(missing_ok=True)
generate_models_yaml(stack, str(path))
except (ProfileDoesNotFitError, ValueError) as e:
raise SystemExit(f"MSHIP_MODEL_STACK={stack}: {e}") from e
except OSError as e:
# Read-only / permission-denied config dir, etc. — fail cleanly instead
# of dumping a traceback.
raise SystemExit(f"MSHIP_MODEL_STACK={stack}: cannot write {path}: {e}") from e
return str(path)

default = config_dir / "models.yaml"
if default.exists():
return str(default)

raise FileNotFoundError(
f"{default} not found. Set MSHIP_MODEL_STACK=<profile> (or pass --model-stack) to "
f"auto-generate one, or copy an example config from config/examples/ to config/models.yaml."
)
raise FileNotFoundError(f"{default} not found. Copy an example config from config/examples/ to config/models.yaml.")


def load_yaml_config(arg_path: str | None) -> ModelshipConfig:
Expand All @@ -78,9 +43,7 @@ def load_raw_models(arg_path: str | None) -> list[dict]:

The effective-config store keeps raw dicts (not validated configs, which don't
round-trip through num_gpus/tp normalization), so the deploy path merges at the
raw-dict level and validates only the merged result. Shares resolve_config_path
with load_yaml_config so MSHIP_MODEL_STACK generation runs at most once per
deploy (callers should use one or the other, not both)."""
raw-dict level and validates only the merged result."""
with open(resolve_config_path(arg_path)) as f:
data = yaml.safe_load(f) or {}
if not isinstance(data, dict):
Expand Down
12 changes: 0 additions & 12 deletions modelship/deploy/profiles/__init__.py

This file was deleted.

103 changes: 0 additions & 103 deletions modelship/deploy/profiles/budget.py

This file was deleted.

Loading
Loading