Feat/bench llama server and no preflight

AGENTS.md

@@ -105,6 +105,7 @@ Commit messages matter: use Conventional Commits prefixes so the changelog gener
 - **GGUF is not supported on the `vllm` loader.** 0.24 moved GGUF out of tree; the only external `vllm-gguf-plugin` (`0.0.2`) has a stale `override_quantization_method` signature incompatible with 0.24's quantization API (it breaks *all* quantized models, not just GGUF), so it is deliberately not installed. `resolve_all_model_sources` rejects a `.gguf` on the vllm loader at driver preflight and points to `llama_server`. For GGUF use `loader: llama_server`; feed the vllm loader safetensors or an AWQ/GPTQ/FP8 quant.
 - `llama_server` loader (GGUF) launches a `llama-server` subprocess — found via `MSHIP_LLAMA_SERVER_BIN`, pinned in the Docker images at `/opt/llama.cpp/llama-server.sh` — and proxies its native OpenAI API instead of parsing output in-process. `num_gpus` must be `0` or a whole integer — fractional is rejected at config time (llama.cpp has no VRAM-fraction knob). `num_gpus >= 1` honors `n_gpu_layers`; `stable_diffusion_cpp` remains forced to `num_gpus: 0` in `mship_deploy.py:build_deployment_options`. `--parallel` slots give real request concurrency instead of serializing behind a single lock. Tool-call/reasoning parsing is llama-server's own, auto-detected per chat template: named-function `tool_choice` forcing is unsupported globally (silently falls back to `auto`), and `tool_choice: required` is grammar-enforced for harmony-style templates but a silent no-op for hermes-style ones (e.g. Qwen3); bare `response_format: {"type": "json_object"}` (no `schema` key) is also unenforced despite llama-server's own docs claiming support — `type: json_schema` requests (what modelship sends whenever a schema is given) are unaffected. No persistent on-disk prompt cache. See `docs/model-configuration.md`'s llama_server section for the full field table and examples.
 - Metrics are on by default on port **8079** (not 8000). Disable with `--no-metrics` or `MSHIP_METRICS=false`.
+- Preflight hardware auto-sizing is on by default. Disable with `--no-preflight` or `MSHIP_PREFLIGHT=false` to run models on loader/library defaults plus explicit `models.yaml` config only — useful for benchmarking across hardware.
 - Log level `TRACE` (below `DEBUG`) is a custom level and logs full request/response payloads.
 - Docker images are multi-arch (amd64 + arm64). The CPU image uses the unified `Dockerfile` with `--build-arg MSHIP_VARIANT=cpu` and has a different tag suffix (`:latest-cpu`).
 

Dockerfile

@@ -76,24 +76,33 @@ RUN apt-get update -y && \
         curl \
         espeak-ng \
         gcc \
+        g++ \
         gnupg \
         gosu \
         libc6-dev \
-        libgomp1 && \
+        libgomp1 \
+        libnuma1 \
+        ninja-build && \
     rm -rf /var/lib/apt/lists/*
 
-# Register the NVIDIA CUDA apt repo and install cuda-cudart (GPU variant only).
-# gcc + libc6-dev stay because torch/triton JIT-compile kernels at model-load
-# time and shell out to $CC; without them, vllm crashes in _inductor with
-# "Failed to find C compiler".
+# Register the NVIDIA CUDA apt repo and install cuda-cudart, cuda-nvcc, and
+# cuda-cuobjdump (GPU variant only). gcc/g++ + libc6-dev and ninja-build stay
+# because torch/triton and flashinfer JIT-compile kernels at model-load time
+# and shell out to $CC/nvcc; without them, vllm crashes in _inductor (needs
+# g++ for its CPU codegen backend, e.g. the vllm CPU loader's torch.compile
+# path) or flashinfer on newer architectures (such as Blackwell).
 RUN if [ "$MSHIP_VARIANT" = "gpu" ]; then \
     CUDA_VERSION_DASH=$(echo $CUDA_VERSION | cut -d. -f1,2 | tr '.' '-') && \
     curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \
         | gpg --dearmor -o /usr/share/keyrings/cuda-keyring.gpg && \
     echo "deb [signed-by=/usr/share/keyrings/cuda-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /" \
         > /etc/apt/sources.list.d/cuda.list && \
     apt-get update -y && \
-    apt-get install -y --no-install-recommends cuda-cudart-${CUDA_VERSION_DASH} && \
+    apt-get install -y --no-install-recommends \
+        cuda-cudart-${CUDA_VERSION_DASH} \
+        cuda-nvcc-${CUDA_VERSION_DASH} \
+        cuda-cuobjdump-${CUDA_VERSION_DASH} \
+        libcurand-dev-${CUDA_VERSION_DASH} && \
     apt-get purge -y --auto-remove gnupg && \
     rm -f /etc/apt/sources.list.d/cuda.list /usr/share/keyrings/cuda-keyring.gpg && \
     rm -rf /var/lib/apt/lists/*; \

README.md

@@ -14,7 +14,7 @@ Most self-hosted inference tools focus on running a single model. Modelship is f
 - **GPU memory control** — allocate exact GPU fractions per model (e.g. 70% for the LLM, 5% for TTS) so everything fits on your hardware
 - **Mix and match backends** — use vLLM for high-throughput GPU inference, vLLM or llama.cpp for CPU-only workloads, Diffusers for images, and plugins for custom backends — in the same deployment
 - **Agentic-ready** — reasoning (`<think>` → `reasoning_content`), universal tool/function calling, and the `/v1/responses` API work across the vLLM and llama.cpp (`llama_server`) loaders — the OpenAI-spec surface agents already target
-- **Drop-in OpenAI replacement** — any OpenAI SDK client works out of the box, making it easy to integrate with existing apps and tools like [Home Assistant](docs/home-assistant.md)
+- **Drop-in OpenAI replacement** — any OpenAI SDK client works out of the box, making it easy to integrate with existing apps and tools
 
 ## Architecture
 
@@ -179,7 +179,6 @@ For a full guide on writing your own plugin, see [Plugin Development](docs/plugi
 - [Model Configuration](docs/model-configuration.md) — full `models.yaml` reference, GPU pinning, environment variables
 - [Architecture](docs/architecture.md) — system design, request lifecycle, plugin loading
 - [Plugin Development](docs/plugins.md) — writing custom TTS/STT backends
-- [Home Assistant Integration](docs/home-assistant.md) — Wyoming protocol setup for voice automation
 - [Monitoring & Logging](docs/monitoring.md) — Prometheus metrics, Grafana dashboard, structured logging, health checks
 - [Troubleshooting](docs/troubleshooting.md) — common first-run errors and fixes