feat: add llama_server loader (Stage B1 of the parser-migration roadmap)

.github/workflows/ci.yml

@@ -55,21 +55,6 @@ jobs:
       - name: Set up Python
         run: uv python install 3.12
 
-      # The gpu extra's llama-cpp-python wheel (cu130) statically links libcuda.so.1
-      # and fails to import at all without it, even for CPU-only test configs. This
-      # runner has no GPU/driver, so provide NVIDIA's official stub libcuda.so (from
-      # cuda-driver-dev, meant for driverless build/CI environments): its cuInit()
-      # returns CUDA_ERROR_STUB_LIBRARY instead of crashing, which is enough to let
-      # the module load. See CLAUDE.md sharp edges for details.
-      - name: Install CUDA driver stub for llama-cpp-python import
-        run: |
-          curl -sSL -o /tmp/cuda-driver-dev.deb \
-            https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-driver-dev-13-0_13.0.96-1_amd64.deb
-          dpkg-deb -x /tmp/cuda-driver-dev.deb /tmp/cuda-stub
-          stub_dir=/tmp/cuda-stub/usr/local/cuda-13.0/targets/x86_64-linux/lib/stubs
-          ln -s libcuda.so "$stub_dir/libcuda.so.1"
-          echo "LD_LIBRARY_PATH=$stub_dir:${LD_LIBRARY_PATH}" >> "$GITHUB_ENV"
-
       - name: Install dependencies
         run: uv sync --extra dev --extra gpu
 

AGENTS.md

@@ -23,12 +23,12 @@ make lint-fix    # ruff check --fix + ruff format
 make test        # uv run pytest tests/ -v
 
 # Run a single test
-uv run pytest tests/test_config.py::TestLlamaCppConfig::test_defaults -v
+uv run pytest tests/test_config.py::TestLlamaServerConfig::test_defaults -v
 ```
 
 CI (`.github/workflows/ci.yml`) runs `uv sync --extra dev --extra gpu` on Linux, then `ruff check`, `ruff format --check`, `pyright`, and `pytest tests/ -v`. Match that locally before pushing.
 
-`make lint` requires `--extra gpu` to be installed. Pyright resolves imports against the active venv, and `vllm`, `gguf`, `diffusers`, and `psutil` only ship under the gpu extra, so lint on a cpu-only sync fails with `reportMissingImports`. Tests run fine on either extra (the gpu extra is a superset).
+`make lint` requires `--extra gpu` to be installed. Pyright resolves imports against the active venv, and `gguf`, `diffusers`, and `psutil` only ship under the gpu extra, so lint on a cpu-only sync fails with `reportMissingImports`. (`vllm` is importable under both extras as of the Stage E0 CPU wheel wiring — it's no longer gpu-only, just not enough on its own to make lint pass cpu-only.) Tests run fine on either extra (the gpu extra is a superset).
 
 Agents: when running tests on your own initiative (sanity-checking a change, verifying a bump), skip the slow `integration`-marked suite by default — `uv run pytest tests/ -v -m "not integration"`. Only run the full `make test` (which includes integration) when explicitly requested.
 
@@ -69,7 +69,7 @@ The Docker image's `CMD` is `uv run --no-sync mship_deploy.py` (against the venv
 - `modelship/openai/api.py` — FastAPI gateway. Uses `RequestWatcher` + a single shared `DisconnectRegistry` Ray actor (keyed by request id) to propagate client disconnects across process boundaries.
 - `modelship/infer/model_deployment.py` — the single `@serve.deployment` actor class; lazily imports the right backend based on `config.loader`.
 - `modelship/infer/infer_config.py` — pydantic config schemas **and** `RawRequestProxy` / `DisconnectRegistry`. `RawRequestProxy` exists because FastAPI `Request` cannot cross Ray process boundaries; any new attribute vLLM reads from `raw_request` must be added there.
-- `modelship/infer/{vllm,transformers,diffusers,llama_cpp,custom}/` — one subdir per loader. Each has an `*_infer.py` and (for non-custom) an `openai/` adapter subpackage.
+- `modelship/infer/{vllm,diffusers,custom}/` — one subdir per loader. Each has an `*_infer.py` and (for non-custom) an `openai/` adapter subpackage. `modelship/infer/llama_server/llama_server_infer.py` is a flat file with no `openai/` subpackage — it proxies a `llama-server` subprocess's own OpenAI-compatible HTTP API rather than parsing output in-process.
 - `modelship/plugins/base_plugin.py` — `BasePlugin` ABC that plugin packages subclass as `ModelPlugin`.
 - `plugins/*` — workspace packages, each opt-in via a root extra. The plugin module name and the extra name must match (`ensure_plugin()` calls `importlib.import_module(config.plugin)` and the error message says `uv sync --extra <plugin>`).
 
@@ -102,8 +102,8 @@ Commit messages matter: use Conventional Commits prefixes so the changelog gener
 
 - `config/models.yaml` is gitignored; `mship_deploy.py` errors out with a pointer to `config/examples/` if missing.
 - vLLM version is pinned (`vllm==0.24.0`). Do not bump casually — the TP scheduling logic in `mship_deploy.py:build_deployment_options` defaults to the Ray V2 executor, and the loader imports vLLM-internal `entrypoints.*` module paths that upstream restructures between minors.
-- **GGUF is not supported on the `vllm` loader.** 0.24 moved GGUF out of tree; the only external `vllm-gguf-plugin` (`0.0.2`) has a stale `override_quantization_method` signature incompatible with 0.24's quantization API (it breaks *all* quantized models, not just GGUF), so it is deliberately not installed. `resolve_all_model_sources` rejects a `.gguf` on the vllm loader at driver preflight and points to `llama_cpp`. For GGUF use `loader: llama_cpp`; feed the vllm loader safetensors or an AWQ/GPTQ/FP8 quant.
-- `llama_cpp` loader supports CPU or whole-GPU offload. The gpu extra installs a prebuilt CUDA (cu130) wheel from `abetlen.github.io/llama-cpp-python/whl/cu130`; the cpu extra keeps the plain PyPI CPU-only wheel. `num_gpus` must be `0` or a whole integer for llama_cpp — fractional is rejected at config time (llama.cpp has no VRAM-fraction knob). `num_gpus >= 1` honors `n_gpu_layers`; `stable_diffusion_cpp` remains forced to `num_gpus: 0` in `mship_deploy.py:build_deployment_options`.
+- **GGUF is not supported on the `vllm` loader.** 0.24 moved GGUF out of tree; the only external `vllm-gguf-plugin` (`0.0.2`) has a stale `override_quantization_method` signature incompatible with 0.24's quantization API (it breaks *all* quantized models, not just GGUF), so it is deliberately not installed. `resolve_all_model_sources` rejects a `.gguf` on the vllm loader at driver preflight and points to `llama_server`. For GGUF use `loader: llama_server`; feed the vllm loader safetensors or an AWQ/GPTQ/FP8 quant.
+- `llama_server` loader (GGUF) launches a `llama-server` subprocess — found via `MSHIP_LLAMA_SERVER_BIN`, pinned in the Docker images at `/opt/llama.cpp/llama-server.sh` — and proxies its native OpenAI API instead of parsing output in-process. `num_gpus` must be `0` or a whole integer — fractional is rejected at config time (llama.cpp has no VRAM-fraction knob). `num_gpus >= 1` honors `n_gpu_layers`; `stable_diffusion_cpp` remains forced to `num_gpus: 0` in `mship_deploy.py:build_deployment_options`. `--parallel` slots give real request concurrency instead of serializing behind a single lock. Tool-call/reasoning parsing is llama-server's own, auto-detected per chat template: named-function `tool_choice` forcing is unsupported globally (silently falls back to `auto`), and `tool_choice: required` is grammar-enforced for harmony-style templates but a silent no-op for hermes-style ones (e.g. Qwen3); bare `response_format: {"type": "json_object"}` (no `schema` key) is also unenforced despite llama-server's own docs claiming support — `type: json_schema` requests (what modelship sends whenever a schema is given) are unaffected. No persistent on-disk prompt cache. See `docs/model-configuration.md`'s llama_server section for the full field table and examples.
 - Metrics are on by default on port **8079** (not 8000). Disable with `--no-metrics` or `MSHIP_METRICS=false`.
 - Log level `TRACE` (below `DEBUG`) is a custom level and logs full request/response payloads.
 - Docker images are multi-arch (amd64 + arm64). The CPU image uses the unified `Dockerfile` with `--build-arg MSHIP_VARIANT=cpu` and has a different tag suffix (`:latest-cpu`).

CLAUDE.md

@@ -23,12 +23,12 @@ make lint-fix    # auto-fix ruff issues
 make test        # uv run pytest tests/ -v
 
 # Single test
-uv run pytest tests/test_config.py::TestLlamaCppConfig::test_defaults -v
+uv run pytest tests/test_config.py::TestLlamaServerConfig::test_defaults -v
 ```
 
 CI mirrors `make lint` + `pytest tests/ -v`. Match it locally before pushing.
 
-`make lint` requires the `gpu` extra — pyright fails with `reportMissingImports` for `vllm`, `gguf`, `diffusers`, and `psutil` under the cpu sync. Tests pass on either extra.
+`make lint` requires the `gpu` extra — pyright fails with `reportMissingImports` for `gguf`, `diffusers`, and `psutil` under the cpu sync. (`vllm` is now importable under both extras — Stage E0 wired a CPU wheel index — but that alone doesn't unblock a cpu-only lint.) Tests pass on either extra.
 
 When running tests on your own initiative, skip the slow integration suite: `uv run pytest tests/ -v -m "not integration"`. Only run full `make test` when explicitly requested.
 
@@ -45,11 +45,11 @@ Docker's `CMD` is `uv run --no-sync mship_deploy.py` (auto-detecting CPUs/GPUs u
 
 ## Architecture map
 
-- `mship_deploy.py` — Ray init + deploy loop. `build_deployment_options` (in `modelship/deploy/actor_options.py`) handles GPU allocation: multi-slot vLLM deploys (`tp*pp > 1`) always build a Ray Serve placement group (one whole-GPU bundle per slot, STRICT_PACK) that vLLM's ray executor inherits via `get_current_placement_group()`. Single-slot deploys use a scalar `num_gpus` on the outer actor (fractional sharing supported). Fractional `num_gpus` with `tp*pp > 1` is rejected at config time — Ray packs fractional PG bundles onto the same physical GPU. `llama_cpp` loader supports whole-GPU offload (`num_gpus` must be `0` or an integer — fractional is rejected, llama.cpp has no VRAM-fraction knob); `stable_diffusion_cpp` remains CPU-only, forced to `num_gpus: 0`.
+- `mship_deploy.py` — Ray init + deploy loop. `build_deployment_options` (in `modelship/deploy/actor_options.py`) handles GPU allocation: multi-slot vLLM deploys (`tp*pp > 1`) always build a Ray Serve placement group (one whole-GPU bundle per slot, STRICT_PACK) that vLLM's ray executor inherits via `get_current_placement_group()`. Single-slot deploys use a scalar `num_gpus` on the outer actor (fractional sharing supported). Fractional `num_gpus` with `tp*pp > 1` is rejected at config time — Ray packs fractional PG bundles onto the same physical GPU. `llama_server` loader supports whole-GPU offload (`num_gpus` must be `0` or an integer — fractional is rejected, llama.cpp has no VRAM-fraction knob); `stable_diffusion_cpp` remains CPU-only, forced to `num_gpus: 0`.
 - `modelship/openai/api.py` — FastAPI gateway. Uses `RequestWatcher` + a single shared `DisconnectRegistry` Ray actor (keyed by request id) to propagate client disconnects across process boundaries and cancel in-flight inference.
 - `modelship/infer/model_deployment.py` — the single `@serve.deployment` actor class; lazily imports the right backend from `config.loader`.
 - `modelship/infer/infer_config.py` — pydantic config schemas plus `RawRequestProxy` / `DisconnectRegistry`. `RawRequestProxy` exists because FastAPI `Request` can't cross Ray process boundaries. **Any new attribute vLLM reads from `raw_request` must be added there.**
-- `modelship/infer/{vllm,transformers,diffusers,llama_cpp,custom}/` — one subdir per loader, each with an `*_infer.py` and (for non-custom) an `openai/` adapter subpackage.
+- `modelship/infer/{vllm,diffusers,custom}/` — one subdir per loader, each with an `*_infer.py` and (for non-custom) an `openai/` adapter subpackage. `modelship/infer/llama_server/llama_server_infer.py` is the exception: a single flat file (no `openai/` subpackage) — it proxies a `llama-server` subprocess's own OpenAI-compatible HTTP API rather than running modelship's parsers in-process.
 - `modelship/plugins/base_plugin.py` — `BasePlugin` ABC that each plugin package subclasses as `ModelPlugin`.
 - `plugins/*` — workspace packages, each opt-in via a matching root extra. The plugin module name and extra name **must match** (`ensure_plugin()` does `importlib.import_module(config.plugin)`).
 
@@ -66,12 +66,12 @@ Under `tests/`, `pytest-asyncio` for async. Tests **mock out Ray Serve** — the
 ## Sharp edges
 
 - `vllm==0.24.0` is pinned. Don't bump casually — TP scheduling in `mship_deploy.py:build_deployment_options` defaults to the Ray V2 executor, and the loader binds to vLLM-internal `entrypoints.*` module paths that upstream restructures between minors (the `vllm_infer.py` imports moved in 0.22/0.23).
-- **GGUF is unsupported on the `vllm` loader.** 0.24 moved GGUF out of tree, and the only external `vllm-gguf-plugin` (`0.0.2`) has a stale `override_quantization_method` signature that breaks *every* quantized model on 0.24 — so it's deliberately not installed. `resolve_all_model_sources` (in `deploy/config.py`) rejects a `.gguf` on the vllm loader at driver preflight with a pointer to `llama_cpp`. Use `loader: llama_cpp` for GGUF; the vllm loader takes safetensors or AWQ/GPTQ/FP8 quants.
-- `llama_cpp` GPU support relies on the prebuilt CUDA wheel from the `llama-cpp-cu130` index (`abetlen.github.io/llama-cpp-python/whl/cu130`), scoped to the `gpu` extra via `tool.uv.sources`. The `cpu` extra has no source entry, so it resolves the plain PyPI sdist (built from source) as before. Both extras pin the same `llama-cpp-python` floor so they stay in lockstep.
-- The cu130 wheel's `libllama.so` statically links `libcuda.so.1` (the real NVIDIA driver library), unlike torch/vllm which resolve CUDA lazily — so merely `import llama_cpp` crashes on any driverless machine, including CI's `ubuntu-latest` runner. `ci.yml`'s `test` job installs NVIDIA's `cuda-driver-dev` stub `libcuda.so` (via a pinned `.deb`, extracted with `dpkg-deb`, no apt repo added) and symlinks it to `libcuda.so.1` on `LD_LIBRARY_PATH` — its `cuInit()` returns `CUDA_ERROR_STUB_LIBRARY` instead of crashing, which is enough for the import to succeed. If that pinned `.deb` URL ever 404s, grab a current `cuda-driver-dev-13-*` build from `developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/`.
+- **GGUF is unsupported on the `vllm` loader.** 0.24 moved GGUF out of tree, and the only external `vllm-gguf-plugin` (`0.0.2`) has a stale `override_quantization_method` signature that breaks *every* quantized model on 0.24 — so it's deliberately not installed. `resolve_all_model_sources` (in `deploy/config.py`) rejects a `.gguf` on the vllm loader at driver preflight with a pointer to `llama_server`. Use `loader: llama_server` for GGUF; the vllm loader takes safetensors or AWQ/GPTQ/FP8 quants. **This is unconditional on GPU vs. CPU** — the CPU wheel (below) doesn't relax it either; a GGUF gemma still needs `llama_server`, or a non-GGUF checkpoint to run on `vllm`.
+- `vllm==0.24.0+cpu` is installable on the `cpu` extra via an explicit `vllm-cpu` index (`wheels.vllm.ai/0.24.0/cpu`, scoped through `tool.uv.sources`) — the URL embeds the vLLM version, so a future bump must update it. On vLLM's CPU backend, `gpu_memory_utilization` is repurposed to mean *fraction of host RAM* reserved for KV cache, not VRAM; `normalize_num_gpus_and_tp` (`infer_config.py`) lowers its default to `0.4` for `num_gpus: 0` vllm deploys so a naive CPU config doesn't ask to reserve 90% of node RAM and crash at worker init — an explicit value always overrides it. See `config/examples/vllm-cpu.yaml`.
 - Metrics live on port **8079** (not 8000). `MSHIP_METRICS=false` or `--no-metrics` disables. When `mship_deploy` starts its own head (no `--use-existing-ray-cluster`), `connect_ray` pins that port via `ray.init(_metrics_export_port=…)` — a **private** Ray kwarg (accepted through `**kwargs`). A `TestConnectRay` test guards it so a Ray bump that drops it fails loudly.
 - `TRACE` is a custom log level below `DEBUG`; it logs full request/response payloads.
 - Docker CPU image uses the unified `Dockerfile` with `--build-arg MSHIP_VARIANT=cpu` and has a `:latest-cpu` tag suffix.
+- `llama_server` loader launches a `llama-server` subprocess (found via `MSHIP_LLAMA_SERVER_BIN`, pinned in the Docker images at `/opt/llama.cpp`) and proxies its native OpenAI API — concurrency comes from `--parallel` slots instead of a single `asyncio.Lock`. `num_gpus` must be `0` or a whole integer (fractional is rejected, llama.cpp has no VRAM-fraction knob). Tool-call/reasoning parsing is llama-server's own, auto-detected per chat template: named-function `tool_choice` forcing is globally unsupported (silently falls back to `auto`), and `tool_choice: required` is grammar-enforced for harmony-style templates but a silent no-op for hermes-style ones (e.g. Qwen3); bare `response_format: {"type": "json_object"}` (no `schema` key) is also unenforced despite llama-server's own docs claiming support (verified against the b9859 binary directly) — `type: json_schema` requests, which is what modelship actually sends whenever a schema is given, are unaffected and correctly constrained. See `docs/model-configuration.md`'s llama_server section. No persistent on-disk prompt cache.
 
 ## Further reading