diff --git a/.claude/skills/adding-a-model-profile/SKILL.md b/.claude/skills/adding-a-model-profile/SKILL.md new file mode 100644 index 0000000..82f6db5 --- /dev/null +++ b/.claude/skills/adding-a-model-profile/SKILL.md @@ -0,0 +1,195 @@ +--- +name: adding-a-model-profile +description: "Add or change a vLLM model profile (ansible/models/.yml) in inference-gpu-spinner. Use when asked to add, port or define a model, checkpoint or quantization for bin/spin — \"add Qwen3.9 to the spinner\", \"make a profile for this HF repo\", \"support the NVFP4 build\", \"why is this model on B200?\", \"can this run on the RTX card?\" — or to change an existing profile's plan, context, image or gates. Offline and free: this skill never provisions anything. Testing the result is a separate skill, testing-a-model-profile." +--- + +# Adding a model profile + +A profile is one YAML file in `ansible/models/`. It is the *only* thing that decides which GPU +plan a model lands on, what vLLM is told to do, and whether the deploy is allowed at all. Getting +it wrong costs real money, because most mistakes surface only after a paid server has downloaded +several hundred gigabytes. + +**This skill is offline. You must not run `bin/spin up`, `bin/spin prefetch`, `tofu apply` or any +Ansible play against a host.** Finish the profile, run the free gates, and hand off. + +## Rule one + +**Write down only what you measured. Everything else is `false`, absent, or the recipe's number +clearly labelled as such.** `validated:` defaults to **true** in the role +(`ansible/roles/vllm/tasks/main.yml`), so a new profile that omits it silently claims a live run it +never had. `glm52.yml` shipped that way and nobody noticed for months. + +## 1. Research before you write anything + +```bash +REPO=owner/name +curl -s "https://huggingface.co/api/models/$REPO" | jq '{gated, pipeline_tag}' +curl -s "https://huggingface.co/$REPO/resolve/main/config.json" | jq '{architectures, num_hidden_layers, max_position_embeddings, quantization_config}' +# The per-layer attention map — this is what tells you the checkpoint is hybrid. +curl -s "https://huggingface.co/$REPO/resolve/main/config.json" \ + | jq '{model_type, linear_attn_config, full_attn_layers, full_attention_interval, + layer_types: (.layer_types // [] | group_by(.) | map({(.[0]): length}) | add)}' +# Real download size — sum the file listing. Avoid `usedStorage`: it counts every revision. +curl -s "https://huggingface.co/api/models/$REPO?blobs=true" \ + | jq '[.siblings[].size] | add / 1073741824 | floor | tostring + " GB"' +``` + +- `architectures[0]` is what `tests/check-vllm-compat.py` checks against the pinned image. +- `max_position_embeddings` is the checkpoint's **native** context. It is a starting point, not a + measured ceiling. +- **Is it hybrid?** Any per-layer type that is not full attention — `linear_attention` / KDA, + Gated DeltaNet, mamba — carries recurrent state across a prefill chunk, and the profile then + declares `hybrid_recurrent_state: true` and pins prefix caching off (§4). GLM-5.3-Flash is 45 + layers with 34 KDA. With no per-layer map in the config, the model card and the architecture + name usually say so ("Next", "Flash", "Linear", "Hybrid"); vllm#51562 (closed 2026-09-22) names + the models sharing the GatedDeltaNet metadata builder: Qwen3-Next, Qwen3.5, OLMo-hybrid, + InternS2-Mobius, Bailing-MoE-v3, Kimi-Linear. +- `gated: true` means you must also set `gated: true` in the profile, or the download 401s + mid-deploy on a billing GPU. +- Then read the model's own serving recipe (model card / vLLM release notes). The recipe usually + names a **higher** minimum vLLM version than the release that first carried the architecture. + +## 2. Pick the quantization and the GPU family + +| Checkpoint | Runs on | Notes | +|---|---|---| +| FP8 | L40S (8.9) and newer — everything here | The safe default | +| NVFP4, **dense** | Blackwell: B200 (cc 10.0) and RTX PRO 6000 (cc 12.0) | `min_compute_capability: 10.0` | +| NVFP4, **MoE** | Blackwell, both families | Measured 2026-09-20 on RTX PRO 6000 (cc 12.0) on the shared cu129 pin: `gemma4-26b-nvfp4` via `VLLM_CUTLASS` and `qwen36-35b-nvfp4` via `MARLIN`, both correct under a graded soak. The old "grouped block-scaled GEMM returns garbage on cc 12.0" gate was stale — flashinfer#2708 and vllm#54189 had closed months earlier. Gate a new profile only on evidence you have. | +| MXFP4 / compressed-tensors | check the recipe | | +| DeepSeek-style **sparse MLA** | B200 only | SM120 has no working kernel (vllm#55757). The failure hides: short prompts pass, realistic lengths fail. | + +Compute capabilities here: L40S 8.9, H100 9.0, B200 10.0, RTX PRO 6000 12.0. Note 12.0 > 10.0 but +is a *different* family — a `min_compute_capability` alone never keeps a model off the RTX card. + +## 3. Size it, then choose the plan + +``` +per-GPU budget = (GPU VRAM GB) x gpu_memory_utilization (0.90) +tensor_parallel_size = smallest N where (weights GB / N) leaves room for a useful KV pool +``` + +Per-GPU VRAM as the role sees it (MiB floor-divided by 1024): L40S 44, H100 79, RTX PRO 6000 95, +B200 179. Pick `min_plan` from `tests/plans.txt` — **and nothing else**; CI checks it, and +`tests/check-plans.sh` checks `tests/plans.txt` against UpCloud's live listing. The identifiers +move: on 2026-09-20 all three B200 ids in this repo were dead. + +Multi-GPU RTX PRO 6000 plans have **no NVLink** — TP runs over PCIe. Mention it in the comment and +point at `NCCL_P2P_DISABLE: 1` as the fallback if init hangs; do not set it pre-emptively. + +## 4. Write the file + +Required (the compose template interpolates these bare; a missing one is an undefined-variable +failure): `model_repo`, `served_name`, `max_model_len`, `tensor_parallel_size`, `dtype`. +`gpu_memory_utilization` falls back to 0.90 from `group_vars/all.yml`. + +Required by this repo's own gates: + +| Key | How to choose it | +|---|---| +| `min_vram_gb` | Per-GPU floor. Set it high enough to keep out a plan that would OOM *after* the download. | +| `weights_gb` | The real download size: sum `.siblings[].size` from the HF API. The deploy-time disk preflight reads this key. | +| `min_disk_gb` | The disk size to provision, not the free space a deploy needs: `(weights_gb + 80) / 0.934`, rounded up to 10. The 80 GB is the headroom `ansible/prefetch.yml` asserts (container image plus Docker's data-root, which share the disk); the 0.934 is ext4's 5% root reserve, so a 500 GB volume offers 467 GB (measured 2026-09-21). Skipping the reserve leaves a profile too small to pass its own prefetch preflight. | +| `validated: false` | Always, on a new profile. Remove it only when a dated `docs/validation.md` row exists. | +| `health_wait_retries` | `(weights_GB / 7.56 + engine_init_minutes) x 1.5 x 6`. 7.56 GB/min is the measured authenticated download rate. Default 90 = 15 min, which is too short for anything over ~100 GB. | + +Optional gates: `min_compute_capability`, `unsupported_compute_capabilities` + +`unsupported_reason`, `untested_compute_capabilities`, `min_driver_major`, `gated`, +`vllm_image_override`, `extra_env`, `extra_vllm_args`, `vllm_plugins`, `enable_lora`, +`hybrid_recurrent_state`. + +**A hybrid checkpoint declares `hybrid_recurrent_state: true` and puts `--no-enable-prefix-caching` +in `extra_vllm_args`.** vLLM defaults `enable_prefix_caching` to true, and on a model whose layers +carry recurrent state that default returns confidently wrong answers on a long cached prefix +(vllm#56960; measured 2026-09-22 — [docs/gpu-spinner.md](../../../docs/gpu-spinner.md#hybrid-checkpoints-and-prefix-caching)). The key is +what arms the gates: `tests/check-prefix-caching.sh` (inside `make test`) fails any profile that +declares it without the flag, and `roles/llama_swap/tasks/refusals.yml` asserts the same at deploy +time. Copy the comment from `ansible/models/custom_model.yml`. + +Cost needs no key. `bin/spin` prices the resolved `min_plan` from `tests/plans.txt` and refuses +anything above `MAX_EUR_PER_HOUR` (default 10) until the operator passes `--allow-expensive`, so an +expensive profile is gated the moment its plan is. (`requires_review`, which used to do this per +profile, was retired 2026-09-21.) `--allow-unvalidated` is unrelated: it turns off **every** +hardware preflight. + +Pin `vllm_image_override` **by digest** when the tag is mutable — several of these tags have +already moved: +```bash +curl -s "https://hub.docker.com/v2/repositories/vllm/vllm-openai/tags/" | jq -r .digest +``` + +## 5. max_model_len: start at the recipe, never at a guess + +Set it to the value the model's serving recipe gives, or to native if the recipe is silent. **Do +not cap it lower without a measurement, and do not claim native without one either.** Add a comment +saying which it is. A measured cap comes from `bin/spin ceiling` and belongs to the other skill. + +A hybrid-model context number in this repo dated before 2026-09-22 predates the unique per-request session tag in +`_gs_prompt` (`bin/lib/checks.sh`) — re-measure it if it came from a wrong-answer failure. A crash +boundary caught by the container restart counter is unaffected, and a recorded pass stays a pass: a +cache hit yields a wrong answer, so it can never manufacture one. + +## 6. Comment the way this repo comments + +Read `ansible/models/glm53-flash-nvfp4.yml`, the house style, and copy its shape: + +- what was measured, on what hardware, **on what date**; +- `FLAGS THAT ARE NOT PREFERENCES` — every non-obvious flag with the failure it prevents; +- upstream issue numbers for every claim; +- a `ruled out, so nobody re-chases them` list, each entry naming the control that made it a real + test **and the fault it was ruled out _of_** — an unscoped "ruled out" reads as a blanket + clearance and stops the next person looking. `glm53-flash-nvfp4.yml` rules prefix caching out of + its **crash** boundary (a unique randomised haystack per request, still crashing at 131,072), + which says nothing about wrong answers: a separate fault in the same layers (vllm#56960); +- the disk figure, and what happens if the disk is too small. + +**Check the upstream issues you cite are still open.** Several in this repo were fixed months +before anyone re-read them (vllm#54189 and flashinfer#2708 both closed while the profiles still +described them as blockers): +```bash +curl -s https://api.github.com/repos/vllm-project/vllm/issues/54189 | jq '{state, updated_at}' +``` + +## 7. Swap presets: only if the profile fits + +`ansible/swap-profiles/*.yml` presets are served by a bare `docker run` that carries image, repo, +context, memory fraction, TP, dtype and `extra_vllm_args` — **and nothing else**. A profile with +`extra_env` or `vllm_plugins` is now refused outright (`roles/llama_swap/tasks/refusals.yml`), +because it would otherwise start and serve silently-wrong output. Do not try to work around that. +A `hybrid_recurrent_state` entry whose `extra_vllm_args` omit `--no-enable-prefix-caching` is +refused by the same file, for the same reason (vllm#56960). + +Also note: in swap mode the client-facing name is the **profile filename**, not `served_name`. + +## 8. The free gates — run all of them before asking for money + +```bash +make test # everything below, plus lint, renders and terraform +# or individually: +python3 tests/check-vllm-compat.py +ansible-playbook tests/render-profile.yml -e profile= +tests/check-plans.sh +``` + +A registry hit in `check-vllm-compat.py` means the architecture *exists* in that build. Whether the +build serves the model correctly is what a live run answers — the other skill. + +## 9. Also update + +- `README.md` — the model table: "Main lane" is `min_plan`, "Measured on" is a dated + `docs/validation.md` row. A new profile has **no** "Measured on" entry. +- `docs/validation.md` — add the row now, marked not-yet-run. +- `tests/plans.txt` — only if you reference a plan it does not list. + +## Finish with this check + +```bash +for k in model_repo served_name max_model_len tensor_parallel_size dtype min_vram_gb min_disk_gb; do + grep -q "^$k:" ansible/models/.yml || echo "MISSING: $k" +done +grep -q '^validated:' ansible/models/.yml || echo "MISSING: validated (defaults to TRUE — it would claim a live run)" +``` + +Then hand off: *"the profile passes every offline gate; it has never been run. Testing it costs +about €X on — see the testing-a-model-profile skill."* diff --git a/.claude/skills/testing-a-model-profile/SKILL.md b/.claude/skills/testing-a-model-profile/SKILL.md new file mode 100644 index 0000000..eb27db7 --- /dev/null +++ b/.claude/skills/testing-a-model-profile/SKILL.md @@ -0,0 +1,233 @@ +--- +name: testing-a-model-profile +description: "Live-test a model profile in inference-gpu-spinner on real, billed UpCloud GPUs: pre-spend gates, prefetching weights on a cheap box, bin/spin up, reading the vLLM engine log, validate, soak, finding the max_model_len token ceiling, debugging a wrong or broken deploy, and recording the docs/validation.md row. Use BEFORE any command that provisions, starts or resizes a server — \"test this model\", \"validate the profile\", \"soak it\", \"what context can it take?\", \"why is it answering garbage?\", \"is it ready to ship?\". Every step here spends money." +--- + +# Testing a model profile + +Everything in this skill costs money by the hour. The GPU plans run from €1.11/h to €36/h, and +sessions are usually wasted on something dull: downloading weights on a GPU, giving up 15 minutes +into a 90-minute pull, or recording a number nobody measured. + +## Rules + +1. **`bin/spin status` is step zero**, before anything else. This repo has sat with a €6.60/h + server record in tofu state and nobody knew. Check before you plan, and again before you stop. +2. **State the euros before every spend**, keep a running total, and print it at each batch + boundary. Stop and ask when you reach the ceiling the user gave you. `bin/spin` enforces its own + floor here: a plan above `MAX_EUR_PER_HOUR` (default €10) needs `--allow-expensive`. +3. **Every number you record must come from output you actually saw.** Paste the command and its + output into the run log. Never carry a figure over from another tier, another model, or a + profile comment. +4. **Never download weights on a GPU.** Use `bin/spin prefetch` (below). This is ~150x cheaper. +5. **Do not leave the session without saying what is still running**, and what it costs per hour. + +## Pre-spend checklist + +```bash +bin/spin status # is anything already billing? +make test # every free gate; must be green +bash --version # soak's tally needs bash 4+; never hardcode an interpreter path +grep -n 'min_disk_gb\|health_wait_retries\|validated' ansible/models/.yml +grep -E '^GPU-' tests/plans.txt # the price you are about to pay +``` + +Then size the disk. `min_disk_gb` is now enforced by a preflight, but the preflight runs *on the +box you already paid for* — get it right first: +```bash +# WEIGHTS_SIZE_GB in .env, then: +bin/spin persistent-init # grows the disk. UpCloud CANNOT shrink one afterwards. +``` + +Two flags that prevent expensive accidents: +- `--shutdown-at HH:MM` past your session end. The 21:00 Europe/Zurich timer has already killed a + 4×B200 mid-engine-load. +- `bin/spin down --keep-disk` between runs in a campaign. A plain `down` **deletes** any disk + ≥150 GB and the next run re-pulls everything. + +## 1. Prefetch (cheap box), then spin up (expensive box) + +```bash +bin/spin prefetch [...] # ~€0.045/h CPU box; pulls weights AND images, then self-destructs +bin/spin up --model --plan --shutdown-at 23:30 +``` + +An UpCloud disk attaches to one server at a time, so prefetch and GPU sessions are strictly +sequential; `bin/spin prefetch` refuses while a GPU server exists. + +Use the **spot** plan id for validation runs — it is a plan id, not a flag, and it is 25% off: +`GPU-SPOT-16xCPU-80GB-1xRTXPRO6000` (€1.24/h), `GPU-SPOT-64xCPU-320GB-4xRTXPRO6000` (€4.95/h). A +spot box can be reclaimed; with weights already on `/data` that costs minutes, not a re-download. + +**Chaining models on one live box** saves the provision+teardown cycle (~€0.15 on L40S, ~€2.50 on +4×B200): +```bash +bin/spin up --model --plan "$(tofu -chdir=terraform/providers/upcloud show -json \ + | jq -r '.values.root_module.resources[]|select(.type=="upcloud_server").values.plan')" +``` +You must pass `--plan` with **exactly the value already in state**: without it the mismatch guard +dies, and with a *different* value `tofu apply -auto-approve` resizes the live server. Chains can +only stay inside one `min_plan` lane — the compute-capability gates forbid the rest. And if the box +is *stopped*, `bin/spin start` first: the resume path refuses `--model`. + +## 2. Read the engine log before believing anything + +```bash +bin/spin ssh 'docker logs gpu-spinner-vllm-1 2>&1 | grep -iE "attention backend|MoE backend|Capturing|graph|GPU KV cache size|Maximum concurrency|enable_prefix_caching="' +``` + +Check, in order: the attention backend is the one the profile expects · the MoE backend line names +what actually ran (it is often not what the flags suggest) · CUDA graphs captured unless the +profile sets `--enforce-eager` · the KV pool and max concurrency, which are two of the columns in +`docs/validation.md` · the EngineCore config line prints `enable_prefix_caching=True|False`, which +is the only proof a correctness flag in `extra_vllm_args` reached the engine. On a hybrid model it +must read `False`; if it reads `True`, every long-context correctness number from this box is a +prefix-cache measurement (vllm#56960, see §5). + +## 3. validate, then soak + +```bash +bin/spin validate +bin/spin soak --rounds 3 --concurrency 20 --context 32768 +``` + +`validate` proves it serves and returns coherent text. **It grades nothing.** A short prompt +answering correctly proves very little: vllm#55757 is specifically a path that passes short +prompts and fails at realistic lengths. + +`soak` is the real check — a factual battery, fabrication traps, then graded needle-in-haystack +under parallel load at three depths. What its output means: + +- **a wrong needle fails the command**; rejections and timeouts do not (those are admission + control, not correctness). +- `--concurrency` below 3 exercises only the head depth. Never go below 3. +- the ~8-request factual battery runs on **every** invocation. On a €36/h box ten probes is ~€70 of + battery alone — budget it. +- p50/p95 latency is **not a throughput number**: all requests at one context share a single + haystack, so tail-depth requests share most of their prefill. +- **the context clamp is a false pass.** Ask for more than `max_model_len - 2048` and soak silently + reduces it and warns. A green run at "131072" may have been ~96k. Record + `usage.prompt_tokens`, never the number you asked for. +- **a green soak is a cold-prefill result.** Every needle prompt opens with a unique session tag + (`_gs_prompt`), so soak always forces a real prefill. It therefore says nothing about a long + prefix served from cache, which is what multi-turn chat and a shared system prompt are by + construction; on a hybrid model that gap is the whole of vllm#56960 (§5). + +In **swap mode**, `bin/spin validate` reports idle VRAM and a null `max_model_len` with no error — +do not paste it. Pre-warm each entry with a throwaway curl, pass `--model ` to soak +explicitly, and take real numbers from `docker logs vllm-swap-` inside the idle TTL (the +container is `--rm`; the logs vanish after it). + +## 4. The token ceiling + +Four different ceilings. `bin/spin ceiling` searches two of them, and the restart counter is what +tells those two apart: + +| Kind | How you find it | Cost | +|---|---|---| +| **Capacity** — the KV pool cannot hold one full-length request | read "GPU KV cache size" from `validate`, divide by the context. vLLM refuses at startup rather than crashing | free | +| **Crash** — the engine dies above some length, KV barely touched. `ceiling` prints `CRASH` and the restart counter has moved | `bin/spin ceiling` | one engine reload per failure | +| **Wrong answer** — the request is admitted and answered in normal time or faster, and the needle is absent. `ceiling` prints `WRONG` (HTTP 200) and the restart counter is unmoved | `bin/spin ceiling` | one probe, no reload | +| **Mode floor** — the checkpoint needs a minimum context for a documented mode | read the recipe | free | + +A crash ceiling and a wrong-answer ceiling are different faults, and neither explains the other. +`glm53-flash-nvfp4`'s 98,304 sits under a crash — a CUDA illegal memory access in KDA linear +attention (vllm#54317) — and nothing about wrong answers touches it. A wrong answer on a hybrid +model is usually the prefix cache (vllm#56960, §5), which never crashes an engine. Because a cache +hit makes the model answer *wrongly*, it can never manufacture a false pass: recorded passing +results stay trustworthy. What deserves re-checking is a recorded failure and any cap derived from +one. + +Check capacity **first**: deploying at native when native does not fit burns a reload and teaches +you nothing. + +```bash +# No request can exceed the deployed max_model_len, so redeploy high before searching: +bin/spin up --model --plan -e max_model_len= +bin/spin ceiling --start 32768 +``` + +`ceiling` ascends (a pass costs a minute; a crash costs a full reload, so bisecting is the +expensive way) and reads the container's **restart counter** around every probe — `restart: +unless-stopped` brings a crashed engine back, and a fast reload finishes inside soak's own tally, +so soak files the crash as a "rejected" request and still prints `Soak OK`. The counter decides +`CRASH` and nothing else: a probe that returns 200 is then graded on the needle, so the verdict is +one of `pass`, `WRONG`, `timeout` (curl code 000) or `rejected()`. Read which one it printed +before you record a number. `--keep-going` runs the whole ladder instead of stopping at the first +failure — what you want when you are testing a *flag* rather than searching for a boundary. + +Branch on the verdict before you cap anything: +- `CRASH` — a real boundary. Set `max_model_len` below the last passing `prompt_tokens`, record the + whole pass/fail matrix and the fault from the engine log in the profile comment. +- `WRONG` — **do not cap.** A cap trades context away and leaves the defect in place below it. + Redeploy with `--no-enable-prefix-caching` and re-run the same ladder. On 2026-09-22 every rung of + `custom_model`'s ladder (98,138 / 196,188 / 392,306 / 784,545 / 1,044,002 `prompt_tokens`) failed + warm and passed cold; stopping at the first warm failure would have recorded ~98k for a model that + reaches 1,044,002. If the rungs still fail with the flag on, it is a genuine long-context + correctness ceiling — record it by that name, so nobody later reads it as a crash. +- `timeout` / `rejected()` — inconclusive. Neither is a ceiling; re-probe. + +Then re-soak at whatever you settled on: +```bash +bin/spin soak --context --concurrency 10 +``` + +After an `-e max_model_len=N` redeploy the file and the running config disagree. **State the +deployed value in every row you record**, and redeploy cleanly from the profile before teardown. + +## 5. Failure signatures + +| Symptom | Cause | Action | +|---|---|---| +| One token repeated forever | NVFP4 MoE activation scale read 0.0 — every expert output multiplied by zero (vllm#54189, **closed 2026-08-30**) | Check the image carries the fix, or that the profile's `extra_env` scale took effect | +| Garbage on cc 12.0, NVFP4 MoE | CUTLASS grouped block-scaled GEMM (flashinfer#2708, **closed 2026-04-20**) | Needs a cu130 image. No cu130 *release* tag exists — only model-specific builds | +| Short prompts fine, long prompts wrong | Sparse MLA on sm_120 (vllm#55757, open) | This is exactly what soak's graded needles catch. Do not clear it with a smoke test. On a hybrid model rule out the prefix cache (row below) first — it presents identically and costs one redeploy to test | +| Long prompts wrong, restart counter unmoved, and repeating the same prompt answers wrong *faster* than the first attempt | Prefix caching on a hybrid (linear-attention) model — vllm#56960, open. Mechanism, measurement and the affected model families: [docs/gpu-spinner.md](../../../docs/gpu-spinner.md#hybrid-checkpoints-and-prefix-caching) | Redeploy with `--no-enable-prefix-caching` and re-probe. It corrupts answers and never crashes an engine, so it neither explains nor invalidates a crash ceiling | +| `CUDA error: an illegal memory access` in `chunk_kda_with_fused_gate` | KDA linear attention above ~100k tokens (vllm#54317, open) | Cap `max_model_len` below the boundary you measure | +| Xid 13 warp errors after hours of load | Marlin MoE on RTX PRO 6000 (vllm#52225, open) | Only a long run surfaces it. Record the duration you actually exercised | +| 100% GPU, no output, NCCL watchdog kills it | flashinfer autotune | `--no-enable-flashinfer-autotune` | +| Engine never becomes healthy | `health_wait_retries` too short for the download | Recompute: `(GB / 7.56 + init_min) x 1.5 x 6` | +| OOM at load | wrong plan, or `gpu_memory_utilization` | CUDA graph capture allocates *outside* the budget; 0.95 fails | +| Hangs in NCCL at init | no NVLink on multi-GPU RTX PRO 6000 | `NCCL_P2P_DISABLE: 1` in `extra_env`, at a throughput cost | +| `No space left on device` mid-pull | `min_disk_gb` > free space | The preflight catches it now; if it fired, grow the disk or delete a cached model | + +**Before you trust a profile comment, check the ticket.** Several claims in this repo were stale by +months. `curl -s https://api.github.com/repos/vllm-project/vllm/issues/N | jq '{state,updated_at}'`. +When a fresh measurement disagrees with a recorded one, **suspect the record**, name both numbers, +and do not silently overwrite. Treat a **"ruled out"** line the same way and check what it was +ruled out *of*: a scoped exclusion written up as a blanket one reads as clearance and stops the +next person looking. `glm53-flash-nvfp4`'s ruled-out prefix caching is that case — a real control +for its **crash** boundary, and no evidence at all about wrong answers, which are a separate fault +in the same layers (vllm#56960). + +## 6. Record it + +Add the `docs/validation.md` row: profile, model, quant, tier measured, `max_model_len`, KV pool, +concurrency at max, VRAM used/total, date, **whether prefix caching was on** (the vLLM default) or +off (`--no-enable-prefix-caching`), and the evidence. Without that last field nobody can tell +afterwards whether a context number is a cold-prefill measurement or a warm-prefix one. Then update +the profile: flip `validated:`, and write the evidence comment in the `glm53-flash-nvfp4.yml` house +style — measured figures with dates, the flags that are not preferences, upstream issue numbers, +and a "ruled out, so nobody re-chases them" list that says what each entry was ruled out *of*. + +If it did **not** pass, that is a result too. Record what broke, at what length, with the fault +from the log — and leave `validated: false`. + +## 7. Wind down + +```bash +bin/spin down --keep-disk # more runs coming within ~2 days +bin/spin down # done: also deletes a disk >=150 GB; keeps IP, hostname, TLS cert +bin/spin persistent-destroy --yes # €0 standing, but LOSES the IP, hostname and cert. Rarely right. +``` + +Then verify by hand — no `bin/spin` command reports standing resources: +```bash +set -a; . ./.env; set +a +curl -fsS -H "Authorization: Bearer $UPCLOUD_TOKEN" https://api.upcloud.com/1.3/server | jq '.servers.server[]|{hostname,plan,state}' +curl -fsS -H "Authorization: Bearer $UPCLOUD_TOKEN" https://api.upcloud.com/1.3/storage | jq '.storages.storage[]|select(.type=="normal")|{title,size,tier}' +``` +Watch for a disk left in the 110–149 GB band (kept forever by design) and an orphaned 100 GB +`terraform-gpu-spinner-disk` boot disk. **Never delete `gpu-spinner-weights` by hand.** + +A **stopped** server still bills its disks. Stopped is not wound down. diff --git a/.env.example b/.env.example index 80bfb02..af6686a 100644 --- a/.env.example +++ b/.env.example @@ -47,6 +47,12 @@ OPERATOR_CIDRS= WEIGHTS_SIZE_GB= WEIGHTS_TIER= +# Refuse to deploy a plan costing more than this per hour. bin/spin reads the price of the resolved +# plan from tests/plans.txt and stops before anything is provisioned; `--allow-expensive` on a +# single run, or a higher value here, opts in. Unset means 10. +# MAX_EUR_PER_HOUR=10 +MAX_EUR_PER_HOUR= + # Auto-decommission the persistent disk on `bin/spin down` to stop its standing cost. By DEFAULT # (auto), a disk >= DECOMMISSION_THRESHOLD_GB (default 150) is deleted on teardown — a visible warning # is printed first, and the floating IP is KEPT (stable hostname; next `bin/spin up` recreates the disk diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 8ea4ad1..c72be8b 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -28,7 +28,8 @@ jobs: with: python-version: "3.12" - name: Install ansible + ansible-lint - run: pip install ansible ansible-lint jmespath + # From requirements.txt, so CI installs the same pinned versions the guide documents. + run: pip install -r requirements.txt - name: Install collections run: ansible-galaxy collection install -r requirements.yml - name: ansible-lint @@ -37,15 +38,64 @@ jobs: run: | ansible-playbook ansible/site.yml --syntax-check -e vllm_model=qwen36 -e sslip_host=example ansible-playbook ansible/site.yml --syntax-check -e vllm_model=qwen36 -e sslip_host=example -e multi_model_enabled=true + ansible-playbook ansible/prefetch.yml --syntax-check -i ansible/inventory/placeholder.ini + - name: the inventory directory actually parses + # `.ini` is in Ansible's default ignore list, so a directory inventory of .ini files + # resolves to nothing and every play then matches no hosts while exiting 0. Only + # placeholder.ini is committed, and an empty group is reported under all.children rather + # than as a top-level key — check there, so this passes with or without a live hosts.ini. + run: | + ansible-inventory -i ansible/inventory --list > /tmp/inv.json + jq -e '(.all.children // []) | index("gpu")' /tmp/inv.json > /dev/null - name: bin/spin default profile exists + every profile renders run: | - default=$(sed -n 's/.*VLLM_MODEL:-\([a-z0-9-]*\).*/\1/p' bin/spin | head -1) + default=$(sed -n 's/.*VLLM_MODEL:-\([a-z0-9-]*\).*/\1/p' bin/spin bin/lib/*.sh | head -1) echo "bin/spin default profile: $default" test -f "ansible/models/$default.yml" || { echo "::error::bin/spin defaults to missing profile '$default'"; exit 1; } for f in ansible/models/*.yml; do ansible-playbook tests/render-profile.yml -e "profile=$(basename "$f" .yml)" >/dev/null || { echo "::error::profile $f fails to render"; exit 1; } echo "rendered: $f" done + - name: every profile's plan is a real UpCloud plan + # bin/spin deploys min_plan / swap_plan when --plan is omitted, so a typo here reaches + # `tofu apply` on a paid resource instead of just printing a wrong message. + run: | + rc=0 + check() { # check + plan=$(sed -n "s/^$2: *\"\{0,1\}\([A-Za-z0-9-]*\)\"\{0,1\}.*/\1/p" "$1" | head -1) + [ -n "$plan" ] || return 0 # absent is allowed: bin/spin falls through to the default + if ! grep -qE "^${plan}[[:space:]]" tests/plans.txt; then + echo "::error file=$1::$2 '$plan' is not listed in tests/plans.txt"; rc=1 + else + echo "$(basename "$1") -> $plan" + fi + } + for f in ansible/models/*.yml; do check "$f" min_plan; done + for f in ansible/swap-profiles/*.yml; do check "$f" swap_plan; done + exit $rc + + - name: the plan cost gate behaves + # Replaced the per-profile requires_review flag, which had its own regression test. + run: tests/check-cost-gate.sh + + - name: hybrid profiles keep prefix caching off + # Measured 2026-09-22: with prefix caching on, a KDA/GatedDeltaNet checkpoint answers fast + # and wrong on a cached long prefix, with no crash and nothing in the log (vllm#56960). + # Nothing else in CI would notice the flag being deleted. + run: tests/check-prefix-caching.sh + + - name: tests/plans.txt still matches UpCloud's live plan listing + # The check above only proves the repo agrees with itself. This one asks the provider. + # Skips cleanly when the secret is absent, so forks and PRs from forks still pass. + env: + UPCLOUD_TOKEN: ${{ secrets.UPCLOUD_TOKEN }} + run: tests/check-plans.sh + + - name: llama-swap refuses presets it cannot serve faithfully + # The backend command carries no per-model env and no plugin mounts, so such a profile would + # start and serve silently-wrong output. Proves the refusal, not just the happy path. + run: ansible-playbook tests/swap-refusals.yml + - name: every swap preset resolves + renders (llama-swap) run: | for f in ansible/swap-profiles/*.yml; do @@ -54,11 +104,23 @@ jobs: echo "rendered swap: $f" done + vllm-compat: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: every profile matches the vLLM build its image pins + # Reads vLLM's source at the ref each pinned image corresponds to and checks the + # checkpoint's architecture, the named parsers and every CLI flag. Catches the mistakes + # the render tests cannot: a flag that does not exist, a parser that is not registered, + # or an architecture the pinned release predates. + run: python3 tests/check-vllm-compat.py + shell: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: bash syntax check - run: bash -n bin/spin + run: for f in bin/spin bin/lib/*.sh; do bash -n "$f"; done - name: shellcheck - run: shellcheck bin/spin # ubuntu-latest ships shellcheck + # -x follows the `. bin/lib/*.sh` sources, so the CLI is checked as one unit. + run: shellcheck -x bin/spin # ubuntu-latest ships shellcheck diff --git a/.gitignore b/.gitignore index db12997..6a02837 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,9 @@ override.tf.json *_override.tf.json # NOTE: .terraform.lock.hcl is intentionally NOT ignored (commit it for reproducible providers). +# --- Tests --- +tests/.vllm-compat-cache/ + # --- Ansible --- ansible/inventory/hosts.ini *.retry @@ -38,3 +41,5 @@ __pycache__/ *~ TODO.txt plan.md + +ansible/models/custom_model.yml diff --git a/Makefile b/Makefile index bb862fc..b462df2 100644 --- a/Makefile +++ b/Makefile @@ -1,15 +1,19 @@ # Thin aliases over bin/spin. Pass extra args via ARGS, e.g.: # make up ARGS="--model qwen36 --plan GPU-12xCPU-240GB-1xH100" -.PHONY: help persistent-init up down status logs ssh plan fmt lint +# make down ARGS="--keep-disk" +# Every target that takes flags must pass $(ARGS): without it `make down ARGS=--keep-disk` +# silently dropped the flag and bin/spin deleted the weights disk by default. +.PHONY: help persistent-init up down status logs ssh plan fmt lint test help: ; @bin/spin help -persistent-init: ; @bin/spin persistent-init +persistent-init: ; @bin/spin persistent-init $(ARGS) up: ; @bin/spin up $(ARGS) -down: ; @bin/spin down +down: ; @bin/spin down $(ARGS) status: ; @bin/spin status logs: ; @bin/spin logs ssh: ; @bin/spin ssh $(ARGS) -plan: ; @bin/spin plan +plan: ; @bin/spin plan $(ARGS) fmt: ; tofu -chdir=terraform/persistent fmt && tofu -chdir=terraform/providers/upcloud fmt lint: ; ansible-lint +test: ; @tests/run-offline-gates.sh $(ARGS) diff --git a/README.md b/README.md index a529fe2..359828c 100644 --- a/README.md +++ b/README.md @@ -42,11 +42,14 @@ Each variable, and what to set when your egress IP rotates, is covered in ```bash bin/spin persistent-init # once: create the disk + floating IP bin/spin up # default: qwen36 (Qwen3.6-27B FP8) on an L40S -bin/spin up --model qwen36-35b --plan GPU-12xCPU-240GB-1xH100 # a bigger model on an H100 -bin/spin up --swap-profile b200-nvfp4 --plan GPU-24xCPU-240GB-1xB200 # several models, one endpoint +bin/spin up --model qwen38-flash-next # each profile brings its own plan (4x RTX PRO 6000 here) +bin/spin up --model qwen36-35b --plan GPU-16xCPU-80GB-1xRTXPRO6000 # --plan overrides it +bin/spin up --swap-profile b200-nvfp4 # several models, one endpoint +bin/spin prefetch qwen38 qwen38-nvfp4 # pull weights + images on a ~EUR 0.045/h CPU box bin/spin status # GPU + service status bin/spin validate # capture context/KV/VRAM/test-gen evidence bin/spin soak # answer-quality + concurrent-load check +bin/spin ceiling # find the longest single request the model actually survives bin/spin down # destroy the GPU (keeps the IP) — stops the compute bill ``` @@ -62,27 +65,108 @@ curl https://.sslip.io/v1/chat/completions \ ## Models -Pick a profile with `--model` (or `-e vllm_model=`); profiles live in `ansible/models/`. FP8 runs on -L40S/H100; the **NVFP4** variants are **Blackwell/B200-only** (~2× smaller + faster). - -| Profile | Model | Quant | Tier | -|---|---|---|---| -| `qwen36` (default) | Qwen3.6-27B | FP8 | L40S | -| `gemma4-26b` | Gemma 4 26B-A4B (MoE) | FP8 | L40S | -| `gemma4-31b` | Gemma 4 31B | FP8 | L40S | -| `qwen36-35b` | Qwen3.6-35B-A3B (MoE) | FP8 | H100 | -| `qwen36-27b-nvfp4`, `qwen36-35b-nvfp4`, `gemma4-26b-nvfp4`, `gemma4-31b-nvfp4` | (same models) | NVFP4 | B200 | -| `deepseek-v4-flash` | DeepSeek-V4-Flash (284B-A13B MoE) | NVFP4 | B200 | -| `glm52-nvfp4` | GLM-5.2 (753B MoE) | NVFP4 | 4×B200 | -| `glm52` | GLM-5.2 | FP8 | 8×B200 — `requires_review` (kept gated: ~36 €/h to test) | - -All profiles are live-validated except `glm52` FP8, which stays behind `requires_review` because an -8×B200 node is too costly to test (pass `--allow-unvalidated` to run it anyway). The exact, -dated matrix — context, KV pool, concurrency, VRAM — is the system of record: -**[docs/validation.md](docs/validation.md)**. - -All current model repos are **ungated** on Hugging Face, so no `HF_TOKEN` is required (but setting one -in `.env` avoids the anonymous download throttle). **LoRA** adapter serving and **multi-model swap** +Pick a profile with `--model` (or `-e vllm_model=`); profiles live in `ansible/models/`. To serve something with no profile here, copy [`ansible/models/custom_model.yml`](ansible/models/custom_model.yml) — it is a working example with four `EDIT ME` markers (repo, served name, vLLM args, sizing) and no registration step: the filename is the model name. FP8 runs +anywhere from an L40S up; **NVFP4** needs Blackwell (~2× smaller + faster), and *which* Blackwell +depends on whether the checkpoint is dense or MoE — see below the table. + +**Main lane** is the plan `bin/spin up --model ` deploys when you don't pass `--plan`; it +comes from the profile's `min_plan`. **Measured on** is where a dated live run exists in +[docs/validation.md](docs/validation.md). The two differ whenever the cheapest plan that runs a +model is still waiting for someone to measure it. + +| Profile | Model | Quant | Main lane (default plan) | Measured on | +|---|---|---|---|---| +| `qwen36` (default) | Qwen3.6-27B | FP8 | L40S | L40S | +| `qwen38` | Qwen3.8-27B | FP8 | L40S | **L40S** and RTX PRO 6000 | +| `qwen38-nvfp4` | Qwen3.8-27B | NVFP4 | RTX PRO 6000 | **RTX PRO 6000** | +| `qwen38-flash-next` | Qwen3.8-Flash-Next (125B-A6B MoE + 51B N-gram) | FP8 | **4×RTX PRO 6000** | **4×RTX PRO 6000** | +| `glm53-flash-nvfp4` | GLM-5.3-Flash (321B-A18B MoE, **multimodal**) | NVFP4 | **4×RTX PRO 6000** | **4×RTX PRO 6000** (context capped at 98304) | +| `gemma4-26b` | Gemma 4 26B-A4B (MoE) | FP8 | L40S | L40S | +| `gemma4-31b` | Gemma 4 31B | FP8 | L40S | L40S | +| `qwen36-35b` | Qwen3.6-35B-A3B (MoE) | FP8 | H100 | **H100** | +| `qwen36-27b-nvfp4`, `gemma4-31b-nvfp4` | (same models, dense) | NVFP4 | B200 | B200; also RTX PRO 6000 via the `rtxpro6000` preset | +| `qwen36-35b-nvfp4`, `gemma4-26b-nvfp4` | (same models, MoE) | NVFP4 | **RTX PRO 6000** (was B200) | B200 **and RTX PRO 6000** | +| `glm52-nvfp4` | GLM-5.2 (753B MoE) | NVFP4 | 4×B200 | 4×B200 | +| `glm53-flash` | GLM-5.3-Flash, FP8 build | FP8 | 4×B200 — €18/h | — (new) | +| `glm53` | GLM-5.3 (743B-A39B MoE) | FP8 | 8×B200 — €36/h | — (new) | +| `k2-horizon-375b` | K2-Horizon-375B-A23B (379B-A27B MoE) | FP8 | **8×RTX PRO 6000** (TP=8) — €13.20/h | — (new) | +| `custom_model` | **the editable custom recipe** | - | 4×RTX PRO 6000 | 2026-09-22 | +| `kimi-k3` | Kimi K3 (93L, 896 experts, **multimodal**) | compressed-tensors | **no UpCloud plan fits** | — (new) | +| `deepseek-v4-flash-vision` | DeepSeek-V4-Flash-Vision-Exp (285B-A13B MoE, **multimodal**) | FP4+FP8 | 4×B200 — €18/h | — (new) | +| `deepseek-v41-flash` | DeepSeek-V4.1-Flash (552B + 196B Engram MoE, **multimodal**) | MXFP4+MXFP8 | 4×B200 — €18/h | — (new) | +| `deepseek-v4-pro-0813` | DeepSeek-V4-Pro-0813 (1.6T-A49B MoE) | FP4+FP8 | 8×B200 — €36/h | — (new) | +| `glm52` | GLM-5.2 | FP8 | 8×B200 — €36/h, too costly to validate | — | + +**The RTX PRO 6000 line.** 96 GB, compute capability 12.0 — Blackwell, but a different family from +the B200's 10.0. At €1.65/h for a single card and €6.60 for four, against €4.50 and €18.00 for the +B200 equivalents, it is the default lane wherever it works, and it usually has capacity when H100 +and B200 are sold out. Every FP8 profile runs on it, and so does **dense** NVFP4. Multi-GPU plans +have **no NVLink**, so tensor parallelism runs over PCIe. As of 2026-09-20 there are measured runs +for `glm53-flash-nvfp4`, `qwen38-nvfp4`, `qwen38`, `qwen38-flash-next`, `gemma4-26b-nvfp4`, +`qwen36-35b-nvfp4` and the whole `rtxpro6000` preset. + +**NVFP4 MoE runs on this card too.** `gemma4-26b-nvfp4` and `qwen36-35b-nvfp4` both pass a full +graded soak on the *existing* cu129 pin — one via `VLLM_CUTLASS`, one via `MARLIN` — so their +default lane moved from a €4.50/h B200 to this €1.65/h card (€1.24 spot). Both had refused cc 12.0 +until 2026-09-20, citing a CUTLASS grouped GEMM that returns invalid output; the gate was banning +the whole family over one suspect kernel, and the upstream fixes +([flashinfer#2708](https://github.com/flashinfer-ai/flashinfer/issues/2708), +[vllm#54189](https://github.com/vllm-project/vllm/issues/54189)) had landed months before anyone +re-read the tickets. Check the ticket before trusting a gate. + +**Sparse attention is what keeps the rest on B200.** Every remaining large model here is a +DeepSeek-style sparse MLA architecture, and SM120 has no working kernel for it +([vllm#55757](https://github.com/vllm-project/vllm/issues/55757)). The failure hides: short prompts +answer correctly and realistic lengths break. `glm53-flash-nvfp4` gets around it with a pinned +third-party kernel overlay; the DeepSeek profiles keep their cc 12.0 gate until the upstream fixes +land, each recording its own trigger. Quantisation is the settled part: cu130 fixes the CUTLASS +grouped-GEMM fault behind the old reports of NVFP4 MoE misbehaving on this card. Profiles +declare their limits via `min_compute_capability`, `unsupported_compute_capabilities` and +`untested_compute_capabilities`. Detail: +[docs/gpu-spinner.md](docs/gpu-spinner.md#what-runs-on-which-gpu). + +**Size the weights disk before the first pull.** The default 150 GB `/data` does not hold the large +profiles — `qwen38-flash-next` is ~186 GB, `deepseek-v41-flash` ~511 GB, `deepseek-v4-pro-0813` +~893 GB. Each profile states its own figure: set `WEIGHTS_SIZE_GB` and re-run +`bin/spin persistent-init`, and remember the disk bills 24/7 until you remove it. + +Where a main lane has no measured row, it should still work and warns when you deploy it: treat the +run as a validation, capture `bin/spin validate` **and `bin/spin soak`**, and add the row. Run +`bin/spin prefetch ` first — a weights pull is billed at whatever compute holds the disk, +and doing it on a CPU box instead of a GPU is about 150× cheaper. If the profile's `max_model_len` +is the recipe's number rather than a measurement, `bin/spin ceiling` finds the real boundary. +On a hybrid checkpoint (KDA, Gated DeltaNet, mamba state), check the engine log for +`enable_prefix_caching=` before believing any long-context result — a cache hit makes the model +answer fast and wrong ([vllm#56960](https://github.com/vllm-project/vllm/issues/56960); +[what to do about it](docs/gpu-spinner.md#hybrid-checkpoints-and-prefix-caching)). +Two skills in [`.claude/skills/`](.claude/skills/) walk an agent through both jobs: +`adding-a-model-profile` (offline, free) and `testing-a-model-profile` (live, billed). + +**The price of the plan is what gates a deploy.** `bin/spin up` reads the resolved plan's hourly +rate from [`tests/plans.txt`](tests/plans.txt) and refuses anything above `MAX_EUR_PER_HOUR` +(default €10) until you pass `--allow-expensive`; the refusal comes before anything is provisioned. +That covers the 8×B200 profiles at €36/h (`glm52`, `glm53`, `deepseek-v4-pro-0813`), the 4×B200 +ones at €18/h (`glm53-flash`, `deepseek-v41-flash`, `deepseek-v4-flash-vision`) and +`k2-horizon-375b` at €13.20/h. `kimi-k3` fits no plan here at all, so its preflights refuse it +whatever you pass. `--allow-unvalidated` is a different flag: it overrides the hardware preflights +(VRAM, compute capability, known-bad GPU list, driver, disk) and says nothing about cost. + +**Why two profiles serve the same model.** `glm53-flash` (FP8, 4×B200, €18/h) and +`glm53-flash-nvfp4` (4×RTX PRO 6000, €6.60/h) are the same checkpoint, and so are `glm52` and +`glm52-nvfp4` — but they are not interchangeable, so both stay. The NVFP4 builds are cheaper and +are the validated path; the FP8 builds carry a longer context (`glm53-flash` 1048576 against a +*measured* 98304 cap on the NVFP4 sibling; `glm52` 1048576 against 786432), higher-precision +weights, and no dependency on a pinned third-party kernel overlay. Use the NVFP4 lane unless you +need one of those. Note the FP8 context figures are the recipes' numbers and unmeasured, so +ceiling-search before relying on 1M. The crash frame in +[vllm#54317](https://github.com/vllm-project/vllm/issues/54317) is not an architecture-wide limit: +on 2026-09-22 `custom_model` reached 1,044,002 prompt tokens on the same geometry, image digest and +overlay commit that caps its NVFP4 sibling near 100k, pointing at the NVFP4 build rather than at KDA +in general. + +All 21 model repos are **ungated** on Hugging Face (re-verified 2026-09-20), so no `HF_TOKEN` is +required — but set one in `.env` anyway: anonymous downloads throttle to ~50 KB/s after a few +hundred GB in a day, which costs far more than the token on a billing GPU. **LoRA** adapter serving and **multi-model swap** are supported — see below and `docs/gpu-spinner.md`. ## Multi-model serving @@ -92,16 +176,18 @@ demand, chosen by the OpenAI `model` field: ```bash bin/spin swap-profiles # list presets -bin/spin up --swap-profile l40s --plan GPU-8xCPU-64GB-1xL40S +bin/spin up --swap-profile l40s # each preset names its own plan (swap_plan) +bin/spin up --swap-profile rtxpro6000 # ...pass --plan only to override ``` -Presets live in `ansible/swap-profiles/` (just a list of model names). Details: +Presets live in `ansible/swap-profiles/` (just a list of model names): `l40s`, `rtxpro6000` and +`b200-nvfp4`. Details: [docs/gpu-spinner.md](docs/gpu-spinner.md#multi-model-serving-llama-swap). ## Cost Compute bills only while the GPU server exists (L40S ≈ €1.11/h, up to €36/h for an 8×B200 node); -the persistent disk and floating IP bill 24/7 whether or not a GPU exists (≈ €37/mo at the default +the persistent disk and floating IP bill 24/7 whether or not a GPU exists (≈ €36/mo at the default 150 GB MaxIOPS disk). Rates per tier, disk sizing, and measured per-session costs: [docs/gpu-spinner.md](docs/gpu-spinner.md#cost) and [docs/validation.md](docs/validation.md). @@ -114,6 +200,29 @@ at a fixed local time (default 21:00 Europe/Zurich; tune with `--shutdown-at` / `--no-shutdown`). `bin/spin persistent-destroy --yes` is the only path to €0 standing cost, and is irreversible. Full detail: [docs/gpu-spinner.md](docs/gpu-spinner.md#cost). +## Checking a profile before you pay for it + +For every profile, `tests/check-vllm-compat.py` reads vLLM's source at the ref its pinned image +corresponds to and verifies that the checkpoint's architecture is in that build's model registry, +that the parsers it names are registered, and that every flag it passes exists. No GPU needed; it +runs in CI: + +```bash +make test # every free gate: lint, renders, compat, plans, terraform +python3 tests/check-vllm-compat.py # all profiles +python3 tests/check-vllm-compat.py qwen38 # just one +tests/check-plans.sh # do the plan ids still exist? (needs UPCLOUD_TOKEN) +``` + +`tests/check-plans.sh` exists because the identifiers move. On 2026-09-20 all three B200 plan ids +in this repo were dead — UpCloud had doubled the vCPU count in the name — and eleven profiles +pointed at plans that no longer existed. CI was green throughout, because it only compared the +profiles to `tests/plans.txt`, which was stale too. That check asks the provider instead. + +A registry hit means the code exists, while the recipe says which version actually serves the +model — often a higher floor than the release that first carried the architecture. Read the recipe, +then pin accordingly. + ## Other providers `terraform/providers/{runpod,lambda,vast,generic-openstack}/` are stubs documenting the provider diff --git a/ansible.cfg b/ansible.cfg index 5442f66..2ebd058 100644 --- a/ansible.cfg +++ b/ansible.cfg @@ -2,6 +2,10 @@ # (Independent of any surrounding repo — does not read parent config or roles.) [defaults] inventory = ansible/inventory +# `.ini` is in Ansible's default inventory_ignore_extensions. With `inventory` pointing at a +# directory, that silently skips every file in it — including the hosts.ini bin/spin writes — +# and the run then matches no hosts while still exiting 0. Keep the default list minus `.ini`. +inventory_ignore_extensions = .pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, .rst, .orig, .cfg, .retry roles_path = ansible/roles callbacks_enabled = profile_tasks, timer interpreter_python = auto_silent diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml index 71abcba..08d75b4 100644 --- a/ansible/group_vars/all.yml +++ b/ansible/group_vars/all.yml @@ -14,8 +14,23 @@ weights_fs_type: ext4 # --- vLLM container (pin the exact tag for reproducible spin-ups) --- vllm_image: "vllm/vllm-openai:v0.23.0-cu129-ubuntu2404" vllm_port: 8000 -# Fallback GPU memory fraction. Every model profile sets its own gpu_memory_utilization (which wins); -# this is the default for a llama-swap backend whose swap_models entry omits it. +# Fallback GPU memory fraction, used only where nothing more specific applies: every model profile +# sets its own, and so does any swap_models entry that cares. +# +# The fraction caps what vLLM itself allocates — weights, KV cache, the profiling forward pass. CUDA +# graph capture, NCCL buffers, the CUDA context and fragmentation come out of the remainder, so the +# card always ends up fuller than the number suggests. Measured at 0.90 (docs/validation.md): +# 41.0-41.6 of 46.1 GiB on an L40S (89-90%), 74.2 of 81.6 on an H100 (91%), 170.8 of 183.4 on a +# B200 (93%). That remaining headroom is what 0.90 buys. +# +# Raising it pays badly. On a B200, 0.95 adds ~9 GB of KV to a pool already holding millions of +# tokens, and the extra allocation has to come from the same remainder that graph capture needs — +# on a GPU billed by the hour, an OOM at startup costs more than the KV is worth. Lowering it does +# not help either: one recipe (Qwen3.8 on a 31.4 GiB card) measured 0.80 and 0.93 leaving identical +# free memory, since capture allocates outside the budget in both cases. +# +# So tune it per profile, with a measurement: deploy, read the KV pool from `bin/spin validate`, +# adjust, redeploy, compare. qwen38-flash-next uses the 0.85 its recipe measured on 4x H100. gpu_memory_utilization: 0.90 # --- Caddy (reverse proxy + automatic TLS) --- diff --git a/ansible/models/custom_model.yml b/ansible/models/custom_model.yml new file mode 100644 index 0000000..d1765c2 --- /dev/null +++ b/ansible/models/custom_model.yml @@ -0,0 +1,74 @@ +--- +# CUSTOM MODEL — the recipe to copy when you want to serve something this repo has no profile for. +# The checkpoint set here really runs, so the file doubles as a working example. +# +# Copy it to ansible/models/.yml rather than editing in place, change the four marks +# below (repo, served name, serving args, sizing), and run the offline gates: +# +# make test +# python3 tests/check-vllm-compat.py +# +# Nothing registers a profile — the filename is the name, so `bin/spin up --model ` picks +# yours up with no other change. Read .claude/skills/adding-a-model-profile/SKILL.md before changing +# the sizing keys: most mistakes there surface only after a paid server has downloaded several +# hundred gigabytes. +# +# EDIT ME (1): the Hugging Face repo id. +model_repo: "EDIT/ME" +# EDIT ME (2): the name clients send in the OpenAI `model` field. bin/spin prints it after a deploy; +# /v1/models is authoritative. In llama-swap mode the profile filename is used instead. +served_name: "custom-model" +# For your own model, check with tests/check-vllm-compat.py and +# pin by digest whenever the tag is mutable. +vllm_image_override: "xxxxxx" +tensor_parallel_size: 4 +max_model_len: 1048576 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +vllm_plugins: + - name: sparse-mla-blackwell + repo: "https://github.com/Libertai/vllm-sparse-mla-blackwell" + ref: "b1f20638d7f82f880e4bcb2136ee2f19c06f38a4" + build_env: + GLM53_ARCHS: "120a" # example +extra_env: + VLLM_GLM53_CUDA_SPARSE_MLA: 1 # example +# EDIT ME (3): the serving command from your model's own recipe. +# FLAGS THAT ARE NOT PREFERENCES: +# --no-enable-flashinfer-autotune : inherited from glm53-flash-nvfp4, where autotune spun the GPU +# at 100% with no output until the NCCL watchdog killed it. Not re-measured for this quant, and +# kept because the failure it prevents is expensive and the flag costs nothing. +# --kv-cache-dtype fp8 : the overlay kernel supports it; bf16 also works at about half the KV. +# parsers glm47 : this is a GLM-5.3 derivative and uses the base model's reasoning/tool format. +# MTP is left off: config.json has num_nextn_predict_layers: 1, so a draft head exists, but a first +# validation should measure the model alone. Enable with: +# - "--speculative-config" +# - '{"method":"mtp","num_speculative_tokens":3}' +extra_vllm_args: + - "--no-enable-flashinfer-autotune" +# - "--no-enable-prefix-caching" + - "--kv-cache-dtype" + - "fp8" + - "--reasoning-parser" + - "glm47" + - "--tool-call-parser" + - "glm47" + - "--enable-auto-tool-choice" +# EDIT ME (4): sizing. min_plan must appear in tests/plans.txt — CI checks it, and bin/spin refuses +# any plan above MAX_EUR_PER_HOUR (default 10) until you pass --allow-expensive. +# Hybrid attention: most layers carry recurrent state, which is what makes +# --no-enable-prefix-caching mandatory above. tests/check-prefix-caching.sh keys on this. +hybrid_recurrent_state: true +min_plan: "GPU-64xCPU-320GB-4xRTXPRO6000" +min_vram_gb: 90 # per-GPU floor: RTX PRO 6000 reports 95, B200 179; keeps H100 (79) out +weights_gb: 181 # measured from the HF file listing (sum of .siblings[].size) 2026-09-21 +min_disk_gb: 280 # (181 + 80) / 0.934, rounded up to 10 — see the skill for the derivation +min_compute_capability: 10.0 # Blackwell; the overlay kernel covers sm_120 and sm_121 +min_driver_major: 580 # the image is a cu130 build +# Deliberately false even though the example checkpoint has a dated row in docs/validation.md: this +# file is the copy-me template, so `validated: true` here would become a claim about whatever model +# the next person points it at. It only warns, never blocks. Set it true in your copy, once your row +# exists. +validated: false +health_wait_retries: 300 # x10s = 50 min: ~181 GB to pull, plugin build, then TP=4 init diff --git a/ansible/models/deepseek-v4-flash-vision.yml b/ansible/models/deepseek-v4-flash-vision.yml new file mode 100644 index 0000000..0e1460c --- /dev/null +++ b/ansible/models/deepseek-v4-flash-vision.yml @@ -0,0 +1,78 @@ +--- +# DeepSeek-V4-Flash-Vision-Exp — experimental multimodal V4 (285B total / 13B active MoE, 43 layers +# plus a 32-layer ViT and two-layer aligner). Mixed precision: MoE experts FP4, attention/norm/router +# FP8, vision tower BF16 — 48 shards, ~168 GB on disk. The recipe budgets ~202 GB of VRAM before KV, +# so this is a multi-GPU profile (TP=4). +# +# PINNED IMAGE (required): vision support is not in a stable release — it lives in +# vllm-project/vllm#54566. The official wheel routes this checkpoint to the text-only class and +# fails on the vision tensors, so this profile overrides the shared image pin. +# +# HARDWARE: B200 for now, though 4x RTX PRO 6000 would fit comfortably (~202 GB across 380 GB, at +# 6.60 EUR/h against 18.00) and min_vram_gb 90 already clears a 95 GB card. The cc 12.0 gate below +# is the only thing holding this on B200. +# +# WHY THE GATE IS THERE: sparse attention. vllm-project/vllm#55757 reports DeepSeek-V4 unservable +# on sm_120, and the failure hides — short prompts answer correctly and realistic prompt lengths +# break, so a smoke test signs off on a broken build. +# +# The FP4 experts are fine on a cu130 build: flashinfer-ai/flashinfer#2708 fixes the CUTLASS grouped +# block-scaled GEMM and is in vLLM's FlashInfer pin. Much of the "NVFP4 MoE returns garbage" +# reporting turned out to be vllm-project/vllm#54189 — an uninitialised w13_input_scale that zeroes +# every expert output on a weight-only checkpoint. +# +# WHEN TO OPEN THE LANE. vllm-project/vllm#41834 adds SM12x support for DeepSeek V4 Flash, but as of +# 2026-09-19 it is 288 commits over 221 files with conflicts, and its tagged preview +# (sm120-pr-41834-stable-preview-20260809, aa0d513027) is validated on +# deepseek-ai/DeepSeek-V4-Flash-0731 — a text checkpoint with no profile here, so not direct +# evidence for the vision build. Re-check that tag, then drop unsupported_compute_capabilities, set +# min_plan to GPU-64xCPU-320GB-4xRTXPRO6000, and add the overlay to vllm_plugins the way +# glm53-flash-nvfp4.yml does. +# +# Expensive tier: only NVIDIA GB200 NVL4 (TP4 + expert parallel) and AMD MI350X have a published +# vision run, and UpCloud has neither. The nearest verified-equivalent topology is 4x B200 with +# NVLink at ~18 EUR/h — too costly to validate speculatively. 2x B200 (TP=2, 384 GB, ~9 EUR/h) also +# clears the 202 GB budget and is what the ROCm run used, but has no published NVIDIA run. +# +# max_model_len 32768 is the recipe's measured bound; the checkpoint advertises 1,048,576. A +# 1M-token KV cache needs far more memory than the weights do, so raise this only after reading the +# KV pool vLLM reports at startup. +# --allowed-local-media-path is omitted — it is only needed for file:// image URLs, and clients here +# send http(s):// or base64. The repo ships no Jinja template; vLLM resolves --tokenizer-mode +# deepseek_v4 itself. +model_repo: "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp" +served_name: "deepseek-v4-flash-vision" +vllm_image_override: "vllm/vllm-openai:deepseekv4-flash-vision" +tensor_parallel_size: 4 +max_model_len: 32768 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + - "--enable-expert-parallel" + - "--kv-cache-dtype" + - "fp8" + - "--block-size" + - "256" + - "--tokenizer-mode" + - "deepseek_v4" + - "--reasoning-parser" + - "deepseek_v4" + - "--tool-call-parser" + - "deepseek_v4" + - "--enable-auto-tool-choice" + # DSpark: the draft module is fused into this same checkpoint, so there is no separate draft model. + - "--speculative-config" + - '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}' +min_plan: "GPU-96xCPU-960GB-4xB200" +min_vram_gb: 90 # per-GPU floor: ~202 GB budget across TP=4, plus KV headroom at 32K +min_disk_gb: 260 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 156 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 +unsupported_compute_capabilities: [12.0] +unsupported_reason: >- + DeepSeek-style sparse MLA has no working attention path on RTX Blackwell (cc 12.0) — + vllm-project/vllm#55757, where a short-prompt smoke test passes and realistic prompts fail. + vllm-project/vllm#41834 is the open SM12x enabler; see the note at the top of this profile +validated: false # newly added — no live run on this account yet +health_wait_retries: 150 # x10s = 25 min — ~168 GB of weights plus TP=4 engine init and DSpark setup diff --git a/ansible/models/deepseek-v4-flash.yml b/ansible/models/deepseek-v4-flash.yml deleted file mode 100644 index aaa4503..0000000 --- a/ansible/models/deepseek-v4-flash.yml +++ /dev/null @@ -1,29 +0,0 @@ ---- -# DeepSeek-V4-Flash (284B-A13B MoE, text-only) in NVFP4 — Blackwell (B200) ONLY. -# Official NVIDIA ModelOpt checkpoint (157 GB on disk). VALIDATED LIVE on a SINGLE B200 at TP1 -# (2026-07-04): serves at 262144 ctx with fp8 KV — KV pool 347,227 tok, 1.32x concurrency at full -# context, 170.7/179 GiB. Tight but real; lower max_model_len if you need more concurrency. -# (The FP8 build ~320 GB still needs a multi-GPU node — NVFP4-on-B200 is the single-GPU path.) -# MoE NVFP4 backend: vLLM auto-selects CUTLASS (do NOT set VLLM_USE_FLASHINFER_MOE_FP4 — it raises -# "no FlashInfer NVFP4 MoE backend supports the configuration" on this image). -model_repo: "nvidia/DeepSeek-V4-Flash-NVFP4" -served_name: "deepseek-v4-flash-nvfp4" -tensor_parallel_size: 1 -max_model_len: 262144 -gpu_memory_utilization: 0.92 -dtype: "auto" -enable_lora: false -extra_vllm_args: - - "--trust-remote-code" - - "--kv-cache-dtype" - - "fp8" - - "--tokenizer-mode" - - "deepseek_v4" - - "--reasoning-parser" - - "deepseek_v4" - - "--tool-call-parser" - - "deepseek_v4" - - "--enable-auto-tool-choice" -min_plan: "GPU-24xCPU-240GB-1xB200" -min_vram_gb: 175 # B200 reports 179 GB after floor-div (183,359 MiB) — measured live 2026-07-04 -health_wait_retries: 120 # x10s = 20 min — 157 GB weights; download dominates a cold start diff --git a/ansible/models/deepseek-v4-pro-0813.yml b/ansible/models/deepseek-v4-pro-0813.yml new file mode 100644 index 0000000..4e309b6 --- /dev/null +++ b/ansible/models/deepseek-v4-pro-0813.yml @@ -0,0 +1,81 @@ +--- +# DeepSeek-V4-Pro-0813 — flagship of the V4 family: 1.6T total / 49B active MoE, 61 layers at hidden +# 7168, 384 routed experts with 6 active plus one shared. Hybrid attention (Compressed Sparse + +# Heavily Compressed) with Manifold-Constrained Hyper-Connections, reaching 27% of V3.2's per-token +# FLOPs and 10% of its KV cache at 1M context. FP4+FP8 mixed: MoE experts FP4, attention / norm / +# router FP8. 1,048,576 native context via YaRN. 0813 supersedes the preview and adds fused DSpark. +# +# THE MOST EXPENSIVE PROFILE HERE: eight GPUs and ~893 GB of weights, i.e. +# GPU-192xCPU-1920GB-8xB200 at ~36 EUR/h — the glm52 tier, so bin/spin refuses it until you pass +# --allow-expensive (or raise MAX_EUR_PER_HOUR). Read the disk note below before the first run; the +# standing cost of that disk is the largest in this repo. +# +# PINNED IMAGE. Fused-DSpark checkpoints need vLLM 0.25.0+; the shared pin is v0.23.0, so this uses +# v0.29.0, the newest release. +# +# WHY EIGHT GPUs. The recipe is explicit that the mixed-precision checkpoint does not fit one GB200 +# NVL4 tray (4 GPUs, 768 GB) and must run across eight; its recommended deployments are all 8-GPU +# (B300, H200, MI355X). 8x B200 gives 1536 GB against ~893 GB of weights. +# +# TOPOLOGY. Plain TP=8, the shape the MI355X reference command and glm52 here both use. The recipe +# recommends DP + EP (--data-parallel-size 8) for 8-GPU nodes instead; that is the change to make +# once throughput matters more than a first successful serve. +# +# B200 ONLY. The FP4 MoE experts take the grouped block-scaled GEMM path that returns invalid output +# on RTX Blackwell (cc 12.0) with a cu129 build, and an 8x RTX PRO 6000 node would be 768 GB with no +# NVLink — too small for this checkpoint and the wrong interconnect for TP=8. +# +# CONTEXT. max_model_len 393216 is the documented floor for the "max" reasoning mode: below 384K +# that mode truncates. The recipe caps 8-GPU H200 nodes at 800000 for KV headroom; 8x B200 has more +# room, so raising toward 800000 is the first thing to try once a live run shows the KV pool. Native +# is 1,048,576. +# +# REASONING: three effort levels on 0813 (low / high / max), selected per request. DeepSeek +# recommends temperature 1.0, top_p 0.95 for agentic use and 1.0 otherwise, and up to 384K output +# tokens. +# +# DSpark: the fused draft module is left off so the first validation run tests the model alone — at +# this hourly rate a failure that could be either half is expensive. Enable with: +# extra_vllm_args += ["--speculative-config", +# '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'] +# +# DISK: ~893 GB of weights. Set WEIGHTS_SIZE_GB to at least 1000 and re-run bin/spin persistent-init +# before the first pull. At MaxIOPS a 1000 GB disk is ~EUR 220/mo standing, billed whether or not a +# GPU exists. Either budget for that or let bin/spin down decommission it each evening and accept a +# fresh ~2 h pull. +model_repo: "deepseek-ai/DeepSeek-V4-Pro-0813" +served_name: "deepseek-v4-pro-0813" +vllm_image_override: "vllm/vllm-openai:v0.29.0-cu129-ubuntu2404" +tensor_parallel_size: 8 +max_model_len: 393216 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_env: + # Pulling and loading ~893 GB takes hours; the engine's readiness timeout has to outlast it. + VLLM_ENGINE_READY_TIMEOUT_S: 7200 +extra_vllm_args: + - "--trust-remote-code" + - "--kv-cache-dtype" + - "fp8" + - "--tokenizer-mode" + - "deepseek_v4" + - "--reasoning-parser" + - "deepseek_v4" + - "--tool-call-parser" + - "deepseek_v4" + - "--enable-auto-tool-choice" + - "--max-num-seqs=512" + - "--max-num-batched-tokens=8192" +min_plan: "GPU-192xCPU-1920GB-8xB200" +min_vram_gb: 175 # per-GPU floor (B200 reports 179 GB); ~893 GB of weights across TP=8 +min_disk_gb: 1000 # disk size to provision: (weights_gb + 80) / 0.934; raised to this profile's stated floor +weights_gb: 831 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 +unsupported_compute_capabilities: [12.0] +unsupported_reason: >- + size decides this one regardless of kernels: ~893 GB of weights against 768 GB on an 8x RTX PRO + 6000 node, which is also PCIe-only with no NVLink for TP=8. DeepSeek-style sparse MLA has no + working cc 12.0 attention path either (vllm-project/vllm#55757) +validated: false # newly added — no live run on this account yet +health_wait_retries: 900 # x10s = 150 min — ~893 GB to pull, then TP=8 init across eight GPUs diff --git a/ansible/models/deepseek-v41-flash.yml b/ansible/models/deepseek-v41-flash.yml new file mode 100644 index 0000000..c73b8b0 --- /dev/null +++ b/ansible/models/deepseek-v41-flash.yml @@ -0,0 +1,91 @@ +--- +# DeepSeek-V4.1-Flash — vision-language MoE, 552B backbone + 196B Engram memory, 8B active per +# prompt token and 16B per output token. 40 layers at hidden 5120, split into a 20-layer encoder and +# a 20-layer decoder (prefill runs only the encoder half), with a 32-layer ViT and aligner in front. +# Two-tier sparse attention with a V3.2-style indexer, Hyper-Connections, and a fused DSpark draft +# head. Routing is 6 of 384 experts plus one shared, with a separate routing bias for image spans. +# 1,048,576 context via YaRN factor 16 over a 65,536-token training window. +# +# PINNED IMAGE (required): no pip wheel serves this architecture. It landed on vLLM main in +# vllm-project/vllm#56228, so any nightly from 2026-09-10 carries it; v0.29.0 predates it. The tag +# below is an immutable dated nightly, not the floating :nightly, so a spin-up two weeks from now +# gets the same engine. Replace it with a v0.30.0 release tag once one exists. +# +# MEMORY. ~511 GB on disk (476 GiB) against the recipe's 614 GB VRAM floor — one GB200 NVL4 tray at +# TP4, which 4x B200 (768 GB) matches here. The Engram tables alone are 196.6B parameters (~183 GiB) +# and dominate everything but the experts, so weights and batch size set the capacity limit before +# KV does. +# +# B200 FOR NOW, and the closest DeepSeek profile to an RTX PRO 6000 lane: 8x RTX PRO 6000 is 760 GB +# against the 614 GB floor at 13.20 EUR/h versus 18.00, so the lane would be TP=8 on +# GPU-128xCPU-640GB-8xRTXPRO6000. 4x RTX PRO 6000 (380 GB) does not fit. +# +# WHAT BLOCKS IT: sparse attention. vllm-project/vllm#55757 reports DeepSeek-style sparse MLA +# unservable on sm_120 — it passes a short-prompt smoke test and fails at realistic lengths. The +# MXFP4 experts are clear: cu130 fixes the grouped block-scaled GEMM fault, and much of the "4-bit +# MoE returns garbage" history traces to vllm-project/vllm#54189, an uninitialised activation scale +# zeroing every expert. +# +# WHEN TO OPEN THE LANE. vllm-project/vllm#54929 covers this model's geometry (512 NoPE + 64 RoPE = +# 576 head layout) and, unlike #41834, is a small mergeable PR (12 files, mergeable 2026-09-19). +# When it lands in a release image: drop unsupported_compute_capabilities, set +# tensor_parallel_size 8, min_plan GPU-128xCPU-640GB-8xRTXPRO6000, min_vram_gb 90. The overlay route +# in glm53-flash-nvfp4.yml is NoPE-specific and will not serve this geometry. +# +# --language-model-only skips the vision encoder, freeing its VRAM for KV; the verified GB200 runs +# (TP4 and 1P1D) were text-only, so that is what this profile reproduces. For image input, remove it +# and consider --mm-encoder-tp-mode data: at 32 layers and hidden 1024 the ViT is small enough that +# tensor-parallel communication costs more than it saves. +# +# --tokenizer-mode / --reasoning-parser / --tool-call-parser are all deepseek_v41, not deepseek_v4. +# +# REASONING is on by default at effort 50. Callers set chat_template_kwargs: thinking (bool) and +# reasoning_effort — low (25), high (50), xhigh (75), max (100) or an integer 1-100. The DeepSeek +# API maps low/high/max to 50/75/100 while vLLM maps low/high/xhigh/max to 25/50/75/100, so pass an +# integer if you need parity. +# +# DSpark: the fused draft head adds roughly 14B parameters to the load. Left off so the first +# validation run tests the model alone. Enable with: +# extra_vllm_args += ["--speculative-config", +# '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}'] +# +# DISK: ~511 GB against a default 150 GB /data. Set WEIGHTS_SIZE_GB to at least 600 and re-run +# bin/spin persistent-init before the first pull. At MaxIOPS that is ~EUR 132/mo while the disk +# exists — bin/spin down removes it by default, at the cost of a fresh ~70 min pull. +model_repo: "deepseek-ai/DeepSeek-V4.1-Flash" +served_name: "deepseek-v41-flash" +vllm_image_override: "vllm/vllm-openai:cu129-nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3" +tensor_parallel_size: 4 +max_model_len: 1048576 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_env: + # First load is very long: weights, Engram tables and CUDA graph capture. The engine's own + # readiness timeout has to outlast it or vLLM aborts before it ever serves. + VLLM_ENGINE_READY_TIMEOUT_S: 3600 +extra_vllm_args: + - "--language-model-only" + - "--tokenizer-mode" + - "deepseek_v41" + - "--reasoning-parser" + - "deepseek_v41" + - "--tool-call-parser" + - "deepseek_v41" + - "--enable-auto-tool-choice" + - "--max-num-seqs=128" + - "--max-num-batched-tokens=16384" + # Covers expected decode batches including DSpark draft tokens: 128 * (1 + 5) = 768, rounded up. + - "--max-cudagraph-capture-size=1024" +min_plan: "GPU-96xCPU-960GB-4xB200" +min_vram_gb: 175 # per-GPU floor (B200 reports 179 GB); 614 GB total across TP=4 +min_disk_gb: 600 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 475 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 +unsupported_compute_capabilities: [12.0] +unsupported_reason: >- + DeepSeek-style sparse MLA has no working attention path on RTX Blackwell (cc 12.0) — + vllm-project/vllm#55757, where a short-prompt smoke test passes and realistic prompts fail. + vllm-project/vllm#54929 is the open fix for this geometry; see the note at the top of this profile +validated: false # newly added — no live run on this account yet +health_wait_retries: 600 # x10s = 100 min — ~511 GB to pull, then TP=4 init and graph capture diff --git a/ansible/models/gemma4-26b-nvfp4.yml b/ansible/models/gemma4-26b-nvfp4.yml index 0712b64..a482149 100644 --- a/ansible/models/gemma4-26b-nvfp4.yml +++ b/ansible/models/gemma4-26b-nvfp4.yml @@ -1,9 +1,23 @@ --- -# Gemma 4 26B-A4B in NVFP4 — Blackwell (B200) ONLY. Official NVIDIA ModelOpt checkpoint (~14 GB). -# Quant auto-detected (do NOT pass --quantization). MoE NVFP4 backend: let vLLM auto-select (CUTLASS). Do NOT set VLLM_USE_FLASHINFER_MOE_FP4=1 — on this -# image it raises "no FlashInfer NVFP4 MoE backend supports the configuration" (measured live 2026-07-04). -# B200 (NVFP4 will not run on L40S/H100). Live-validated on B200 2026-07-05 (262144, KV 10.64M tok, -# 40.59x, 165.9/183 GiB; soak 6/6; CUTLASS NVFP4-MoE path confirmed). +# Gemma 4 26B-A4B in NVFP4 — Blackwell only: B200 (cc 10.0) or RTX PRO 6000 (cc 12.0). Official +# NVIDIA ModelOpt checkpoint (~14 GB). Quantization is auto-detected (do not pass --quantization). +# Let vLLM pick the NVFP4 MoE backend: VLLM_USE_FLASHINFER_MOE_FP4=1 raises "no FlashInfer NVFP4 MoE +# backend supports the configuration" on this image (measured live 2026-07-04). +# +# VALIDATED on both lanes. +# B200, 2026-07-05: 262144 ctx, KV 10.64M tokens, 40.59x concurrency, 165.9/183 GiB, soak 6/6, +# CUTLASS NVFP4-MoE path confirmed. +# RTX PRO 6000, 2026-09-20: GPU-SPOT-16xCPU-80GB-1xRTXPRO6000, driver 595.58.03, on the same +# cu129 pin. vLLM chose the 'VLLM_CUTLASS' NvFp4 MoE path and answered correctly throughout. +# KV pool 4,822,910 tokens, 18.40x at 262,144, 88,921/97,887 MiB. soak 3x20 @32768: factual +# 6/6, traps 2/2 declined, 60/60 needles, 0 wrong, p50 14.1 s. The graded long-context load is +# what clears it — vllm-project/vllm#55757 is the failure shape where short prompts pass and +# realistic lengths break. No cu130 image was needed. +# +# Until that run this profile refused cc 12.0, on the grounds that NVFP4 MoE takes a CUTLASS grouped +# block-scaled GEMM returning invalid output on cu129. Both upstream fixes had already landed: +# flashinfer-ai/flashinfer#2708 closed 2026-04-20 and vllm-project/vllm#54189 closed 2026-08-30, +# before this image was built on 2026-06-13. Check a ticket before trusting a gate. model_repo: "nvidia/Gemma-4-26B-A4B-NVFP4" served_name: "gemma4-26b-a4b-nvfp4" tensor_parallel_size: 1 @@ -22,5 +36,9 @@ extra_vllm_args: - "gemma4" - "--kv-cache-dtype" - "fp8" -min_plan: "GPU-24xCPU-240GB-1xB200" -min_vram_gb: 175 # B200 reports 179 GB after floor-div (183,359 MiB) — measured live 2026-07-04 +min_plan: "GPU-16xCPU-80GB-1xRTXPRO6000" # 1.65/h list, 1.24 spot; the 1xB200 lane (4.50/h) + # also serves this profile. +min_vram_gb: 40 # real weight+KV floor; the capability gates below pin the family +min_disk_gb: 110 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 18 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # NVFP4 needs Blackwell — no L4/L40S/H100 diff --git a/ansible/models/gemma4-26b.yml b/ansible/models/gemma4-26b.yml index 233167a..2a8102a 100644 --- a/ansible/models/gemma4-26b.yml +++ b/ansible/models/gemma4-26b.yml @@ -4,7 +4,8 @@ # Fits a single L40S 48GB at the EDGE (~40-44 GB, text-only); H100 80GB is the safe choice. # --limit-mm-per-prompt image=0 drops the vision tower for text-only serving (this arch's # --language-model-only equivalent). Live-validated on a single L40S 2026-07-05 (KV 986,681 tok, -# 7.53x @131072, 41.6/46 GiB; soak 6/6 + traps declined). L40S edge fit — --plan ...1xH100 for headroom. +# 7.53x @131072, 41.6/46 GiB; soak 6/6 + traps declined). L40S edge fit — for headroom use +# --plan GPU-16xCPU-80GB-1xRTXPRO6000 (96 GB, 1.65 EUR/h) or GPU-12xCPU-240GB-1xH100 (80 GB). model_repo: "RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic" served_name: "gemma4-26b-a4b-fp8" tensor_parallel_size: 1 @@ -23,4 +24,7 @@ extra_vllm_args: - "--kv-cache-dtype" - "fp8" min_plan: "GPU-8xCPU-64GB-1xL40S" +untested_compute_capabilities: [12.0] # also fits RTX PRO 6000 / H100; only L40S is measured min_vram_gb: 40 +min_disk_gb: 120 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 27 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) diff --git a/ansible/models/gemma4-31b-nvfp4.yml b/ansible/models/gemma4-31b-nvfp4.yml index af36da9..befd230 100644 --- a/ansible/models/gemma4-31b-nvfp4.yml +++ b/ansible/models/gemma4-31b-nvfp4.yml @@ -1,8 +1,10 @@ --- -# Gemma 4 31B in NVFP4 — Blackwell (B200) ONLY. Official NVIDIA ModelOpt checkpoint (~21 GB). -# Dense, so no FlashInfer-MoE env needed. Quant auto-detected (do NOT pass --quantization). -# min_vram_gb=175 pins this to the B200 (NVFP4 will not run on L40S/H100). Live-validated on B200 -# 2026-07-05 (262144, KV 2.42M tok, 9.22x, 166/183 GiB; soak 6/6). +# Gemma 4 31B in NVFP4 — Blackwell-class GPUs (B200 or RTX PRO 6000). Official NVIDIA ModelOpt +# checkpoint (~21 GB). Dense, so it uses the NVFP4 scaled-mm path rather than the grouped-GEMM MoE +# kernels that misbehave on RTX Blackwell — hence this one runs on cc 12.0 and the NVFP4 *MoE* +# profiles do not. Quant auto-detected (do not pass --quantization). min_compute_capability keeps it +# off L40S/H100, which have no NVFP4 support at all. Live-validated on B200 2026-07-05 (262144, +# KV 2.42M tok, 9.22x, 166/183 GiB; soak 6/6). Not yet validated on RTX PRO 6000. model_repo: "nvidia/Gemma-4-31B-IT-NVFP4" served_name: "gemma4-31b-nvfp4" tensor_parallel_size: 1 @@ -20,5 +22,9 @@ extra_vllm_args: - "gemma4" - "--kv-cache-dtype" - "fp8" -min_plan: "GPU-24xCPU-240GB-1xB200" -min_vram_gb: 175 # B200 reports 179 GB after floor-div (183,359 MiB) — measured live 2026-07-04 +min_plan: "GPU-24xCPU-240GB-1xB200" # the validated tier; 1xRTXPRO6000 also passes, untested +min_vram_gb: 40 # ~21 GB weights + KV; keeps an L4/L40S-sized card out +min_disk_gb: 120 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 30 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # Blackwell or newer: B200 (cc 10.0) and RTX PRO 6000 (cc 12.0) +untested_compute_capabilities: [12.0] # dense NVFP4 should run on RTX PRO 6000; not yet measured diff --git a/ansible/models/gemma4-31b.yml b/ansible/models/gemma4-31b.yml index 96c223a..4534c5f 100644 --- a/ansible/models/gemma4-31b.yml +++ b/ansible/models/gemma4-31b.yml @@ -1,7 +1,8 @@ --- # Gemma 4 31B (Google) — dense, multimodal (image+text) — in FP8. # FP8-Dynamic compressed-tensors (~33 GB weights); quant auto-detected (no --quantization). -# Fits a single L40S 48GB but TIGHT (~43 GB); H100 80GB is comfortable (and allows longer context). +# Fits a single L40S 48GB but TIGHT (~43 GB). For comfort and longer context use a 1x RTX PRO 6000 +# (96 GB, 1.65 EUR/h) or an H100 80GB (1.79 EUR/h). # --limit-mm-per-prompt image=0 drops the vision tower for text-only serving. Live-validated on a # single L40S 2026-07-05 (65536 ctx, KV 105,859 tok, 1.62x, 41.5/46 GiB; soak 6/6). Tight but real. model_repo: "RedHatAI/gemma-4-31B-it-FP8-Dynamic" @@ -23,4 +24,7 @@ extra_vllm_args: - "--kv-cache-dtype" - "fp8" min_plan: "GPU-8xCPU-64GB-1xL40S" +untested_compute_capabilities: [12.0] # also fits RTX PRO 6000 / H100; only L40S is measured min_vram_gb: 40 +min_disk_gb: 120 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 31 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) diff --git a/ansible/models/glm52-nvfp4.yml b/ansible/models/glm52-nvfp4.yml index c56d2d7..cc60f11 100644 --- a/ansible/models/glm52-nvfp4.yml +++ b/ansible/models/glm52-nvfp4.yml @@ -1,6 +1,6 @@ --- # GLM-5.2 (Z.ai, 753B total / ~40B active MoE, text-only, 1M native ctx) in NVFP4 — the runnable -# multi-GPU path on this account: 4x B200 (GPU-96xCPU-960GB-4xB200, TP=4). NVFP4 weights ~377 GB +# multi-GPU path on this account: 4x B200 (GPU-96xCPU-960GB-4xB200, TP=4). NVFP4 weights ~433 GB (measured from the HF file listing 2026-09-20) # fit 4x179 GB VRAM with headroom (the FP8 build ~756 GB needs 8x B200 — see glm52.yml). # Quant auto-detected (no --quantization). MoE NVFP4 backend: vLLM auto-selects CUTLASS — do NOT # set VLLM_USE_FLASHINFER_MOE_FP4 (raises "no FlashInfer NVFP4 MoE backend..." on this image). @@ -26,4 +26,14 @@ extra_vllm_args: - "fp8_e4m3" min_plan: "GPU-96xCPU-960GB-4xB200" min_vram_gb: 175 # per-GPU floor (B200 reports 179 GB after floor-div) +min_disk_gb: 550 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 433 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # NVFP4 needs Blackwell +unsupported_compute_capabilities: [12.0] +unsupported_reason: >- + GLM-5.2 is a DSA (DeepSeek-style sparse MLA) model, and that has no working attention path on + RTX Blackwell (cc 12.0) — vllm-project/vllm#55757, where a short-prompt smoke test passes and + realistic prompts fail. That outlives the NVFP4 MoE GEMM fault this reason used to cite, which is + reported fixed on cu130. 8x RTX PRO 6000 would also be 768 GB of PCIe-attached VRAM with no + NVLink, a poor fit for TP=4/8 on a 433 GB checkpoint health_wait_retries: 150 # x10s = 25 min — measured ~18 min cold start (447 GB load + 457 s engine init) diff --git a/ansible/models/glm52.yml b/ansible/models/glm52.yml index e7780bc..e7aa6b8 100644 --- a/ansible/models/glm52.yml +++ b/ansible/models/glm52.yml @@ -1,10 +1,12 @@ --- -# GLM-5.2 (Z.ai / zai-org) FP8 — 753B total / ~40B active MoE, text-only, 1M ctx. INTENTIONALLY kept -# requires_review: an 8x B200 node is ~36 €/h, too expensive to validate — gated ON PURPOSE (cost), -# not a reliability doubt. Run the validated NVFP4 path (glm52-nvfp4.yml, 4x B200 ~18 €/h) instead. -# WARNING: FP8 weights ~756 GB fit ONLY an 8x B200 node (8x179 GB) — NOT 8x H100 (8x80 = 640 GB is -# too small). requires_review gates it (the vllm role refuses without -e allow_unvalidated_model=true). -# The RUNNABLE GLM path on this account is glm52-nvfp4.yml (NVFP4 on 4x B200 at half the price). +# GLM-5.2 (Z.ai / zai-org) FP8 — 753B total / ~40B active MoE, text-only, 1M ctx. +# INTENTIONALLY NOT VALIDATED: an 8x B200 node is ~36 EUR/h, too expensive to justify — a cost +# decision, not a reliability doubt. Run the validated NVFP4 path instead (glm52-nvfp4.yml, +# 4x B200 ~18 EUR/h), which is the runnable GLM-5.2 lane on this account. +# WARNING: FP8 weights ~704 GB fit ONLY an 8x B200 node (8x179 GB) — NOT 8x H100 (8x80 = 640 GB is +# too small), and not an 8x RTX PRO 6000 node either: 8x96 = 768 GB leaves no working headroom and +# has no NVLink for TP=8. bin/spin refuses any plan above MAX_EUR_PER_HOUR (default 10) unless +# --allow-expensive is passed, so this tier needs a deliberate opt-in. model_repo: "zai-org/GLM-5.2-FP8" served_name: "glm52" tensor_parallel_size: 8 @@ -23,5 +25,15 @@ extra_vllm_args: - "--kv-cache-dtype" - "fp8_e4m3" min_plan: "GPU-192xCPU-1920GB-8xB200" -min_vram_gb: 76 -requires_review: true +min_vram_gb: 175 # per-GPU floor (B200 reports 179 GB). Not 76: at that value an 8x RTX PRO 6000 + # node (95 GB/GPU) clears the gate and then OOMs on ~704 GB of weights against a + # 684 GB budget, after the whole download. The file comment above says 8x96 does + # not fit; this is the check that enforces it. +validated: false # never run live. WITHOUT this key the vllm role defaults `validated` to true and + # the "UNTESTED MODEL" warning never prints, so this profile silently claimed a + # live run it has never had. Added 2026-09-20 during the RTX validation campaign. +health_wait_retries: 800 # x10s = 133 min. 704 GB of shards at ~126 MB/s is ~93 min of download + # alone, before TP=8 init. The default of 90 (15 min) gave up long before + # the pull finished, on a 36 EUR/h node. +min_disk_gb: 840 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 704 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) diff --git a/ansible/models/glm53-flash-nvfp4.yml b/ansible/models/glm53-flash-nvfp4.yml new file mode 100644 index 0000000..9d38878 --- /dev/null +++ b/ansible/models/glm53-flash-nvfp4.yml @@ -0,0 +1,199 @@ +--- +# GLM-5.3-Flash (Z.ai) in NVFP4 — the RTX PRO 6000 lane for the model that glm53-flash.yml serves +# in FP8 on 4x B200. Same architecture: ~321B total / 18B active multimodal MoE, 45 layers mixing +# KDA linear attention with NoPE sparse MLA, 8 of 288 experts per token, 1,048,576 native context. +# +# VALIDATED 2026-09-20 on 4x RTX PRO 6000 Blackwell Server Edition (driver 595.58.03, cc 12.0), +# plan GPU-64xCPU-320GB-4xRTXPRO6000, fi-hel2. Measured: KV pool 4,552,849 tokens, 4.34x +# concurrency at 1,048,576 (~46x at the capped 98,304), 85,219/97,887 MiB per card, init 185 s +# (graph capture 79 s PIECEWISE + 16 s FULL). validate OK; soak 6/6 factual, 2/2 fabrication +# traps declined, 60/60 needles under 20-way graded long-context load. Needle-in-haystack also +# retrieved a head-of-prompt fact at 24,455 and 80,455 prompt tokens (11.8 s / 7.9 s). That is the +# check this model needs: vllm-project/vllm#55757 reports a broken sparse-MLA path on sm_120 still +# answering short prompts correctly and failing only at realistic lengths. +# +# WHY THIS EXISTS. 4x RTX PRO 6000 is 6.60 EUR/h against 18.00 for 4x B200, and the NVFP4 build is +# 175 GB of weights against 328 GB for FP8. Weights are 155.8 GB packed NVFP4 plus 19.3 GB BF16 +# (attention, vision and MTP stay BF16), so at TP=4 each rank holds ~43.8 GB against an 85.5 GB +# budget at 0.90 — about 41.7 GB of KV per rank. +# +# 2x RTX PRO 6000 (3.30 EUR/h) does not fit at this utilisation: 175 GB against 190 GB physical +# needs 0.97 and a short context. +# +# SPARSE MLA ON SM120, AND THE OVERLAY THAT FIXES IT. +# This model's MLA layers are rope-free (qk_rope_head_dim: 0), and no stock vLLM build has an +# SM120 kernel for that shape. Three failure modes are catalogued in vllm-project/vllm#53963: the +# fp8_ds_mla cache writer asserts pe_dim == 64; bf16 KV finds no backend at all because +# platforms/cuda.py hardcodes the cc 12.0 MLA candidates to [TRITON_MLA, +# FLASHINFER_MLA_SPARSE_SM120]; and widening the SM100 lane by hand stops at a shape guard, because +# the SM120 kernel only covers the DeepSeek 576-wide layout. vllm-project/vllm#54929 does not help +# either — its scope is that same DeepSeek geometry. +# +# vllm_plugins below closes the gap with an out-of-tree kernel. It is third-party code +# (github.com/Libertai/vllm-sparse-mla-blackwell, Apache-2.0), pinned to a commit, built inside this +# profile's own image and mounted read-only. It ships two vllm.general_plugins entry points, each +# inert unless its environment variable is set, so nothing in vLLM is patched. It fixes: +# 1. NoPE sparse MLA on sm_120/121 — a hand-written kernel taking head_size = kv_lora_rank +# directly, so no fabricated rope block is needed. Verified upstream on RTX PRO 6000 Blackwell +# Server, RTX 5090 and GB10, at 8/16/32 heads per rank (TP=8/4/2 on this 64-head model). +# 2. vllm-project/vllm#54189 — ModelOptNvFp4FusedMoE leaves w13_input_scale uninitialised and +# expects the checkpoint to fill it. A weight-only NVFP4 checkpoint (input_activations absent, +# which this one is) never does, it reads 0.0, and every expert output is multiplied by zero +# with no error. The tell is a model that loads and serves but emits one token repeatedly. +# Remove the overlay once vllm-project/vllm#55277 and flashinfer-ai/flashinfer#5075 land in a +# release image; both were open as of 2026-09-19. +# +# IMAGE. The cu130 tag, pinned by digest because the tag is mutable and has already moved — the +# working reports in #53963 ran sha256:2e771fa6..., which is the fallback if the digest below +# regresses. cu130 needs an r580+ host driver, hence min_driver_major. +# +# FLAGS THAT ARE NOT PREFERENCES. +# --no-enable-flashinfer-autotune : without it the engine spins at 100% GPU with no output until +# the NCCL watchdog kills it. The autotune step is the culprit, +# not the CUTLASS MoE backend, so --moe-backend marlin is not +# needed here. +# Measured 2026-09-20: with autotune off, vLLM auto-selects Marlin anyway — +# "Using 'MARLIN' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', +# 'FLASHINFER_CUTEDSL', ...]" +# So the flag stays unnecessary, but Marlin is what runs: the W4A16 dequant path (weights +# decompressed to FP16 before the GEMM). vllm-project/vllm#52225 reports Xid 13 warp errors on +# RTX PRO 6000 under sustained multi-hour load with Marlin MoE, which a short validate cannot +# surface — watch for it during long soaks and record the run length actually exercised. +# STATUS 2026-09-20: NOT MEASURED, AND DELIBERATELY SO. Reaching that regime needs 45+ minutes +# of continuous graded load at 6.60 EUR/h list (4.95 spot), roughly 4 EUR, and the repo owner +# declined the spend. Both the original validation and the 2026-09-20 re-run exercised MINUTES, +# not hours. This is a decision, not a TODO. For multi-hour production traffic on this profile +# treat Xid 13 as an unquantified risk and watch the container restart counter: +# bin/spin ssh "docker inspect -f '{{.RestartCount}}' gpu-spinner-vllm-1" +# no --enforce-eager : CUDA graphs capture normally with this overlay, and eager was +# measured 5-6x slower on this hardware. Do not add it. +# fp8 KV is supported by the overlay kernel (bf16 also works, at roughly half the KV capacity). +# +# DISK: ~195 GB to download — 121 safetensors shards totalling 194.6 GB on the main revision. That +# is larger than the 175 GB of loaded weights above, because the on-disk shards carry block scales +# and padding the VRAM figure does not; size from 195, not 175. (The HF repo reports ~587 GB of +# usedStorage, which counts every revision, not what a pull fetches.) +# Set WEIGHTS_SIZE_GB to at least 250 and re-run bin/spin persistent-init before the first pull, or +# the download dies part-way with "No space left on device" on a GPU billed by the hour. +# Measured 2026-09-20: /data was 29 GB used before any weights, because the docker role moves +# Docker's data-root onto this disk and the cu130 image is ~8.7 GB. A 250 GB disk leaves ~206 GB +# for a ~195 GB pull — ~11 GB of margin. Size from weights + ~30 GB of images: 250 is the floor, +# prefer 300 if the disk is staying. +# At MaxIOPS a 250 GB disk is ~EUR 55/mo standing. It is also above DECOMMISSION_THRESHOLD_GB +# (default 150), so `bin/spin down` will remove it by default and the next run re-pulls 195 GB — +# use `bin/spin down --keep-disk` or DECOMMISSION_ON_DOWN=never between test runs. +# +# CONTEXT CEILING. max_model_len is capped at 98304, well under the checkpoint's native 1,048,576, +# because one over-long request kills the engine for every other user on the box. +# +# Measured 2026-09-20, by prompt tokens per request: +# 1 x 102,400 -> crash 10 x 98,100 -> pass 10/10 (p50 57 s) +# 1 x 106,496 -> crash 10 x 65,403 -> pass 10/10 (p50 50 s) +# 1 x 131,072 -> crash 20 x 51,670 -> pass 20/20 (p50 27 s) +# 2 x 250,000 -> crash 20 x 25,861 -> pass 60/60 needles over 3 rounds (p50 27.4 s, +# 20 x ~103,000 -> crash p95 31.0 s, head/middle/tail depths) +# The boundary is between 98,100 and 102,400 prompt tokens per request, independent of how many +# requests are in flight. Record prompt_tokens rather than the `--context` you asked for: the +# haystack sizer aims by words, so the requested figure runs ~25% high. +# +# Fault: CUDA illegal memory access in the KDA linear-attention path. +# vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py:1214 chunk_kda_with_fused_gate +# -> kernels.py:1126 chunk_kda_with_fused_gate_fwd -> prepare_chunk_indices(cu_seqlens, ...) +# -> vllm/third_party/flash_linear_attention/ops/index.py:27 +# torch.AcceleratorError: CUDA error: an illegal memory access was encountered +# 34 of the 45 layers are KDA, i.e. stock vLLM code that the sparse-MLA overlay never touches. +# +# Ruled out, so nobody re-chases them: +# * prefix caching AS A CAUSE OF THIS CRASH (vllm-project/vllm#55600 names KDA): 1 x 131,072 with +# a unique randomised haystack per request, no cache hits possible, crashes identically. +# SCOPE, added 2026-09-22: that clears prefix caching for the CRASH only. It is not a clearance +# for correctness -- prefix caching corrupts ANSWERS on this architecture (vllm#56960, see the +# --no-enable-prefix-caching note above), which is a separate fault that never crashes anything. +# * concurrency: one request at 102,400 is enough. +# * KV capacity: peaked at 17.8% during a crash; the 4.55M-token pool would hold 10 x 455k. +# * memory exhaustion: VRAM flat at 86,981/97,887 MiB per GPU (that is the pre-allocated 0.90 +# pool, ~10.9 GB still free per card), host RAM 16 of 314 GB, no OOM kills. The exception is +# an out-of-bounds index, not OutOfMemoryError, so gpu_memory_utilization will not move it. +# * config: chunked prefill is already on with max_num_batched_tokens=8192. +# * this GPU and this overlay: vllm-project/vllm#54317 reports the same faulting frame on 4x B200 +# (SM100) with the stock day-0 image, so the FP8 glm53-flash profile is exposed too. That report +# is sporadic under sustained traffic; the length-triggered repro above is deterministic and is +# worth adding to the thread. +# +# With the cap, vLLM rejects an over-long request with HTTP 400 instead of dying. It bounds the +# blast radius but does not guarantee stability, since #54317 also fires sporadically under long +# uptime — watch container restarts. Raise it only after the upstream kernel fix, and re-run +# `bin/spin soak --context --concurrency 10` first. +# +# Concurrency at 98,304: 4,552,849 KV tokens / 98,304 = ~46 by KV math, 10-way measured clean. +# 5-10 concurrent users at ~98k each is the supported envelope. +# +# NO NVLINK on this plan: TP=4 runs over PCIe. If engine init hangs in NCCL rather than failing, +# add NCCL_P2P_DISABLE: 1 to extra_env — it costs throughput, so do not set it pre-emptively. +# +# WHAT TO CHECK ON A RE-RUN, in order: +# 1. The driver major the gpu role prints ("Report the detected GPU topology"). This image is +# cu130 and needs r580+; it also answers whether other profiles could move to a cu130 tag. +# Record it in docs/validation.md either way. +# 2. The engine log says: Using FLASHINFER_MLA_SPARSE_SM120 attention backend. The overlay +# replaces that slot, so no --attention-backend flag is needed; if the line is missing, the +# plugin did not load and the run is not testing what you think. +# 3. CUDA graphs capture (no --enforce-eager in the log). +# 4. `bin/spin validate` returns coherent text. One token repeated over and over is the signature +# of the zeroed-expert fault (#54189) — it means VLLM_GLM53_MOE_INPUT_SCALE did not take, not +# that the model is bad. +# 5. `bin/spin soak` — the concurrency and answer-quality pass. Do not skip it because step 4 +# looked fine; it is also what exercises long prompts, which a smoke test does not. Broken +# sparse-MLA paths on sm_120 serve short prompts correctly (vllm-project/vllm#55757). +# Then add the row to docs/validation.md with KV pool, concurrency and VRAM. +# +# PROVENANCE: a third-party NVFP4 conversion (LibertAIDAI), not a vendor build, so its quality may +# differ from the original weights. glm53-flash.yml serves the same model from Z.ai's FP8 release. +model_repo: "LibertAIDAI/GLM-5.3-Flash-NVFP4" +served_name: "glm53-flash-nvfp4" +vllm_image_override: "vllm/vllm-openai:glm53-flash-x86_64-cu130@sha256:0674a3ea3c971706cea310bca71daadff68ea1717c758d2f54affa5def001b86" +tensor_parallel_size: 4 +max_model_len: 98304 # not the model's 1,048,576 — see CONTEXT CEILING above +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +vllm_plugins: + - name: sparse-mla-blackwell + repo: "https://github.com/Libertai/vllm-sparse-mla-blackwell" + ref: "b1f20638d7f82f880e4bcb2136ee2f19c06f38a4" + build_env: + GLM53_ARCHS: "120a" +extra_env: + VLLM_GLM53_CUDA_SPARSE_MLA: 1 # the NoPE sparse-MLA backend + VLLM_GLM53_MOE_INPUT_SCALE: 1.0 # the missing NVFP4 MoE activation scale (#54189) +extra_vllm_args: + - "--no-enable-flashinfer-autotune" + # Correctness flag, not a tuning knob: this checkpoint is hybrid, and a prefix-cache hit resumes + # its KDA layers from a recycled block's recurrent state, so a long prompt is answered fast and + # wrong (vllm#56960). Measured 2026-09-22 — see docs/gpu-spinner.md, "Hybrid checkpoints and + # prefix caching". + # Separate from the 98,304 crash cap below: that is an illegal memory access caught by + # RestartCount, and this defect never crashes an engine. The recorded soak passes also stand -- + # a cache hit returns a wrong answer, so it cannot manufacture a pass. + - "--no-enable-prefix-caching" + - "--kv-cache-dtype" + - "fp8" + - "--reasoning-parser" + - "glm47" + - "--tool-call-parser" + - "glm47" + - "--enable-auto-tool-choice" +# Hybrid attention: most layers carry recurrent state, which is what makes +# --no-enable-prefix-caching mandatory above. tests/check-prefix-caching.sh keys on this. +hybrid_recurrent_state: true +min_plan: "GPU-64xCPU-320GB-4xRTXPRO6000" +min_vram_gb: 90 # per-GPU floor: RTX PRO 6000 reports 95, B200 179; keeps H100 (79) out +min_disk_gb: 280 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 181 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # Blackwell; the overlay kernel covers sm_120 and sm_121 +min_driver_major: 580 # the image is a cu130 build +validated: true # live run 2026-09-20 — see docs/validation.md +# At 6.60 EUR/h the 4-GPU plan sits under the default MAX_EUR_PER_HOUR of 10, so a bare +# `bin/spin up --model glm53-flash-nvfp4` deploys it without an opt-in. Lower the limit if that is +# more per hour than you want a stray command to be able to spend. +health_wait_retries: 300 # x10s = 50 min — ~175 GB to pull, plugin build, then TP=4 init diff --git a/ansible/models/glm53-flash.yml b/ansible/models/glm53-flash.yml new file mode 100644 index 0000000..aa00023 --- /dev/null +++ b/ansible/models/glm53-flash.yml @@ -0,0 +1,69 @@ +--- +# GLM-5.3-Flash (Z.ai) — ~321B total / 18B active multimodal MoE, 45 layers mixing KDA linear +# attention with NoPE sparse MLA, 8 of 288 experts per token. Image and video input, 1,048,576 +# native context, one MTP draft layer. Native FP8 weights, ~306 GiB. +# +# PINNED IMAGE (required): Glm5NextForConditionalGeneration is in neither v0.23.0 nor v0.29.0 — the +# recipe says to use Docker until the integration reaches the public repo, so this pins +# vllm/vllm-openai:glm53-flash. That build also satisfies the FlashInfer 0.6.17 floor NoPE sparse +# MLA needs. +# +# 4x B200 matches the recipe's verified TP4-on-one-GB200-tray (768 GB) configuration — ~18 EUR/h, +# hence the cost gate. Hopper is supported by the implementation but cannot use an FP8 KV cache +# for this model and must run BF16 KV, so min_compute_capability pins the Blackwell path the flags +# below assume. +# +# MTP: one draft layer is in the checkpoint and the recipe's TP4 command enables it with 5 tokens. +# Left off so the first validation run tests the model alone. Enable with: +# extra_vllm_args += ["--speculative-config", '{"method":"mtp","num_speculative_tokens":5}'] +# +# Vision stays on: the recipe's command has no --language-model-only equivalent, and the multimodal +# path is part of what this model is for. Serving is text-first either way. +# +# DISK: ~328 GB of weights. Set WEIGHTS_SIZE_GB to at least 400 and re-run bin/spin persistent-init +# before the first pull. At MaxIOPS that is ~EUR 88/mo standing. +# +# NOT ON RTX PRO 6000 — use glm53-flash-nvfp4.yml, which serves the same model on 4x RTX PRO 6000 at +# 6.60 EUR/h by carrying an out-of-tree kernel. This profile stays the plain, overlay-free FP8 path. +# The MLA layers are rope-free (qk_rope_head_dim: 0) and no stock vLLM build has an SM120 kernel for +# that shape (three failure modes in vllm-project/vllm#53963). The failure also hides: +# vllm-project/vllm#55757 reports DSA models answering short prompts correctly on sm_120 and dying +# at realistic lengths, so unsupported_compute_capabilities refuses the GPU up front. +model_repo: "zai-org/GLM-5.3-Flash" +served_name: "glm53-flash" +vllm_image_override: "vllm/vllm-openai:glm53-flash" +tensor_parallel_size: 4 +max_model_len: 1048576 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + # Correctness flag, not a tuning knob: this checkpoint is hybrid, and a prefix-cache hit resumes + # its KDA layers from a recycled block's recurrent state, so a long prompt is answered fast and + # wrong (vllm#56960). Measured 2026-09-22 — see docs/gpu-spinner.md, "Hybrid checkpoints and + # prefix caching". + # This profile has never had a live run, so nothing here is being retracted -- the flag is set + # before the first EUR 18/h session rather than after it. + - "--no-enable-prefix-caching" + - "--kv-cache-dtype" + - "fp8" + - "--reasoning-parser" + - "glm47" + - "--tool-call-parser" + - "glm47" + - "--enable-auto-tool-choice" +# Hybrid attention: most layers carry recurrent state, which is what makes +# --no-enable-prefix-caching mandatory above. tests/check-prefix-caching.sh keys on this. +hybrid_recurrent_state: true +min_plan: "GPU-96xCPU-960GB-4xB200" +min_vram_gb: 175 # per-GPU floor (B200 reports 179 GB); ~306 GiB of weights across TP=4 +min_disk_gb: 420 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 306 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # FP8 KV cache for this model is Blackwell-only; Hopper needs BF16 KV +unsupported_compute_capabilities: [12.0] +unsupported_reason: >- + its rope-free sparse MLA (qk_rope_head_dim 0) has no SM120 attention or KV path in any stock vLLM + build (vllm-project/vllm#53963), and the failure only appears at realistic prompt lengths + (vllm-project/vllm#55757). glm53-flash-nvfp4 is the RTX PRO 6000 path +validated: false # newly added — no live run on this account yet +health_wait_retries: 400 # x10s = 67 min — ~328 GB to pull, then TP=4 init diff --git a/ansible/models/glm53.yml b/ansible/models/glm53.yml new file mode 100644 index 0000000..d67b510 --- /dev/null +++ b/ansible/models/glm53.yml @@ -0,0 +1,50 @@ +--- +# GLM-5.3 (Z.ai) — ~743B total / 39B active MoE, text-only, 1M native context, 5-token MTP. +# Architecture and serving flags are identical to GLM-5.2 (glm52.yml); the packaging changed, and +# the default zai-org/GLM-5.3 checkpoint is now native FP8 rather than a separate build. BF16 lives +# under zai-org/GLM-5.3-BF16. ~756 GB of weights. +# +# PINNED IMAGE. GlmMoeDsaForCausalLM is already in the shared v0.23.0 pin, but the recipe's floor +# for this model is vLLM 0.28.0: a registry hit means the code exists, the recipe says which version +# serves it. This uses v0.29.0, the newest release. +# +# 8x B200. The recipe's single-node options are 8x H200 or 8x H20 (141 GB each) for FP8, and 8x B200 +# (180 GB each) to reach 1M context with an FP8 KV cache. UpCloud has neither H200 nor H20, so +# 8x B200 it is — ~36 EUR/h, the glm52 tier, hence the cost gate. 8x H100 would be 640 GB against +# ~756 GB of weights, and 8x RTX PRO 6000 768 GB with nothing left for KV and no NVLink; +# min_vram_gb keeps both out. +# +# MTP: the 5-token draft is left off so the first validation run tests the model alone — at this +# hourly rate a failure that could be either half is expensive. Enable with: +# extra_vllm_args += ["--speculative-config", '{"method":"mtp","num_speculative_tokens":5}'] +# +# DISK: ~756 GB of weights. Set WEIGHTS_SIZE_GB to at least 800 and re-run bin/spin persistent-init +# before the first pull. At MaxIOPS that is ~EUR 176/mo standing, billed whether or not a GPU +# exists. Either budget for it or let bin/spin down remove the disk each evening and accept a fresh +# ~100 min pull. +# +# An NVFP4 checkpoint exists (Inferact/GLM-5.3-NVFP4) and is not used here: NVFP4 MoE takes the +# CUTLASS grouped block-scaled GEMM path that misbehaves on RTX Blackwell, and on B200 the native +# FP8 build is what the recipe verifies. +model_repo: "zai-org/GLM-5.3" +served_name: "glm53" +vllm_image_override: "vllm/vllm-openai:v0.29.0-cu129-ubuntu2404" +tensor_parallel_size: 8 +max_model_len: 1048576 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + - "--kv-cache-dtype" + - "fp8" + - "--reasoning-parser" + - "glm47" + - "--tool-call-parser" + - "glm47" + - "--enable-auto-tool-choice" +min_plan: "GPU-192xCPU-1920GB-8xB200" +min_vram_gb: 175 # per-GPU floor (B200 reports 179 GB); ~756 GB of weights across TP=8 +min_disk_gb: 840 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 704 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +validated: false # newly added — no live run on this account yet +health_wait_retries: 800 # x10s = 133 min — ~756 GB to pull, then TP=8 init across eight GPUs diff --git a/ansible/models/k2-horizon-375b.yml b/ansible/models/k2-horizon-375b.yml new file mode 100644 index 0000000..da29d45 --- /dev/null +++ b/ansible/models/k2-horizon-375b.yml @@ -0,0 +1,61 @@ +--- +# K2-Horizon-375B-A23B (IFM) in FP8 — frontier-scale MoE, 379B stored parameters including +# embeddings, 26.7B active per token. 61 layers at hidden 6144, 192 experts with 8 active, +# text-only, 524,288 native context. The FP8 build is ~391 GB across 20 shards +# (compressed-tensors); the BF16 original is ~758 GB. +# +# PINNED IMAGE (required): K2HorizonForCausalLM is in neither v0.23.0 nor v0.29.0 — it reached vLLM +# main after v0.29.0, so this profile uses the same dated nightly as deepseek-v41-flash. Replace +# both with a release tag once one carries the architecture. +# +# WHY FP8 AT TP8 ON RTX PRO 6000. recipes.vllm.ai/IFM/K2-Horizon-375B-A23B serves the BF16 repo at +# --tensor-parallel-size 8 --dtype bfloat16, verified only on an 8-GPU MI355X node. At ~758 GB that +# would need 8x B200 here, ~36 EUR/h. The FP8 build is ~391 GB (367.8B in FP8 plus 11.3B in BF16), +# which changes what fits: +# 8x RTX PRO 6000 768 GB, 13.20 EUR/h <- this profile: ~48.8 GB/rank, ~36 GB of KV per rank +# 4x B200 768 GB, 18.00 EUR/h <- the previous lane, at TP=4 +# 4x RTX PRO 6000 384 GB <- does not fit: 391 GB of weights alone +# TP=8 is cheaper than the 4x B200 lane and closer to the published run. That lane is still one +# command away, since extra vars outrank a profile: +# bin/spin up --model k2-horizon-375b --allow-expensive \ +# --plan GPU-96xCPU-960GB-4xB200 -e tensor_parallel_size=4 +# +# NO NVLINK on the RTX PRO 6000 plans: TP=8 over PCIe across eight cards is the heaviest +# communication pattern in this repo, and unmeasured. If engine init hangs inside NCCL rather than +# failing, add NCCL_P2P_DISABLE: 1 to extra_env — it costs throughput, so not pre-emptively. +# Attention here is standard partial-RoPE, so the SM120 sparse-MLA gap that blocks the GLM and +# DeepSeek profiles (vllm-project/vllm#55757) leaves this one alone. +# +# max_model_len 131072 is what the verified MI355X run used; native is 524,288. With ~48.8 GB of +# weights per rank against 95 GB there is headroom to raise it once a live run shows the KV pool. +# +# --trust-remote-code and --enable-expert-parallel come from the recipe's launch command. Reasoning +# effort is per request via chat_template_kwargs {"reasoning_effort": "high|medium|low"} (high is +# the default) or server-wide via --default-chat-template-kwargs. The model card suggests +# temperature 1.0 with top_p 0.95. +# +# DISK: ~391 GB against a default 150 GB /data. Set WEIGHTS_SIZE_GB to at least 450 and re-run +# bin/spin persistent-init before the first pull. At MaxIOPS that is ~EUR 99/mo standing. +model_repo: "IFM/K2-Horizon-375B-A23B-FP8" +served_name: "k2-horizon-375b" +vllm_image_override: "vllm/vllm-openai:cu129-nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3" +tensor_parallel_size: 8 +max_model_len: 131072 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + - "--trust-remote-code" + - "--enable-expert-parallel" + - "--reasoning-parser" + - "k2_horizon" + - "--tool-call-parser" + - "k2_horizon" + - "--enable-auto-tool-choice" +min_plan: "GPU-128xCPU-640GB-8xRTXPRO6000" +min_vram_gb: 90 # per-GPU floor: RTX PRO 6000 reports 95; ~48.8 GB per rank at TP=8 +min_disk_gb: 480 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 364 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +validated: false # newly added — no live run on this account yet +untested_compute_capabilities: [12.0] # RTX PRO 6000 is the target tier; nothing measured there +health_wait_retries: 450 # x10s = 75 min — ~391 GB to pull, then TP=8 init diff --git a/ansible/models/kimi-k3.yml b/ansible/models/kimi-k3.yml new file mode 100644 index 0000000..5936524 --- /dev/null +++ b/ansible/models/kimi-k3.yml @@ -0,0 +1,53 @@ +--- +# Kimi K3 (Moonshot AI) — 93-layer multimodal MoE with 896 experts, 1M native context. +# ~1561 GB of weights across 96 shards, compressed-tensors quantized. +# +# NO UPCLOUD PLAN FITS THIS PROFILE. It is here so the requirement is recorded and so it starts +# working the day a large enough tier appears; the preflights refuse it on everything UpCloud sells +# today rather than letting it fail an hour into a paid deploy. Three things block it: +# +# 1. VRAM. The recipe's floor is "at least 8x GB300" — 8x 288 GB = 2304 GB. UpCloud's largest GPU +# plan is 8x B200, ~1538 GB usable, and the weights alone are ~1561 GB before KV or +# activations. min_vram_gb: 280 encodes the per-GPU class, so a B200 (179 GB) is refused. +# UpCloud lists B300 (288 GB) on its pricing page but publishes no plan identifier, and +# min_plan now has to be a real one — bin/spin reads it to choose the plan to deploy. With the +# key absent, bin/spin falls back to the OpenTofu default and the preflights refuse the run, +# which is the right outcome here. +# 2. DRIVER. The kimi-k3 image is CUDA 13 (cu130) only — no cu129 tag, and the K3 wheels are not +# on the cu129 nightly index. It needs an r580+ host driver, which the UpCloud GPU template +# decides; min_driver_major: 580 checks that before the container starts. +# 3. DISK. ~1561 GB of weights needs WEIGHTS_SIZE_GB around 1700. At MaxIOPS that is ~EUR 374/mo +# standing, billed whether or not a GPU exists — by far the largest number in this repo. +# +# If a suitable tier appears, the recipe's cross-node notes are: --all2all-backend deepep_v2 for +# RDMA and flashinfer_nvlink_one_sided for NVLink, with --moe-backend deep_gemm_mega_moe for DEP +# over cross-node NVLink (GB200/GB300). DeepGEMM MegaMoE is not compatible with cross-node RDMA. +# +# max_model_len 262144 is a conservative quarter of the native 1,048,576; the recipe gives no +# measured context figure to justify starting at full length. +# +# Neither rebuild helps the fit: nvidia/Kimi-K3-NVFP4 is larger at ~1610 GB and RedHatAI's FP8-BLOCK +# larger still at ~2820 GB. Moonshot's own checkpoint is the smallest of the three. +model_repo: "moonshotai/Kimi-K3" +served_name: "kimi-k3" +vllm_image_override: "vllm/vllm-openai:kimi-k3" +tensor_parallel_size: 8 +max_model_len: 262144 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + - "--kv-cache-dtype" + - "fp8" + - "--reasoning-parser" + - "kimi_k3" + - "--tool-call-parser" + - "kimi_k3" + - "--enable-auto-tool-choice" +min_vram_gb: 280 # per-GPU floor: GB300/B300 class. A B200 (179 GB) is refused. +min_disk_gb: 1700 # disk size to provision: (weights_gb + 80) / 0.934; raised to this profile's stated floor +weights_gb: 1454 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 +min_driver_major: 580 # the kimi-k3 image is a cu130 build and needs an r580+ host driver +validated: false # newly added — no live run on this account yet +health_wait_retries: 1500 # x10s = 250 min — ~1561 GB to pull, then TP=8 init diff --git a/ansible/models/qwen36-27b-nvfp4.yml b/ansible/models/qwen36-27b-nvfp4.yml index b68ae08..f84ac4d 100644 --- a/ansible/models/qwen36-27b-nvfp4.yml +++ b/ansible/models/qwen36-27b-nvfp4.yml @@ -1,10 +1,13 @@ --- -# Qwen3.6-27B in NVFP4 — Blackwell (B200) ONLY. Official NVIDIA ModelOpt checkpoint -# (nvidia/Qwen3.6-27B-NVFP4, Apache-2.0, "ready for inference with vLLM"). Dense, so no -# FlashInfer-MoE env needed. Quant auto-detected (no --quantization). --language-model-only drops -# the vision tower; --enforce-eager guards the hybrid (Mamba) state cache (drop once stable on -# Blackwell — B200 has ample headroom for a 27B at NVFP4). min_vram_gb=175 pins to B200. Live-validated -# on B200 2026-07-04 (262144, KV 4.41M tok, 16.84x, 169/183 GiB); re-served via llama-swap 2026-07-05. +# Qwen3.6-27B in NVFP4 — Blackwell-class GPUs (B200 or RTX PRO 6000). Official NVIDIA ModelOpt +# checkpoint (nvidia/Qwen3.6-27B-NVFP4, Apache-2.0, "ready for inference with vLLM"). Dense, so it +# uses the NVFP4 scaled-mm path rather than the grouped-GEMM MoE kernels that misbehave on RTX +# Blackwell — which is why this profile runs on cc 12.0 and the NVFP4 *MoE* profiles do not. +# Quant auto-detected (no --quantization). --language-model-only drops the vision tower; +# --enforce-eager guards the hybrid (Mamba) state cache (drop once stable on Blackwell). +# Live-validated on B200 2026-07-04 (262144, KV 4.41M tok, 16.84x, 169/183 GiB); re-served via +# llama-swap 2026-07-05. Not yet validated on RTX PRO 6000: ~21 GB of weights leave roughly 70 GB +# of the 96 GB card for KV, so full context should hold, but the numbers above are B200 numbers. model_repo: "nvidia/Qwen3.6-27B-NVFP4" served_name: "qwen3.6-27b-nvfp4" tensor_parallel_size: 1 @@ -19,5 +22,9 @@ extra_vllm_args: - "--enforce-eager" - "--kv-cache-dtype" - "fp8" -min_plan: "GPU-24xCPU-240GB-1xB200" -min_vram_gb: 175 # B200 reports 179 GB after floor-div (183,359 MiB) — measured live 2026-07-04 +min_plan: "GPU-24xCPU-240GB-1xB200" # the validated tier; 1xRTXPRO6000 also passes, untested +min_vram_gb: 40 # ~21 GB weights + KV; keeps an L4/L40S-sized card out +min_disk_gb: 110 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 20 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # Blackwell or newer: B200 (cc 10.0) and RTX PRO 6000 (cc 12.0) +untested_compute_capabilities: [12.0] # dense NVFP4 should run on RTX PRO 6000; not yet measured diff --git a/ansible/models/qwen36-35b-nvfp4.yml b/ansible/models/qwen36-35b-nvfp4.yml index 875a522..9b3f918 100644 --- a/ansible/models/qwen36-35b-nvfp4.yml +++ b/ansible/models/qwen36-35b-nvfp4.yml @@ -1,9 +1,23 @@ --- -# Qwen3.6-35B-A3B in NVFP4 — Blackwell (B200) ONLY. Official NVIDIA ModelOpt checkpoint (~19 GB). -# MoE NVFP4 backend: let vLLM auto-select (CUTLASS). Do NOT set VLLM_USE_FLASHINFER_MOE_FP4=1 — on this -# auto-detected (no --quantization). --language-model-only drops the vision tower; --enforce-eager -# guards the hybrid state cache (drop once stable on Blackwell). min_vram_gb=175 pins to B200. -# Live-validated on B200 2026-07-04 (262144, KV 13.9M tok, 53.02x, 168/183 GiB); llama-swap 2026-07-05. +# Qwen3.6-35B-A3B in NVFP4 — Blackwell only: B200 (cc 10.0) or RTX PRO 6000 (cc 12.0). Official +# NVIDIA ModelOpt checkpoint (~19 GB). Quantization is auto-detected (no --quantization); let vLLM +# pick the NVFP4 MoE backend, since VLLM_USE_FLASHINFER_MOE_FP4=1 raises "no FlashInfer NVFP4 MoE +# backend supports the configuration" on this image. --language-model-only drops the vision tower; +# --enforce-eager guards the hybrid state cache (drop it once that is stable on Blackwell). +# +# VALIDATED on both lanes. +# B200, 2026-07-04: 262144 ctx, KV 13.9M tokens, 53.02x concurrency, 168/183 GiB; re-served via +# llama-swap 2026-07-05. +# RTX PRO 6000, 2026-09-20: GPU-SPOT-16xCPU-80GB-1xRTXPRO6000, driver 595.58.03, on the shared +# cu129 pin. vLLM chose the 'MARLIN' NvFp4 MoE kernel and answered correctly throughout. KV pool +# 6,269,102 tokens, 23.91x at 262,144, 90,283/97,887 MiB. soak 3x20 @32768: factual 6/6, traps +# 2/2 declined, 60/60 needles, 0 wrong, p50 17.3 s. +# +# Until that run this profile refused cc 12.0, on the grounds that NVFP4 MoE takes a CUTLASS grouped +# block-scaled GEMM returning invalid output on cu129. gemma4-26b-nvfp4 passed the same test the +# same day through 'VLLM_CUTLASS', so the gate had been banning the whole family over one suspect +# kernel. Both upstream fixes had already landed: flashinfer-ai/flashinfer#2708 closed 2026-04-20 +# and vllm-project/vllm#54189 closed 2026-08-30, before this image was built on 2026-06-13. model_repo: "nvidia/Qwen3.6-35B-A3B-NVFP4" served_name: "qwen3.6-35b-a3b-nvfp4" tensor_parallel_size: 1 @@ -22,5 +36,9 @@ extra_vllm_args: - "--enforce-eager" - "--kv-cache-dtype" - "fp8" -min_plan: "GPU-24xCPU-240GB-1xB200" -min_vram_gb: 175 # B200 reports 179 GB after floor-div (183,359 MiB) — measured live 2026-07-04 +min_plan: "GPU-16xCPU-80GB-1xRTXPRO6000" # 1.65/h list, 1.24 spot; the 1xB200 lane (4.50/h) + # also serves this profile. +min_vram_gb: 40 # real weight+KV floor; the capability gates below pin the family +min_disk_gb: 110 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 22 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # NVFP4 needs Blackwell — no L4/L40S/H100 diff --git a/ansible/models/qwen36-35b.yml b/ansible/models/qwen36-35b.yml index 572f089..adf2587 100644 --- a/ansible/models/qwen36-35b.yml +++ b/ansible/models/qwen36-35b.yml @@ -5,6 +5,9 @@ # --language-model-only drops the vision tower; --enforce-eager guards the hybrid linear-attention # (Mamba) state cache. On B200 you can drop --enforce-eager and raise ctx/seqs. Live-validated on # H100 2026-07-04 (262144 ctx, KV 3.42M tok, 13.03x, 74/81 GiB) — see docs/validation.md. +# H100 80 GB (1.79 EUR/h) is the validated floor and stays the recommended tier. A 1x RTX PRO 6000 +# (96 GB, 1.65 EUR/h) also clears the VRAM gate and is slightly cheaper with more headroom, but has +# not been measured for this profile — the deploy warns and serves. GPU prices move; keep both. model_repo: "Qwen/Qwen3.6-35B-A3B-FP8" served_name: "qwen3.6-35b-a3b-fp8" tensor_parallel_size: 1 @@ -25,4 +28,7 @@ extra_vllm_args: - "fp8" min_plan: "GPU-12xCPU-240GB-1xH100" min_vram_gb: 76 +min_disk_gb: 130 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 35 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +untested_compute_capabilities: [12.0] # runs on RTX PRO 6000, not yet measured there health_wait_retries: 120 # x10s = 20 min — 37.5 GB download + MoE load took ~15 min cold diff --git a/ansible/models/qwen36.yml b/ansible/models/qwen36.yml index 24f68ad..39e2adf 100644 --- a/ansible/models/qwen36.yml +++ b/ansible/models/qwen36.yml @@ -14,7 +14,8 @@ tensor_parallel_size: 1 # --max-num-seqs 16 : bound concurrency -> bounded KV/scheduler memory # --enforce-eager : skip cudagraph capture for headroom + faster start # FP8 quantization is auto-detected from the checkpoint config. For more throughput on a -# larger GPU (e.g. H100 80 GB), drop --enforce-eager and raise max-model-len / batched-tokens. +# larger GPU (H100 80 GB, or 1x RTX PRO 6000 96 GB at 1.65 EUR/h), drop --enforce-eager and raise +# max-model-len / batched-tokens — the L40S-specific caps above exist only to fit 48 GB. max_model_len: 262144 gpu_memory_utilization: 0.90 dtype: "auto" @@ -22,6 +23,14 @@ enable_lora: false extra_vllm_args: - "--language-model-only" - "--reasoning-parser=qwen3" + # Prefix caching stays on, reviewed 2026-09-22. This is a hybrid checkpoint, the class where a + # cache hit can answer fast and wrong (see docs/gpu-spinner.md, "Hybrid checkpoints and prefix + # caching") — but qwen38 and + # qwen38-flash-next both hold graded long-context passes with caching on, so the Qwen hybrids have + # counter-evidence. This profile has none of its own: its row predates the graded soak. Being the + # repo default on the cheapest lane, it is the first thing to re-measure on the next GPU: + # bin/spin soak --rounds 1 --concurrency 3 --context 262144 + # A wrong needle there with RestartCount unmoved means flip this flag. - "--enable-prefix-caching" - "--enforce-eager" - "--max-num-batched-tokens=2048" @@ -31,5 +40,8 @@ extra_vllm_args: - "--kv-cache-dtype" - "fp8" min_plan: "GPU-8xCPU-64GB-1xL40S" +untested_compute_capabilities: [12.0] # also fits RTX PRO 6000 / H100; only L40S is measured # Preflight floor (GB): an L40S reports ~44 GB. 40 blocks an accidental L4 run (which would OOM). min_vram_gb: 40 +min_disk_gb: 120 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 29 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) diff --git a/ansible/models/qwen38-flash-next.yml b/ansible/models/qwen38-flash-next.yml new file mode 100644 index 0000000..a345fc2 --- /dev/null +++ b/ansible/models/qwen38-flash-next.yml @@ -0,0 +1,108 @@ +--- +# Qwen3.8-Flash-Next in FP8 (official Qwen block-scaled build, ~186 GB on disk / 172.78 GiB loaded). +# Qwen4 architecture preview: ultra-sparse MoE, 125B parameters with 6B active per token, plus a +# separate 51B N-gram embedding table and a 4B MTP head. 48 layers as +# 12 x (3 x (Gated DeltaNet -> MoE) -> 1 x (Qwen Sparse Attention -> MoE)), 512 experts with 10 +# routed + 1 shared active. Native 262,144 context, 1M with YaRN. Multimodal. +# +# PINNED IMAGE (required): the recipe states that PyPI installation is not supported for this model +# and that the dedicated vllm/vllm-openai:qwen38-flash-next build must be used. The shared image pin +# does not carry the qwen4_exp architecture. +# +# WHY 4x RTX PRO 6000. recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next verified 4x H100 at ~1,430 output +# tok/s (concurrency 64, 1024-in/256-out), sold here as GPU-48xCPU-960GB-4xH100 at ~7.16 EUR/h. +# GPU-64xCPU-320GB-4xRTXPRO6000 is 6.60 EUR/h for 96 GB cards instead of 80, so it is the default +# lane; the H100 tier is a one-flag fallback (--plan GPU-48xCPU-960GB-4xH100) and the only one with +# measured numbers behind it. +# +# TWO FLAGS BELOW, both of which stay on this lane: +# VLLM_PLE_CPU_OFFLOAD=1 — the 51B N-gram table is replicated per rank, not sharded, so it fits +# neither an 80 GB H100 nor a 96 GB card: 95 x 0.85 = 80.75 GB per rank +# against ~51 GB of table plus ~33.6 GB of sharded weights. Offloaded, +# the table sits in host RAM (320 GB here, ~51 GB needed) with rows +# prefetched asynchronously, leaving ~47 GB per rank for KV. Optional +# from 141 GB (H200) up. +# --moe-backend triton — the recipe's Hopper MoE backend. It works on every generation, and +# dropping it on RTX Blackwell is a measurement nobody has taken. Try +# that once a clean run exists to compare against. +# +# NO NVLINK on the RTX PRO 6000 plans: TP=4 runs over PCIe. If engine init hangs inside NCCL rather +# than failing, add NCCL_P2P_DISABLE: 1 to extra_env. Qwen Sparse Attention sidesteps the SM120 +# sparse-MLA gap that blocks the GLM and DeepSeek profiles — vllm-project/vllm#55430 is a prefill +# tuning PR enabled on SM121 only. +# +# OTHER TOPOLOGIES. TP2 is the minimum validated FP8 deployment (on GB300), TP4 the recommended full +# tray. On eight GPUs use TEP8 (TP8 plus --enable-expert-parallel); plain TP8 is incompatible with +# this checkpoint's 128-wide quantization blocks. --enable-expert-parallel helps TP4 too. +# +# MTP: the checkpoint carries a Multi-Token Prediction head, left off because the recipe measured it +# making throughput worse at every concurrency on 4x H100 (8-36% lower request throughput, 32-173% +# higher per-token latency, ~36% acceptance). Verify before enabling on other hardware: +# extra_vllm_args += ["--speculative-config", '{"method":"mtp","num_speculative_tokens":3}'] +# +# Leave --max-num-seqs at 256: lowering it triggers a Mamba-cache capacity error at startup. +# --gpu-memory-utilization 0.85 is the verified H100 value (the recipe advises 0.90 for large +# multimodal batches). --tool-call-parser is qwen3_coder here, the qwen3_xml one belongs to +# Qwen3.8-27B. --reasoning-parser qwen3 is required: the chat template opens every turn with +# . +# +# max_model_len 262144 is the native limit. Startup and bounded evaluation were validated there; a +# single full-length 262K request was not tested upstream. +# +# DISK: ~186 GB against a default 150 GB /data. Raise WEIGHTS_SIZE_GB (e.g. 300) and re-run +# bin/spin persistent-init before the first pull, or the download dies with "No space left on +# device". +# +# An NVFP4 build exists (nvidia/Qwen3.8-Flash-Next-NVFP4, ~133 GB) and is not offered here: the +# recipe covers only FP8 and BF16, the dedicated image has no published NVFP4 run for this +# architecture, and NVFP4 MoE would be B200-only anyway. +model_repo: "Qwen/Qwen3.8-Flash-Next-FP8" +served_name: "qwen3.8-flash-next" +vllm_image_override: "vllm/vllm-openai:qwen38-flash-next" +tensor_parallel_size: 4 +max_model_len: 262144 +gpu_memory_utilization: 0.85 +dtype: "auto" +enable_lora: false +# CAP_SYS_PTRACE is not optional for this profile. VLLM_PLE_CPU_OFFLOAD spawns a PLE offload +# worker that hands CUDA tensors to the TP workers over CUDA IPC, and that calls pidfd_getfd(2). +# Docker's default seccomp profile gates that syscall behind CAP_SYS_PTRACE and returns EPERM +# without it, so engine init fails at worker registration with +# RuntimeError: pidfd_getfd: Operation not permitted (torch reductions.rebuild_cuda_tensor) +# after the checkpoint has already loaded — about 8 minutes into a 4-GPU start. Measured live +# 2026-09-20 on 4x RTX PRO 6000. Narrower than seccomp=unconfined or pid=host. +cap_add: + - SYS_PTRACE +extra_env: + VLLM_PLE_CPU_OFFLOAD: 1 +extra_vllm_args: + - "--moe-backend" + - "triton" + - "--max-num-seqs=256" + # Prefix caching stays on, reviewed 2026-09-22. This is a hybrid checkpoint, the class where a + # cache hit can answer fast and wrong (see docs/gpu-spinner.md, "Hybrid checkpoints and prefix + # caching"), but this + # profile's ceiling rungs are counter-evidence: 65,429 / 130,803 / 259,512 prompt_tokens all + # passed with the flag set, under the pre-2026-09-22 soak that reused one haystack per context, so + # they ran with cache hits and still retrieved correctly. A cache hit answers wrongly, so a pass + # cannot be an artefact of one. Re-test before changing it. + - "--enable-prefix-caching" + - "--no-enable-flashinfer-autotune" + - "--reasoning-parser" + - "qwen3" + - "--tool-call-parser" + - "qwen3_coder" + - "--enable-auto-tool-choice" +min_plan: "GPU-64xCPU-320GB-4xRTXPRO6000" +min_vram_gb: 76 # per-GPU floor: ~43 GiB per rank at TP=4; keeps both 4xH100 and 4xRTXPRO6000 +min_disk_gb: 300 # disk size to provision: (weights_gb + 80) / 0.934; raised to this profile's stated floor +weights_gb: 173 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +validated: true # live run 2026-09-20 on 4x RTX PRO 6000 (spot) — see docs/validation.md +# MEASURED 2026-09-20 on GPU-SPOT-64xCPU-320GB-4xRTXPRO6000 (driver 595.58.03, cc 12.0): +# KV pool 3,640,845 tokens · 13.89x concurrency at 262,144 · ~85,340/97,887 MiB per card +# TRITON Fp8 MoE · model load 32.72 GiB/rank in ~29 s · engine init 110.7 s +# soak 3x20 @32768: factual 6/6, 60/60 needles, 0 wrong/rejected/timed-out, p50 20.8 s +# ceiling: 65,429 / 130,803 / 259,512 prompt_tokens all PASS — 262144 is measured, not the recipe. +# NCCL over PCIe came up first try on this 4-GPU plan; NCCL_P2P_DISABLE was not needed. # newly added — no live run on this account yet +untested_compute_capabilities: [12.0] # RTX PRO 6000 is now the default lane; nothing measured there +health_wait_retries: 200 # x10s = 33 min — ~186 GB download, TP=4 init, N-gram table offload diff --git a/ansible/models/qwen38-nvfp4.yml b/ansible/models/qwen38-nvfp4.yml new file mode 100644 index 0000000..8875cfa --- /dev/null +++ b/ansible/models/qwen38-nvfp4.yml @@ -0,0 +1,67 @@ +--- +# Qwen3.8-27B in NVFP4 — Blackwell-class GPUs (RTX PRO 6000 or B200). ~22 GB weights. Same dense +# hybrid backbone as qwen38.yml (64 layers, 48 of them linear attention), native 262,144 context. +# Quantization is auto-detected (no --quantization flag). +# +# CHECKPOINT CHOICE. The other NVFP4 profiles here use nvidia/ ModelOpt builds; this one uses +# unsloth/, the build the vLLM recipe verified on consumer Blackwell (sm120) and the one that leaves +# most room for KV. Measured at 262,144 context on RTX 5090 (TP2), from +# recipes.vllm.ai/Qwen/Qwen3.8-27B: +# build KV tokens weights/GPU MTP acceptance +# Qwen FP8 377,456 14.28 GiB 0.771 +# Inferact NVFP4 445,875 12.02 GiB 0.897 +# unsloth NVFP4 920,517 10.64 GiB 0.788 +# unsloth is mixed-precision (FP8 channel-wise alongside the 4-bit groups), hence roughly double the +# KV pool; the uniform-W4A4 Inferact build drafts better. Both serve correctly. +# nvidia/Qwen3.8-27B-NVFP4 (ModelOpt v0.48.0, mixed NVFP4/FP8) exists but has no published sm120 +# run — swap model_repo to stay on the ModelOpt line. +# +# RUNS ON RTX PRO 6000, unlike the NVFP4 *MoE* profiles here, because it is dense: vLLM selects +# FlashInferCutlassNvFp4LinearKernel on sm120, a real CUTLASS kernel rather than an emulation +# fallback, and the grouped block-scaled GEMM that breaks NVFP4 MoE on cc 12.0 is not on this path. +# vLLM also auto-disables DeepGemm for model_type qwen3_5_text on Blackwell, so no VLLM_USE_DEEP_GEMM=0 +# workaround is needed. +# +# No --enforce-eager: the recipe needed it only on a single 31.4 GiB RTX 5090, where ~22 GB of +# weights left nothing for CUDA graph capture. A 96 GB RTX PRO 6000 or 179 GB B200 has room. Add it +# back for a smaller card — capture allocates outside the --gpu-memory-utilization budget and OOMs +# there however that fraction is set. +# +# --reasoning-parser qwen3 is not optional (the chat template opens every turn with ). +# --tool-call-parser is qwen3_xml for this generation. --language-model-only drops the vision tower. +# +# MTP: the checkpoint carries a Multi-Token Prediction head (measured acceptance 0.788 here). Left +# off so the first validation run tests the model alone. Enable with: +# extra_vllm_args += ["--speculative-config", '{"method":"mtp","num_speculative_tokens":3}'] +# Unsloth suggests 2 speculative tokens for NVFP4; the recipe measured 3. Read +# vllm:spec_decode_num_{accepted,draft}_tokens_total to confirm the drafter is used. +model_repo: "unsloth/Qwen3.8-27B-NVFP4" +served_name: "qwen3.8-27b-nvfp4" +tensor_parallel_size: 1 +max_model_len: 262144 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + - "--language-model-only" + - "--reasoning-parser" + - "qwen3" + - "--tool-call-parser" + - "qwen3_xml" + - "--enable-auto-tool-choice" + - "--kv-cache-dtype" + - "fp8" +min_plan: "GPU-16xCPU-80GB-1xRTXPRO6000" +min_vram_gb: 40 # ~22 GB weights + KV; keeps an L4/L40S-sized card out +min_disk_gb: 110 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 22 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +min_compute_capability: 10.0 # NVFP4 needs Blackwell: B200 (cc 10.0) or RTX PRO 6000 (cc 12.0) +validated: true # live run 2026-09-20 on 1x RTX PRO 6000 (spot) — see docs/validation.md +# MEASURED 2026-09-20 on GPU-SPOT-16xCPU-80GB-1xRTXPRO6000 (driver 595.58.03, cc 12.0): +# KV pool 1,902,460 tokens · 7.26x concurrency at 262,144 · 88,087/97,887 MiB +# FLASHINFER attention backend · CUDA graphs PIECEWISE+FULL in 9 s (1.55 GiB) · engine init 123.9 s +# soak 3x20 @32768: factual 6/6, traps 2/2 declined, 60/60 needles, 0 wrong/rejected/timed-out +# ceiling: 65,429 / 130,803 / 259,512 prompt_tokens all PASS, container restart count unchanged. +# max_model_len 262144 is therefore MEASURED, not inherited from the recipe. The dense NVFP4 path +# takes a real CUTLASS kernel on sm120 as this profile predicted — no sign of vllm#54189 (which +# would show as one token repeated), and none expected: that issue closed 2026-08-30. # newly added — no live run on this account yet diff --git a/ansible/models/qwen38.yml b/ansible/models/qwen38.yml new file mode 100644 index 0000000..55ec6b1 --- /dev/null +++ b/ansible/models/qwen38.yml @@ -0,0 +1,73 @@ +--- +# Qwen3.8-27B in FP8 (official Qwen block-scaled build, ~31 GB weights, Apache-2.0). +# Dense 27B on the Qwen3.5 hybrid backbone: 64 layers as 16 x (3 x Gated DeltaNet -> FFN, then +# 1 x Gated Attention -> FFN), so only 16 run full attention (full_attention_interval 4). Native +# 262,144 context, 1M with YaRN. Multimodal (image + video) and thinking-capable. Quantization is +# auto-detected from the checkpoint config (no --quantization flag). +# +# --reasoning-parser qwen3 is not optional: the chat template opens every assistant turn with +# , so without it the reasoning block lands in message.content and a short max_tokens budget +# is spent before the answer starts (recipes.vllm.ai/Qwen/Qwen3.8-27B). +# --tool-call-parser is qwen3_xml for this generation, not the qwen3_coder qwen36-35b uses. +# --language-model-only drops the vision tower and measurably enlarges the KV pool (recipe: +# 91,022 -> 135,926 tokens at 32K on a 31.4 GiB card). +# --enforce-eager is here for the same reason as on qwen36: on a 48 GB L40S, CUDA graph capture +# allocates outside the --gpu-memory-utilization budget, so a tight fit dies in capture however that +# fraction is set. Drop it on a 96 GB RTX PRO 6000 or a B200. +# +# max_model_len 131072 against a native 262,144 is a measured choice, not a cautious one — the +# MEASURED block at the foot of this file has both settings on the L40S lane. For context, FP8 +# weights are ~31 GB against an L40S's ~46 GiB usable, between qwen36 (27 GB, serves 262144 here) +# and gemma4-31b (33 GB, capped to 65536). +# +# MTP: the checkpoint carries a Multi-Token Prediction head that works in every precision (measured +# acceptance 0.771 for FP8). Left off so the first validation run tests the model alone — a failure +# with speculation on does not tell you which half broke. Enable with: +# extra_vllm_args += ["--speculative-config", '{"method":"mtp","num_speculative_tokens":3}'] +# and read vllm:spec_decode_num_{accepted,draft}_tokens_total to confirm the drafter is used. +# +# Sampling (model card, for callers — not set here): thinking mode temperature 1.0, top_p 0.95, +# top_k 20; non-thinking 0.7 / 0.80 / 20 with presence_penalty 1.5. Reasoning depth is per request +# via chat_template_kwargs {"reasoning_effort": "xhigh|medium|low"}. +model_repo: "Qwen/Qwen3.8-27B-FP8" +served_name: "qwen3.8-27b-fp8" +tensor_parallel_size: 1 +max_model_len: 131072 +gpu_memory_utilization: 0.90 +dtype: "auto" +enable_lora: false +extra_vllm_args: + - "--language-model-only" + - "--reasoning-parser" + - "qwen3" + - "--tool-call-parser" + - "qwen3_xml" + - "--enable-auto-tool-choice" + - "--enable-prefix-caching" + - "--enforce-eager" + - "--max-num-batched-tokens=2048" + - "--max-num-seqs=16" + - "--kv-cache-dtype" + - "fp8" +min_plan: "GPU-8xCPU-64GB-1xL40S" +min_vram_gb: 40 # ~31 GB weights + KV; an L4's 24 GB cannot hold it +min_disk_gb: 120 # disk size to provision: (weights_gb + 80) / 0.934 +weights_gb: 29 # measured 2026-09-20 from the HF file listing (sum of .siblings[].size) +validated: true # live runs 2026-09-20 (RTX PRO 6000) and 2026-09-21 (L40S, its own lane) +# MEASURED 2026-09-21 on GPU-SPOT-8xCPU-64GB-1xL40S (46,068 MiB, cc 8.9, driver 595.58.03), the +# lane min_plan names: +# KV pool 359,719 tokens · 2.74x concurrency at 131,072 · 41,023/46,068 MiB +# load 27.57 GiB in 72.9 s · engine init 51.2 s +# soak 3x20 @32768: factual 6/6, traps 2/2, 60/60 needles, 0 wrong · p50 73.2 s +# ceiling: 128,766 prompt_tokens passes — the full deployed context serves. +# +# WHY max_model_len STAYS 131072. Deployed at the native 262144 on the same card: KV pool 372,123, +# concurrency 1.42x, one 259,512-token request passes in 173.8 s, and three concurrent 260k requests +# all returned the correct needle with none rejected — vLLM queued a 2.1x oversubscribed pool rather +# than refusing. The slowest of those three waited 9.8 minutes, so the full window costs latency +# rather than correctness, and turns a 3-user box into a 1-user box. Take it deliberately for +# single-user or batch work: +# bin/spin up --model qwen38 --plan GPU-8xCPU-64GB-1xL40S -e max_model_len=262144 +# The 2026-09-20 run on a 96 GiB RTX PRO 6000 (KV 1,743,257, 13.30x) has 4.8x more KV and cannot be +# used to size this lane. +health_wait_retries: 120 # x10s = 20 min — ~31 GB download plus a 64-layer hybrid load diff --git a/ansible/prefetch.yml b/ansible/prefetch.yml new file mode 100644 index 0000000..3123cb8 --- /dev/null +++ b/ansible/prefetch.yml @@ -0,0 +1,206 @@ +--- +# Fill the persistent weights disk from a cheap CPU box, before any GPU exists. +# +# Why: a weights download is billed at whatever compute is attached to the disk, and it is the +# largest single line item in a validation campaign. 354 GB at ~126 MB/s is ~47 minutes: 5.17 EUR +# on 4x RTX PRO 6000 (6.60/h), or 0.035 EUR on 2xCPU-4GB (0.0446/h). Nothing about a download is +# GPU work. The GPU box then starts warm and spends its expensive minutes on inference. +# +# Run by `bin/spin prefetch ...`, which provisions terraform/prefetch, runs this, and +# destroys the box. An UpCloud storage device attaches to one server at a time, so this can never +# run alongside a GPU server — bin/spin refuses rather than letting tofu discover it. +- name: Prefetch model weights and container images onto the persistent disk + hosts: prefetch + become: true + gather_facts: true + vars: + # No GPU here; the docker role's nvidia-ctk assert would fail on an otherwise healthy box. + docker_require_nvidia: false + prefetch_models: [] + tasks: + - name: Require a list of profiles to fetch + ansible.builtin.assert: + that: (prefetch_models | default([])) | length > 0 + fail_msg: "Nothing to prefetch. Pass -e '{\"prefetch_models\":[\"qwen38\",...]}' (bin/spin prefetch does this)." + + - name: Mount and grow the persistent weights disk + ansible.builtin.import_tasks: roles/gpu/tasks/mount_data.yml + + # Image prefetch is optional and best-effort. The docker role is verification-first — it + # assumes the UpCloud GPU template's preinstalled Docker — and this box runs plain Ubuntu, so + # Docker is usually absent. Installing a second Docker here is not worth it: it would write the + # shared /data data-root with a different build from the one the GPU box runs, and a data-root + # the GPU box cannot read costs far more than the ~EUR 0.65 of image pulls it would save. + # Weights are 560 GB and the whole point; images are ~74 GB and pull fine at GPU rate. + - name: Check whether Docker is available (image prefetch is optional) + ansible.builtin.command: docker info + register: pf_docker + changed_when: false + failed_when: false + + - name: Set up Docker's data-root on the persistent disk (only if Docker is already present) + ansible.builtin.include_role: + name: docker + when: pf_docker.rc == 0 + + - name: Note that images are not being prefetched + ansible.builtin.debug: + msg: >- + No Docker on this box, so only weights are prefetched. The GPU box + pulls its own image on first deploy (~5-10 min, a fraction of the weight download this + run is saving). To prefetch images too, give terraform/prefetch an os_template that ships + Docker. + when: pf_docker.rc != 0 + + - name: Load each requested profile + ansible.builtin.set_fact: + _pf: "{{ (_pf | default([])) + [lookup('ansible.builtin.file', playbook_dir ~ '/models/' ~ item ~ '.yml') | from_yaml | combine({'_name': item})] }}" + loop: "{{ prefetch_models }}" + loop_control: + label: "{{ item }}" + + - name: Report what will be fetched + ansible.builtin.debug: + msg: >- + {{ _pf | length }} profile(s): + {% for p in _pf %}{{ p._name }} ({{ p.model_repo }}, needs ~{{ p.min_disk_gb | default('?') }} GB){% if not loop.last %}, {% endif %}{% endfor %} + + # `du -L` follows huggingface_hub's symlinks into its shared blob store and dedups by inode. + # Plain `du` and `find -L` both get it wrong — see the same measurement in + # roles/vllm/tasks/main.yml. + # `du -L` follows the shared blob store and dedups by inode, so each figure is what that model + # already occupies rather than what the disk happens to hold. + # + # The question is how much still has to be downloaded, plus 80 GB of headroom for the container + # image and Docker's data-root — not how big the disk is. + # + # Count per model, not against the whole cache: a disk holding something you are not fetching + # otherwise reads as free space you do not have. Measured 2026-09-21 — a 500 GB disk with 364 GB + # of k2-horizon cached had 102 GB free, and a 181 GB model needs 261 GB; whole-cache accounting + # would have compared 102 + 364 against 261, passed, and died at ~102 GB. + - name: Measure free space and what still has to be downloaded + ansible.builtin.shell: | + set -o pipefail + avail=$(df -BG --output=avail {{ data_root | quote }} | tail -1 | tr -dc '0-9') + still=0 + {% for m in _pf %} + have=$(du -sBG -L {{ (hf_cache_dir ~ '/hub/models--' ~ (m.model_repo | replace('/', '--'))) | quote }} \ + 2>/dev/null | cut -f1 | tr -dc '0-9'); have=${have:-0} + want={{ m.weights_gb | int }} + [ "$have" -lt "$want" ] && still=$(( still + want - have )) + {% endfor %} + printf '%s %s' "$avail" "$still" + args: + executable: /bin/bash + register: pf_space + changed_when: false + + - name: Preflight — the disk has room for what still has to be downloaded + vars: + _free: "{{ (pf_space.stdout.split() | first) | int }}" + _still: "{{ (pf_space.stdout.split() | last) | int }}" + _need: "{{ (_still | int) + 80 }}" + ansible.builtin.assert: + that: + - (_free | int) >= (_need | int) + fail_msg: >- + {{ _free }} GB free on {{ data_root }}, but {{ _pf | length }} requested profile(s) still + need {{ _still }} GB of weights downloaded, plus 80 GB of headroom = {{ _need }} GB. + Grow the disk (WEIGHTS_SIZE_GB={{ ((_need | int) + (_free | int) * 0 + 40) }} or more, + then bin/spin persistent-init — it can grow but never shrink), fetch fewer profiles, or + free space: du -sh -L {{ data_root }}/hf-cache/hub/models--* then rm -rf one you no + longer need. + success_msg: >- + {{ _free }} GB free; {{ _still }} GB still to download for {{ _pf | length }} profile(s) + (+80 GB headroom = {{ _need }} GB needed) + + # Plain Ubuntu has no pip. The GPU template does, which is why the rest of this repo never + # needed it. + - name: Install pip (absent on a stock Ubuntu image) + ansible.builtin.apt: + name: python3-pip + state: present + update_cache: true + cache_valid_time: 3600 + retries: 3 + delay: 10 + + # The Xet downloader buffers chunks in RAM and was OOM-killed repeatedly on a 4 GB box with no + # swap (measured 2026-09-20: ~3.5 GB RSS, five of ten profiles killed with rc -9). Swap is the + # safety net under the bigger plan: a spike stalls instead of losing a half-finished + # multi-hundred-GB download. + - name: Create a swap file (the download is memory-hungry and the box has none) + ansible.builtin.shell: | + set -e + fallocate -l 8G /swapfile + chmod 600 /swapfile + mkswap /swapfile + swapon /swapfile + args: + executable: /bin/bash + creates: /swapfile + + - name: Install the Hugging Face CLI (with the fast transfer backend) + ansible.builtin.pip: + name: + - "huggingface_hub[cli,hf_transfer]" + state: present + extra_args: --break-system-packages + retries: 3 + delay: 10 + + # hf_transfer is what turns a ~50 MB/s single-stream pull into the ~126 MB/s this repo's + # timings assume. HF_TOKEN is not required (every current repo is ungated) but avoids the + # anonymous throttle, which drops to ~50 KB/s after a few hundred GB in a day. + - name: Download each profile's weights to the shared HF cache + ansible.builtin.command: + cmd: "hf download {{ item.model_repo }} --quiet" + environment: + HF_HOME: "{{ hf_cache_dir }}" + HF_TOKEN: "{{ hf_token | default('') }}" + # Xet replaced hf_transfer in huggingface_hub 1.x, which now ignores + # HF_HUB_ENABLE_HF_TRANSFER and warns about it. Xet's concurrency drives memory, so cap it: + # the default of 16 concurrent range GETs exhausted a 4 GB box. + HF_XET_NUM_CONCURRENT_RANGE_GETS: "8" + loop: "{{ _pf }}" + loop_control: + label: "{{ item._name }} -> {{ item.model_repo }}" + register: pf_dl + changed_when: true + async: 21600 # 6 h: deepseek-v4-pro is ~831 GB + poll: 30 + + # The container image is 9-14 GB and is pulled at GPU rate today. Docker's data-root is on + # /data, so pulling here means the GPU box starts it from local disk. + # Plain `docker pull`, not community.docker: that collection is not in requirements.yml and + # this needs nothing it offers. `docker pull` is already idempotent — a present image reports + # "Image is up to date". + - name: Pull each profile's container image onto the shared Docker data-root + ansible.builtin.command: + cmd: "docker pull {{ item.vllm_image_override | default(vllm_image) }}" + loop: "{{ _pf }}" + loop_control: + label: "{{ item.vllm_image_override | default(vllm_image) }}" + register: pf_pull + changed_when: "'Image is up to date' not in pf_pull.stdout" + async: 3600 + poll: 15 + when: pf_docker.rc == 0 + + - name: Report what is now cached, and what is left + ansible.builtin.shell: | + set -o pipefail + echo "--- {{ data_root }} ---"; df -h {{ data_root }} | tail -1 + echo "--- cached models ---"; du -sh {{ hf_cache_dir }}/hub/models--* 2>/dev/null || echo "(none)" + echo "--- docker images ---" + command -v docker >/dev/null \ + && docker images --format '{{ "{{.Repository}}:{{.Tag}} {{.Size}}" }}' \ + || echo "(docker not installed — images not prefetched)" + args: + executable: /bin/bash + register: pf_report + changed_when: false + + - name: Prefetch summary + ansible.builtin.debug: + msg: "{{ pf_report.stdout_lines }}" diff --git a/ansible/roles/docker/defaults/main.yml b/ansible/roles/docker/defaults/main.yml index 402bf91..dadc075 100644 --- a/ansible/roles/docker/defaults/main.yml +++ b/ansible/roles/docker/defaults/main.yml @@ -8,3 +8,7 @@ docker_gpu_smoketest: false # threshold (so it won't break a not-yet-resized disk). Needs the weights disk >= the threshold. docker_data_root_on_data: true docker_data_root_min_gb: 100 + +# The NVIDIA Container Toolkit check is mandatory on a GPU box and meaningless on the prefetch box +# (ansible/prefetch.yml), which only pulls images and weights onto the shared /data disk. +docker_require_nvidia: true diff --git a/ansible/roles/docker/tasks/main.yml b/ansible/roles/docker/tasks/main.yml index 38787b6..34f2791 100644 --- a/ansible/roles/docker/tasks/main.yml +++ b/ansible/roles/docker/tasks/main.yml @@ -12,6 +12,7 @@ register: nvidia_ctk changed_when: false failed_when: false + when: docker_require_nvidia | default(true) | bool - name: Require the NVIDIA Container Toolkit (GPU passthrough depends on it) ansible.builtin.assert: @@ -20,6 +21,7 @@ nvidia-ctk not found — GPU passthrough to containers will fail. The UpCloud GPU (NVIDIA+CUDA) template should ship it. Install it and run `nvidia-ctk runtime configure --runtime=docker && systemctl restart docker`. + when: docker_require_nvidia | default(true) | bool - name: Measure /data capacity (GB) for Docker image persistence ansible.builtin.shell: | diff --git a/ansible/roles/gpu/defaults/main.yml b/ansible/roles/gpu/defaults/main.yml index 487d152..0436626 100644 --- a/ansible/roles/gpu/defaults/main.yml +++ b/ansible/roles/gpu/defaults/main.yml @@ -1,5 +1,7 @@ --- -# Minimum NVIDIA driver major version required on Blackwell-class GPUs (B100/B200/GB200). -# Older GPUs (L4/L40S/H100) are not checked against this. The UpCloud GPU template ships a -# recent driver; this guard catches a stale template before it fails obscurely inside vLLM/CUDA. +# Minimum NVIDIA driver major for Blackwell-class GPUs, detected by compute capability (>= 10.0) +# rather than model name, so it covers B200 (cc 10.0) and RTX PRO 6000 (cc 12.0) without a name list +# to maintain. Older GPUs (L4/L40S cc 8.9, H100 cc 9.0) are not checked. The guard catches a stale +# UpCloud template before it fails obscurely inside vLLM/CUDA. gpu_min_driver_blackwell: 570 +gpu_blackwell_min_compute_cap: 10.0 diff --git a/ansible/roles/gpu/tasks/main.yml b/ansible/roles/gpu/tasks/main.yml index 21aa562..a524f04 100644 --- a/ansible/roles/gpu/tasks/main.yml +++ b/ansible/roles/gpu/tasks/main.yml @@ -12,95 +12,37 @@ ansible.builtin.debug: msg: "{{ gpu_nvidia_smi.stdout_lines | select('search', 'NVIDIA') | list }}" -- name: Query GPU model, driver version, and total VRAM (MiB) for preflight checks - ansible.builtin.command: nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader,nounits +# compute_cap is what the model-profile preflights gate on: VRAM alone cannot distinguish a +# 96 GB RTX PRO 6000 (cc 12.0, PCIe, no NVLink) from a 192 GB B200 (cc 10.0), and some kernels +# are built for one family and not the other. gpu_count drives the multi-GPU (tensor-parallel) +# checks — nvidia-smi prints one line per GPU. +- name: Query GPU model, driver version, total VRAM (MiB) and compute capability for preflight checks + ansible.builtin.command: nvidia-smi --query-gpu=name,driver_version,memory.total,compute_cap --format=csv,noheader,nounits register: gpu_query changed_when: false -- name: Record GPU facts (model, driver major, per-GPU VRAM in GB) +- name: Record GPU facts (model, driver major, per-GPU VRAM in GB, compute capability, count) ansible.builtin.set_fact: gpu_name: "{{ (gpu_query.stdout_lines[0] | split(','))[0] | trim }}" gpu_driver_major: "{{ ((gpu_query.stdout_lines[0] | split(','))[1] | trim | regex_search('[0-9]+')) | int }}" gpu_total_vram_gb: "{{ (((gpu_query.stdout_lines[0] | split(','))[2] | trim | int) // 1024) | int }}" + gpu_compute_cap: "{{ (gpu_query.stdout_lines[0] | split(','))[3] | trim | float }}" + gpu_count: "{{ gpu_query.stdout_lines | length }}" + +- name: Report the detected GPU topology + ansible.builtin.debug: + msg: >- + {{ gpu_count }}x {{ gpu_name }} — {{ gpu_total_vram_gb }} GB per GPU, + compute capability {{ gpu_compute_cap }}, driver {{ gpu_driver_major }} - name: Require a recent driver on Blackwell-class GPUs (CUDA/vLLM needs >= {{ gpu_min_driver_blackwell }}) ansible.builtin.assert: that: (gpu_driver_major | int) >= (gpu_min_driver_blackwell | int) fail_msg: >- - {{ gpu_name }} is Blackwell-class but the driver is {{ gpu_driver_major }} - (< {{ gpu_min_driver_blackwell }}). The UpCloud GPU template should ship a recent driver; - this one is too old for this GPU. - when: gpu_name is search('B100') or gpu_name is search('B200') or gpu_name is search('GB200') - -- name: Check whether the persistent weights filesystem already exists (re-attach case) - ansible.builtin.command: "blkid -L {{ weights_fs_label }}" - register: weights_blkid - changed_when: false - failed_when: false - -- name: Discover the raw data disk (only when no labelled FS exists yet) - ansible.builtin.shell: | - set -o pipefail - root_part="$(findmnt -no SOURCE /)" - root_pk="$(lsblk -no PKNAME "$root_part" 2>/dev/null | head -1)" - root_disk="" - [ -n "$root_pk" ] && root_disk="/dev/$root_pk" - # Whole disks with no filesystem signature (NF==1), excluding the OS disk. - cands="$(lsblk -dpno NAME,FSTYPE | awk -v root="$root_disk" '$1 != root && NF == 1 { print $1 }')" - n="$(printf '%s\n' "$cands" | grep -c .)" - if [ "$n" -eq 1 ]; then printf '%s' "$cands"; - elif [ "$n" -eq 0 ]; then printf ''; - else printf 'MULTIPLE:%s' "$(printf '%s' "$cands" | tr '\n' ',')"; fi - args: - executable: /bin/bash - register: weights_disk - changed_when: false - when: weights_blkid.rc != 0 - -- name: Refuse to format unless exactly one unformatted data disk is found - ansible.builtin.fail: - msg: >- - Could not safely identify the persistent data disk (got: - '{{ weights_disk.stdout | default('') | trim }}'). Expected exactly one - unformatted disk besides the OS disk; refusing to format. Inspect with `lsblk`. - when: - - weights_blkid.rc != 0 - - ((weights_disk.stdout | default('') | trim) == '') or ((weights_disk.stdout | default('')) is search('^MULTIPLE:')) - -- name: Create the ext4 filesystem on the data disk (first spin-up only; never reformats) - community.general.filesystem: - fstype: "{{ weights_fs_type }}" - dev: "{{ weights_disk.stdout | trim }}" - opts: "-L {{ weights_fs_label }}" - force: false - when: - - weights_blkid.rc != 0 - - (weights_disk.stdout | default('') | trim) != '' - - not ((weights_disk.stdout | default('')) is search('^MULTIPLE:')) - -- name: Mount the persistent data disk by label at {{ data_root }} - ansible.posix.mount: - path: "{{ data_root }}" - src: "LABEL={{ weights_fs_label }}" - fstype: "{{ weights_fs_type }}" - opts: defaults,nofail - state: mounted - -- name: Resolve the data disk device - ansible.builtin.command: "findmnt -no SOURCE {{ data_root }}" - register: data_device - changed_when: false - -- name: Grow the data filesystem to fill the disk (after a persistent-disk resize) - ansible.builtin.command: "resize2fs {{ data_device.stdout | trim }}" - register: data_resize - changed_when: "'Nothing to do' not in (data_resize.stdout + data_resize.stderr)" + {{ gpu_name }} is Blackwell-class (compute capability {{ gpu_compute_cap }}) but the driver + is {{ gpu_driver_major }} (< {{ gpu_min_driver_blackwell }}). The UpCloud GPU template should + ship a recent driver; this one is too old for this GPU. + when: (gpu_compute_cap | float) >= (gpu_blackwell_min_compute_cap | float) -- name: Ensure HF cache and Caddy data directories exist on the persistent disk - ansible.builtin.file: - path: "{{ item }}" - state: directory - mode: "0755" - loop: - - "{{ hf_cache_dir }}" - - "{{ caddy_data_dir }}" +- name: Mount and grow the persistent weights disk + ansible.builtin.import_tasks: mount_data.yml diff --git a/ansible/roles/gpu/tasks/mount_data.yml b/ansible/roles/gpu/tasks/mount_data.yml new file mode 100644 index 0000000..0c9c93e --- /dev/null +++ b/ansible/roles/gpu/tasks/mount_data.yml @@ -0,0 +1,81 @@ +--- +# Attach, format-if-virgin, mount and grow the persistent weights disk at {{ data_root }}. +# +# Split out of the gpu role's main.yml so the prefetch play (ansible/prefetch.yml) can mount the +# SAME disk on a cheap CPU box — there is no GPU there, so it cannot run the whole role, and a +# second copy of this logic would be a second chance to format the wrong device. +# +# Never reformats: it keys off the filesystem label and only creates a filesystem when exactly one +# unformatted non-OS disk is present. +- name: Check whether the persistent weights filesystem already exists (re-attach case) + ansible.builtin.command: "blkid -L {{ weights_fs_label }}" + register: weights_blkid + changed_when: false + failed_when: false + +- name: Discover the raw data disk (only when no labelled FS exists yet) + ansible.builtin.shell: | + set -o pipefail + root_part="$(findmnt -no SOURCE /)" + root_pk="$(lsblk -no PKNAME "$root_part" 2>/dev/null | head -1)" + root_disk="" + [ -n "$root_pk" ] && root_disk="/dev/$root_pk" + # Whole disks with no filesystem signature (NF==1), excluding the OS disk. + cands="$(lsblk -dpno NAME,FSTYPE | awk -v root="$root_disk" '$1 != root && NF == 1 { print $1 }')" + n="$(printf '%s\n' "$cands" | grep -c .)" + if [ "$n" -eq 1 ]; then printf '%s' "$cands"; + elif [ "$n" -eq 0 ]; then printf ''; + else printf 'MULTIPLE:%s' "$(printf '%s' "$cands" | tr '\n' ',')"; fi + args: + executable: /bin/bash + register: weights_disk + changed_when: false + when: weights_blkid.rc != 0 + +- name: Refuse to format unless exactly one unformatted data disk is found + ansible.builtin.fail: + msg: >- + Could not safely identify the persistent data disk (got: + '{{ weights_disk.stdout | default('') | trim }}'). Expected exactly one + unformatted disk besides the OS disk; refusing to format. Inspect with `lsblk`. + when: + - weights_blkid.rc != 0 + - ((weights_disk.stdout | default('') | trim) == '') or ((weights_disk.stdout | default('')) is search('^MULTIPLE:')) + +- name: Create the ext4 filesystem on the data disk (first spin-up only; never reformats) + community.general.filesystem: + fstype: "{{ weights_fs_type }}" + dev: "{{ weights_disk.stdout | trim }}" + opts: "-L {{ weights_fs_label }}" + force: false + when: + - weights_blkid.rc != 0 + - (weights_disk.stdout | default('') | trim) != '' + - not ((weights_disk.stdout | default('')) is search('^MULTIPLE:')) + +- name: Mount the persistent data disk by label at {{ data_root }} + ansible.posix.mount: + path: "{{ data_root }}" + src: "LABEL={{ weights_fs_label }}" + fstype: "{{ weights_fs_type }}" + opts: defaults,nofail + state: mounted + +- name: Resolve the data disk device + ansible.builtin.command: "findmnt -no SOURCE {{ data_root }}" + register: data_device + changed_when: false + +- name: Grow the data filesystem to fill the disk (after a persistent-disk resize) + ansible.builtin.command: "resize2fs {{ data_device.stdout | trim }}" + register: data_resize + changed_when: "'Nothing to do' not in (data_resize.stdout + data_resize.stderr)" + +- name: Ensure HF cache and Caddy data directories exist on the persistent disk + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ hf_cache_dir }}" + - "{{ caddy_data_dir }}" diff --git a/ansible/roles/llama_swap/tasks/main.yml b/ansible/roles/llama_swap/tasks/main.yml index 0a0a5fa..16a1f61 100644 --- a/ansible/roles/llama_swap/tasks/main.yml +++ b/ansible/roles/llama_swap/tasks/main.yml @@ -15,16 +15,10 @@ multi_model_enabled=true but no models resolved. Use --swap-profile (see: bin/spin swap-profiles), or pass swap_profiles=[] / a swap_models list. -- name: Validate each swap_models entry has the required keys - ansible.builtin.assert: - that: - - item.name is defined and (item.name | string | length) > 0 - - item.model_repo is defined and (item.model_repo | string | length) > 0 - - item.max_model_len is defined - fail_msg: "swap_models entry is missing name/model_repo/max_model_len: {{ item }}" - loop: "{{ swap_models | default([]) }}" - loop_control: - label: "{{ item.name | default('') }}" +# Per-entry preflights (hardware gates, and shapes llama-swap cannot faithfully express). +# In their own file so tests/swap-refusals.yml can import and prove them without a GPU. +- name: Preflight every entry in the preset + ansible.builtin.import_tasks: refusals.yml - name: Download the llama-swap release tarball (checksum-verified) ansible.builtin.get_url: @@ -59,10 +53,12 @@ force: true notify: Restart gpu-swap -- name: Pre-pull the vLLM image (so the first model request isn't a cold ~30 GB pull) +# Every distinct image across the preset — a profile may pin its own (vllm_image_override). +- name: Pre-pull the vLLM image(s) (so the first model request isn't a cold ~30 GB pull) ansible.builtin.command: - cmd: "docker pull {{ vllm_image }}" + cmd: "docker pull {{ item }}" changed_when: false + loop: "{{ (swap_models | default([]) | map(attribute='image') | list + [vllm_image]) | unique }}" - name: Require the rendered compose .env (the vllm role writes it; supplies VLLM_API_KEY) ansible.builtin.stat: diff --git a/ansible/roles/llama_swap/tasks/refusals.yml b/ansible/roles/llama_swap/tasks/refusals.yml new file mode 100644 index 0000000..a1b48eb --- /dev/null +++ b/ansible/roles/llama_swap/tasks/refusals.yml @@ -0,0 +1,116 @@ +--- +# Per-entry preflights for a multi-model swap preset. Split out of main.yml so tests/swap-refusals.yml +# exercises the same asserts the role runs; a copy inside the test would drift from the role. +# +# Imported by roles/llama_swap/tasks/main.yml after resolve_models.yml has built `swap_models`. +- name: Validate each swap_models entry has the required keys + ansible.builtin.assert: + that: + - item.name is defined and (item.name | string | length) > 0 + - item.model_repo is defined and (item.model_repo | string | length) > 0 + - item.max_model_len is defined + fail_msg: "swap_models entry is missing name/model_repo/max_model_len: {{ item }}" + loop: "{{ swap_models | default([]) }}" + loop_control: + label: "{{ item.name | default('') }}" + +# A preset is a menu of profiles, and llama-swap starts each one on this GPU. Single-model mode +# preflights its one profile in the vllm role; without the same check here a preset could include a +# model this GPU cannot serve, and the failure would only surface on the first request to it. +- name: Preflight — every model in the preset can run on this GPU + ansible.builtin.assert: + that: + - (gpu_total_vram_gb is not defined) or + ((gpu_total_vram_gb | int) >= (item.min_vram_gb | default(0) | int)) + - (gpu_compute_cap is not defined) or + ((gpu_compute_cap | float) >= (item.min_compute_capability | default(0) | float)) + - (gpu_compute_cap is not defined) or + ((gpu_compute_cap | float) not in + (item.unsupported_compute_capabilities | default([]) | map('float') | list)) + - (gpu_count is not defined) or + ((gpu_count | int) >= (item.tensor_parallel_size | default(1) | int)) + fail_msg: >- + Swap preset entry '{{ item.name }}' cannot run on this server + ({{ gpu_count | default('?') }}x {{ gpu_name | default('?') }}, + {{ gpu_total_vram_gb | default('?') }} GB per GPU, compute capability + {{ gpu_compute_cap | default('?') }}). It needs >= {{ item.min_vram_gb | default(0) }} GB, + compute capability >= {{ item.min_compute_capability | default(0) }} + (excluding {{ item.unsupported_compute_capabilities | default([]) }}) and + {{ item.tensor_parallel_size | default(1) }} GPU(s). Pick a preset that matches the plan + (bin/spin swap-profiles), or drop that model from the preset. + loop: "{{ swap_models | default([]) }}" + loop_control: + label: "{{ item.name | default('') }}" + when: not (allow_unvalidated_model | default(false) | bool) + +# The llama-swap backend is a bare `docker run` (llama-swap.yaml.j2). It carries the image, repo, +# context, memory fraction, TP, dtype and extra_vllm_args — and nothing else. A profile that needs a +# per-model environment variable or an out-of-tree plugin overlay would start and serve, silently +# missing the thing that makes it correct. That is strictly worse than not deploying: the failure +# signature for a zeroed NVFP4 MoE (vllm-project/vllm#54189) is one token repeated, which reads as a +# bad model rather than a bad deployment. Refuse instead. +- name: Preflight — the preset contains no profile llama-swap cannot faithfully express + ansible.builtin.assert: + that: + - (item.extra_env | default({})) | length == 0 + - (item.vllm_plugins | default([])) | length == 0 + fail_msg: >- + Swap preset entry '{{ item.name }}' cannot be served faithfully by llama-swap: it declares + {{ 'extra_env ' ~ (item.extra_env | default({}) | list | join(', ')) if (item.extra_env | default({})) | length > 0 else '' }} + {{ 'vllm_plugins ' ~ (item.vllm_plugins | default([]) | map(attribute='name') | join(', ')) if (item.vllm_plugins | default([])) | length > 0 else '' }} + and the generated backend command carries neither. The model would load and answer, with the + setting silently absent. Deploy it single-model instead + (bin/spin up --model {{ item.name }}), or drop it from the preset. + loop: "{{ swap_models | default([]) }}" + loop_control: + label: "{{ item.name | default('') }}" + +# llama-swap does carry extra_vllm_args, so a profile that sets --no-enable-prefix-caching keeps it +# in the backend command. The hole is a profile that declares the architecture and forgets the flag: +# the entry serves, and on a long cached prefix it answers fast and wrong with no crash and nothing +# logged (vllm#56960; see docs/gpu-spinner.md, "Hybrid checkpoints and prefix caching"). +# tests/check-prefix-caching.sh catches that in CI, and this is the same assertion at deploy time, +# for a preset assembled outside CI. +- name: Preflight — no hybrid entry reaches the swap backend with prefix caching on + ansible.builtin.assert: + that: + - not (item.hybrid_recurrent_state | default(false) | bool) + or ('--no-enable-prefix-caching' in (item.args | default([]))) + - not (item.hybrid_recurrent_state | default(false) | bool) + or ('--enable-prefix-caching' not in (item.args | default([]))) + fail_msg: >- + Swap preset entry '{{ item.name }}' declares hybrid_recurrent_state (its attention carries + recurrent state) but does not pin prefix caching off. vLLM defaults enable_prefix_caching to + true, and a cache hit resumes those layers from a recycled block's state, so the model answers + confidently wrong on a long prompt — no crash, nothing logged (vllm#56960). Add + "--no-enable-prefix-caching" to that profile's extra_vllm_args, or drop it from the preset. + loop: "{{ swap_models | default([]) }}" + loop_control: + label: "{{ item.name | default('') }}" + +- name: Warn when a preset entry needs multiple GPUs (swap-in will be slow) + ansible.builtin.debug: + msg: >- + '{{ item.name }}' runs at tensor-parallel size {{ item.tensor_parallel_size }}. It is served + correctly, but every swap-in re-initialises {{ item.tensor_parallel_size }} ranks, so the + first request after an idle period waits minutes rather than seconds. Consider single-model + mode for this profile. + loop: "{{ swap_models | default([]) }}" + loop_control: + label: "{{ item.name | default('') }}" + when: (item.tensor_parallel_size | default(1) | int) > 1 + +- name: Warn about preset entries with no live run on this GPU yet + ansible.builtin.debug: + msg: >- + UNTESTED: '{{ item.name }}' has no recorded run on {{ gpu_name | default('this GPU') }} + (compute capability {{ gpu_compute_cap | default('?') }}). It should serve; the numbers in + docs/validation.md are from other hardware. + loop: "{{ swap_models | default([]) }}" + loop_control: + label: "{{ item.name | default('') }}" + when: + - gpu_compute_cap is defined + - (not (item.validated | default(true) | bool)) or + ((gpu_compute_cap | float) in + (item.untested_compute_capabilities | default([]) | map('float') | list)) diff --git a/ansible/roles/llama_swap/tasks/resolve_models.yml b/ansible/roles/llama_swap/tasks/resolve_models.yml index a628c3b..df89295 100644 --- a/ansible/roles/llama_swap/tasks/resolve_models.yml +++ b/ansible/roles/llama_swap/tasks/resolve_models.yml @@ -17,6 +17,30 @@ max_model_len: "{{ _profile.max_model_len }}" gpu_memory_utilization: "{{ _profile.gpu_memory_utilization | default(gpu_memory_utilization | default(0.90)) }}" args: "{{ _profile.extra_vllm_args | default([]) }}" + image: "{{ _profile.vllm_image_override | default(vllm_image) }}" + # Hardware gates, carried so multi-model mode can preflight each entry the same way + # single-model mode preflights its one profile (see the llama_swap role). + min_vram_gb: "{{ _profile.min_vram_gb | default(0) }}" + min_compute_capability: "{{ _profile.min_compute_capability | default(0) }}" + unsupported_compute_capabilities: "{{ _profile.unsupported_compute_capabilities | default([]) }}" + untested_compute_capabilities: "{{ _profile.untested_compute_capabilities | default([]) }}" + validated: "{{ _profile.validated | default(true) }}" + tensor_parallel_size: "{{ _profile.tensor_parallel_size | default(1) }}" + dtype: "{{ _profile.dtype | default('auto') }}" + health_wait_retries: "{{ _profile.health_wait_retries | default(90) }}" + # Carried so the role can REFUSE them: the llama-swap backend is a bare `docker run` with no + # per-model environment and no plugin mounts (see llama-swap.yaml.j2), so a profile relying on + # either would start and serve wrong output rather than fail. glm53-flash-nvfp4 is the live + # example — without VLLM_GLM53_MOE_INPUT_SCALE every expert output is multiplied by zero. + extra_env: "{{ _profile.extra_env | default({}) }}" + vllm_plugins: "{{ _profile.vllm_plugins | default([]) }}" + # Carried so refusals.yml can check it: a hybrid checkpoint must pin prefix caching off + # (vllm#56960). The flag itself travels in `args` above; this key is what makes it mandatory. + hybrid_recurrent_state: "{{ _profile.hybrid_recurrent_state | default(false) }}" + # NOT carried, by design: `served_name`. In swap mode the client-facing name is the profile + # FILENAME (llama-swap.yaml.j2 passes `--served-model-name `), because that is the key + # a preset lists and the string a client puts in the OpenAI `model` field. So `qwen36` serves + # as "qwen36" here and as "qwen3.6-27b-fp8" in single-model mode. /v1/models is authoritative. loop: "{{ swap_profiles | default([]) }}" loop_control: label: "{{ item }}" diff --git a/ansible/roles/llama_swap/templates/llama-swap.yaml.j2 b/ansible/roles/llama_swap/templates/llama-swap.yaml.j2 index 2121b5b..a78166b 100644 --- a/ansible/roles/llama_swap/templates/llama-swap.yaml.j2 +++ b/ansible/roles/llama_swap/templates/llama-swap.yaml.j2 @@ -3,7 +3,14 @@ # {{ compose_dir }}/.env, which sets VLLM_API_KEY), so it never appears in the backend's argv / # `docker inspect`. Clients pick a model via the OpenAI "model" field; an idle backend is unloaded # after `ttl` seconds. -healthCheckTimeout: 600 +{# A literal 600 s was shorter than most profiles' own health_wait_retries (x10 s), so llama-swap + gave up on a backend that was still loading and reported it as failed. Derive it from the + slowest entry, with a 600 s floor. #} +{% set _hct = [600] %} +{%- for _m in swap_models | default([]) %} +{%- set _ = _hct.append((_m.health_wait_retries | default(90) | int) * 10) %} +{%- endfor %} +healthCheckTimeout: {{ _hct | max }} models: {% for m in swap_models %} {#- Shell-quote each vLLM arg (map('quote')) so JSON-ish args like --limit-mm-per-prompt '{"image":0}' @@ -11,7 +18,8 @@ models: valid YAML (a bare double-quoted scalar would break on the '"' in the JSON arg). ${PORT} is a llama-swap macro and is left literal for llama-swap to expand. #} {% set _args = (m.args | default([]) | map('quote') | join(' ')) %} -{% set _cmd = "docker run --rm --name vllm-swap-" ~ m.name ~ " --network host --gpus all --ipc=host --env-file " ~ compose_dir ~ "/.env -v " ~ hf_cache_dir ~ ":/root/.cache/huggingface -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e HF_HOME=/root/.cache/huggingface " ~ vllm_image ~ " " ~ m.model_repo ~ " --host 127.0.0.1 --port ${PORT} --served-model-name " ~ m.name ~ " --max-model-len " ~ (m.max_model_len | string) ~ " --gpu-memory-utilization " ~ (m.gpu_memory_utilization | default(gpu_memory_utilization | default(0.90)) | string) ~ ((" " ~ _args) if (_args | length > 0) else "") %} +{% set _image = m.image | default(vllm_image) %} +{% set _cmd = "docker run --rm --name vllm-swap-" ~ m.name ~ " --network host --gpus all --ipc=host --env-file " ~ compose_dir ~ "/.env -v " ~ hf_cache_dir ~ ":/root/.cache/huggingface -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e HF_HOME=/root/.cache/huggingface " ~ _image ~ " " ~ m.model_repo ~ " --host 127.0.0.1 --port ${PORT} --served-model-name " ~ m.name ~ " --max-model-len " ~ (m.max_model_len | string) ~ " --gpu-memory-utilization " ~ (m.gpu_memory_utilization | default(gpu_memory_utilization | default(0.90)) | string) ~ " --tensor-parallel-size " ~ (m.tensor_parallel_size | default(1) | string) ~ " --dtype " ~ (m.dtype | default('auto') | string) ~ ((" " ~ _args) if (_args | length > 0) else "") %} {{ m.name | to_json }}: # --network host is REQUIRED on UpCloud: containers on the DEFAULT docker bridge have no working # DNS (they get the UpCloud resolvers in resolv.conf but can't reach them from the bridge), so diff --git a/ansible/roles/vllm/defaults/main.yml b/ansible/roles/vllm/defaults/main.yml index d0dedf0..617094a 100644 --- a/ansible/roles/vllm/defaults/main.yml +++ b/ansible/roles/vllm/defaults/main.yml @@ -1,5 +1,6 @@ --- -# Override with -e if you really intend to run a profile flagged requires_review. +# Override with -e to bypass the HARDWARE preflights (VRAM, compute capability, known-bad GPU +# list, driver, disk). Cost is gated separately by bin/spin via MAX_EUR_PER_HOUR. allow_unvalidated_model: false # LoRA serving (enable per-model in the model profile). Adapters are picked at request time by @@ -8,3 +9,9 @@ enable_lora: false max_lora_rank: 16 max_loras: 4 lora_modules: [] # e.g. [{ name: "my-adapter", path: "org/my-lora-hf-repo" }] + +# Out-of-tree vLLM plugins (see tasks/plugins.yml). A profile sets vllm_plugins when its GPU needs a +# kernel no released image carries; each entry is built once per pinned commit under this root, on +# the persistent disk, and mounted read-only into the container. +vllm_plugins: [] +vllm_plugin_root: "{{ data_root }}/ext" diff --git a/ansible/roles/vllm/tasks/main.yml b/ansible/roles/vllm/tasks/main.yml index 0c1996a..762762e 100644 --- a/ansible/roles/vllm/tasks/main.yml +++ b/ansible/roles/vllm/tasks/main.yml @@ -14,14 +14,11 @@ - name: Load the selected model profile ansible.builtin.include_vars: "{{ playbook_dir }}/models/{{ vllm_model }}.yml" -- name: Guard against an unvalidated (too-large) model profile - ansible.builtin.assert: - that: not (requires_review | default(false) | bool) or (allow_unvalidated_model | default(false) | bool) - fail_msg: >- - Model profile '{{ vllm_model }}' is marked requires_review — its floor is - {{ min_plan | default('a multi-GPU plan') }}, larger than the default - GPU-8xCPU-64GB-1xL40S. Choose an appropriate --plan and re-run with - --allow-unvalidated (or -e allow_unvalidated_model=true). +# NOTE: the `requires_review` gate that used to live here was retired 2026-09-21. It meant "this +# needs an expensive plan, confirm", but it read as "this is unvalidated", and its override +# (allow_unvalidated_model) also switched off every check below. Cost is now gated where cost is +# known — bin/spin compares the resolved plan against MAX_EUR_PER_HOUR using tests/plans.txt, +# before anything is provisioned. `validated:` remains the honest validation-status flag. - name: Preflight — this GPU has enough VRAM for the selected model profile ansible.builtin.assert: @@ -34,6 +31,82 @@ is {{ min_plan | default('an L40S or larger') }}) or override with -e allow_unvalidated_model=true. (Running anyway would OOM at vLLM start on a GPU you are already paying for.) +# VRAM is not enough on its own: an RTX PRO 6000 (96 GB, cc 12.0) and a B200 (192 GB, cc 10.0) are +# both Blackwell, but NVFP4 kernels are not equally usable on both. Profiles set +# min_compute_capability for the generation they need, and list capabilities the kernels mis-handle +# in unsupported_compute_capabilities. +- name: Preflight — this GPU generation is new enough for the selected model profile + ansible.builtin.assert: + that: + - (gpu_compute_cap is not defined) or (allow_unvalidated_model | default(false) | bool) or + ((gpu_compute_cap | float) >= (min_compute_capability | default(0) | float)) + fail_msg: >- + Model '{{ vllm_model }}' needs compute capability + {{ min_compute_capability | default('?') }} or newer, but this GPU + ({{ gpu_name | default('?') }}) reports {{ gpu_compute_cap | default('?') }}. Pick a plan with + a newer GPU (this profile's floor is {{ min_plan | default('a newer tier') }}) or override with + -e allow_unvalidated_model=true. + +- name: Preflight — this GPU generation is not on the profile's known-bad list + ansible.builtin.assert: + that: + - (gpu_compute_cap is not defined) or (allow_unvalidated_model | default(false) | bool) or + ((gpu_compute_cap | float) not in + (unsupported_compute_capabilities | default([]) | map('float') | list)) + fail_msg: >- + Model '{{ vllm_model }}' does not run correctly on compute capability + {{ gpu_compute_cap | default('?') }} ({{ gpu_name | default('?') }}) — + {{ unsupported_reason | default('its kernels are known to fail or return wrong output on this + GPU family') }}. Pick a plan this profile supports (floor: + {{ min_plan | default('see the profile') }}) or override with -e allow_unvalidated_model=true. + +# tensor_parallel_size larger than the GPU count fails deep inside the engine after the weights +# have already downloaded; catch it before anything is paid for. +- name: Preflight — the plan has enough GPUs for this profile's tensor-parallel size + ansible.builtin.assert: + that: + - (gpu_count is not defined) or + ((gpu_count | int) >= (tensor_parallel_size | default(1) | int)) + fail_msg: >- + Model '{{ vllm_model }}' runs at tensor-parallel size + {{ tensor_parallel_size | default(1) }} but this server has {{ gpu_count | default('?') }} GPU(s). + Pick a multi-GPU plan (this profile's floor is {{ min_plan | default('a multi-GPU plan') }}). + +# Some images are built against a newer CUDA than the shared pin and need a newer host driver. The +# UpCloud template ships whatever it ships, so check before the container fails with a CUDA error +# that does not name the real cause. +- name: Preflight — the host driver is new enough for this profile's image + ansible.builtin.assert: + that: + - (gpu_driver_major is not defined) or (allow_unvalidated_model | default(false) | bool) or + ((gpu_driver_major | int) >= (min_driver_major | default(0) | int)) + fail_msg: >- + Model '{{ vllm_model }}' needs NVIDIA driver {{ min_driver_major | default('?') }} or newer + (its image is a newer CUDA build), but this host reports {{ gpu_driver_major | default('?') }}. + The UpCloud GPU template decides the driver, so this needs a template that ships a newer one. + +# Untested and unsupported are separate states: an unmeasured model or GPU deploys with a warning, +# so the operator knows the run is evidence for docs/validation.md. Unsupported combinations fail +# the asserts above instead. +- name: Warn when this profile has no live run on this account yet + ansible.builtin.debug: + msg: >- + UNTESTED MODEL — '{{ vllm_model }}' has no recorded live run. Treat this deployment as a + validation run: capture `bin/spin validate` and `bin/spin soak` output and add the row to + docs/validation.md. + when: not (validated | default(true) | bool) + +- name: Warn when this GPU generation has no live run for this profile yet + ansible.builtin.debug: + msg: >- + UNTESTED GPU — '{{ vllm_model }}' has a live run on {{ min_plan | default('another tier') }} + only. This server is {{ gpu_name | default('an unmeasured GPU') }} (compute capability + {{ gpu_compute_cap | default('?') }}). It should run; the numbers in docs/validation.md come + from other hardware. Record what you measure here. + when: + - gpu_compute_cap is defined + - (gpu_compute_cap | float) in (untested_compute_capabilities | default([]) | map('float') | list) + - name: Require HF_TOKEN for a gated model profile (fail clearly, not a mid-deploy 401) ansible.builtin.assert: that: not (gated | default(false) | bool) or ((hf_token | default('') | length) > 0) @@ -43,6 +116,81 @@ Hugging Face, then re-run. (All current profiles are ungated; check a repo's gate with `curl https://huggingface.co/api/models/ | jq .gated`.) +# Disk space is the common failure that used to have no machine check: profiles stated their +# download size in a comment that nothing read, so an under-sized WEIGHTS_SIZE_GB surfaced as +# "No space left on device" 20 to 120 minutes into a pull, on a GPU billed by the hour. `weights_gb` +# in each profile is that figure, measured from the Hugging Face file listing (sum of +# `.siblings[].size` — `usedStorage` counts every revision and reads several times too high). +# Free space is the thing to check, since /data also carries Docker's data-root and any previously +# cached model. Measured here rather than reused from the docker role so `--tags vllm` still checks. +# Free space plus what this model already occupies. A cache filled ahead of time (bin/spin +# prefetch) means the pull is already paid for, so comparing free space alone to min_disk_gb +# refuses a deploy with nothing left to download — glm53-flash-nvfp4 asks for 250 GB and its +# weights are 181 GB of that. +# Measure with `du -L`: it follows the symlinks into huggingface_hub's shared blob store and dedups +# by inode, giving 22G/182G/18G against an expected 22/181/18 (2026-09-20). Plain `du` reads the +# symlinks themselves (1 GB for a 22 GB model) and `find -L` double-counts blobs/ and snapshots/ +# (43 GB for that same model). ansible/prefetch.yml relies on the same measurement. +- name: Measure free space and this model's existing cache + ansible.builtin.shell: | + set -o pipefail + df -BG --output=avail {{ data_root | quote }} | tail -1 | tr -dc '0-9' + printf ' ' + du -sBG -L {{ (hf_cache_dir ~ '/hub/models--' ~ (model_repo | replace('/', '--'))) | quote }} \ + 2>/dev/null | cut -f1 | tr -dc '0-9' || printf 0 + args: + executable: /bin/bash + register: data_space + changed_when: false + failed_when: (data_space.stdout | trim | length) == 0 + +# This checks what a deploy needs — the weights still missing, plus room for the image — which is a +# different question from `min_disk_gb`. That key is the size to provision for a cold start, +# (weights_gb + 80) / 0.934: 80 GB of headroom for the image and Docker's data-root, divided by +# what ext4 leaves usable after its 5% root reserve. +# +# Measured live 2026-09-20: qwen38-flash-next has min_disk_gb 300, its 173 GB of weights were +# already prefetched, and 72 GB was free — enough for its ~30 GB image several times over. The +# earlier form of this check (free + cached >= min_disk_gb) refused it, because the profile's 300 +# is a floor chosen for a disk that also has to hold other things. Requiring cold-start headroom +# on a warm cache blocks exactly the workflow `bin/spin prefetch` exists to enable. +- name: Preflight — the weights disk has room for this profile + vars: + _free: "{{ (data_space.stdout.split() | first) | int }}" + _cached: "{{ (data_space.stdout.split() | last) | int }}" + # 60 GB covers the largest image here unpacked (~35 GB) plus working room. Nothing in this + # repo needs more at deploy time once the weights are down. + _still_to_pull: "{{ [0, (weights_gb | default(0) | int) - (_cached | int)] | max }}" + _need_free: "{{ (_still_to_pull | int) + 60 }}" + ansible.builtin.assert: + that: + - (weights_gb is not defined) or + (allow_unvalidated_model | default(false) | bool) or + ((_free | int) >= (_need_free | int)) + fail_msg: >- + Model '{{ vllm_model }}' needs {{ _need_free }} GB free on {{ data_root }} + ({{ _still_to_pull }} GB of weights still to pull, plus 60 GB for the container image and + working room) but only {{ _free }} GB is free. The download would die part-way with + "No space left on device" on a GPU you are paying for by the hour. + Either pull the weights ahead of time on a cheap box (`bin/spin prefetch {{ vllm_model }}`), + grow the disk (set WEIGHTS_SIZE_GB={{ (min_disk_gb | default(0) | int) + 50 }} and re-run + `bin/spin persistent-init` — UpCloud can grow a disk but never shrink it), or free space: + `bin/spin ssh 'du -sh -L {{ data_root }}/hf-cache/hub/models--*'` then rm -rf one you no + longer need. + success_msg: >- + {{ _free }} GB free on {{ data_root }}; '{{ vllm_model }}' needs {{ _need_free }} GB + ({{ _cached }} of {{ weights_gb | default('?') }} GB of weights already cached) + +- name: Warn when the weights disk has little headroom left + ansible.builtin.debug: + msg: >- + Only {{ (data_space.stdout.split() | first) | int }} GB free on {{ data_root }}. The deploy + fits, but there is little room for another model's weights or a second container image — + check `du -sh {{ data_root }}/hf-cache/hub/models--*` before the next deploy. + when: + - min_disk_gb is defined + - ((data_space.stdout.split() | first) | int) < 60 + - name: Require the sslip hostname (bin/spin passes it from the tofu output) ansible.builtin.assert: that: @@ -56,7 +204,7 @@ - acme_email | default('') | length > 0 - "'@' in (acme_email | default(''))" # Let's Encrypt REJECTS placeholder domains (example.com/org/net, .invalid, .local, .test) as - # a forbidden contact — Caddy then never gets a cert and HTTPS silently fails AFTER a full + # a forbidden contact — Caddy then never gets a cert and HTTPS silently fails after a full # deploy. Catch it upfront (learned live 2026-07-05: gpu-spinner@example.com blocked the cert). - not (acme_email | default('') | lower | regex_search('@(example\.(com|org|net)|[^@]*\.(invalid|local|test))$')) - vllm_api_key | default('') | length > 0 @@ -65,6 +213,12 @@ Let's Encrypt refuses placeholder domains like example.com/.local/.test, so Caddy would never obtain a TLS cert. Current ACME_EMAIL='{{ acme_email | default('') }}'. +# Built before the compose render, which needs the paths they produce, and after the preflights, so +# a profile that cannot run on this GPU fails before anything is cloned or compiled. +- name: Build any out-of-tree vLLM plugins this profile pins + ansible.builtin.include_tasks: plugins.yml + when: (vllm_plugins | default([])) | length > 0 + - name: Ensure the compose directory exists ansible.builtin.file: path: "{{ compose_dir }}" diff --git a/ansible/roles/vllm/tasks/plugin_build.yml b/ansible/roles/vllm/tasks/plugin_build.yml new file mode 100644 index 0000000..b2634f4 --- /dev/null +++ b/ansible/roles/vllm/tasks/plugin_build.yml @@ -0,0 +1,69 @@ +--- +# One plugin from vllm_plugins. Included per item by plugins.yml. + +- name: Set the build paths for {{ plugin.name }} + ansible.builtin.set_fact: + plugin_dest: "{{ vllm_plugin_root }}/{{ plugin.name }}-{{ plugin.ref[:12] }}" + plugin_src: "{{ vllm_plugin_root }}/src/{{ plugin.name }}-{{ plugin.ref[:12] }}" + +# Gate on a marker written only after the build is verified. A bare directory check would treat a +# half-finished build as done: `docker run -v dest:/out` creates the directory on the host even when +# the build inside it fails. +- name: Check for an existing pinned build of {{ plugin.name }} + ansible.builtin.stat: + path: "{{ plugin_dest }}/.build-complete" + register: plugin_built + +# Everything below is skipped on a re-run, so a redeploy costs nothing once the commit is built. +- name: Clone the pinned source of {{ plugin.name }} + ansible.builtin.git: + repo: "{{ plugin.repo }}" + dest: "{{ plugin_src }}" + version: "{{ plugin.ref }}" + when: not plugin_built.stat.exists + +- name: Pull the image that will build {{ plugin.name }} + ansible.builtin.command: + cmd: "docker pull {{ vllm_image_override | default(vllm_image) }}" + changed_when: false + register: plugin_image_pull + retries: 5 + delay: 15 + until: plugin_image_pull.rc == 0 + when: not plugin_built.stat.exists + +# The image ships cuda-nvcc (vLLM installs it in its runtime base stage), so the extension compiles +# against the exact torch and Python ABI it will run under. --target keeps the result a plain +# directory that goes on PYTHONPATH rather than an install into the image. +- name: Compile the CUDA extension for {{ plugin.name }} + ansible.builtin.command: + cmd: >- + docker run --rm + -v {{ plugin_src }}:/src + -v {{ plugin_dest }}:/out + {% for k, v in (plugin.build_env | default({})).items() %}-e {{ k }}={{ v }} {% endfor %} + --entrypoint /bin/sh + {{ vllm_image_override | default(vllm_image) }} + -c "pip install --no-build-isolation --target /out /src" + changed_when: true + when: not plugin_built.stat.exists + +- name: Confirm a compiled artifact exists for {{ plugin.name }} + ansible.builtin.find: + paths: "{{ plugin_dest }}" + patterns: "*.so" + recurse: true + register: plugin_artifacts + failed_when: plugin_artifacts.files | length == 0 + when: not plugin_built.stat.exists + +- name: Write the build marker for {{ plugin.name }} + ansible.builtin.copy: + content: "{{ plugin.repo }}@{{ plugin.ref }}\n" + dest: "{{ plugin_dest }}/.build-complete" + mode: "0644" + when: not plugin_built.stat.exists + +- name: Record the container mount path for {{ plugin.name }} + ansible.builtin.set_fact: + vllm_plugin_mounts: "{{ (vllm_plugin_mounts | default([])) + [plugin_dest] }}" diff --git a/ansible/roles/vllm/tasks/plugins.yml b/ansible/roles/vllm/tasks/plugins.yml new file mode 100644 index 0000000..d466e5c --- /dev/null +++ b/ansible/roles/vllm/tasks/plugins.yml @@ -0,0 +1,25 @@ +--- +# Out-of-tree vLLM plugins, for GPU generations needing a kernel no released image carries yet. +# A profile lists the package in vllm_plugins; each entry is cloned at a pinned commit, built inside +# that profile's own image (the .so is libtorch- and Python-ABI-tagged, so it has to be), and left +# on the persistent disk, so the build survives teardown and happens once per commit. +# compose.yml.j2 mounts the result read-only on PYTHONPATH. +# +# Plugins register through vLLM's vllm.general_plugins entry point, so no vLLM file is patched, and +# a well-behaved one is inert unless the profile also sets its environment variable. +# +# THIRD-PARTY CODE. Read the source at the pinned commit before adding an entry, pin a full SHA not +# a branch, and record in the profile why the overlay is there and what would retire it. + +- name: Ensure the plugin build root exists on the persistent disk + ansible.builtin.file: + path: "{{ vllm_plugin_root }}" + state: directory + mode: "0755" + +- name: Build each pinned plugin that is not already present + ansible.builtin.include_tasks: plugin_build.yml + loop: "{{ vllm_plugins | default([]) }}" + loop_control: + loop_var: plugin + label: "{{ plugin.name }}@{{ plugin.ref[:12] }}" diff --git a/ansible/roles/vllm/templates/compose.yml.j2 b/ansible/roles/vllm/templates/compose.yml.j2 index 65f64e4..1617d0b 100644 --- a/ansible/roles/vllm/templates/compose.yml.j2 +++ b/ansible/roles/vllm/templates/compose.yml.j2 @@ -3,9 +3,25 @@ services: {% if not (multi_model_enabled | default(false) | bool) %} vllm: - image: {{ vllm_image }} + # Profiles may pin their own image (vllm_image_override) when a checkpoint needs a build the + # shared pin does not carry yet; everything else uses the validated global pin. + image: {{ vllm_image_override | default(vllm_image) }} restart: unless-stopped ipc: host # required by vLLM (shared memory); avoids cryptic CUDA errors +{% if (cap_add | default([])) | length > 0 %} + # Extra Linux capabilities this profile needs. Docker's default seccomp profile does not simply + # allow-list syscalls — it gates some behind a matching capability, and denies them with EPERM + # otherwise. Measured 2026-09-20: qwen38-flash-next sets VLLM_PLE_CPU_OFFLOAD=1, whose offload + # worker passes CUDA tensors between processes with CUDA IPC. That path calls pidfd_getfd(2), + # which is gated behind CAP_SYS_PTRACE, so engine init died with + # RuntimeError: pidfd_getfd: Operation not permitted + # in torch/multiprocessing/reductions.py -> rebuild_cuda_tensor. Granting the one capability is + # far narrower than seccomp=unconfined or pid=host, which are the other ways round it. + cap_add: +{% for c in cap_add %} + - {{ c }} +{% endfor %} +{% endif %} command: # model as a positional arg (vllm serve deprecates --model) - "{{ model_repo }}" @@ -31,6 +47,12 @@ services: environment: - HF_HOME=/root/.cache/huggingface - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True +{% if (vllm_plugin_mounts | default([])) | length > 0 %} + # Out-of-tree plugin overlays (roles/vllm/tasks/plugins.yml). vLLM loads vllm.general_plugins + # entry points from PYTHONPATH in every process; each plugin stays inert unless the profile + # also sets its own environment variable below. + - PYTHONPATH={{ vllm_plugin_mounts | map('regex_replace', '^.*/', '/opt/vllm-plugins/') | join(':') }} +{% endif %} {% for k, v in (extra_env | default({})).items() %} - {{ k }}={{ v }} {% endfor %} @@ -41,6 +63,9 @@ services: - "{{ compose_dir }}/.env" volumes: - {{ hf_cache_dir }}:/root/.cache/huggingface +{% for m in vllm_plugin_mounts | default([]) %} + - {{ m }}:/opt/vllm-plugins/{{ m | basename }}:ro +{% endfor %} ports: - "127.0.0.1:{{ vllm_port }}:8000" # never exposed publicly; Caddy fronts it deploy: diff --git a/ansible/swap-profiles/b200-nvfp4.yml b/ansible/swap-profiles/b200-nvfp4.yml index 735a61a..a4a2214 100644 --- a/ansible/swap-profiles/b200-nvfp4.yml +++ b/ansible/swap-profiles/b200-nvfp4.yml @@ -1,7 +1,7 @@ --- # Multi-model swap preset — single B200 (179 GB), NVFP4 checkpoints. Each model loads on demand, one # in VRAM at a time. NVFP4 runs only on Blackwell (B200). -# Deploy: bin/spin up --swap-profile b200-nvfp4 --plan GPU-24xCPU-240GB-1xB200 +# Deploy: bin/spin up --swap-profile b200-nvfp4 # # `swap_profiles` references ansible/models/.yml by name (repo/context/args come from there). # All four are small NVFP4 weights (~14–21 GB each), so the whole set caches comfortably on /data. @@ -11,3 +11,6 @@ swap_profiles: - gemma4-26b-nvfp4 # Gemma 4 26B-A4B (MoE) - gemma4-31b-nvfp4 # Gemma 4 31B (dense) # swap_idle_ttl: 1800 # optional override (seconds a backend stays warm after its last request) + +# Plan bin/spin deploys for this preset when --plan is not given (see tests/plans.txt). +swap_plan: "GPU-24xCPU-240GB-1xB200" diff --git a/ansible/swap-profiles/l40s.yml b/ansible/swap-profiles/l40s.yml index 765ff95..1946b2e 100644 --- a/ansible/swap-profiles/l40s.yml +++ b/ansible/swap-profiles/l40s.yml @@ -1,7 +1,7 @@ --- # Multi-model swap preset — single L40S (48 GB). Each model loads on demand, one in VRAM at a time # (llama-swap starts/stops a vLLM backend per request; an idle one unloads after swap_idle_ttl). -# Deploy: bin/spin up --swap-profile l40s --plan GPU-8xCPU-64GB-1xL40S +# Deploy: bin/spin up --swap-profile l40s # # `swap_profiles` lists ansible/models/.yml profiles by name — each model's repo, context and # vLLM args are taken from that profile (defined once), so this file is just the menu. Clients pick a @@ -11,3 +11,6 @@ swap_profiles: - gemma4-26b # Gemma 4 26B-A4B FP8 (MoE) - gemma4-31b # Gemma 4 31B FP8 (dense, tight fit — loads alone) # swap_idle_ttl: 1800 # optional override (seconds a backend stays warm after its last request) + +# Plan bin/spin deploys for this preset when --plan is not given (see tests/plans.txt). +swap_plan: "GPU-8xCPU-64GB-1xL40S" diff --git a/ansible/swap-profiles/rtxpro6000.yml b/ansible/swap-profiles/rtxpro6000.yml new file mode 100644 index 0000000..9b3ae72 --- /dev/null +++ b/ansible/swap-profiles/rtxpro6000.yml @@ -0,0 +1,20 @@ +--- +# Multi-model swap preset — single RTX PRO 6000 (96 GB GDDR7, compute capability 12.0). Each model +# loads on demand, one in VRAM at a time (llama-swap starts/stops a vLLM backend per request; an +# idle one unloads after swap_idle_ttl). +# Deploy: bin/spin up --swap-profile rtxpro6000 +# +# The 96 GB card holds any one of these comfortably — twice the L40S, and enough headroom to serve +# the FP8 models without the tight-fit caps they need on a 48 GB L40S. NVFP4 entries here are dense +# checkpoints only: NVFP4 *MoE* goes through a CUTLASS grouped GEMM that returns invalid output on +# RTX Blackwell, so those profiles stay B200-only (see their unsupported_compute_capabilities). +swap_profiles: + - qwen36 # Qwen3.6-27B FP8 (dense) + - gemma4-26b # Gemma 4 26B-A4B FP8 (MoE — FP8 MoE is fine on this GPU) + - gemma4-31b # Gemma 4 31B FP8 (dense) + - qwen36-27b-nvfp4 # Qwen3.6-27B NVFP4 (dense) + - gemma4-31b-nvfp4 # Gemma 4 31B NVFP4 (dense) +# swap_idle_ttl: 1800 # optional override (seconds a backend stays warm after its last request) + +# Plan bin/spin deploys for this preset when --plan is not given (see tests/plans.txt). +swap_plan: "GPU-16xCPU-80GB-1xRTXPRO6000" diff --git a/bin/lib/checks.sh b/bin/lib/checks.sh new file mode 100644 index 0000000..71121c4 --- /dev/null +++ b/bin/lib/checks.sh @@ -0,0 +1,452 @@ +# shellcheck shell=bash +# Sourced by bin/spin. Evidence commands: validate and soak. + +# Capture exact, reproducible validation evidence for the currently-deployed (single-model) box: +# served model + max_model_len, vLLM's reported KV-cache size + max concurrency, VRAM, and a test +# generation. Paste the result into docs/validation.md so the matrix stays exact, not eyeballed. +# --------------------------------------------------------------------------------------------- +# Shared probe helpers, used by `soak` (many requests, graded) and `ceiling` (one request at +# a time, looking for the length that breaks the engine). They were nested inside cmd_soak and +# closed over its locals; hoisted so ceiling reuses the exact same request shape, haystack sizer +# and grading rule. A second implementation would drift, and the two commands' numbers must be +# comparable — they end up in the same docs/validation.md row. +# --------------------------------------------------------------------------------------------- + +# Resolve endpoint + served model + the model's deployed max_model_len. +# Sets _GS_EP, _GS_MODEL, _GS_MAXLEN. Args: [model-name] +_gs_resolve() { + local models + _GS_EP="$(tofu_out endpoint)" || die "server is down." + [ -n "${VLLM_API_KEY:-}" ] || die "VLLM_API_KEY not set (needed to query the model)." + models="$(curl -fsS --max-time 25 "$_GS_EP/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" 2>/dev/null)" \ + || die "could not reach $_GS_EP/v1/models (still loading, or firewall/key?)." + _GS_MODEL="${1:-}" + [ -n "$_GS_MODEL" ] || _GS_MODEL="$(printf '%s' "$models" | jq -r '.data[0].id // empty')" + [ -n "$_GS_MODEL" ] || die "could not resolve served model from $_GS_EP/v1/models (still loading?)." + _GS_MAXLEN="$(printf '%s' "$models" | jq -r --arg m "$_GS_MODEL" '.data[] | select(.id==$m) | .max_model_len // empty' | head -1)" + case "$_GS_MAXLEN" in ''|*[!0-9]*) _GS_MAXLEN="" ;; esac +} + +# One haystack of roughly $1 tokens into $2. The 19/20 ratio is calibrated against this filler and +# a single tokenizer: 24,576 words -> 25,861 prompt_tokens, 49,152 -> 51,670 (~1.052 tokens per +# word), which lands the requested length within a couple of percent for that model. +# +# The ratio does not carry to other tokenizers. GLM-5.3-Flash packs this filler at ~0.83 tokens per +# word, so a request for 32768 produced 25,861 prompt_tokens there, 21% short (the same gap is +# recorded from the other side in ansible/models/glm53-flash-nvfp4.yml). Treat the requested length +# as a request and read the achieved length back from usage.prompt_tokens — that is the figure to +# record. A ceiling certified against the requested number can be a fifth away from the length +# actually exercised. +_gs_haystack() { + local want_tok="$1" out="$2" words line i + line="survey logistics inventory corridor ledger transcript appendix rotation calibration sequence manifest interval schedule tolerance baseline procedure checkpoint envelope threshold gradient" + words=$(( want_tok * 19 / 20 )) + : > "$out" + for (( i = 0; i < words / 20 + 1; i++ )); do printf '%s\n' "$line"; done >> "$out" +} + +# Build a needle-in-haystack prompt. Args: haystack-file depth-percent needle out-file +# +# Every prompt opens with a unique session tag, which is what makes these probes measure the model +# rather than the prefix cache: the haystack is one line repeated tens of thousands of times and is +# reused across probes, so without the tag the second and later probes hit cached blocks. A tag at +# position 0 invalidates the whole prefix chain. On a hybrid checkpoint a cached probe answers fast +# and wrong (docs/gpu-spinner.md, "Hybrid checkpoints and prefix caching"). +_gs_prompt() { + local hay="$1" frac="$2" needle="$3" out="$4" lines cut_at + lines="$(wc -l < "$hay")"; cut_at=$(( lines * frac / 100 )); [ "$cut_at" -lt 1 ] && cut_at=1 + _GS_SEQ=$(( ${_GS_SEQ:-0} + 1 )) + { + printf 'Session tag: %s-%s-%s\n\n' "$$" "$_GS_SEQ" "${RANDOM}${RANDOM}" + printf 'Read the following archive dump carefully.\n\n' + head -n "$cut_at" "$hay" + printf '\nThe archival reference code for this cabinet is %s.\n\n' "$needle" + tail -n +$(( cut_at + 1 )) "$hay" + printf '\nQuestion: what is the archival reference code for this cabinet? Reply with just the code.\n' + } > "$out" +} + +# Post one long prompt; write "statussecondsprompt_tokensfound|missing" to $3. +# Args: prompt-file needle out-file timeout-seconds +# Differs from a plain chat call on purpose: +# * the body goes through a file — a 128k-token prompt as argv blows ARG_MAX; +# * no -f, so an HTTP rejection stays distinguishable from a wrong answer (-f empties both); +# * the timeout scales with context — a 128k prefill at 20-way does not fit in 90 s; +# * max_tokens 256, not 32: some models inline reasoning into content, so a tighter budget would +# score a working model wrong. Trivial retrieval measured 51 completion tokens. +_gs_probe() { + local pf="$1" needle="$2" of="$3" tmo="$4" body resp code secs ptok got + body="$(mktemp)"; resp="$(mktemp)" + jq -n --rawfile p "$pf" --arg m "$_GS_MODEL" \ + '{model:$m,messages:[{role:"user",content:$p}],max_tokens:256,temperature:0, + chat_template_kwargs:{enable_thinking:false}}' > "$body" + code="$(curl -sS -o "$resp" -w '%{http_code} %{time_total}' --max-time "$tmo" \ + "$_GS_EP/v1/chat/completions" -H "Authorization: Bearer $VLLM_API_KEY" \ + -H 'Content-Type: application/json' --data-binary "@$body" 2>/dev/null || printf '000 0')" + secs="${code#* }"; code="${code%% *}" + ptok="$(jq -r '.usage.prompt_tokens // 0' < "$resp" 2>/dev/null || printf 0)" + got="$(jq -r '(.choices[0].message.content // "") + " " + (.choices[0].message.reasoning_content // "")' < "$resp" 2>/dev/null || printf '')" + if printf '%s' "$got" | grep -qF "$needle"; then + printf '%s\t%s\t%s\tfound\n' "$code" "$secs" "$ptok" > "$of" + else + printf '%s\t%s\t%s\tmissing\n' "$code" "$secs" "$ptok" > "$of" + fi + rm -f "$body" "$resp" +} + +# How many times has the vLLM container restarted? This is the crash detector. +# `restart: unless-stopped` (compose.yml.j2) brings a crashed engine back by itself, and a fast +# reload can complete inside a soak's own tally — the run then reports the crash as a "rejected" +# request and still prints "Soak OK". The restart counter is the only signal that does not lie. +# Prints an integer, or nothing if it cannot be read. +_gs_restarts() { + local host; host="$(tofu_out ssh_host 2>/dev/null)" || return 0 + [ -n "$host" ] || return 0 + ssh "${SSH_OPTS[@]}" -o ConnectTimeout=15 "root@$host" \ + "docker inspect -f '{{.RestartCount}}' $VLLM_CONTAINER 2>/dev/null" 2>/dev/null | tr -dc '0-9' +} + +cmd_validate() { + local host ep + host="$(tofu_out ssh_host)" || die "server is down." + ep="$(tofu_out endpoint)" + [ -n "${VLLM_API_KEY:-}" ] || die "VLLM_API_KEY not set (needed to query the model)." + log "Validation evidence for $ep:" + local models served + models="$(curl -fsS --max-time 25 "$ep/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" 2>/dev/null)" \ + || die "could not reach $ep/v1/models (still loading, or firewall/key?)." + printf '%s\n' "$models" | jq -r '.data[] | " served=\(.id) max_model_len=\(.max_model_len)"' + served="$(printf '%s\n' "$models" | jq -r '.data[0].id')" + echo " --- vLLM KV cache + GPU (from the box) ---" + ssh "${SSH_OPTS[@]}" "root@$host" \ + 'docker logs '"$VLLM_CONTAINER"' 2>&1 | grep -iE "GPU KV cache size|Maximum concurrency" | tail -2; nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv,noheader' 2>/dev/null \ + | sed 's/^/ /' + echo " --- test generation ---" + # enable_thinking=false: thinking models (Qwen) otherwise spend the whole budget on reasoning and + # return empty content; templates that don't know the kwarg simply ignore it. + curl -fsS --max-time 60 "$ep/v1/chat/completions" -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \ + -d "{\"model\":\"$served\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: validation ok\"}],\"max_tokens\":32,\"temperature\":0,\"chat_template_kwargs\":{\"enable_thinking\":false}}" 2>/dev/null \ + | jq -r '.choices[0].message.content // .choices[0].message.reasoning_content // .error.message // "(no response)"' | sed 's/^/ /' + echo " (now record this row in docs/validation.md)" +} + +# Soak + answer-quality check for the currently-deployed model (single-model, or a llama-swap +# backend by --model NAME). Reusable version of the docs/validation.md procedure: +# 1) a temp-0 factual battery (auto-graded by substring) + fabrication traps (soft-checked), +# 2) a graded long-context load (does the box stay healthy and still answer correctly). +# Usage: bin/spin soak [--model NAME] [--rounds N] [--concurrency M] [--context T] +cmd_soak() { + local ep model="" rounds=3 conc=20 ctx=32768 maxlen models + while [ $# -gt 0 ]; do + case "$1" in + --model) model="$2"; shift 2 ;; + --rounds) rounds="$2"; shift 2 ;; + --concurrency) conc="$2"; shift 2 ;; + --context) ctx="$2"; shift 2 ;; + *) die "soak: unknown option '$1'" ;; + esac + done + case "$rounds$conc$ctx" in *[!0-9]*) die "soak: --rounds/--concurrency/--context must be integers." ;; esac + _gs_resolve "$model"; ep="$_GS_EP"; model="$_GS_MODEL"; maxlen="$_GS_MAXLEN" + # Clamp the haystack to what the model accepts, so the defaults stay safe on a short-context + # profile instead of every request failing on context length. + # + # Read the clamp warning, or the run is a false pass: ask for 131072 against a model deployed at + # 98304 and you get a green run at ~96k. Recording that as "passed at 131072" certifies a length + # nobody tested. `bin/spin ceiling` probes above the deployed value instead — it redeploys first, + # because no request can exceed the engine's own limit. + if [ -n "$maxlen" ] && [ "$ctx" -gt $(( maxlen - 2048 )) ]; then + ctx=$(( maxlen - 2048 )) + warn "soak: --context clamped to $ctx (model max_model_len=$maxlen, leaving room for the answer)." + warn "soak: record the achieved prompt_tokens below, NOT the length you asked for." + fi + log "Soak: model=$model endpoint=$ep" + + # Ask one prompt at temp 0 with thinking suppressed; print the assistant text. Args: prompt [max_tokens] + _soak_ask() { + curl -fsS --max-time 90 "$ep/v1/chat/completions" -H "Authorization: Bearer $VLLM_API_KEY" \ + -H 'Content-Type: application/json' \ + -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":$(printf '%s' "$1" | jq -Rs .)}],\"max_tokens\":${2:-48},\"temperature\":0,\"chat_template_kwargs\":{\"enable_thinking\":false}}" 2>/dev/null \ + | jq -r '.choices[0].message.content // .choices[0].message.reasoning_content // ""' + } + + # 1) Factual battery — each entry is "prompt|=|answer-substring" (grep -i, no false-match traps). + echo " --- factual battery (temp 0, auto-graded) ---" + local battery=( + "What is the capital of Australia? Answer with one word.|=|canberra" + "What is 47 multiplied by 89? Reply with just the number.|=|4183" + "In what year did the Berlin Wall fall? Reply with just the year.|=|1989" + "Who wrote 'One Hundred Years of Solitude'? Reply with the author's full name.|=|rquez" + "Who is the first-listed author of 'Attention Is All You Need'? Reply with one surname.|=|vaswani" + "What is the largest planet in our solar system? Answer with one word.|=|jupiter" + ) + local pass=0 total=0 qa q expect got + for qa in "${battery[@]}"; do + q="${qa%%|=|*}"; expect="${qa##*|=|}"; total=$((total+1)) + got="$(_soak_ask "$q" 40)" + if printf '%s' "$got" | grep -qi "$expect"; then + pass=$((pass+1)); printf ' \033[32m✓\033[0m %s\n' "$q" + else + printf ' \033[31m✗\033[0m %s → got: %s\n' "$q" "$(printf '%s' "$got" | tr '\n' ' ' | head -c 90)" + fi + done + echo " factual: $pass/$total correct" + + # Fabrication traps — soft-checked: pass if the reply signals uncertainty rather than inventing. + echo " --- fabrication traps (should decline / express uncertainty) ---" + local traps=( + "Summarize the plot of the 1987 novel 'The Crimson Meridian' by Eleanor Vance." + "What did Marie Curie say about the 2024 Paris Olympics?" + ) + local t ans + for t in "${traps[@]}"; do + ans="$(_soak_ask "$t" 120)" + # Strip markdown emphasis first. Measured 2026-09-20: qwen38-flash-next correctly refused with + # "Marie Curie **did not** say anything about the 2024 Paris Olympics", and the pattern + # `did not (say|exist|write)` could not match across the bold markers, so a correct refusal was + # flagged. Soft-checked either way, but the warning was simply wrong. + ans="$(printf '%s' "$ans" | sed 's/\*\{1,3\}//g; s/_\{2\}//g; s/`//g')" + if printf '%s' "$ans" | grep -qiE "cannot|can't|could ?n't|no record|not aware|unable|don't have|do not have|does ?n't exist|does not exist|fiction|no (information|evidence|widely|known|such)|i'm not|unaware|there (is|are|was) no|did ?n't (say|exist|write)|did not (say|exist|write)|passed away|misunderstand|not (a )?(real|recognized|find)|couldn't find|could not find"; then + printf ' \033[32m✓ declined\033[0m: %s\n' "$(printf '%s' "$ans" | tr '\n' ' ' | head -c 120)" + else + printf ' \033[33m⚠ check\033[0m: %s\n' "$(printf '%s' "$ans" | tr '\n' ' ' | head -c 120)" + fi + done + + # 2) Graded long-context load. Each request carries its own haystack with a needle at a known + # depth and is graded on retrieving it, because counting non-empty replies to one-line prompts + # measures availability only: vllm-project/vllm#55757 reports a broken sparse-MLA path on sm_120 + # answering short prompts correctly and failing only at realistic lengths. Needles are synthetic + # ASCII (nothing to fold or normalise) and unique per request, so one request cannot be graded + # correct on another's needle. Keeping the prefix cache out of the measurement is a separate job, + # done by _gs_prompt's session tag: the needle sits deep in the prompt, so requests still share a + # long identical prefix ahead of it. + local tmp; tmp="$(mktemp -d)" + + echo " --- graded long-context load: $rounds rounds x $conc parallel, ~$ctx tok each ---" + local depths=(head middle tail) r i d frac hay pf needle + for r in $(seq 1 "$rounds"); do + for i in $(seq 1 "$conc"); do + d="${depths[$(( (i - 1) % 3 ))]}" + case "$d" in head) frac=10 ;; middle) frac=50 ;; *) frac=90 ;; esac + needle="MERIDIAN-${r}${i}${frac}" + pf="$tmp/p-$r-$i"; hay="$tmp/h-$ctx" + [ -f "$hay" ] || _gs_haystack "$ctx" "$hay" + _gs_prompt "$hay" "$frac" "$needle" "$pf" + ( _gs_probe "$pf" "$needle" "$tmp/r-$r-$i-$d" "$(( 120 + ctx / 200 ))" ) & + done + wait + done + + # Tally. A rejection is capacity; a wrong answer is a correctness bug. Keep them apart. + local reqs=$((rounds*conc)) nfound=0 nwrong=0 nrej=0 ntmo=0 f code res + local lats=() + for f in "$tmp"/r-*; do + [ -f "$f" ] || continue + code="$(cut -f1 "$f")"; res="$(cut -f4 "$f")" + case "$code" in + 200) if [ "$res" = found ]; then nfound=$((nfound+1)); else nwrong=$((nwrong+1)); fi + lats+=("$(cut -f2 "$f")") ;; + 000) ntmo=$((ntmo+1)) ;; + *) nrej=$((nrej+1)) ;; + esac + done + # Per-depth accuracy: an aggregate would hide a path that only fails at one depth. + local dd dtot dok + for dd in head middle tail; do + dtot="$(find "$tmp" -name "r-*-$dd" 2>/dev/null | wc -l | tr -d ' ')" + [ "$dtot" -gt 0 ] || continue + dok="$(cat "$tmp"/r-*-"$dd" 2>/dev/null | awk -F'\t' '$1==200 && $4=="found"' | wc -l | tr -d ' ')" + if [ "$dok" = "$dtot" ]; then + printf ' \033[32m✓\033[0m depth %-6s %s/%s retrieved\n' "$dd" "$dok" "$dtot" + else + printf ' \033[31m✗\033[0m depth %-6s %s/%s retrieved\n' "$dd" "$dok" "$dtot" + fi + done + local p50="-" p95="-" pmax="-" n idx + if [ "${#lats[@]}" -gt 0 ]; then + # Not `mapfile`: that is bash 4+, and macOS still ships /bin/bash 3.2, where it exits 127 under + # `set -e` once every request has run and before the verdict prints — discarding a result you + # just paid GPU time for. This loop is 3.2-compatible. + local _sorted=() _l + while IFS= read -r _l; do _sorted+=("$_l"); done < <(printf '%s\n' "${lats[@]}" | sort -n) + lats=(${_sorted[@]+"${_sorted[@]}"}) + n="${#lats[@]}" + p50="${lats[$(( n / 2 ))]}" + idx=$(( n * 95 / 100 )); [ "$idx" -ge "$n" ] && idx=$(( n - 1 )) + p95="${lats[$idx]}"; pmax="${lats[$(( n - 1 ))]}" + fi + local ptok_med + ptok_med="$(cut -f3 "$tmp"/r-* 2>/dev/null | sort -n | awk '{a[NR]=$1} END{if (NR) print a[int(NR/2)+1]}')" + echo " load: $nfound/$reqs correct, $nwrong wrong, $nrej rejected, $ntmo timed out" + echo " prompt_tokens (median): ${ptok_med:-?} latency p50/p95/max: ${p50}s / ${p95}s / ${pmax}s" + rm -rf "$tmp" + + curl -fsS --max-time 20 "$ep/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" >/dev/null 2>&1 \ + || die "Soak: endpoint NOT healthy after load — investigate (docker logs / nvidia-smi via bin/spin ssh)." + # A wrong answer from an admitted request fails the command; rejections and timeouts are + # admission control and say nothing about correctness. + # Two known causes needing different fixes, so name both rather than the first one found. + [ "$nwrong" -eq 0 ] || die "Soak: $nwrong admitted request(s) returned the wrong needle — long-context retrieval is broken. Passing short prompts does not clear it. + Two known causes: + 1. Prefix caching on a hybrid / linear-attention checkpoint (KDA, Gated DeltaNet, mamba + state): a cache hit resumes those layers from a recycled block and the answer comes back + fast and wrong (vllm#56960). Check this first, it is cheap — grep the engine log for + enable_prefix_caching=, and if it says True add --no-enable-prefix-caching and re-run. + 2. A broken sparse-MLA path on sm_120 (vllm#55757), which answers short prompts correctly and + fails only at realistic lengths." + log "Soak OK — factual $pass/$total, long-context $nfound/$reqs correct ($nrej rejected, $ntmo timed out)." +} + +# Find the longest single request a deployed model actually survives, so max_model_len can be set +# from a measurement instead of from the recipe's number. +# +# WHY A SEPARATE COMMAND. `soak` cannot do this. It clamps the requested length to +# max_model_len - 2048 and only warns, so a probe "above the ceiling" silently becomes a probe +# below it and passes. And a crash is invisible to it: `restart: unless-stopped` brings the engine +# back, a fast reload finishes inside the tally, the dead request is filed as "rejected", and soak +# prints "Soak OK". This command probes one request at a time and gates every result on the +# container's restart counter. +# +# FOUR CEILINGS, and only two of them need this command: +# 1. CAPACITY — the KV pool cannot hold one full-length request. Free to find: read "GPU KV cache +# size" from `bin/spin validate` and divide by the context. vLLM refuses at startup rather +# than crashing, so there is nothing to search for. +# 2. CRASH — the engine dies above some length, with the KV pool barely touched. This is what +# `ceiling` searches. glm53-flash-nvfp4 is the worked example: clean at 98,100 prompt tokens, +# CUDA illegal memory access at 102,400, KV peak 17.8% (vllm-project/vllm#54317). +# 3. WRONG ANSWER — the engine stays up, returns HTTP 200 and the needle is absent. `ceiling` +# prints WRONG, and the restart counter is what separates it from a crash. Find the cause +# before capping max_model_len: the usual one is prefix caching on a hybrid checkpoint +# (vllm#56960), where every rung fails warm. On custom_model the whole ladder, 98,138 to +# 1,044,002, passed cold and failed warm on the same engine, so capping would have shipped +# 98,304 as a limit that does not exist. +# 4. MODE FLOOR — the checkpoint needs a minimum context for a documented mode (deepseek-v4-pro's +# 393216 is its "max" reasoning mode). Capping below it changes what the model is; the right +# answer there is fewer concurrent sequences, not a smaller context. +# +# DEPLOY HIGH FIRST. No request can exceed the engine's own max_model_len, so a search on a box +# deployed at the cautious value measures the cautious value. Check that native even fits first +# (ceiling 1 above), then redeploy: +# bin/spin up --model --plan -e max_model_len= +# and run this. It costs one engine reload, not a new server. +# +# ASCEND, DO NOT BISECT. A passing probe costs a minute or two; a crash costs a full engine reload +# (minutes, and the whole weight load again for a big model). Ascending to the first failure costs +# one crash. Bisecting costs three or four — real money on a multi-GPU box. +cmd_ceiling() { + local model="" start=0 max=0 step=2 keep="" + while [ $# -gt 0 ]; do + case "$1" in + --model) model="$2"; shift 2 ;; + --start) start="$2"; shift 2 ;; + --max) max="$2"; shift 2 ;; + --step) step="$2"; shift 2 ;; # multiplier between probes (integer, >=2 means doubling) + --keep-going) keep=1; shift ;; # do not stop at the first failure + *) die "ceiling: unknown option '$1'" ;; + esac + done + case "$start$max$step" in *[!0-9]*) die "ceiling: --start/--max/--step must be integers." ;; esac + [ "$step" -ge 2 ] || die "ceiling: --step must be 2 or more (it is a multiplier)." + + _gs_resolve "$model" + [ -n "$_GS_MAXLEN" ] || die "ceiling: $_GS_EP does not report max_model_len for '$_GS_MODEL'." + local hard=$(( _GS_MAXLEN - 2048 )) # the engine rejects anything above this + [ "$max" -gt 0 ] && [ "$max" -lt "$hard" ] && hard="$max" + [ "$start" -gt 0 ] || start=$(( hard / 8 )); [ "$start" -lt 4096 ] && start=4096 + + log "Ceiling search: model=$_GS_MODEL deployed max_model_len=$_GS_MAXLEN" + echo " probing single requests from $start up to $hard tokens, x$step each step" + echo " NOTE: nothing here can exceed the DEPLOYED max_model_len. If $_GS_MAXLEN is already the" + echo " cautious value rather than the checkpoint's native one, redeploy higher first:" + echo " bin/spin up --model --plan -e max_model_len=" + + local base_rs; base_rs="$(_gs_restarts)" + if [ -n "$base_rs" ]; then echo " container restarts before we start: $base_rs" + else warn "ceiling: could not read the container restart count over SSH — a crash may be misread as a rejection."; fi + + local tmp; tmp="$(mktemp -d)" + # shellcheck disable=SC2064 + trap "rm -rf '$tmp'" RETURN + + local ctx="$start" last_pass="" last_pass_tok="" first_fail="" first_fail_tok="" verdict="" + local hay pf needle out code secs ptok res rs tmo + while :; do + needle="CEILING-$ctx" + hay="$tmp/h-$ctx"; pf="$tmp/p-$ctx"; out="$tmp/r-$ctx" + _gs_haystack "$ctx" "$hay" + # Depth 50%: a needle in the middle is the hardest for a broken long-context path and the + # cheapest single probe that is not trivially answerable from the head of the prompt. + _gs_prompt "$hay" 50 "$needle" "$pf" + tmo=$(( 180 + ctx / 150 )) + printf ' %8d requested ... ' "$ctx" + _gs_probe "$pf" "$needle" "$out" "$tmo" + code="$(cut -f1 "$out")"; secs="$(cut -f2 "$out")"; ptok="$(cut -f3 "$out")"; res="$(cut -f4 "$out")" + + rs="$(_gs_restarts)" + if [ -n "$base_rs" ] && [ -n "$rs" ] && [ "$rs" != "$base_rs" ]; then + verdict="CRASH"; base_rs="$rs" + elif [ "$code" = "200" ] && [ "$res" = found ]; then + verdict="pass" + elif [ "$code" = "200" ]; then + verdict="WRONG" + elif [ "$code" = "000" ]; then + verdict="TIMEOUT" + else + verdict="rejected($code)" + fi + + case "$verdict" in + pass) printf '\033[32mpass\033[0m %s prompt_tokens, %ss\n' "$ptok" "$secs" + last_pass="$ctx"; last_pass_tok="$ptok" ;; + CRASH) printf '\033[31mCRASH\033[0m engine restarted (restart count now %s)\n' "$rs" + first_fail="$ctx"; first_fail_tok="$ptok" ;; + WRONG) printf '\033[31mWRONG\033[0m %s prompt_tokens, %ss — admitted and answered, needle not returned\n' "$ptok" "$secs" + first_fail="$ctx"; first_fail_tok="$ptok" ;; + TIMEOUT) printf '\033[33mtimeout\033[0m after %ss — not a crash by itself; treat as inconclusive\n' "$tmo" + first_fail="$ctx"; first_fail_tok="$ptok" ;; + *) printf '\033[33m%s\033[0m — refused by admission control, not a crash\n' "$verdict" + first_fail="$ctx"; first_fail_tok="$ptok" ;; + esac + + if [ "$verdict" != "pass" ]; then + # Wait for the engine to come back before doing anything else; a crashed container is + # restarted by compose and the next probe would otherwise measure a cold engine. + if [ "$verdict" = "CRASH" ]; then + echo " waiting for the engine to come back..." + local i=0 + until curl -fsS --max-time 15 "$_GS_EP/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" >/dev/null 2>&1; do + i=$(( i + 1 )); [ "$i" -gt 120 ] && { warn "engine did not return within 20 min — investigate with bin/spin logs"; break; } + sleep 10 + done + echo " engine is serving again" + fi + [ -n "$keep" ] || break + fi + + [ "$ctx" -ge "$hard" ] && break + ctx=$(( ctx * step )); [ "$ctx" -gt "$hard" ] && ctx="$hard" + done + + echo + echo " --- ceiling result for $_GS_MODEL ---" + if [ -n "$last_pass" ]; then + echo " last PASSING request: $last_pass_tok prompt_tokens (requested $last_pass)" + else + echo " last PASSING request: none — even $start failed. Search downward, or fix the deploy first." + fi + if [ -n "$first_fail" ]; then + echo " first FAILING request: $first_fail_tok prompt_tokens (requested $first_fail)" + else + echo " first FAILING request: none up to $hard — the ceiling is at or above the deployed max_model_len." + fi + echo + echo " Record prompt_tokens, NOT the requested figure: the haystack sizer is calibrated to one" + echo " tokenizer and lands up to ~20% off on others." + if [ -n "$last_pass_tok" ] && [ -n "$first_fail_tok" ] && [ "$first_fail_tok" -gt 0 ]; then + echo " Suggested max_model_len: a round value at or below $last_pass_tok, e.g." + echo " $(( (last_pass_tok / 8192) * 8192 )) (largest multiple of 8192 that still passed)" + echo " Put the whole pass/fail matrix in the profile comment, with the fault from the engine log." + fi + echo " Then re-soak at the cap before believing it: bin/spin soak --context --concurrency 10" +} diff --git a/bin/lib/common.sh b/bin/lib/common.sh new file mode 100644 index 0000000..b16e68d --- /dev/null +++ b/bin/lib/common.sh @@ -0,0 +1,45 @@ +# shellcheck shell=bash +# Sourced by bin/spin. Logging, the tofu output/init wrappers, and the UpCloud API caller. + +log() { printf '\033[1;34m==>\033[0m %s\n' "$*"; } +warn() { printf '\033[1;33mwarn:\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1;31merror:\033[0m %s\n' "$*" >&2; exit 1; } + +require_token() { [ -n "${UPCLOUD_TOKEN:-}" ] || die "UPCLOUD_TOKEN not set (export it or put it in .env)."; } +# `tofu output -raw NAME` prints a "No outputs found" warning to STDOUT and exits 0 when the state +# has no outputs, so a plain capture returns that 548-byte banner instead of an empty string — +# `[ -n "$x" ]` then passes and the banner lands in a URL or a -var. `output -json NAME` exits +# non-zero with empty stdout instead, which is what every call site here wants. +tofu_output() { + local dir="$1" name="$2" + "$TOFU" -chdir="$dir" output -json "$name" 2>/dev/null \ + | jq -r 'if type == "string" then . else tojson end' 2>/dev/null || true +} +tofu_out() { tofu_output "$EPHEMERAL_DIR" "$1"; } + +# Init a stack, skipping when already initialized (avoids the flaky public registry on every run) +# and retrying when it must hit the network. Force a re-init with TOFU_REINIT=1. +tofu_init() { + local dir="$1" i + if [ -z "${TOFU_REINIT:-}" ] && [ -d "$dir/.terraform" ] && [ -f "$dir/.terraform.lock.hcl" ]; then + return 0 + fi + for i in 1 2 3; do + "$TOFU" -chdir="$dir" init -input=false && return 0 + warn "tofu init failed (attempt $i/3) — registry.opentofu.org flaky? retrying in 10s..." + sleep 10 + done + die "tofu init failed for $dir after 3 attempts (provider registry unreachable?)." +} + +# --- UpCloud API helpers (for stop/start/resume) --- +api() { # api METHOD /path [json-body] (bounded: a hung API call must not hang the CLI) + local method="$1" path="$2" body="${3:-}" + if [ -n "$body" ]; then + curl -fsS --max-time 30 --retry 2 -X "$method" -H "Authorization: Bearer ${UPCLOUD_TOKEN:-}" -H "Content-Type: application/json" \ + --data-binary "$body" "https://api.upcloud.com/1.3${path}" + else + curl -fsS --max-time 30 --retry 2 -X "$method" -H "Authorization: Bearer ${UPCLOUD_TOKEN:-}" "https://api.upcloud.com/1.3${path}" + fi +} +server_state() { api GET "/server/$1" 2>/dev/null | jq -r '.server.state // empty'; } diff --git a/bin/lib/net.sh b/bin/lib/net.sh new file mode 100644 index 0000000..4a3848a --- /dev/null +++ b/bin/lib/net.sh @@ -0,0 +1,119 @@ +# shellcheck shell=bash +# Sourced by bin/spin. Operator allowlist: reachability check, -var plumbing, firewall refresh, SSH wait. + +# Build operator-allowlist -var args from the environment, used by BOTH up and down so neither +# depends on the api.ipify.org autodetect when OPERATOR_IP/OPERATOR_CIDRS is set. +OPERATOR_TF_ARGS=() +# --- Operator reachability ------------------------------------------------------------------- +# The firewall allowlists SSH(22) and HTTPS(443) to the operator CIDRs only. Deploying from an +# address outside them creates the server, starts billing it, and then times out waiting for SSH — +# the box is unreachable for the whole run and nothing tells you why until 10 minutes later. +# Check before anything is provisioned. Pure bash so this adds no dependency beyond curl. +ip_to_int() { local IFS=. a b c d; read -r a b c d < 0 when covered + local net bits ipi neti mask + net="${1%%/*}"; bits="${1##*/}" + [ "$bits" = "$1" ] && bits=32 + case "$net" in *[!0-9.]*|"") return 1 ;; esac + case "$bits" in ''|*[!0-9]*) return 1 ;; esac + [ "$bits" -ge 0 ] && [ "$bits" -le 32 ] || return 1 + [ "$bits" -eq 0 ] && return 0 + ipi="$(ip_to_int "$2")"; neti="$(ip_to_int "$net")" + mask=$(( (0xFFFFFFFF << (32 - bits)) & 0xFFFFFFFF )) + [ $(( ipi & mask )) -eq $(( neti & mask )) ] +} + +# Refuse to provision a box this machine cannot reach. Only meaningful when OPERATOR_CIDRS or +# OPERATOR_IP pins the allowlist — with neither set the stack auto-detects this machine's address, +# so it is reachable by construction. +assert_operator_reachable() { + [ -n "${SKIP_REACHABILITY_CHECK:-}" ] && return 0 + local allow="${OPERATOR_CIDRS:-${OPERATOR_IP:-}}" + [ -n "$allow" ] || return 0 + local me; me="$(curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)" + case "$me" in ''|*[!0-9.]*) warn "could not determine this machine's public IP; skipping the reachability check."; return 0 ;; esac + local c ok="" + local IFS=, + for c in $allow; do + c="$(printf '%s' "$c" | tr -d '[:space:]')" + [ -n "$c" ] || continue + if cidr_covers "$c" "$me"; then ok=1; break; fi + done + unset IFS + [ -n "$ok" ] && { log "Reachability OK — this machine ($me) is inside the operator allowlist."; return 0; } + die "this machine's public IP ($me) is not in the operator allowlist ($allow). + The firewall would drop SSH/HTTPS from here, so the server would bill while every connection + timed out. Add it to OPERATOR_CIDRS in .env, e.g. + OPERATOR_CIDRS=$allow,$me/32 + as a SINGLE assignment (a later duplicate line silently wins). Editing the rule in the provider + console does not work: tofu owns the firewall and reverts it on the next apply. + To deploy anyway (e.g. from a jump host you reach another way): SKIP_REACHABILITY_CHECK=1." +} + +build_operator_tf_args() { + OPERATOR_TF_ARGS=() + if [ -n "${OPERATOR_CIDRS:-}" ]; then + local cidr_json + cidr_json="$(printf '%s' "$OPERATOR_CIDRS" | awk -F, '{printf "["; for(i=1;i<=NF;i++){gsub(/^[ \t]+|[ \t]+$/,"",$i); printf "%s\"%s\"",(i>1?",":""),$i}; printf "]"}')" + OPERATOR_TF_ARGS+=(-var "operator_cidrs=$cidr_json") + elif [ -n "${OPERATOR_IP:-}" ]; then + OPERATOR_TF_ARGS+=(-var "operator_ip=$OPERATOR_IP") + fi +} + +# Refresh only the firewall allowlist for the current operator IP. A server can sit stopped for a +# while; if your egress IP rotated meanwhile, a plain start would resume behind a stale allowlist +# and lock you out (SSH/443 dropped). We -target the firewall resource, but it references the server +# (server_id), so a -target apply pulls the server into the plan too. To avoid an UNINTENDED RESIZE +# back to the default plan, we read the server's CURRENT plan from state and pass it as -var plan=. +refresh_firewall() { + tofu_init "$EPHEMERAL_DIR" + build_operator_tf_args + local fw_args=() + fw_args+=(${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"}) + local cur_plan; cur_plan="$(state_plan)" + if [ -z "$cur_plan" ] || [ "$cur_plan" = "null" ]; then + warn "could not read the server's current plan from state — skipping firewall refresh to avoid an unintended resize." + warn "if your IP changed, run a full 'bin/spin up' (or set OPERATOR_CIDRS) to refresh the allowlist." + return 0 + fi + fw_args+=(-var "plan=$cur_plan") + log "Refreshing firewall allowlist for your current IP (plan=$cur_plan, no resize)..." + "$TOFU" -chdir="$EPHEMERAL_DIR" apply -auto-approve ${fw_args[@]+"${fw_args[@]}"} -target=upcloud_firewall_rules.gpu +} + +# Two failure modes hide behind "waiting for SSH", and both otherwise cost the full timeout on a +# billing GPU: +# * key not installed -> the box answers and refuses (Permission denied). The wait can never +# succeed, so fail immediately and name the missing key. +# * still booting -> no answer at all. Worth waiting for: a 4-GPU tier with a large attached +# disk has been measured over 10 minutes, hence the ~20 min default (SSH_WAIT_TRIES). +wait_ssh() { + local host="$1" i out tries="${SSH_WAIT_TRIES:-240}" + [ -n "$host" ] || die "no ssh_host available." + log "Waiting for SSH on $host (up to ~$(( tries * 5 / 60 )) min) ..." + # stderr is captured into a variable rather than a temp file: a `trap ... RETURN` that removes a + # `local` temp path fires after the variable has left scope, which under `set -u` kills the + # script after a successful deploy. + for i in $(seq 1 "$tries"); do + if out="$(ssh "${SSH_OPTS[@]}" -o BatchMode=yes -o ConnectTimeout=5 "root@$host" true 2>&1)"; then + return 0 + fi + case "$out" in *"Permission denied"*) + die "SSH reached $host but AUTHENTICATION failed (Permission denied (publickey)). + The box is up and the firewall is letting you through — it just does not trust this machine's + key. ssh_public_keys is create-time metadata, so editing terraform.tfvars will not fix a server + that already exists. To fix the running box, append this machine's public key to + /root/.ssh/authorized_keys there from a machine that can already log in; add it to + terraform.tfvars as well so the next build keeps it. This machine offers: + $(ssh-keygen -lf ~/.ssh/id_ed25519.pub 2>/dev/null | awk '{print $2}')" ;; + esac + sleep 5 + done + die "timed out waiting for SSH after ~$(( tries * 5 / 60 )) min. + If this tier is simply slow to boot, raise it: SSH_WAIT_TRIES=360 bin/spin up ..." +} diff --git a/bin/lib/plan.sh b/bin/lib/plan.sh new file mode 100644 index 0000000..0346cad --- /dev/null +++ b/bin/lib/plan.sh @@ -0,0 +1,92 @@ +# shellcheck shell=bash +# Sourced by bin/spin. Where the GPU plan comes from, and the provider tfvars sanity check. + +# The plan of the server currently in tofu state ("" if none) — used to avoid unintended resizes. +state_plan() { + "$TOFU" -chdir="$EPHEMERAL_DIR" show -json 2>/dev/null \ + | jq -r '.values.root_module.resources[]? | select(.address=="upcloud_server.gpu") | .values.plan // empty' 2>/dev/null | head -1 +} + +# Each profile / preset names the plan it runs on (min_plan / swap_plan), which `up` deploys when +# no --plan is given, so the tier need not be remembered per model. Read with sed, not a YAML +# parser: the dependency list is curl/jq/tofu/ansible-playbook/ssh and must not gain python3. +# Anything that is not a well-formed plan identifier is ignored, so a profile may omit the field +# (kimi-k3 does) and callers fall through to the next source. +profile_plan() { + local file="$1" key="$2" value + [ -f "$file" ] || return 0 + value="$(sed -n "s/^${key}: *\"\{0,1\}\([A-Za-z0-9-]*\)\"\{0,1\}.*/\1/p" "$file" | head -1)" + case "$value" in + GPU-[0-9]*xCPU-[0-9]*GB-[0-9]*x*) printf '%s' "$value" ;; + *) return 0 ;; + esac +} + +# The provider tfvars are copied from an example with placeholders. Nothing in the persistent stack +# reads them, so a half-finished copy only surfaces much later as an opaque UpCloud API error during +# the ephemeral apply — after a server has been requested. Check them before anything is created. +check_tfvars() { + local f="$EPHEMERAL_DIR/terraform.tfvars" + [ -f "$f" ] || die "$f not found. Copy it from ${f}.example, then set os_template and ssh_public_keys." + if grep -qE '^[[:space:]]*os_template[[:space:]]*=[[:space:]]*"REPLACE_ME"' "$f"; then + die "$f still has os_template = \"REPLACE_ME\". + List the GPU templates and paste the UUID or exact title: + set -a; . ./.env; set +a + curl -fsS -H \"Authorization: Bearer \$UPCLOUD_TOKEN\" https://api.upcloud.com/1.3/storage/template \\ + | jq -r '.storages.storage[] | select(.title|test(\"GPU|NVIDIA|CUDA\";\"i\")) | \"\\(.uuid) \\(.title)\"'" + fi + # A key inside the ssh_public_keys list, ignoring comment lines. + if ! grep -vE '^[[:space:]]*#' "$f" | grep -qE '(ssh-(rsa|ed25519|dss)|ecdsa-sha2-)'; then + die "$f has no SSH public key in ssh_public_keys. + Without one you cannot reach the server you are about to pay for. Add yours, e.g. + ssh_public_keys = [\"\$(cat ~/.ssh/id_ed25519.pub)\"]" + fi +} + +# Price of a plan in EUR/hour, from tests/plans.txt. Prints nothing if the plan is not listed. +# Spot ids are not separate rows: GPU-SPOT- reads the spot column of the GPU- row. +# awk, not python — bin/spin's dependency list deliberately has no python3 (see check_tfvars). +plan_price() { + local p="$1" row col + case "$p" in + GPU-SPOT-*) row="GPU-${p#GPU-SPOT-}"; col=3 ;; + *) row="$p"; col=2 ;; + esac + awk -v r="$row" -v c="$col" '$1 == r { print $c; exit }' tests/plans.txt 2>/dev/null +} + +# Refuse a plan above MAX_EUR_PER_HOUR unless the operator opts in. +# +# This replaces the old per-profile `requires_review` flag, retired 2026-09-21. That flag meant +# "this needs an expensive plan, confirm" but read as "this is unvalidated", and its override +# (`allow_unvalidated_model`) also switched off the VRAM floor, the compute-capability checks, the +# driver floor and the disk check — six gates behind one badly-named flag. Its original premise was +# also gone: it guarded against deploying a huge model onto the default L40S, which cannot happen +# now that bin/spin derives the plan from min_plan. +# +# Price is the thing worth confirming, so confirm price. Reading it from tests/plans.txt means +# every plan is covered automatically, including ones added later, with no per-profile flag to +# forget — and tests/check-plans.sh keeps those prices honest against the provider. +assert_plan_affordable() { # assert_plan_affordable + local plan="$1" what="$2" allow="$3" price max + [ -n "$plan" ] || return 0 + [ -z "$allow" ] || return 0 + max="${MAX_EUR_PER_HOUR:-10}" + price="$(plan_price "$plan")" + if [ -z "$price" ] || [ "$price" = "?" ]; then + warn "no recorded price for $plan — cannot check it against MAX_EUR_PER_HOUR=$max. Add it to tests/plans.txt." + return 0 + fi + if awk -v p="$price" -v m="$max" 'BEGIN { exit !(p > m) }'; then + local hint="see tests/plans.txt for every plan and both prices" + case "$plan" in + GPU-SPOT-*) : ;; # already the cheap id; suggesting one again would read as GPU-SPOT-SPOT- + *) hint="cheaper spot id: GPU-SPOT-${plan#GPU-} — $hint" ;; + esac + die "'$what' deploys on $plan at EUR $price/hour, above the MAX_EUR_PER_HOUR limit of $max. + That is about EUR $(awk -v p="$price" 'BEGIN{printf "%.0f", p*24}')/day if you leave it up. + If you mean it: add --allow-expensive (or raise MAX_EUR_PER_HOUR) + $hint." + fi + log "Plan $plan costs EUR $price/hour (limit $max)." +} diff --git a/bin/lib/stack.sh b/bin/lib/stack.sh new file mode 100644 index 0000000..b12e15d --- /dev/null +++ b/bin/lib/stack.sh @@ -0,0 +1,519 @@ +# shellcheck shell=bash +# Sourced by bin/spin. Lifecycle commands: persistent stack, up/down, start/stop, status/logs/ssh/plan. + +# Which persistent data disk this invocation targets, and its UUID. +# +# WEIGHTS_DISK=alt selects the optional second disk (terraform/persistent, weights_alt_size_gb), +# for running one model without growing or evicting the shared cache. Everything that touches a +# disk goes through here, so `down` can never decommission the disk you were not using — that +# mistake would silently delete someone else's cached weights. +weights_output_name() { + case "${WEIGHTS_DISK:-primary}" in + alt) printf 'weights_alt_storage_id' ;; + primary) printf 'weights_storage_id' ;; + *) die "WEIGHTS_DISK must be 'primary' or 'alt', got '${WEIGHTS_DISK}'." ;; + esac +} + +# Pass the selection to the ephemeral / prefetch stacks so they attach the same disk. +weights_disk_tf_args() { + printf -- '-var\nweights_disk=%s\n' "${WEIGHTS_DISK:-primary}" +} + +cmd_persistent_init() { + require_token + # Disk size/tier are the standing-cost knobs — let the operator set them in .env instead of + # remembering tofu -var syntax. Explicit -var passed after (in "$@") still wins. See .env.example. + local pv=() + [ -n "${WEIGHTS_SIZE_GB:-}" ] && pv+=(-var "weights_size_gb=$WEIGHTS_SIZE_GB") + [ -n "${WEIGHTS_TIER:-}" ] && pv+=(-var "weights_tier=$WEIGHTS_TIER") + + # Warn before a size change: growing the disk raises the 24/7 bill, and UpCloud CANNOT shrink a + # disk (only decommission + re-init smaller). Compare the requested size to the live disk. + local want="${WEIGHTS_SIZE_GB:-500}" diskid cur + diskid="$(tofu_output "$PERSIST_DIR" "$(weights_output_name)")" + if [ -n "$diskid" ] && printf '%s' "$want" | grep -qE '^[0-9]+$'; then + # || true: advisory size comparison. Under `set -e` with pipefail a failed API call here + # aborted the command substitution and killed bin/spin with a bare exit code (seen: 56). + cur="$(api GET "/storage/$diskid" 2>/dev/null | jq -r '.storage.size // empty' || true)" + if printf '%s' "$cur" | grep -qE '^[0-9]+$'; then + local rate=226; [ "${WEIGHTS_TIER:-maxiops}" = "standard" ] && rate=86 + if [ "$want" -gt "$cur" ]; then + warn "weights disk will GROW ${cur}→${want} GB (≈ +€$(( (want-cur)*rate/1000 ))/mo at ${WEIGHTS_TIER:-maxiops}). Disks cannot shrink later." + elif [ "$want" -lt "$cur" ]; then + warn "requested ${want} GB < current ${cur} GB — UpCloud cannot SHRINK a disk, so apply will fail." + warn "to reduce: 'bin/spin persistent-destroy' then re-init at the smaller size (re-downloads weights)." + fi + fi + fi + + tofu_init "$PERSIST_DIR" + + # Report the failure rather than letting `set -e` end the script with nothing on screen. A silent + # return here was indistinguishable from success, and `up` called this internally and ended the + # same way. + local rc=0 + "$TOFU" -chdir="$PERSIST_DIR" apply ${pv[@]+"${pv[@]}"} "$@" || rc=$? + if [ "$rc" -ne 0 ]; then + die "tofu apply exited $rc in $PERSIST_DIR. Nothing was created. + If it printed nothing above, run it directly to see the reason: + $TOFU -chdir=$PERSIST_DIR apply" + fi + + # An apply can exit 0 having created nothing. The outputs are what the rest of the CLI reads, so + # check those rather than the exit status alone. + local ip + ip="$(tofu_output "$PERSIST_DIR" floating_ip)" + if [ -z "$ip" ]; then + die "apply finished but $PERSIST_DIR has no floating_ip output, so the stack is not up. + Check what the state holds: $TOFU -chdir=$PERSIST_DIR state list" + fi + log "Persistent stack ready. Floating IP: $ip" +} + +# Decommission the persistent stack → €0 standing cost. prevent_destroy (main.tf) blocks a plain +# `tofu destroy` (and must be a literal, so it can't be toggled by a var), so this deletes the disk + +# releases the IP via the API, then drops them from tofu state (the stack stays declarable — a later +# persistent-init recreates both fresh). IRREVERSIBLE: loses cached weights + the stable IP/cert. +cmd_persistent_destroy() { + require_token + local yes=""; [ "${1:-}" = "--yes" ] && yes=1 + local srv; srv="$(tofu_out server_uuid 2>/dev/null || true)" + # tofu prints a "no outputs found" warning (not a UUID) once the server is down — only a real UUID + # means a server still exists; anything else (warning text / empty) is treated as "no server". + printf '%s' "$srv" | grep -qE '^[0-9a-fA-F-]{36}$' || srv="" + [ -z "$srv" ] || die "an ephemeral GPU server still exists ($srv) — run 'bin/spin down' first." + local diskid ip + diskid="$(tofu_output "$PERSIST_DIR" "$(weights_output_name)")" + ip="$(tofu_output "$PERSIST_DIR" floating_ip)" + [ -n "$diskid$ip" ] || die "no persistent resources in state — nothing to decommission." + warn "PERSISTENT DECOMMISSION — IRREVERSIBLE. This will:" + printf ' \033[1;33m•\033[0m DELETE weights disk %s (cached models + Caddy TLS cert)\n' "${diskid:-?}" >&2 + printf ' \033[1;33m•\033[0m RELEASE floating IP %s (the stable .sslip.io hostname is lost)\n' "${ip:-?}" >&2 + printf ' Next spin-up re-downloads ALL weights and gets a NEW IP + cert. Standing cost → €0.\n' >&2 + [ -n "$yes" ] || die "refusing without confirmation. Re-run: bin/spin persistent-destroy --yes" + if [ -n "$diskid" ]; then + if api DELETE "/storage/$diskid" >/dev/null 2>&1; then log "deleted weights disk $diskid" + else warn "could not delete disk $diskid (already gone / still attached?)"; fi + fi + if [ -n "$ip" ]; then + if api DELETE "/ip_address/$ip" >/dev/null 2>&1; then log "released floating IP $ip" + else warn "could not release IP $ip (already gone?)"; fi + fi + "$TOFU" -chdir="$PERSIST_DIR" state rm upcloud_storage.weights upcloud_floating_ip_address.vip >/dev/null 2>&1 || true + log "Persistent stack decommissioned. Standing cost → €0. Re-create anytime with 'bin/spin persistent-init'." +} + +cmd_up() { + require_token + local model="${VLLM_MODEL:-qwen36}" plan="" flags_explicit="" swap_profile="" allow_expensive="" + local tf_args=() ans_args=() + local sd_at="${AUTO_SHUTDOWN_AT:-}" sd_tz="${AUTO_SHUTDOWN_TZ:-}" sd_enabled="${AUTO_SHUTDOWN:-}" + while [ $# -gt 0 ]; do + case "$1" in + --model) model="$2"; flags_explicit=1; shift 2 ;; + --plan) plan="$2"; flags_explicit=1; shift 2 ;; + --swap-profile) swap_profile="$2"; flags_explicit=1; shift 2 ;; + --allow-unvalidated) ans_args+=(-e allow_unvalidated_model=true); shift ;; + --allow-expensive) allow_expensive=1; shift ;; + # Pass an Ansible extra-var straight through. Extra vars outrank the profile's own + # include_vars, so this is how you deploy the same profile at a different setting without + # editing the file: `-e max_model_len=262144` for a token-ceiling search, or the fallbacks + # several profiles document in their own comments (`-e tensor_parallel_size=4`, + # `-e vllm_image_override=...`). Repeatable. The deployed value then differs from the file, + # so record which one was live, and re-run without -e before you tear down. + -e|--set) [ $# -ge 2 ] || die "$1 needs a key=value argument." + case "$2" in *=*) ;; *) die "$1 expects key=value, got '$2'." ;; esac + ans_args+=(-e "$2"); shift 2 ;; + --shutdown-at) sd_at="$2"; shift 2 ;; + --shutdown-tz) sd_tz="$2"; shift 2 ;; + --no-shutdown) sd_enabled="false"; shift ;; + --shutdown) sd_enabled="true"; shift ;; + *) die "unknown option: $1" ;; + esac + done + + # Multi-model swap: turn --swap-profile into the ansible vars that enable llama-swap. The + # preset (ansible/swap-profiles/.yml) is loaded whole via -e @file (Ansible reads its + # swap_profiles list). Validate the name here so a typo fails before any tofu/GPU spend. + local swap_file="" + if [ -n "$swap_profile" ]; then + swap_file="ansible/swap-profiles/${swap_profile}.yml" + [ -f "$swap_file" ] || die "unknown swap profile '$swap_profile'. See: bin/spin swap-profiles" + ans_args+=(-e multi_model_enabled=true -e "@$swap_file") + fi + + # Resume an existing STOPPED server (e.g. after the auto-off safety) instead of recreating — + # the image + model are already on the intact disks, so this is fast. Run `bin/spin down` first + # if you want a fresh box. + local existing; existing="$(tofu_out server_uuid 2>/dev/null || true)" + if [ -n "$existing" ] && [ "$(server_state "$existing")" = "stopped" ]; then + # Resume can't change the model or plan — refuse rather than silently ignore explicit flags. + [ -z "$flags_explicit" ] || die "a STOPPED server exists; resume can't apply --model/--plan. + Either 'bin/spin start' (resume as-is), or 'bin/spin down' first for a fresh box with new flags." + log "Existing server $existing is stopped — resuming (no rebuild)..." + refresh_firewall + api POST "/server/$existing/start" >/dev/null + wait_ssh "$(tofu_out ssh_host)" + log "Resumed. Endpoint: $(tofu_out endpoint) (vLLM auto-starts on boot; give it ~a minute)" + return 0 + fi + + [ -n "${ACME_EMAIL:-}" ] || die "ACME_EMAIL not set (needed for Let's Encrypt)." + # The vllm role rejects placeholder domains, but only after the server exists and ~40 tasks + # have run. Same test here costs nothing and fails before anything is provisioned. + if printf '%s' "$ACME_EMAIL" | tr '[:upper:]' '[:lower:]' \ + | grep -qE '@(example\.(com|org|net)|[^@]*\.(invalid|local|test))$'; then + die "ACME_EMAIL is still a placeholder ($ACME_EMAIL). + Let's Encrypt refuses these domains, so Caddy would never get a certificate and HTTPS would + fail after an otherwise clean deploy. Set a real, deliverable address in .env." + fi + [ -n "${VLLM_API_KEY:-}" ] || die "VLLM_API_KEY not set (vLLM auth)." + check_tfvars + # Plan resolution, highest priority first: an explicit --plan, then the plan of a server that + # already exists, then the one the profile/preset names, then the OpenTofu default. + local want_plan="" + if [ -n "$swap_profile" ]; then want_plan="$(profile_plan "$swap_file" swap_plan)" + else want_plan="$(profile_plan "ansible/models/${model}.yml" min_plan)"; fi + + # A profile with no parseable min_plan produces an empty want_plan, which silently disables both + # guards below: it inherits whatever box is live (no mismatch check) or falls through to the + # OpenTofu default, and the mistake only surfaces after Ansible has run. kimi-k3 is the live + # example — no UpCloud plan satisfies its 280 GB per-GPU floor, so it deliberately has no + # min_plan, and it is precisely the profile that must not latch onto a running server. + # Refuse unless the operator names the tier themselves. + if [ -z "$plan" ] && [ -z "$want_plan" ] && [ -z "$swap_profile" ]; then + die "profile '${model}' declares no usable min_plan, so bin/spin cannot choose a tier for it. + Either the profile is missing the field, or no plan fits it (kimi-k3 needs ~280 GB per GPU; + nothing sold here has that). Pass --plan to name the tier explicitly and accept the cost. + Plans and prices: tests/plans.txt" + fi + + # Guard against an unintended resize: with no --plan, a re-run on an EXISTING server would plan + # the config default against the live box (auto-approve!). Inherit the current plan from state. + if [ -z "$plan" ] && [ -n "$existing" ]; then + plan="$(state_plan)" + # A live box on the wrong tier is worse than an error here: preflight only refuses after the + # weights download, so an L40S inherited under a 390 GB profile burns the pull first. + if [ -n "$plan" ] && [ -n "$want_plan" ] && [ "$plan" != "$want_plan" ]; then + die "the existing server is $plan but '${swap_profile:-$model}' expects $want_plan. + Pass --plan explicitly to override, or 'bin/spin down' first to rebuild on the right tier." + fi + [ -n "$plan" ] && log "No --plan given; keeping the existing server's plan ($plan)." + fi + if [ -z "$plan" ] && [ -n "$want_plan" ]; then + plan="$want_plan" + log "No --plan given; using $plan from the ${swap_profile:-$model} profile." + fi + [ -n "$plan" ] && tf_args+=(-var "plan=$plan") + # Cost gate. Runs after the plan is settled and before anything is created, so a refusal is free. + assert_plan_affordable "$plan" "${swap_profile:-$model}" "$allow_expensive" + build_operator_tf_args + tf_args+=(${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"}) + assert_operator_reachable + [ -n "$sd_enabled" ] && ans_args+=(-e "auto_shutdown_enabled=$sd_enabled") + [ -n "$sd_at" ] && ans_args+=(-e "auto_shutdown_time=$sd_at") + [ -n "$sd_tz" ] && ans_args+=(-e "auto_shutdown_timezone=$sd_tz") + + # If a previous `down` auto-decommissioned the persistent disk, recreate it first — the server + # can't attach a disk that no longer exists. The weights_storage_id output goes stale after a + # decommission (`state rm` doesn't recompute outputs), so verify against the API. And tell "the + # API says it's gone" from "the API call failed": curl exit 22 is the -f HTTP-error case (404 = + # really gone), anything else is a transport failure that must not trigger a recreate. + local wid disk_ok="" api_rc=0; wid="$(tofu_output "$PERSIST_DIR" "$(weights_output_name)")" + if printf '%s' "$wid" | grep -qE '^[0-9a-fA-F-]{36}$'; then + api GET "/storage/$wid" >/dev/null 2>&1 || api_rc=$? + if [ "$api_rc" -eq 0 ]; then + disk_ok=1 + elif [ "$api_rc" -ne 22 ]; then + warn "could not confirm the weights disk via the UpCloud API (curl exit $api_rc) — assuming it still exists rather than recreating it. Re-run if the deploy then fails on a missing disk." + disk_ok=1 + fi + fi + if [ -z "$disk_ok" ]; then + log "Persistent weights disk missing (decommissioned on a previous down?) — recreating it first..." + # -auto-approve: this runs inside `up`, which is non-interactive throughout. Without it the + # apply stops at tofu's confirmation prompt, and a cancelled apply exits non-zero, so `set -e` + # ends the whole spin-up with nothing but the line above on screen. + cmd_persistent_init -auto-approve + fi + + while IFS= read -r _a; do [ -n "$_a" ] && tf_args+=("$_a"); done < <(weights_disk_tf_args) + [ "${WEIGHTS_DISK:-primary}" = primary ] || log "Using the ALT data disk (WEIGHTS_DISK=alt)." + log "Provisioning GPU server (model=$model)..." + tofu_init "$EPHEMERAL_DIR" + local apply_log; apply_log="$(mktemp)" + trap 'rm -f "$apply_log"' RETURN + if ! "$TOFU" -chdir="$EPHEMERAL_DIR" apply -auto-approve ${tf_args[@]+"${tf_args[@]}"} 2>&1 | tee "$apply_log"; then + if grep -qiE 'SERVER_RESOURCES_UNAVAILABLE|resources?[ _]unavailable|out of stock|no (host|capacity)' "$apply_log"; then + rm -f "$apply_log" + die "UpCloud has no '${plan:-the default plan}' capacity in fi-hel2 right now. + Retry shortly, or override the profile's tier with another that fits the model + (tiers and prices: tests/plans.txt and docs/gpu-spinner.md): + bin/spin up --model $model --plan GPU-16xCPU-80GB-1xRTXPRO6000 # 96 GB, usually available + bin/spin up --model $model --plan GPU-12xCPU-240GB-1xH100 # or ...-1xB200" + fi + rm -f "$apply_log" + die "tofu apply failed (see output above)." + fi + rm -f "$apply_log" + + local host sslip endpoint + host="$(tofu_out ssh_host)"; sslip="$(tofu_out sslip_host)"; endpoint="$(tofu_out endpoint)" + [ -n "$host" ] || die "could not read ssh_host from tofu outputs." + + log "Writing inventory ($host)..." + mkdir -p "$(dirname "$INVENTORY")" + printf '[gpu]\n%s ansible_user=root\n' "$host" > "$INVENTORY" + + wait_ssh "$host" + + # ansible-playbook exits 0 when the host pattern matches nothing, so a broken inventory used to + # produce a "successful" spin-up that configured nothing at all — the endpoint and test commands + # below would print for a box Ansible never touched. Resolve the inventory first and stop here if + # our host is not in it. -i names the file rather than the directory, so this works even where + # ansible.cfg is not picked up. + if ! ansible-inventory -i "$INVENTORY" --list 2>/dev/null \ + | jq -e --arg h "$host" '.gpu.hosts | index($h)' >/dev/null 2>&1; then + die "the 'gpu' group in $INVENTORY does not resolve to $host, so Ansible would configure nothing. + Check what Ansible sees: ansible-inventory -i $INVENTORY --list" + fi + + log "Configuring with Ansible (vLLM + Caddy)..." + ansible-playbook -i "$INVENTORY" ansible/site.yml -e "vllm_model=$model" -e "sslip_host=$sslip" ${ans_args[@]+"${ans_args[@]}"} + + log "Up. Endpoint: $endpoint" + # The API expects the served model name (what /v1/models lists). In swap mode that's a preset + # entry name (first one listed); in single-model mode it's the profile's served_name. + local served + if [ -n "$swap_file" ]; then + served="$(awk '/^swap_profiles:/{f=1;next} f&&/^[[:space:]]*-[[:space:]]/{sub(/^[[:space:]]*-[[:space:]]*/,"");sub(/[[:space:]]*#.*/,"");print;exit} f&&/^[^[:space:]#-]/{exit}' "$swap_file" 2>/dev/null)" + log "Multi-model swap active (preset '$swap_profile'). Models load on demand; run 'bin/spin swap-profiles' to see the menu." + else + served="$(awk -F'"' '/^served_name:/{print $2; exit}' "ansible/models/${model}.yml" 2>/dev/null)" + fi + cat </dev/null | jq -r '.storage.size // empty')" + printf '%s' "$size" | grep -qE '^[0-9]+$' || { warn "could not read disk size — leaving the persistent disk in place."; return 0; } + if [ "$size" -lt "$thr" ]; then + log "Persistent disk ${size} GB (< ${thr} GB threshold) kept — minor standing cost. (DECOMMISSION_THRESHOLD_GB=$thr)" + return 0 + fi + local rate=226; [ "${WEIGHTS_TIER:-maxiops}" = "standard" ] && rate=86 + warn "AUTO-DECOMMISSION: weights disk is ${size} GB (>= ${thr} GB) — deleting it to stop ~EUR $(( size*rate/1000 ))/mo standing cost." + warn " Floating IP is KEPT (stable hostname). Next 'bin/spin up' recreates the disk, re-downloads weights + re-issues the TLS cert (~1 min)." + warn " Keep it warm instead: 'bin/spin down --keep-disk', or set DECOMMISSION_ON_DOWN=never / raise DECOMMISSION_THRESHOLD_GB." + if api DELETE "/storage/$diskid" >/dev/null 2>&1; then + "$TOFU" -chdir="$PERSIST_DIR" state rm upcloud_storage.weights >/dev/null 2>&1 || true + log "Weights disk deleted. Standing cost now ~EUR 3.5/mo (floating IP only)." + else + warn "could not delete weights disk $diskid (still attached? already gone?) — check manually." + fi +} + +cmd_down() { + require_token + local keep_disk="" dargs=() + while [ $# -gt 0 ]; do + case "$1" in + --keep-disk) keep_disk=1; shift ;; + *) dargs+=("$1"); shift ;; + esac + done + log "Destroying GPU server..." + build_operator_tf_args # so destroy doesn't trip on the ipify autodetect data source + "$TOFU" -chdir="$EPHEMERAL_DIR" destroy -auto-approve ${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"} ${dargs[@]+"${dargs[@]}"} + rm -f "$INVENTORY" # stale inventory would point later ansible runs at a recycled IP + log "Down. The GPU is no longer billing." + down_decommission_disk "$keep_disk" +} + +cmd_start() { + require_token + local uuid; uuid="$(tofu_out server_uuid)" + [ -n "$uuid" ] || die "no server in state — use 'bin/spin up' to create one." + local st; st="$(server_state "$uuid")" + case "$st" in + started) log "Server $uuid is already running." ;; + stopped) log "Starting server $uuid..."; refresh_firewall; api POST "/server/$uuid/start" >/dev/null ;; + "") die "could not read server state (API/token reachable?)." ;; + *) die "server state is '$st' — not startable." ;; + esac + wait_ssh "$(tofu_out ssh_host)" + log "Up. Endpoint: $(tofu_out endpoint) (vLLM auto-starts on boot)" +} + +cmd_stop() { + require_token + local uuid; uuid="$(tofu_out server_uuid)" + [ -n "$uuid" ] || die "no server in state." + log "Powering off $uuid (GPU billing stops; disk + IP + on-box config are kept)..." + api POST "/server/$uuid/stop" '{"stop_server":{"stop_type":"soft"}}' >/dev/null + log "Stop requested. Resume with 'bin/spin start'; tear down with 'bin/spin down'." +} + +cmd_status() { + "$TOFU" -chdir="$EPHEMERAL_DIR" output 2>/dev/null || warn "no ephemeral state (server is down?)" + local host; host="$(tofu_out ssh_host || true)" + if [ -n "${host:-}" ]; then + # Report both possible model units: gpu-spinner (single-model compose) and gpu-swap + # (llama-swap multi-model). Either may be the active one depending on how it was deployed. + ssh "${SSH_OPTS[@]}" "root@$host" \ + 'nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total --format=csv 2>/dev/null || echo "nvidia-smi unavailable" + for u in gpu-spinner gpu-swap; do printf "%-12s %s\n" "$u:" "$(systemctl is-active "$u" 2>/dev/null || true)"; done' 2>/dev/null || true + fi +} + +cmd_logs() { + local host; host="$(tofu_out ssh_host)" || die "server is down." + # Follow whichever model layer is running: llama-swap (gpu-swap) in multi-model mode, else the + # single-model docker compose stack. + ssh "${SSH_OPTS[@]}" "root@$host" ' + if systemctl is-active --quiet gpu-swap; then + echo "== llama-swap (multi-model) journal — backends start/stop on demand ==" >&2 + journalctl -u gpu-swap -n 100 -f + else + cd '"$COMPOSE_DIR"' && docker compose logs -f --tail=100 + fi' +} + +cmd_ssh() { + local host; host="$(tofu_out ssh_host)" || die "server is down." + exec ssh "${SSH_OPTS[@]}" "root@$host" "$@" +} + +cmd_plan() { "$TOFU" -chdir="$EPHEMERAL_DIR" plan "$@"; } + +# List the multi-model swap presets and the profiles each serves, so a user can pick one for +# `bin/spin up --swap-profile `. Reads the `- name` lines under swap_profiles: in each preset. +cmd_swap_profiles() { + local dir="ansible/swap-profiles" + [ -d "$dir" ] || die "no swap-profiles directory ($dir)." + local found="" + log "Multi-model swap presets ($dir) — deploy with: bin/spin up --swap-profile --plan " + for f in "$dir"/*.yml; do + [ -e "$f" ] || break + found=1 + printf '\n \033[1m%s\033[0m\n' "$(basename "$f" .yml)" + # models are the '- ' list items; comment lines describe the tier + grep -E '^\s*-\s' "$f" | sed -E 's/^\s*-\s*/ - /' + done + [ -n "$found" ] || warn "no presets found in $dir." +} + +# Fill the persistent weights disk from a cheap CPU box, before any GPU exists. +# +# The arithmetic: a download is billed at whatever compute holds the disk, and nothing about it is +# GPU work. 354 GB at ~126 MB/s is ~47 min: 5.17 EUR on 4x RTX PRO 6000 (6.60/h) against 0.035 EUR +# on 2xCPU-4GB (0.0446/h) — about 150x. The GPU box then starts warm and spends its expensive +# minutes on inference. This also pulls each profile's container image (9-14 GB) onto the same +# disk, because Docker's data-root lives there too. +# +# An UpCloud storage device attaches to one server at a time, so this is mutually exclusive with a +# GPU server by construction. Refuse up front rather than letting tofu fail halfway. +cmd_prefetch() { + require_token + local models=() keep="" + while [ $# -gt 0 ]; do + case "$1" in + --keep) keep=1; shift ;; # leave the box up (debugging); it still costs ~0.045/h + -*) die "prefetch: unknown option '$1'" ;; + *) models+=("$1"); shift ;; + esac + done + [ "${#models[@]}" -gt 0 ] || die "prefetch: name at least one profile, e.g. + bin/spin prefetch qwen38-nvfp4 qwen38-flash-next + Profiles: ls ansible/models/" + + local m + for m in "${models[@]}"; do + [ -f "ansible/models/${m}.yml" ] || die "unknown model profile '$m' (see ansible/models/)." + done + + # The weights disk cannot be attached to two servers. Fail with the reason, not a tofu error. + local gpu_srv; gpu_srv="$(tofu_out server_uuid 2>/dev/null || true)" + if printf '%s' "$gpu_srv" | grep -qE '^[0-9a-fA-F-]{36}$'; then + die "a GPU server still exists ($gpu_srv) and holds the weights disk. + An UpCloud disk attaches to one server at a time. Run 'bin/spin down --keep-disk' first + (keeps the cache), then prefetch, then bring the GPU box back up." + fi + + check_tfvars + local wid; wid="$(tofu_output "$PERSIST_DIR" "$(weights_output_name)")" + printf '%s' "$wid" | grep -qE '^[0-9a-fA-F-]{36}$' \ + || die "no persistent weights disk in state — run 'bin/spin persistent-init' first." + + build_operator_tf_args + assert_operator_reachable + + local tf_args=(${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"}) + # Reuse the ephemeral stack's SSH keys. `ssh_public_keys` is a multi-line HCL list, so + # a line-oriented extraction returns a bare "[" and tofu dies with "Missing expression". Collapse + # the whole bracketed block onto one line instead. Written to a tfvars file rather than passed as + # -var because the value contains spaces and quotes. + local keys + keys="$(awk '/^ssh_public_keys[[:space:]]*=/{f=1} f{printf "%s ", $0} f&&/\]/{exit}' \ + "$EPHEMERAL_DIR/terraform.tfvars" 2>/dev/null | sed 's/ */ /g')" + if [ -n "$keys" ]; then + printf '%s\n' "$keys" > "$PREFETCH_DIR/terraform.tfvars" + else + warn "no ssh_public_keys found in $EPHEMERAL_DIR/terraform.tfvars — the prefetch box will be unreachable." + fi + + while IFS= read -r _a; do [ -n "$_a" ] && tf_args+=("$_a"); done < <(weights_disk_tf_args) + [ "${WEIGHTS_DISK:-primary}" = primary ] || log "Using the ALT data disk (WEIGHTS_DISK=alt)." + log "Provisioning the prefetch box (${#models[@]} profile(s): ${models[*]})..." + tofu_init "$PREFETCH_DIR" + "$TOFU" -chdir="$PREFETCH_DIR" apply -auto-approve ${tf_args[@]+"${tf_args[@]}"} \ + || die "tofu apply failed in $PREFETCH_DIR — nothing was fetched." + + local host; host="$(tofu_output "$PREFETCH_DIR" ssh_host)" + [ -n "$host" ] || die "prefetch stack has no ssh_host output; check: $TOFU -chdir=$PREFETCH_DIR state list" + wait_ssh "$host" + + printf '[prefetch]\n%s ansible_user=root\n' "$host" > "$INVENTORY" + local json; json="$(printf '%s\n' "${models[@]}" | jq -R . | jq -sc '{prefetch_models: .}')" + local rc=0 + ansible-playbook -i "$INVENTORY" ansible/prefetch.yml -e "$json" || rc=$? + rm -f "$INVENTORY" + + if [ -n "$keep" ]; then + warn "--keep: the prefetch box is still running at ~EUR 0.045/h. Destroy it with: + $TOFU -chdir=$PREFETCH_DIR destroy -auto-approve" + else + log "Destroying the prefetch box..." + "$TOFU" -chdir="$PREFETCH_DIR" destroy -auto-approve ${tf_args[@]+"${tf_args[@]}"} \ + || warn "prefetch teardown FAILED — the box is still billing. Destroy it by hand: + $TOFU -chdir=$PREFETCH_DIR destroy" + fi + + [ "$rc" -eq 0 ] || die "the prefetch play failed (exit $rc) — see the output above. The disk keeps whatever did download." + log "Prefetch done. The weights disk is warm; 'bin/spin up --model ' now skips the pull." +} diff --git a/bin/spin b/bin/spin index 5e3ed35..68062f9 100755 --- a/bin/spin +++ b/bin/spin @@ -7,6 +7,7 @@ cd "$ROOT" PERSIST_DIR="terraform/persistent" EPHEMERAL_DIR="terraform/providers/upcloud" +PREFETCH_DIR="terraform/prefetch" INVENTORY="ansible/inventory/hosts.ini" # Must match ansible's compose_dir (group_vars/all.yml); the compose project name (and thus # container names like -vllm-1) derives from this directory's basename. @@ -20,97 +21,24 @@ SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null) # shellcheck source=/dev/null if [ -f .env ]; then set -a; . ./.env; set +a; fi -log() { printf '\033[1;34m==>\033[0m %s\n' "$*"; } -warn() { printf '\033[1;33mwarn:\033[0m %s\n' "$*" >&2; } -die() { printf '\033[1;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +# The command implementations live in bin/lib/*.sh, sourced here. Paths are spelled out so +# `shellcheck -x` follows them and checks the CLI as one unit. +# shellcheck source=bin/lib/common.sh +. "$ROOT/bin/lib/common.sh" +# shellcheck source=bin/lib/net.sh +. "$ROOT/bin/lib/net.sh" +# shellcheck source=bin/lib/plan.sh +. "$ROOT/bin/lib/plan.sh" +# shellcheck source=bin/lib/stack.sh +. "$ROOT/bin/lib/stack.sh" +# shellcheck source=bin/lib/checks.sh +. "$ROOT/bin/lib/checks.sh" # Everything below leans on these; fail fast with a clear message instead of mid-command. for _dep in curl jq "$TOFU" ansible-playbook ssh; do command -v "$_dep" >/dev/null 2>&1 || die "required tool '$_dep' not found in PATH." done -require_token() { [ -n "${UPCLOUD_TOKEN:-}" ] || die "UPCLOUD_TOKEN not set (export it or put it in .env)."; } -tofu_out() { "$TOFU" -chdir="$EPHEMERAL_DIR" output -raw "$1" 2>/dev/null; } - -# Init a stack, skipping when already initialized (avoids the flaky public registry on every run) -# and retrying when it must hit the network. Force a re-init with TOFU_REINIT=1. -tofu_init() { - local dir="$1" i - if [ -z "${TOFU_REINIT:-}" ] && [ -d "$dir/.terraform" ] && [ -f "$dir/.terraform.lock.hcl" ]; then - return 0 - fi - for i in 1 2 3; do - "$TOFU" -chdir="$dir" init -input=false && return 0 - warn "tofu init failed (attempt $i/3) — registry.opentofu.org flaky? retrying in 10s..." - sleep 10 - done - die "tofu init failed for $dir after 3 attempts (provider registry unreachable?)." -} - -# Build operator-allowlist -var args from the environment, used by BOTH up and down so neither -# depends on the api.ipify.org autodetect when OPERATOR_IP/OPERATOR_CIDRS is set. -OPERATOR_TF_ARGS=() -build_operator_tf_args() { - OPERATOR_TF_ARGS=() - if [ -n "${OPERATOR_CIDRS:-}" ]; then - local cidr_json - cidr_json="$(printf '%s' "$OPERATOR_CIDRS" | awk -F, '{printf "["; for(i=1;i<=NF;i++){gsub(/^[ \t]+|[ \t]+$/,"",$i); printf "%s\"%s\"",(i>1?",":""),$i}; printf "]"}')" - OPERATOR_TF_ARGS+=(-var "operator_cidrs=$cidr_json") - elif [ -n "${OPERATOR_IP:-}" ]; then - OPERATOR_TF_ARGS+=(-var "operator_ip=$OPERATOR_IP") - fi -} - -# Refresh ONLY the firewall allowlist for the current operator IP. A server can sit stopped for a -# while; if your egress IP rotated meanwhile, a plain start would resume behind a stale allowlist -# and lock you out (SSH/443 dropped). We -target the firewall resource, but it references the server -# (server_id), so a -target apply pulls the server into the plan too. To avoid an UNINTENDED RESIZE -# back to the default plan, we read the server's CURRENT plan from state and pass it as -var plan=. -# The plan of the server currently in tofu state ("" if none) — used to avoid unintended resizes. -state_plan() { - "$TOFU" -chdir="$EPHEMERAL_DIR" show -json 2>/dev/null \ - | jq -r '.values.root_module.resources[]? | select(.address=="upcloud_server.gpu") | .values.plan // empty' 2>/dev/null | head -1 -} - -refresh_firewall() { - tofu_init "$EPHEMERAL_DIR" - build_operator_tf_args - local fw_args=() - fw_args+=(${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"}) - local cur_plan; cur_plan="$(state_plan)" - if [ -z "$cur_plan" ] || [ "$cur_plan" = "null" ]; then - warn "could not read the server's current plan from state — skipping firewall refresh to avoid an unintended resize." - warn "if your IP changed, run a full 'bin/spin up' (or set OPERATOR_CIDRS) to refresh the allowlist." - return 0 - fi - fw_args+=(-var "plan=$cur_plan") - log "Refreshing firewall allowlist for your current IP (plan=$cur_plan, no resize)..." - "$TOFU" -chdir="$EPHEMERAL_DIR" apply -auto-approve ${fw_args[@]+"${fw_args[@]}"} -target=upcloud_firewall_rules.gpu -} - -# --- UpCloud API helpers (for stop/start/resume) --- -api() { # api METHOD /path [json-body] (bounded: a hung API call must not hang the CLI) - local method="$1" path="$2" body="${3:-}" - if [ -n "$body" ]; then - curl -fsS --max-time 30 --retry 2 -X "$method" -H "Authorization: Bearer ${UPCLOUD_TOKEN:-}" -H "Content-Type: application/json" \ - --data-binary "$body" "https://api.upcloud.com/1.3${path}" - else - curl -fsS --max-time 30 --retry 2 -X "$method" -H "Authorization: Bearer ${UPCLOUD_TOKEN:-}" "https://api.upcloud.com/1.3${path}" - fi -} -server_state() { api GET "/server/$1" 2>/dev/null | jq -r '.server.state // empty'; } - -wait_ssh() { - local host="$1" i - [ -n "$host" ] || die "no ssh_host available." - log "Waiting for SSH on $host ..." - for i in $(seq 1 120); do - ssh "${SSH_OPTS[@]}" -o BatchMode=yes -o ConnectTimeout=5 "root@$host" true 2>/dev/null && return 0 - [ "$i" -eq 120 ] && die "timed out waiting for SSH after ~10 min." - sleep 5 - done -} - usage() { cat <<'EOF' inference-gpu-spinner — on-demand Cloud GPU + vLLM @@ -122,9 +50,18 @@ Usage: bin/spin [options] loses cached weights + the stable IP/cert. Requires --yes. up [--model M] [--plan P] [--swap-profile NAME] [--shutdown-at HH:MM] [--shutdown-tz TZ] [--no-shutdown] Provision the GPU server, then deploy vLLM via Ansible. + --plan is OPTIONAL: without it, the tier comes from the profile's min_plan + (or the preset's swap_plan), so each model lands on the plan it is meant + for. Pass --plan to override. On an existing server the live plan wins, and + a mismatch with the profile's is an error rather than a silent resize. --swap-profile NAME serves several models on one box via llama-swap (one in VRAM at a time); presets live in ansible/swap-profiles/ (see swap-profiles). - --allow-unvalidated lets a requires_review profile deploy (needs a big --plan). + --allow-expensive deploys a plan above MAX_EUR_PER_HOUR (default 10). + --allow-unvalidated overrides the HARDWARE preflights (VRAM, compute + capability, known-bad GPU list, driver, disk). It is not a cost flag. + -e k=v passes an Ansible extra-var through (repeatable). Extra vars beat + the profile, so this deploys the same model at a different setting without + editing it: -e max_model_len=262144, -e vllm_image_override=. Safety: auto power-off defaults ON at 21:00 Europe/Zurich (UpCloud only, where a stopped server isn't billed). --no-shutdown opts out. down [--keep-disk] Destroy the GPU server (the cost saver). By DEFAULT it also decommissions a @@ -138,443 +75,43 @@ Usage: bin/spin [options] ssh [args...] SSH into the GPU box. plan tofu plan for the ephemeral stack. swap-profiles List the multi-model swap presets (ansible/swap-profiles/). + prefetch ... Download those profiles' weights AND container images onto the persistent + disk using a cheap CPU box (~EUR 0.045/h), then destroy it. A download is + billed at whatever compute holds the disk and is not GPU work: ~354 GB + costs ~EUR 5 on 4x RTX PRO 6000 and ~EUR 0.04 here. Run it before + `up` so the GPU box starts warm. Requires no GPU server to exist (a disk + attaches to one server at a time). validate Capture exact validation evidence (context/KV/VRAM/test gen) for docs/validation.md. - soak [--rounds N] [--concurrency M] [--model NAME] - Answer-quality + concurrent-load soak of the deployed model. + soak [--rounds N] [--concurrency M] [--context T] [--model NAME] + Answer-quality soak: temp-0 factual battery, fabrication traps, then a + graded long-context load — each of M parallel requests gets its own + ~T-token haystack with a unique needle at a known depth and is graded on + retrieving it. Default 3 rounds x 20 parallel x 32768 tokens. A wrong + answer from an admitted request fails the command; rejections and timeouts + are reported but do not (admission control, not correctness). T is clamped + to the model's max_model_len. Prompts cross the public endpoint: ~2.5 MB + per round at 32768, ~10 MB at 131072. + ceiling [--model NAME] [--start N] [--max N] [--step K] [--keep-going] + Find the longest SINGLE request the deployed model survives, so + max_model_len comes from a measurement rather than the recipe. Probes one + request at a time, ascending (a crash costs a full engine reload, so + bisecting is the expensive way), and judges each result on the container's + restart counter — a crashed engine is restarted by compose and `soak` + would file it as a mere rejection. Nothing can exceed the DEPLOYED + max_model_len, so redeploy high first: + bin/spin up --model M --plan -e max_model_len= + Report the pass/fail boundary in prompt_tokens, never the requested value. help This message. Env (see .env.example): UPCLOUD_TOKEN, ACME_EMAIL, VLLM_API_KEY, [OPERATOR_IP|OPERATOR_CIDRS], + [MAX_EUR_PER_HOUR=10] (refuse a costlier plan unless --allow-expensive; prices from tests/plans.txt) + [WEIGHTS_DISK=primary|alt] (alt = the optional second data disk; see terraform/persistent) [HF_TOKEN], [AUTO_SHUTDOWN=true|false], [AUTO_SHUTDOWN_AT=HH:MM], [AUTO_SHUTDOWN_TZ=Area/City], [WEIGHTS_SIZE_GB=NNN], [WEIGHTS_TIER=maxiops|standard] (persistent disk size/tier = standing cost) [DECOMMISSION_ON_DOWN=auto|never], [DECOMMISSION_THRESHOLD_GB=NNN] (auto-free a big disk on down) EOF } -cmd_persistent_init() { - require_token - # Disk size/tier are the standing-cost knobs — let the operator set them in .env instead of - # remembering tofu -var syntax. Explicit -var passed after (in "$@") still wins. See .env.example. - local pv=() - [ -n "${WEIGHTS_SIZE_GB:-}" ] && pv+=(-var "weights_size_gb=$WEIGHTS_SIZE_GB") - [ -n "${WEIGHTS_TIER:-}" ] && pv+=(-var "weights_tier=$WEIGHTS_TIER") - - # Warn before a size change: growing the disk raises the 24/7 bill, and UpCloud CANNOT shrink a - # disk (only decommission + re-init smaller). Compare the requested size to the live disk. - local want="${WEIGHTS_SIZE_GB:-500}" diskid cur - diskid="$("$TOFU" -chdir="$PERSIST_DIR" output -raw weights_storage_id 2>/dev/null || true)" - if [ -n "$diskid" ] && printf '%s' "$want" | grep -qE '^[0-9]+$'; then - cur="$(api GET "/storage/$diskid" 2>/dev/null | jq -r '.storage.size // empty')" - if printf '%s' "$cur" | grep -qE '^[0-9]+$'; then - local rate=226; [ "${WEIGHTS_TIER:-maxiops}" = "standard" ] && rate=86 - if [ "$want" -gt "$cur" ]; then - warn "weights disk will GROW ${cur}→${want} GB (≈ +€$(( (want-cur)*rate/1000 ))/mo at ${WEIGHTS_TIER:-maxiops}). Disks cannot shrink later." - elif [ "$want" -lt "$cur" ]; then - warn "requested ${want} GB < current ${cur} GB — UpCloud cannot SHRINK a disk, so apply will fail." - warn "to reduce: 'bin/spin persistent-destroy' then re-init at the smaller size (re-downloads weights)." - fi - fi - fi - - tofu_init "$PERSIST_DIR" - "$TOFU" -chdir="$PERSIST_DIR" apply ${pv[@]+"${pv[@]}"} "$@" - log "Persistent stack ready. Floating IP: $("$TOFU" -chdir="$PERSIST_DIR" output -raw floating_ip 2>/dev/null || echo '?')" -} - -# Decommission the persistent stack → €0 standing cost. prevent_destroy (main.tf) blocks a plain -# `tofu destroy` (and must be a literal, so it can't be toggled by a var), so this deletes the disk + -# releases the IP via the API, then drops them from tofu state (the stack stays declarable — a later -# persistent-init recreates both fresh). IRREVERSIBLE: loses cached weights + the stable IP/cert. -cmd_persistent_destroy() { - require_token - local yes=""; [ "${1:-}" = "--yes" ] && yes=1 - local srv; srv="$(tofu_out server_uuid 2>/dev/null || true)" - # tofu prints a "no outputs found" warning (not a UUID) once the server is down — only a real UUID - # means a server still exists; anything else (warning text / empty) is treated as "no server". - printf '%s' "$srv" | grep -qE '^[0-9a-fA-F-]{36}$' || srv="" - [ -z "$srv" ] || die "an ephemeral GPU server still exists ($srv) — run 'bin/spin down' first." - local diskid ip - diskid="$("$TOFU" -chdir="$PERSIST_DIR" output -raw weights_storage_id 2>/dev/null || true)" - ip="$("$TOFU" -chdir="$PERSIST_DIR" output -raw floating_ip 2>/dev/null || true)" - [ -n "$diskid$ip" ] || die "no persistent resources in state — nothing to decommission." - warn "PERSISTENT DECOMMISSION — IRREVERSIBLE. This will:" - printf ' \033[1;33m•\033[0m DELETE weights disk %s (cached models + Caddy TLS cert)\n' "${diskid:-?}" >&2 - printf ' \033[1;33m•\033[0m RELEASE floating IP %s (the stable .sslip.io hostname is lost)\n' "${ip:-?}" >&2 - printf ' Next spin-up re-downloads ALL weights and gets a NEW IP + cert. Standing cost → €0.\n' >&2 - [ -n "$yes" ] || die "refusing without confirmation. Re-run: bin/spin persistent-destroy --yes" - if [ -n "$diskid" ]; then - if api DELETE "/storage/$diskid" >/dev/null 2>&1; then log "deleted weights disk $diskid" - else warn "could not delete disk $diskid (already gone / still attached?)"; fi - fi - if [ -n "$ip" ]; then - if api DELETE "/ip_address/$ip" >/dev/null 2>&1; then log "released floating IP $ip" - else warn "could not release IP $ip (already gone?)"; fi - fi - "$TOFU" -chdir="$PERSIST_DIR" state rm upcloud_storage.weights upcloud_floating_ip_address.vip >/dev/null 2>&1 || true - log "Persistent stack decommissioned. Standing cost → €0. Re-create anytime with 'bin/spin persistent-init'." -} - -cmd_up() { - require_token - local model="${VLLM_MODEL:-qwen36}" plan="" flags_explicit="" swap_profile="" - local tf_args=() ans_args=() - local sd_at="${AUTO_SHUTDOWN_AT:-}" sd_tz="${AUTO_SHUTDOWN_TZ:-}" sd_enabled="${AUTO_SHUTDOWN:-}" - while [ $# -gt 0 ]; do - case "$1" in - --model) model="$2"; flags_explicit=1; shift 2 ;; - --plan) plan="$2"; flags_explicit=1; shift 2 ;; - --swap-profile) swap_profile="$2"; flags_explicit=1; shift 2 ;; - --allow-unvalidated) ans_args+=(-e allow_unvalidated_model=true); shift ;; - --shutdown-at) sd_at="$2"; shift 2 ;; - --shutdown-tz) sd_tz="$2"; shift 2 ;; - --no-shutdown) sd_enabled="false"; shift ;; - --shutdown) sd_enabled="true"; shift ;; - *) die "unknown option: $1" ;; - esac - done - - # Multi-model swap: turn --swap-profile into the ansible vars that enable llama-swap. The - # preset (ansible/swap-profiles/.yml) is loaded whole via -e @file (Ansible reads its - # swap_profiles list). Validate the name here so a typo fails before any tofu/GPU spend. - local swap_file="" - if [ -n "$swap_profile" ]; then - swap_file="ansible/swap-profiles/${swap_profile}.yml" - [ -f "$swap_file" ] || die "unknown swap profile '$swap_profile'. See: bin/spin swap-profiles" - ans_args+=(-e multi_model_enabled=true -e "@$swap_file") - fi - - # Resume an existing STOPPED server (e.g. after the auto-off safety) instead of recreating — - # the image + model are already on the intact disks, so this is fast. Run `bin/spin down` first - # if you want a fresh box. - local existing; existing="$(tofu_out server_uuid 2>/dev/null || true)" - if [ -n "$existing" ] && [ "$(server_state "$existing")" = "stopped" ]; then - # Resume can't change the model or plan — refuse rather than silently ignore explicit flags. - [ -z "$flags_explicit" ] || die "a STOPPED server exists; resume can't apply --model/--plan. - Either 'bin/spin start' (resume as-is), or 'bin/spin down' first for a fresh box with new flags." - log "Existing server $existing is stopped — resuming (no rebuild)..." - refresh_firewall - api POST "/server/$existing/start" >/dev/null - wait_ssh "$(tofu_out ssh_host)" - log "Resumed. Endpoint: $(tofu_out endpoint) (vLLM auto-starts on boot; give it ~a minute)" - return 0 - fi - - [ -n "${ACME_EMAIL:-}" ] || die "ACME_EMAIL not set (needed for Let's Encrypt)." - [ -n "${VLLM_API_KEY:-}" ] || die "VLLM_API_KEY not set (vLLM auth)." - # Guard against an unintended resize: with no --plan, a re-run on an EXISTING server would plan - # the config default against the live box (auto-approve!). Inherit the current plan from state. - if [ -z "$plan" ] && [ -n "$existing" ]; then - plan="$(state_plan)" - [ -n "$plan" ] && log "No --plan given; keeping the existing server's plan ($plan)." - fi - [ -n "$plan" ] && tf_args+=(-var "plan=$plan") - build_operator_tf_args - tf_args+=(${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"}) - [ -n "$sd_enabled" ] && ans_args+=(-e "auto_shutdown_enabled=$sd_enabled") - [ -n "$sd_at" ] && ans_args+=(-e "auto_shutdown_time=$sd_at") - [ -n "$sd_tz" ] && ans_args+=(-e "auto_shutdown_timezone=$sd_tz") - - # If a previous `down` auto-decommissioned the persistent disk, recreate it first — the server - # can't attach a weights disk that no longer exists. NOTE: the weights_storage_id output can be - # STALE after a decommission (`state rm` doesn't recompute outputs), so verify against the API, - # not just the output value. - local wid disk_ok=""; wid="$("$TOFU" -chdir="$PERSIST_DIR" output -raw weights_storage_id 2>/dev/null || true)" - if printf '%s' "$wid" | grep -qE '^[0-9a-fA-F-]{36}$'; then - api GET "/storage/$wid" >/dev/null 2>&1 && disk_ok=1 - fi - if [ -z "$disk_ok" ]; then - log "Persistent weights disk missing (decommissioned on a previous down?) — recreating it first..." - cmd_persistent_init - fi - - log "Provisioning GPU server (model=$model)..." - tofu_init "$EPHEMERAL_DIR" - local apply_log; apply_log="$(mktemp)" - trap 'rm -f "$apply_log"' RETURN - if ! "$TOFU" -chdir="$EPHEMERAL_DIR" apply -auto-approve ${tf_args[@]+"${tf_args[@]}"} 2>&1 | tee "$apply_log"; then - if grep -qiE 'SERVER_RESOURCES_UNAVAILABLE|resources?[ _]unavailable|out of stock|no (host|capacity)' "$apply_log"; then - rm -f "$apply_log" - die "UpCloud has no '${plan:-GPU-8xCPU-64GB-1xL40S}' capacity in fi-hel2 right now. - Retry shortly, or try another tier: - bin/spin up --model $model --plan GPU-12xCPU-240GB-1xH100 # or ...-1xB200" - fi - rm -f "$apply_log" - die "tofu apply failed (see output above)." - fi - rm -f "$apply_log" - - local host sslip endpoint - host="$(tofu_out ssh_host)"; sslip="$(tofu_out sslip_host)"; endpoint="$(tofu_out endpoint)" - [ -n "$host" ] || die "could not read ssh_host from tofu outputs." - - log "Writing inventory ($host)..." - mkdir -p "$(dirname "$INVENTORY")" - printf '[gpu]\n%s ansible_user=root\n' "$host" > "$INVENTORY" - - wait_ssh "$host" - - log "Configuring with Ansible (vLLM + Caddy)..." - ansible-playbook ansible/site.yml -e "vllm_model=$model" -e "sslip_host=$sslip" ${ans_args[@]+"${ans_args[@]}"} - - log "Up. Endpoint: $endpoint" - # The API expects the served model name (what /v1/models lists). In swap mode that's a preset - # entry name (first one listed); in single-model mode it's the profile's served_name. - local served - if [ -n "$swap_file" ]; then - served="$(awk '/^swap_profiles:/{f=1;next} f&&/^[[:space:]]*-[[:space:]]/{sub(/^[[:space:]]*-[[:space:]]*/,"");sub(/[[:space:]]*#.*/,"");print;exit} f&&/^[^[:space:]#-]/{exit}' "$swap_file" 2>/dev/null)" - log "Multi-model swap active (preset '$swap_profile'). Models load on demand; run 'bin/spin swap-profiles' to see the menu." - else - served="$(awk -F'"' '/^served_name:/{print $2; exit}' "ansible/models/${model}.yml" 2>/dev/null)" - fi - cat </dev/null || true)" - printf '%s' "$diskid" | grep -qE '^[0-9a-fA-F-]{36}$' || return 0 # no disk in state — nothing to do - size="$(api GET "/storage/$diskid" 2>/dev/null | jq -r '.storage.size // empty')" - printf '%s' "$size" | grep -qE '^[0-9]+$' || { warn "could not read disk size — leaving the persistent disk in place."; return 0; } - if [ "$size" -lt "$thr" ]; then - log "Persistent disk ${size} GB (< ${thr} GB threshold) kept — minor standing cost. (DECOMMISSION_THRESHOLD_GB=$thr)" - return 0 - fi - local rate=226; [ "${WEIGHTS_TIER:-maxiops}" = "standard" ] && rate=86 - warn "AUTO-DECOMMISSION: weights disk is ${size} GB (>= ${thr} GB) — deleting it to stop ~EUR $(( size*rate/1000 ))/mo standing cost." - warn " Floating IP is KEPT (stable hostname). Next 'bin/spin up' recreates the disk, re-downloads weights + re-issues the TLS cert (~1 min)." - warn " Keep it warm instead: 'bin/spin down --keep-disk', or set DECOMMISSION_ON_DOWN=never / raise DECOMMISSION_THRESHOLD_GB." - if api DELETE "/storage/$diskid" >/dev/null 2>&1; then - "$TOFU" -chdir="$PERSIST_DIR" state rm upcloud_storage.weights >/dev/null 2>&1 || true - log "Weights disk deleted. Standing cost now ~EUR 3.5/mo (floating IP only)." - else - warn "could not delete weights disk $diskid (still attached? already gone?) — check manually." - fi -} - -cmd_down() { - require_token - local keep_disk="" dargs=() - while [ $# -gt 0 ]; do - case "$1" in - --keep-disk) keep_disk=1; shift ;; - *) dargs+=("$1"); shift ;; - esac - done - log "Destroying GPU server..." - build_operator_tf_args # so destroy doesn't trip on the ipify autodetect data source - "$TOFU" -chdir="$EPHEMERAL_DIR" destroy -auto-approve ${OPERATOR_TF_ARGS[@]+"${OPERATOR_TF_ARGS[@]}"} ${dargs[@]+"${dargs[@]}"} - rm -f "$INVENTORY" # stale inventory would point later ansible runs at a recycled IP - log "Down. The GPU is no longer billing." - down_decommission_disk "$keep_disk" -} - -cmd_start() { - require_token - local uuid; uuid="$(tofu_out server_uuid)" - [ -n "$uuid" ] || die "no server in state — use 'bin/spin up' to create one." - local st; st="$(server_state "$uuid")" - case "$st" in - started) log "Server $uuid is already running." ;; - stopped) log "Starting server $uuid..."; refresh_firewall; api POST "/server/$uuid/start" >/dev/null ;; - "") die "could not read server state (API/token reachable?)." ;; - *) die "server state is '$st' — not startable." ;; - esac - wait_ssh "$(tofu_out ssh_host)" - log "Up. Endpoint: $(tofu_out endpoint) (vLLM auto-starts on boot)" -} - -cmd_stop() { - require_token - local uuid; uuid="$(tofu_out server_uuid)" - [ -n "$uuid" ] || die "no server in state." - log "Powering off $uuid (GPU billing stops; disk + IP + on-box config are kept)..." - api POST "/server/$uuid/stop" '{"stop_server":{"stop_type":"soft"}}' >/dev/null - log "Stop requested. Resume with 'bin/spin start'; tear down with 'bin/spin down'." -} - -cmd_status() { - "$TOFU" -chdir="$EPHEMERAL_DIR" output 2>/dev/null || warn "no ephemeral state (server is down?)" - local host; host="$(tofu_out ssh_host || true)" - if [ -n "${host:-}" ]; then - # Report both possible model units: gpu-spinner (single-model compose) and gpu-swap - # (llama-swap multi-model). Either may be the active one depending on how it was deployed. - ssh "${SSH_OPTS[@]}" "root@$host" \ - 'nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total --format=csv 2>/dev/null || echo "nvidia-smi unavailable" - for u in gpu-spinner gpu-swap; do printf "%-12s %s\n" "$u:" "$(systemctl is-active "$u" 2>/dev/null || true)"; done' 2>/dev/null || true - fi -} - -cmd_logs() { - local host; host="$(tofu_out ssh_host)" || die "server is down." - # Follow whichever model layer is running: llama-swap (gpu-swap) in multi-model mode, else the - # single-model docker compose stack. - ssh "${SSH_OPTS[@]}" "root@$host" ' - if systemctl is-active --quiet gpu-swap; then - echo "== llama-swap (multi-model) journal — backends start/stop on demand ==" >&2 - journalctl -u gpu-swap -n 100 -f - else - cd '"$COMPOSE_DIR"' && docker compose logs -f --tail=100 - fi' -} - -cmd_ssh() { - local host; host="$(tofu_out ssh_host)" || die "server is down." - exec ssh "${SSH_OPTS[@]}" "root@$host" "$@" -} - -cmd_plan() { "$TOFU" -chdir="$EPHEMERAL_DIR" plan "$@"; } - -# List the multi-model swap presets and the profiles each serves, so a user can pick one for -# `bin/spin up --swap-profile `. Reads the `- name` lines under swap_profiles: in each preset. -cmd_swap_profiles() { - local dir="ansible/swap-profiles" - [ -d "$dir" ] || die "no swap-profiles directory ($dir)." - local found="" - log "Multi-model swap presets ($dir) — deploy with: bin/spin up --swap-profile --plan " - for f in "$dir"/*.yml; do - [ -e "$f" ] || break - found=1 - printf '\n \033[1m%s\033[0m\n' "$(basename "$f" .yml)" - # models are the '- ' list items; comment lines describe the tier - grep -E '^\s*-\s' "$f" | sed -E 's/^\s*-\s*/ - /' - done - [ -n "$found" ] || warn "no presets found in $dir." -} - -# Capture exact, reproducible validation evidence for the currently-deployed (single-model) box: -# served model + max_model_len, vLLM's reported KV-cache size + max concurrency, VRAM, and a test -# generation. Paste the result into docs/validation.md so the matrix stays exact, not eyeballed. -cmd_validate() { - local host ep - host="$(tofu_out ssh_host)" || die "server is down." - ep="$(tofu_out endpoint)" - [ -n "${VLLM_API_KEY:-}" ] || die "VLLM_API_KEY not set (needed to query the model)." - log "Validation evidence for $ep:" - local models served - models="$(curl -fsS --max-time 25 "$ep/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" 2>/dev/null)" \ - || die "could not reach $ep/v1/models (still loading, or firewall/key?)." - printf '%s\n' "$models" | jq -r '.data[] | " served=\(.id) max_model_len=\(.max_model_len)"' - served="$(printf '%s\n' "$models" | jq -r '.data[0].id')" - echo " --- vLLM KV cache + GPU (from the box) ---" - ssh "${SSH_OPTS[@]}" "root@$host" \ - 'docker logs '"$VLLM_CONTAINER"' 2>&1 | grep -iE "GPU KV cache size|Maximum concurrency" | tail -2; nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv,noheader' 2>/dev/null \ - | sed 's/^/ /' - echo " --- test generation ---" - # enable_thinking=false: thinking models (Qwen) otherwise spend the whole budget on reasoning and - # return empty content; templates that don't know the kwarg simply ignore it. - curl -fsS --max-time 60 "$ep/v1/chat/completions" -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \ - -d "{\"model\":\"$served\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: validation ok\"}],\"max_tokens\":32,\"temperature\":0,\"chat_template_kwargs\":{\"enable_thinking\":false}}" 2>/dev/null \ - | jq -r '.choices[0].message.content // .choices[0].message.reasoning_content // .error.message // "(no response)"' | sed 's/^/ /' - echo " (now record this row in docs/validation.md)" -} - -# Soak + answer-quality check for the currently-deployed model (single-model, or a llama-swap -# backend by --model NAME). Reusable version of the docs/validation.md procedure: -# 1) a temp-0 factual battery (auto-graded by substring) + fabrication traps (soft-checked), -# 2) a concurrent-load round (does the box stay healthy + keep serving under parallel requests). -# Usage: bin/spin soak [--model NAME] [--rounds N] [--concurrency M] -cmd_soak() { - local ep model="" rounds=3 conc=6 - while [ $# -gt 0 ]; do - case "$1" in - --model) model="$2"; shift 2 ;; - --rounds) rounds="$2"; shift 2 ;; - --concurrency) conc="$2"; shift 2 ;; - *) die "soak: unknown option '$1'" ;; - esac - done - ep="$(tofu_out endpoint)" || die "server is down." - [ -n "${VLLM_API_KEY:-}" ] || die "VLLM_API_KEY not set (needed to query the model)." - [ -n "$model" ] || model="$(curl -fsS --max-time 25 "$ep/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" 2>/dev/null | jq -r '.data[0].id // empty')" - [ -n "$model" ] || die "could not resolve served model from $ep/v1/models (still loading?)." - log "Soak: model=$model endpoint=$ep" - - # Ask one prompt at temp 0 with thinking suppressed; print the assistant text. Args: prompt [max_tokens] - _soak_ask() { - curl -fsS --max-time 90 "$ep/v1/chat/completions" -H "Authorization: Bearer $VLLM_API_KEY" \ - -H 'Content-Type: application/json' \ - -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":$(printf '%s' "$1" | jq -Rs .)}],\"max_tokens\":${2:-48},\"temperature\":0,\"chat_template_kwargs\":{\"enable_thinking\":false}}" 2>/dev/null \ - | jq -r '.choices[0].message.content // .choices[0].message.reasoning_content // ""' - } - - # 1) Factual battery — each entry is "prompt|=|answer-substring" (grep -i, no false-match traps). - echo " --- factual battery (temp 0, auto-graded) ---" - local battery=( - "What is the capital of Australia? Answer with one word.|=|canberra" - "What is 47 multiplied by 89? Reply with just the number.|=|4183" - "In what year did the Berlin Wall fall? Reply with just the year.|=|1989" - "Who wrote 'One Hundred Years of Solitude'? Reply with the author's full name.|=|rquez" - "Who is the first-listed author of 'Attention Is All You Need'? Reply with one surname.|=|vaswani" - "What is the largest planet in our solar system? Answer with one word.|=|jupiter" - ) - local pass=0 total=0 qa q expect got - for qa in "${battery[@]}"; do - q="${qa%%|=|*}"; expect="${qa##*|=|}"; total=$((total+1)) - got="$(_soak_ask "$q" 40)" - if printf '%s' "$got" | grep -qi "$expect"; then - pass=$((pass+1)); printf ' \033[32m✓\033[0m %s\n' "$q" - else - printf ' \033[31m✗\033[0m %s → got: %s\n' "$q" "$(printf '%s' "$got" | tr '\n' ' ' | head -c 90)" - fi - done - echo " factual: $pass/$total correct" - - # Fabrication traps — soft-checked: pass if the reply signals uncertainty rather than inventing. - echo " --- fabrication traps (should decline / express uncertainty) ---" - local traps=( - "Summarize the plot of the 1987 novel 'The Crimson Meridian' by Eleanor Vance." - "What did Marie Curie say about the 2024 Paris Olympics?" - ) - local t ans - for t in "${traps[@]}"; do - ans="$(_soak_ask "$t" 120)" - if printf '%s' "$ans" | grep -qiE "cannot|can't|could ?n't|no record|not aware|unable|don't have|do not have|does ?n't exist|does not exist|fiction|no (information|evidence|widely|known|such)|i'm not|unaware|there (is|are|was) no|did ?n't (say|exist|write)|did not (say|exist|write)|passed away|misunderstand|not (a )?(real|recognized|find)|couldn't find|could not find"; then - printf ' \033[32m✓ declined\033[0m: %s\n' "$(printf '%s' "$ans" | tr '\n' ' ' | head -c 120)" - else - printf ' \033[33m⚠ check\033[0m: %s\n' "$(printf '%s' "$ans" | tr '\n' ' ' | head -c 120)" - fi - done - - # 2) Concurrent load — fire conc parallel requests per round; count how many return content. - echo " --- concurrent load: $rounds rounds x $conc parallel ---" - local tmp; tmp="$(mktemp -d)" - local r i - for r in $(seq 1 "$rounds"); do - for i in $(seq 1 "$conc"); do - ( if [ -n "$(_soak_ask "Write two short sentences about the number $((r*100+i)).")" ]; then echo ok; else echo fail; fi ) > "$tmp/$r-$i" & - done - wait - done - local ok reqs; ok="$(cat "$tmp"/* 2>/dev/null | grep -c '^ok$')"; reqs=$((rounds*conc)); rm -rf "$tmp" - echo " load: $ok/$reqs requests returned content (rejections under load are expected — admission control)" - - if curl -fsS --max-time 20 "$ep/v1/models" -H "Authorization: Bearer $VLLM_API_KEY" >/dev/null 2>&1; then - log "Soak OK — endpoint healthy after load. factual $pass/$total, load $ok/$reqs." - else - die "Soak: endpoint NOT healthy after load — investigate (docker logs / nvidia-smi via bin/spin ssh)." - fi -} - case "${1:-help}" in persistent-init) shift; cmd_persistent_init "$@" ;; persistent-destroy) shift; cmd_persistent_destroy "$@" ;; @@ -586,9 +123,11 @@ case "${1:-help}" in logs) shift; cmd_logs "$@" ;; ssh) shift; cmd_ssh "$@" ;; plan) shift; cmd_plan "$@" ;; + prefetch) shift; cmd_prefetch "$@" ;; swap-profiles) shift; cmd_swap_profiles "$@" ;; validate) shift; cmd_validate "$@" ;; soak) shift; cmd_soak "$@" ;; + ceiling) shift; cmd_ceiling "$@" ;; help|-h|--help) usage ;; *) usage; exit 1 ;; esac diff --git a/docs/getting-started.md b/docs/getting-started.md index 90e1770..bb22829 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -22,9 +22,12 @@ You need: - An UpCloud account with GPU plans enabled. GPUs exist only in the `fi-hel2` zone. - An UpCloud API token, created in the control panel under People → API tokens. +- **Python 3.11, 3.12 or 3.13** on the control machine. Versions outside that range fail; macOS + ships 3.9 as `/usr/bin/python3`, which is the common cause. See step 1. - A machine to drive it from, with `tofu` (OpenTofu 1.7 or newer), `ansible-playbook`, `curl`, `jq` and `ssh` on `PATH`. `bin/spin` checks all five at startup and stops with a clear error if - one is missing. `upctl`, the UpCloud CLI, is optional but makes one setup step much easier. + one is missing. `upctl`, the UpCloud CLI, is optional — step 3 shows how to do the one thing + it is useful for without it. Understand the two cost buckets before you create anything: @@ -34,17 +37,37 @@ Understand the two cost buckets before you create anything: | Standing | 24 hours a day, GPU or no GPU | persistent disk and floating IP | The default L40S tier runs at roughly €1.11/hour. The default 150 GB disk plus a floating IP is -roughly €37/month and keeps billing after `bin/spin down`. Off is cheap, not free. Reaching zero -takes a deliberate `bin/spin persistent-destroy`, covered in step 7. +roughly €36/month and keeps billing after `bin/spin down`. Getting to zero takes a separate +command, `bin/spin persistent-destroy`, covered in step 7. ## Step 1 — Prepare the control machine +Check the interpreter before building the venv, because the failure otherwise arrives as a +wall of every Ansible version ever published rather than a version error: + +```bash +python3 -c 'import sys; v=sys.version_info; print(f"{v.major}.{v.minor}"); assert (3,11) <= v < (3,14), "need Python 3.11-3.13"' +``` + +If that fails, name a suitable interpreter explicitly rather than trusting `python3` — on macOS +`python3` is often the system 3.9, while Homebrew's newer ones sit alongside it: + +```bash +ls /opt/homebrew/bin/python3.1* /usr/local/bin/python3.1* 2>/dev/null +``` + +Then, substituting the version you found: + ```bash -python3 -m venv .venv && . .venv/bin/activate +python3.12 -m venv .venv && . .venv/bin/activate pip install -r requirements.txt ansible-galaxy collection install -r requirements.yml ``` +Why the range: `ansible-core` 2.18 requires 3.11 or newer, and `ansible-lint` 26 refuses to run +on 3.14 while `ansible-core` is below 2.20. Both are pinned exactly in `requirements.txt`, which +CI installs too, so a working control machine here matches a working one there. + Nothing in this step touches the cloud. ## Step 2 — Fill in `.env` @@ -65,7 +88,7 @@ Required: | `ACME_EMAIL` | Let's Encrypt contact address. Must be a real, deliverable address. | | `VLLM_API_KEY` | The bearer token your clients send. Generate one: `openssl rand -hex 32`. | -`ACME_EMAIL` is checked before deployment, not after. Let's Encrypt refuses placeholder domains, +`ACME_EMAIL` is checked before anything is provisioned. Let's Encrypt refuses placeholder domains, and Caddy would then never obtain a certificate while everything else reported success, so the deploy fails fast on an address ending in `example.com`, `example.org`, `example.net`, `.invalid`, `.local` or `.test`. Use an address you actually receive mail at. @@ -90,31 +113,119 @@ cp terraform/providers/upcloud/terraform.tfvars.example \ terraform/providers/upcloud/terraform.tfvars ``` +**Edit the copy before going further.** It lands with an `os_template` of `REPLACE_ME` and an +empty key list. Nothing in the persistent stack reads this file, so an unedited copy goes +unnoticed until the ephemeral apply. `bin/spin up` refuses to start until both are filled in. + Two values are required and have no defaults: -- `os_template` — the UpCloud public template that ships the NVIDIA driver, CUDA and Docker. Find - its exact title or UUID with `upctl storage list --public --template`. The Terraform plan refuses - to run while it is unset. +- `os_template` — the UpCloud public template that ships the NVIDIA driver, CUDA and Docker. The + Terraform plan refuses to run while it is unset. With `upctl` installed: + + ```bash + upctl storage list --public --template + ``` + + Without it, read the same list from the API using the token already in `.env`: + + ```bash + set -a; . ./.env; set +a + curl -fsS -H "Authorization: Bearer $UPCLOUD_TOKEN" \ + https://api.upcloud.com/1.3/storage/template \ + | jq -r '.storages.storage[] | select(.title|test("GPU|NVIDIA|CUDA";"i")) | "\(.uuid) \(.title)"' + ``` + + Either the UUID or the exact title works in `terraform.tfvars`. The list also contains + `UpCloud K8s ... (with NVIDIA drivers & CUDA)` entries — those are Kubernetes node images. + You want the plain `Ubuntu Server ... (with NVIDIA drivers & CUDA)` one. - `ssh_public_keys` — the keys installed for root login on the GPU box. Without one you cannot reach the server you just paid for. +**Deploying from more than one machine.** Both gates that let you reach the box are per-machine, +and both fail in ways that look like something else: + +- **The key must be in `ssh_public_keys` before the server is created.** `login { keys = ... }` in + `terraform/providers/upcloud/main.tf` is create-time metadata, so adding a key to + `terraform.tfvars` does not change a server that already exists — it applies to the next one + built. A machine whose key is missing gets `Permission denied (publickey)`, which `bin/spin` + currently cannot distinguish from a box that is still booting, so it waits the full SSH timeout + first. To add a key to a *running* box, append it to `~/.ssh/authorized_keys` there from a + machine that can already log in, and put it in `terraform.tfvars` as well so the next build + keeps it. +- **The machine's public IP must be in `OPERATOR_CIDRS`.** The firewall allowlists SSH(22) and + HTTPS(443) to those ranges only, so `bin/spin validate` and `bin/spin soak` need it too, not + just the deploy. `bin/spin up` refuses to provision when this machine's address is outside the + list, rather than creating a server it cannot reach. Set it as a **single** assignment in + `.env` — the file is sourced top-to-bottom, so a second `OPERATOR_CIDRS=` line silently wins: + + OPERATOR_CIDRS=198.51.100.0/24,203.0.113.7/32 + + Editing the rule in the provider console does not work: OpenTofu owns + `upcloud_firewall_rules.gpu` and reverts it on the next apply. `SKIP_REACHABILITY_CHECK=1` + bypasses the check for a jump-host setup you reach another way. + +**A changing egress address.** If the control machine's public IP rotates — a mobile or tethered +link, or a VPN with several exits — the allowlist goes stale mid-run and every later step fails as +a timeout rather than as a permission error. Measured on 2026-09-20: switching networks moved the +address twice inside one session. Two consequences: + +- An **in-flight deploy dies** when the address changes, because Ansible multiplexes over one SSH + connection (`ControlMaster` in `ansible.cfg`) and that session cannot survive it. The play ends + with `UNREACHABLE!` and `failed=0` — no task actually failed. Re-running `bin/spin up` resumes: + the weights are on `/data` and the vLLM container runs detached, so a dropped control connection + does not interrupt a download in progress. +- If the address rotates within a known range, allowlisting that range is steadier than chasing + single addresses. Weigh it against the exposure: SSH and HTTPS become reachable from the whole + range, on a box with key-only auth and no password login. Narrow it again once back on a stable + link. + +**Timeouts.** `bin/spin up` waits ~20 minutes for SSH; a 4-GPU tier with a large attached disk has +been measured taking over 10. Raise it with `SSH_WAIT_TRIES=360` (each try is 5s). Authentication +failures are not waited out — they fail immediately with the offending key fingerprint. + Everything else in that file is an optional override, including `plan`, which defaults to `GPU-8xCPU-64GB-1xL40S`. `terraform/persistent/terraform.tfvars` is entirely optional — every variable in that stack has a default, and `WEIGHTS_SIZE_GB` in `.env` covers the one you are likely to change. +### Sizing the weights disk + +Decide this before step 4, because UpCloud cannot shrink a disk afterwards. Two thresholds sit +close together: + +| Size | What changes | +|---|---| +| under ~110 GB | The filesystem reports under 100 GB, so Docker's data-root stays on the boot disk and the ~23 GB vLLM image is pulled again on every spin-up. | +| 110–149 GB | Image and weights both persist on `/data`, and `bin/spin down` **keeps** the disk, so the next spin-up is minutes. | +| 150 GB and above | `bin/spin down` **deletes** the disk by default (`DECOMMISSION_THRESHOLD_GB`), so the next spin-up re-downloads everything. | + +For a first run on the default `qwen36` (~27 GB of weights plus the ~23 GB image), **120** is the +useful choice — large enough to keep the image, small enough that a teardown does not throw the +cache away: + +```bash +WEIGHTS_SIZE_GB=120 +``` + +That is about €30/month while the disk exists, or €0.04/hour. Larger models need much more; each +profile in `ansible/models/` states its own figure in the header comment. + ## Step 4 — Create the persistent layer -Run once, ever: +Run this once: ```bash bin/spin persistent-init ``` +OpenTofu prints a plan and waits for you to type `yes`. Any other answer cancels the apply. The +command stops on the first failure, so a cancelled apply returns to the shell prompt with no +further output and nothing created. Confirm with `tofu -chdir=terraform/persistent output`. + This creates the weights disk and the floating IP, and prints the IP. The dashed form of that IP -becomes your permanent hostname — `.sslip.io` — which is what lets the TLS certificate -survive from one session to the next. +is your permanent hostname, `.sslip.io`. Because the hostname never changes, the TLS +certificate stored on the disk stays valid across sessions. Both resources are guarded against a stray `tofu destroy`. The standing bill starts here. @@ -124,10 +235,22 @@ Both resources are guarded against a stray `tofu destroy`. The standing bill sta bin/spin up ``` -That deploys the default profile, `qwen36`, on an L40S. To pick something else: +That deploys the default profile, `qwen36`, on an L40S. + +**Check the auto-shutdown time before a first run.** Every deploy installs a timer that powers +the box off at 21:00 `Europe/Zurich` by default, and it fires regardless of what the box is +doing — including a deployment still in progress. If you are starting in the evening, or in +another timezone, set it now rather than discovering it mid-download: + +```bash +bin/spin up --shutdown-at 23:30 --shutdown-tz Area/City +``` + +Step 7 covers the rest of that timer. To pick a different model: ```bash bin/spin up --model qwen36-35b --plan GPU-12xCPU-240GB-1xH100 +bin/spin up --model qwen36 --plan GPU-16xCPU-80GB-1xRTXPRO6000 ``` `up` runs through: create the server and firewall, rebind the floating IP, wait for SSH, then @@ -139,7 +262,7 @@ The health wait allows 15 minutes by default, and larger profiles raise their ow spin-up with the same model and a kept disk is much faster, because the weights and the container image are already on `/data`. -Two things are normal rather than broken: +Two things look like failures but are not: - **No capacity.** GPU tiers sell out. `up` recognises the refusal and suggests retrying or choosing another tier. Nothing was created, and nothing is billing. @@ -171,8 +294,13 @@ curl https://.sslip.io/v1/chat/completions \ ``` For a fuller check, `bin/spin validate` reports the served model, context length, KV cache size and -VRAM use, and runs a test generation. `bin/spin soak` grades a short factual battery and fires -parallel requests to confirm the box stays healthy under load. +VRAM use, and runs a test generation. `bin/spin soak` grades a short factual battery, checks the +model declines fabricated premises, then runs a **graded long-context load**: each of 20 parallel +requests gets its own haystack with a unique needle at a known depth and is graded on retrieving +it. A wrong answer from an admitted request fails the command; rejections and timeouts are reported +separately, as admission control rather than a correctness problem. Scale it up with `--context` +and `--rounds` — a model can pass at 32k and kill the engine at 128k. Prompts cross the public +endpoint, so the heavy tiers move real bandwidth: ~2.5 MB per round at 32768, ~10 MB at 131072. ## Step 7 — Shutting down @@ -189,23 +317,22 @@ Three levels, in increasing order of what they destroy: **It also deletes the weights disk by default.** The rule is to remove any disk at or above `DECOMMISSION_THRESHOLD_GB`, which defaults to 150 — the same as the default disk size. So with stock settings, a teardown frees the standing cost and the next spin-up re-downloads every model. -That is the right trade if you spin up rarely. If you work daily, keep the cache: +That suits occasional use. If you spin up daily, keep the cache: ```bash bin/spin down --keep-disk ``` -or set `DECOMMISSION_ON_DOWN=never` in `.env` once and forget about it. Either way the floating IP -is kept, so the hostname and certificate stay valid. +or set `DECOMMISSION_ON_DOWN=never` in `.env`, which applies to every teardown. Either way the +floating IP is kept, so the hostname and certificate stay valid. `bin/spin persistent-destroy --yes` is the only path to zero. It deletes the disk and releases the IP, refuses to run while a GPU server still exists, and refuses without `--yes`. It is irreversible: you lose the cached weights, the IP and the certificate. -There is also a safety net you did not ask for. Every deploy installs a systemd timer that powers -the box off at a fixed local time, defaulting to 21:00 `Europe/Zurich`, so a server you forget -about stops billing. It powers off, never destroys, and `bin/spin start` brings it back. Set your -own time and zone at deploy time: +Every deploy also installs a systemd timer that powers the box off at a fixed local time, +defaulting to 21:00 `Europe/Zurich`, so a forgotten server stops billing. It only powers off, and +`bin/spin start` brings it back. Set your own time and zone at deploy time: ```bash bin/spin up --shutdown-at 23:30 --shutdown-tz Area/City @@ -220,19 +347,40 @@ model that takes an hour to load, start early or turn the timer off for that run Profiles live in `ansible/models/`, one file per model and quantisation. Pick one with `--model`. The README carries the full table; the rules behind it are short: -- FP8 profiles run on L40S and H100. -- NVFP4 profiles run only on Blackwell hardware, meaning B200. They are roughly half the size and - faster, and they will not load anywhere else. -- Every profile declares the VRAM it needs. A deploy onto too small a plan fails in preflight, - before vLLM starts, rather than out-of-memorying on a GPU you are already paying for. -- A profile marked `requires_review` needs `--allow-unvalidated` and a large multi-GPU plan. +- **You usually do not pick a tier.** Each profile names the plan it is meant for, and `bin/spin up + --model ` deploys that. Pass `--plan` only to override it. On a server that already exists + the live plan wins, and a mismatch with the profile's is an error rather than a silent resize. +- FP8 profiles run on any tier from the L40S up. +- NVFP4 profiles need Blackwell. They are roughly half the size and faster, and they will not load + on an L40S or H100 at all. +- Which Blackwell tier matters. The RTX PRO 6000 (96 GB, €1.65/h) is compute capability 12.0; the + B200 (192 GB, €4.50/h) is 10.0. Dense NVFP4 runs on both. What keeps the big MoE models on B200 + is **sparse attention**, which has no working sm_120 path (vllm#55757) — and it fails late, so a + short smoke test passing is not evidence it works. +- Every profile declares the VRAM, GPU generation and GPU count it needs. A deploy onto the wrong + plan fails in preflight, before vLLM starts and before any weights download. It will not run out + of memory on a GPU you are already paying for. +- A plan costing more than `MAX_EUR_PER_HOUR` (default €10/h) needs `--allow-expensive`. The + refusal names the price, the daily cost and the cheaper spot id, and comes before anything is + provisioned. + +Each profile also names the tier it has been measured on. Running it elsewhere is allowed and +prints a warning. Treat that deployment as a validation run: capture `bin/spin validate` and +`bin/spin soak` and add the row to [validation.md](validation.md). + +H100 and B200 both run short of capacity regularly. The RTX PRO 6000 is usually available and covers +every FP8 profile and the two dense NVFP4 ones, and it is the default lane for `qwen38-flash-next`, +`glm53-flash-nvfp4` and `k2-horizon-375b`. It has no measured runs yet, so expect the untested +warning — and when you get one, capture `bin/spin validate` and `bin/spin soak` and add the row. +The H100 remains where `qwen36-35b` was measured. To serve several models from one endpoint, use a swap preset. One model sits in VRAM at a time and the rest load on demand, so size the box for the largest member of the set, not their sum: ```bash bin/spin swap-profiles -bin/spin up --swap-profile l40s --plan GPU-8xCPU-64GB-1xL40S +bin/spin up --swap-profile l40s # the preset names its own plan (swap_plan) +bin/spin up --swap-profile rtxpro6000 ``` Presets are just lists of profile names in `ansible/swap-profiles/`; add one by dropping in a file. @@ -246,7 +394,7 @@ bin/spin logs # follow vLLM and Caddy, or llama-swap in multi-model m bin/spin ssh # root shell on the box bin/spin plan # tofu plan for the ephemeral stack, changes nothing bin/spin validate # served model, context, KV cache, VRAM, test generation -bin/spin soak # answer-quality battery plus a concurrent-load round +bin/spin soak # answer-quality battery plus a graded 20-way long-context load bin/spin help # every command and flag ``` @@ -266,6 +414,10 @@ stays empty in multi-model mode. The rest of it, and all of `soak`, work either | Symptom | Cause | Fix | |---|---|---| +| `persistent-init` returns with no output at all | The OpenTofu confirmation was not answered `yes`, so the apply was cancelled and the command stopped. | Re-run it and type `yes` at the prompt. | +| `up` prints "Persistent weights disk missing — recreating it first..." then stops | Older builds called the interactive `persistent-init` from inside the non-interactive `up`. | Fixed: that path now auto-approves. Run `bin/spin persistent-init` on its own first if you are on an older checkout. | +| `terraform.tfvars still has os_template = "REPLACE_ME"` | Step 3's file was copied but not edited. | Fill in `os_template` and `ssh_public_keys`; the error prints the command that lists templates. | +| `Could not find a version that satisfies the requirement ansible` | The venv was built with a Python older than 3.11 (often macOS's `/usr/bin/python3`, 3.9). | Delete `.venv`, rebuild it naming a 3.11–3.13 interpreter (step 1). | | `required tool 'X' not found in PATH` | Missing dependency on the control machine. | Install it, or point `TOFU` at a non-standard OpenTofu binary. | | `UPCLOUD_TOKEN not set` | `.env` missing or not filled in. | Complete step 2. Run `bin/spin` from the repository root. | | Deploy fails on `ACME_EMAIL must be a REAL deliverable address` | A placeholder contact address. | Use an address you receive mail at. | @@ -274,7 +426,7 @@ stays empty in multi-model mode. The rest of it, and all of `soak`, work either | A deploy fails at `Wait for vLLM to report healthy`, but the box looks fine | A large model outlasted the health wait. | Poll `https://.sslip.io/v1/models` before tearing anything down — the model usually finishes loading. | | `404` from a chat completion | Wrong name in the `model` field. | Query `/v1/models` and use exactly what it returns. | | `needs ~N GB of VRAM but this GPU reports M GB` | Profile too large for the plan. | Use the plan named in the message, or a larger one. | -| `No space left on device` during a download | `/data` is full. It is an LRU cache, not unlimited. | Raise `WEIGHTS_SIZE_GB` and re-run `bin/spin persistent-init`, or trim a swap preset. | +| `No space left on device` during a download | `/data` is full. It is a fixed-size LRU cache. | Raise `WEIGHTS_SIZE_GB` and re-run `bin/spin persistent-init`, or trim a swap preset. | | `Unknown model profile 'X'` | Typo, or a profile that does not exist. | List `ansible/models/`. | If a server ends up in a state Terraform cannot reconcile, `bin/spin status` and the UpCloud diff --git a/docs/gpu-spinner.md b/docs/gpu-spinner.md index 756f24d..a9f65b3 100644 --- a/docs/gpu-spinner.md +++ b/docs/gpu-spinner.md @@ -2,7 +2,8 @@ On-demand GPU inference, built to be **cheap to start and cheap to stop**. The GPU bills by the hour, so creating and destroying it is the daily routine; only a small, cheap layer survives between -sessions. This doc covers the architecture, the security model, multi-model serving, and cost. +sessions. This doc covers the architecture, the security model, multi-model serving, cost, and +which GPU runs what. The live-validated model matrix is the separate system of record: [validation.md](validation.md). ## Architecture: two stacks + Ansible @@ -20,6 +21,10 @@ EPHEMERAL (terraform/providers/) — created + destroyed every session ANSIBLE (push over SSH) gpu(verify) → autostop → docker(verify) → vllm (vLLM + Caddy via docker compose, systemd) └→ llama_swap (multi-model, when enabled) + +CLI (bin/spin) — deps, .env, usage, dispatch; commands live in bin/lib/*.sh, sourced at startup + common (log/tofu/api) · net (allowlist, firewall, ssh wait) · plan (plan + tfvars resolution) + stack (up/down/start/stop/status/…) · checks (validate, soak) ``` **Two stacks, because the split maps to the billing boundary.** The GPU server is disposable @@ -64,13 +69,15 @@ time**. Clients pick a model with the OpenAI `model` field; an idle backend unlo ```bash bin/spin swap-profiles # list presets -bin/spin up --swap-profile l40s --plan GPU-8xCPU-64GB-1xL40S # qwen36 + gemma4-26b/31b -bin/spin up --swap-profile b200-nvfp4 --plan GPU-24xCPU-240GB-1xB200 # the four NVFP4 models +bin/spin up --swap-profile l40s # qwen36 + gemma4-26b/31b +bin/spin up --swap-profile b200-nvfp4 # the four NVFP4 models ``` -- **Presets** live in `ansible/swap-profiles/.yml` — just a list of model-profile names - (`swap_profiles:`). Each model's repo, context, and vLLM args come from its `ansible/models/` - profile, so a model is defined once. Add a preset by dropping in a new file. +- **Presets** live in `ansible/swap-profiles/.yml` — a list of model-profile names + (`swap_profiles:`) plus the plan to deploy on (`swap_plan:`, which `--plan` overrides). Each + model's repo, context, and vLLM args come from its `ansible/models/` profile, so a model is + defined once. Add a preset by dropping in a new file; CI checks its `swap_plan` against + [`tests/plans.txt`](../tests/plans.txt). - **On-demand swap**: the first request for a model loads it (cold: add its weight-download time; warm: seconds). Requests to a different model swap the backend — one in VRAM at a time. Size the box for the **largest** model in the set, not the sum. @@ -101,34 +108,58 @@ until you fully decommission (below). This is provider-agnostic; the numbers bel | Tier | Plan | €/h | Spot €/h | Fits | |---|---|---|---|---| -| L40S 48 GB | `GPU-8xCPU-64GB-1xL40S` | 1.11 | 0.83 | FP8 ≤ ~31B (default) | -| H100 80 GB | `GPU-12xCPU-240GB-1xH100` | 1.79 | 1.78 | FP8 MoE (qwen36-35b) | -| B200 179 GB | `GPU-24xCPU-240GB-1xB200` | 4.50 | 3.38 | all NVFP4, DeepSeek-V4-Flash | -| 4×B200 | `GPU-96xCPU-960GB-4xB200` | 18.00 | — | GLM-5.2 NVFP4 (TP=4) | -| 8×B200 | `GPU-192xCPU-1920GB-8xB200` | 36.00 | — | GLM-5.2 FP8 (TP=8) | - -The L4 (24 GB, €0.58/h) fits none of the current profiles. Spot tiers are ~25 % cheaper but -preemptible. Measured cold/warm timings and per-session costs are in [validation.md](validation.md) +| L40S 48 GB | `GPU-8xCPU-64GB-1xL40S` | 1.11 | 0.83 | FP8 ≤ ~31B (default, validated) | +| RTX PRO 6000 96 GB | `GPU-16xCPU-80GB-1xRTXPRO6000` | 1.65 | 1.24 | every FP8 profile, dense NVFP4 — untested | +| H100 80 GB | `GPU-12xCPU-240GB-1xH100` | 1.79 | 1.78 | FP8 MoE (qwen36-35b, validated) | +| B200 192 GB | `GPU-24xCPU-240GB-1xB200` | 4.50 | 3.38 | all NVFP4, single-GPU | +| 2×RTX PRO 6000 | `GPU-32xCPU-160GB-2xRTXPRO6000` | 3.30 | 2.48 | — (no profile yet; glm53-flash-nvfp4 needs 0.97 util to fit) | +| 4×RTX PRO 6000 | `GPU-64xCPU-320GB-4xRTXPRO6000` | 6.60 | 4.95 | **glm53-flash-nvfp4**, **qwen38-flash-next** | +| 4×H100 | `GPU-48xCPU-960GB-4xH100` | 7.16 | 7.15 | qwen38-flash-next (the upstream-verified tier) | +| 8×RTX PRO 6000 | `GPU-128xCPU-640GB-8xRTXPRO6000` | 13.20 | 9.90 | **k2-horizon-375b** (TP=8) | +| 4×B200 | `GPU-96xCPU-960GB-4xB200` | 18.00 | 13.50 | GLM-5.2 NVFP4 (TP=4), glm53-flash FP8, DeepSeek vision | +| 8×B200 | `GPU-192xCPU-1920GB-8xB200` | 36.00 | 27.00 | GLM-5.2 FP8 (TP=8), GLM-5.3, DeepSeek-V4-Pro | + +`bin/spin up` picks the plan from the profile's `min_plan` (or a preset's `swap_plan`) unless you +pass `--plan`, so the bold rows above are what those models deploy onto by default. The full +allowlist CI checks against is [`tests/plans.txt`](../tests/plans.txt) — keep the two in sync. + +**Spot is a plan id, not a flag.** `GPU-SPOT-` is a real identifier, so +`bin/spin up --model X --plan GPU-SPOT-64xCPU-320GB-4xRTXPRO6000` works today with no code change. +The provider can reclaim a spot server, so use it for validation runs — where losing the box costs +minutes and the weights are already on `/data` — and not for anything serving traffic. The discount +is 25% on L40S, RTX PRO 6000 and B200, and about nothing on H100. + +**These identifiers move.** On 2026-09-20 all three B200 ids in this repo were dead: UpCloud had +doubled the vCPU count in the name (`GPU-12xCPU-240GB-1xB200` → `GPU-24xCPU-240GB-1xB200`, and so +on), eleven profiles pointed at plans that no longer existed, and CI was green because it only +compared the profiles to `tests/plans.txt`, which was stale too. `tests/check-plans.sh` asks the +provider instead — run it after any plan change. + +Plan identifiers come from UpCloud's [GPU Server configurations](https://upcloud.com/docs/products/gpu-servers/configurations/); +the number before `xCPU` is **cores**, not threads. L4 is also offered (24 GB, €0.58/h) and fits +none of the current profiles. No B300 plan is published in `fi-hel2` as of 2026-09-20. + +Measured cold/warm timings and per-session costs are in [validation.md](validation.md) (“Timings & session cost”) — read those before an expensive tier. **Standing — disk + IP (billed 24/7 whether or not a GPU exists):** | Item | Rate | Example | |---|---|---| -| Persistent disk, MaxIOPS | €0.226 / GB·month | 150 GB ≈ **€34/mo**, 500 GB ≈ **€113/mo** | -| Persistent disk, standard (HDD-backed) | €0.086 / GB·month | 500 GB ≈ **€43/mo** | -| Floating IP (IPv4) | €3.51 / month | — | -| Public egress | €0.01 / GB | usually negligible | +| Persistent disk, MaxIOPS | €0.220 / GB·month | 150 GB ≈ **€33/mo**, 500 GB ≈ **€110/mo** | +| Persistent disk, standard | €0.085 / GB·month | 500 GB ≈ **€43/mo** | +| Floating IP (IPv4) | €3.47 / month | — | +| Public egress | €0.00 / GB | zero-cost egress, fair-use policy applies | -So the "off" state is **not free**: the default 150 GB MaxIOPS disk + IP is **~€37/mo** (a 500 GB -disk would be ~€116/mo). Size the disk to what you actually cache — set **`WEIGHTS_SIZE_GB`** and +So the "off" state is **not free**: the default 150 GB MaxIOPS disk + IP is **~€36/mo** (a 500 GB +disk would be ~€114/mo). Size the disk to what you actually cache — set **`WEIGHTS_SIZE_GB`** and **`WEIGHTS_TIER`** in `.env` (used by `bin/spin persistent-init`; e.g. bump to `500` for GLM-5.2, or `WEIGHTS_TIER=standard` for cheaper/slower). An existing disk grows to the new size on the next `persistent-init` (UpCloud can't shrink one — `bin/spin persistent-destroy` then re-init to go smaller). **Big-model tradeoff (e.g. GLM-5.2 NVFP4, ~377 GB weights):** keeping it on a 500 GB MaxIOPS disk -costs ~€113/mo standing, but each spin-up is warm (~18–20 min to serving). Deleting the disk between -uses drops standing cost to just the IP (~€3.5/mo) but every spin-up re-downloads ~447 GB (~1 h, +costs ~€110/mo standing, but each spin-up is warm (~18–20 min to serving). Deleting the disk between +uses drops standing cost to just the IP (~€3.47/mo) but every spin-up re-downloads ~447 GB (~1 h, plus egress). Keep-warm pays off above roughly one spin per few days; otherwise delete and re-pull. **Winding down:** @@ -149,3 +180,109 @@ plus egress). Keep-warm pays off above roughly one spin per few days; otherwise the stable IP/cert are lost, so the next spin-up re-downloads everything and gets a new IP + cert. `prevent_destroy` still guards `terraform/persistent/main.tf` against a stray `tofu destroy`, so the command (which deletes via the API, then drops the resources from state) is the supported path. + +## What runs on which GPU + +**Where RTX PRO 6000 fits.** 96 GB of GDDR7 at 1.6 TB/s, PCIe, **no NVLink**, compute capability +12.0. It is Blackwell, but a different family from the B200's 10.0, which changes what runs: + +- Every **FP8** profile runs on it with far more headroom than the 48 GB L40S, `qwen36-35b` + included. So does **dense NVFP4** (`qwen36-27b-nvfp4`, `gemma4-31b-nvfp4`), which takes the NVFP4 + scaled-mm path and has SM120 kernels. Both were measured elsewhere — H100 and B200 respectively. +- One profile is **measured** here: `glm53-flash-nvfp4`, 4× at TP=4, 2026-09-20 + ([validation.md](validation.md)). Other cc 12.0 lanes list it in + `untested_compute_capabilities`, so the deploy warns and serves. +- **Multi-GPU:** `qwen38-flash-next` and `glm53-flash-nvfp4` default to 4×, `k2-horizon-375b` to 8× + at TP=8, all over PCIe — record throughput on the first runs. If engine init hangs inside NCCL + rather than failing, `NCCL_P2P_DISABLE=1` in a profile's `extra_env` is the fallback; it costs + throughput where P2P works, so not pre-emptively. + +**Sparse attention is what blocks the rest.** + +- **DSA models have no SM120 kernel.** Every remaining large profile is a DeepSeek-style sparse MLA + model, and [vllm#55757](https://github.com/vllm-project/vllm/issues/55757) reports GLM-5.3, + GLM-5.2 and DeepSeek-V4 unservable there, reproduced on 8× RTX PRO 6000. The failure **hides**: + short prompts answer correctly and realistic lengths break. GLM-5.3-Flash is worse still — + rope-free (`qk_rope_head_dim: 0`), with no SM120 kernel at all + ([vllm#53963](https://github.com/vllm-project/vllm/issues/53963)). The fixes + ([vllm#55277](https://github.com/vllm-project/vllm/pull/55277), + [vllm#54929](https://github.com/vllm-project/vllm/pull/54929), + [vllm#41834](https://github.com/vllm-project/vllm/pull/41834)) were all open at 2026-09-19. +- **The NVFP4 MoE GEMM fault is reported fixed** on cu130 builds — the FlashInfer CUTLASS + grouped-GEMM fix from `flashinfer-ai/flashinfer#2708` is in vLLM's FlashInfer pin. Read + "NVFP4 MoE returns invalid output on cc 12.0" as a statement about the **cu129 images this repo + pins** — that is where the fault lives, and a cu130 tag leaves it behind. +- **A second fault produced much of the same symptom.** + [vllm#54189](https://github.com/vllm-project/vllm/issues/54189): `ModelOptNvFp4FusedMoE` leaves + `w13_input_scale` uninitialised and expects the checkpoint to fill it. A weight-only NVFP4 + checkpoint never does, it reads 0.0, and every expert output is multiplied by zero — silently. The + tell is a model that loads and serves but emits one token repeatedly. + +### Hybrid checkpoints and prefix caching + +A checkpoint is **hybrid** when some layers carry recurrent state instead of full attention — KDA / +`linear_attention`, Gated DeltaNet, mamba. `config.json` shows it in `layer_types`, +`linear_attn_config` or `full_attention_interval`; GLM-5.3-Flash is 45 layers of which 34 are KDA. +[vllm#51562](https://github.com/vllm-project/vllm/issues/51562) names the families sharing the +GatedDeltaNet metadata builder: Qwen3-Next, Qwen3.5, OLMo-hybrid, InternS2-Mobius, Bailing-MoE-v3 +and Kimi-Linear. + +On those models vLLM's default `enable_prefix_caching=True` **returns fast, confidently wrong +answers**. Under `mamba-cache-mode=align` the recurrent layers persist state at the end of a prefill +chunk rather than at the reusable cache boundary +([vllm#56960](https://github.com/vllm-project/vllm/issues/56960), open), and mamba state pages are +not zeroed on reallocation ([vllm#51483](https://github.com/vllm-project/vllm/issues/51483), merged +2026-09-16; [vllm#51562](https://github.com/vllm-project/vllm/issues/51562), closed 2026-09-22 by +PR #51565, so the fix is in no image pinned before that date), so +a cache hit resumes 34 layers from whatever the recycled block last held. Measured 2026-09-22 on +4×RTX PRO 6000 with `custom_model`: the identical 196,171-token needle probe came back **wrong in +4 s** served from cache and **right in 20 s** when a unique token at position 0 forced a real +prefill — 2/2 each way, temperature 0, container restart counter unmoved. + +Three consequences worth holding onto: + +- It corrupts answers and never crashes, so it explains no crash ceiling — `glm53-flash-nvfp4`'s + 98,304 cap is a CUDA illegal memory access ([vllm#54317](https://github.com/vllm-project/vllm/issues/54317)) + and untouched by this. +- A cache hit answers *wrongly*, so it can never manufacture a false pass. Recorded passes stand; + recorded failures and any cap derived from one are what deserve re-measuring. +- Multi-turn chat and a shared system prompt are cache hits by construction, so real traffic meets + this sooner than a benchmark does. + +The rule: an affected profile sets `hybrid_recurrent_state: true` and puts +`--no-enable-prefix-caching` in `extra_vllm_args`. `tests/check-prefix-caching.sh` fails CI if the +key is there without the flag, and `roles/llama_swap/tasks/refusals.yml` asserts the same at deploy +time. Drop the flag only once vllm#56960 is in the pinned image and `bin/spin soak` passes twice +without it. + +### cu130 images, and the driver that gates them + +`vllm/vllm-openai` publishes cu130 tags for these models in the repository this repo already pins +from — `glm53-flash-x86_64-cu130`, `qwen38-flash-next-x86_64-cu130`, +`deepseekv4-flash-vision-x86_64-cu130`, among others. Moving a profile to cu130 is a tag change, not +a new image line. + +The gate is the **host driver**: cu130 needs r580+. The UpCloud GPU template shipped **r595 +(595.58.03)** on 2026-09-20, so cu130 tags are usable on this account — `gpu_min_driver_blackwell` +stays at 570 and only `kimi-k3` and `glm53-flash-nvfp4` declare `min_driver_major: 580`. That also +makes `gemma4-26b-nvfp4` and `qwen36-35b-nvfp4` worth re-testing on a single RTX PRO 6000 at +€1.65/h against €4.50 on a B200. The gpu role prints the driver on every deploy (`… driver ` +in "Report the detected GPU topology") — check it if the template changes. + +### Plugin overlays + +When a GPU needs a kernel no released image carries, a profile can pin an out-of-tree package in +`vllm_plugins` (see `ansible/roles/vllm/tasks/plugins.yml`). Each entry is cloned at a **pinned +commit**, built inside that profile's own image — the `.so` is libtorch- and Python-ABI-tagged, so +it has to be — and left under `/data/ext/-`, so the build happens once per commit and +survives teardown. The compose file mounts it read-only on `PYTHONPATH`. + +Plugins register through vLLM's `vllm.general_plugins` entry point, so **no vLLM file is patched**, +and a well-behaved one is inert unless the profile sets its environment variable. The safety +argument rests on that — verify it in the source before adding an entry. + +`glm53-flash-nvfp4` is the only profile using this today. It carries +[Libertai/vllm-sparse-mla-blackwell](https://github.com/Libertai/vllm-sparse-mla-blackwell) +(Apache-2.0) for rope-free sparse MLA on sm_120/121 plus the `vllm#54189` activation-scale fix. +It is third-party code from a small project: read it at the pinned commit rather than trusting it. +Retire the overlay once `vllm#55277` and `flashinfer#5075` land in a release image. diff --git a/docs/validation.md b/docs/validation.md index 06dce6c..c34efdc 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -11,10 +11,45 @@ the served model id + `max_model_len` (`/v1/models`), the vLLM-reported **GPU KV Status legend: ✅ validated live · ⬜ not yet run on this tier (paper-sized) · ❌ known broken · 🚫 not servable on this tier (needs a bigger/multi-GPU node). +**What ⬜ means, as of 2026-09-20.** For every row below it means exactly this and nothing more, +and it is a statement about evidence rather than a queue position: *the offline gate confirms the architecture, the +parsers and every flag exist in the image the profile pins; the preflights confirm the hardware +fits on paper; nobody has started the engine.* The `max_model_len` in such a row is the model's +serving recipe, not a measurement — treat it as untested in both directions. + +Most ⬜ rows sit on a plan above `MAX_EUR_PER_HOUR` (default €10), so reaching one takes a +deliberate `--allow-expensive`. **The cheaper ones do not**: `qwen38-flash-next` and +`glm53-flash-nvfp4` are €6.60/h, under the limit, so a bare `bin/spin up --model ` deploys a +4-GPU box with only a debug warning about the missing validation. Those are the rows that matter +most, because an accidental user reaches them. + +The rest stay ⬜ by budget decision: the campaign that produced the RTX PRO 6000 rows was scoped +to that hardware. Testing the 4×B200 rows is ~€18/h each and the 8×B200 rows +~€36/h; see "Timings & session cost" for what a session buys. + ## Models Numbers are vLLM's own startup report (`kv_cache_utils`), `nvidia-smi`, and `/v1/models`, captured live. +> **What prefix caching means for the numbers below (added 2026-09-22).** On a hybrid checkpoint a +> cache hit makes the model answer fast and wrong — mechanism and measurement in +> [gpu-spinner.md](gpu-spinner.md#hybrid-checkpoints-and-prefix-caching). Until 2026-09-22 +> `bin/spin ceiling` and `bin/spin soak` reused one haystack across probes, so every probe after the +> first was part-served from cache; `_gs_prompt` now opens each prompt with a unique session tag. +> +> A cache hit answers wrongly, so it can never manufacture a false pass: **every PASS recorded here +> stands.** What wants re-measuring is any *failure*, any cap derived from one, and any latency +> figure on the hybrid rows — `qwen36`, `qwen36-27b-nvfp4`, `qwen36-35b`, `qwen36-35b-nvfp4`, +> `qwen38`, `qwen38-nvfp4`, `qwen38-flash-next`, `glm53-flash`, `glm53-flash-nvfp4` — since a cached +> probe skips prefill and those p50s are floors rather than cold-start timings. The +> `glm53-flash-nvfp4` 98,304 cap is outside that set: it is a crash caught by the container restart +> counter, which this defect cannot cause. + +**`Tier (plan)` below is where a run was measured.** Where a profile deploys can differ: `bin/spin` +picks that from the profile's `min_plan`, and for several models the cheapest plan that runs them is +one nobody has measured yet. Rows say so where the two differ, and the main lane per profile is in +the [README table](../README.md#models). + | Profile | Model | Quant | Tier (plan) | `max_model_len` | Status | KV pool | Concurrency @ max | VRAM | Date | Evidence | |---|---|---|---|---|---|---|---|---|---|---| | `qwen36` | Qwen3.6-27B | FP8 | L40S (`…1xL40S`) | 262144 (native) | ✅ | 372,123 tok | 1.42× | 41,023/46,068 MiB | 2026-07-04 | commit `11d6177`; gen "Capital of Finland" | @@ -22,12 +57,22 @@ Numbers are vLLM's own startup report (`kv_cache_utils`), `nvidia-smi`, and `/v1 | `gemma4-26b` | Gemma 4 26B-A4B (MoE) | FP8 | L40S | 131072 | ✅ | 986,681 tok | 7.53× | 41,597/46,068 MiB | 2026-07-04 | `11d6177`; gen "gemma 26b ok" (headroom for 262144) | | `qwen36-35b` | Qwen3.6-35B-A3B (MoE) | FP8 | H100 (`…1xH100`) | 262144 | ✅ | 3,416,000 tok | 13.03× | 74,183/81,559 MiB | 2026-07-04 | `bin/spin validate`: gen "validation ok" / "NVIDIA's flagship AI GPU" (thinking model — pass `enable_thinking:false` for direct answers) | | `qwen36-27b-nvfp4` | Qwen3.6-27B | NVFP4 | B200 (`…1xB200`) | 262144 | ✅ | 4,413,523 tok | 16.84× | 169,304/183,359 MiB | 2026-07-04 | `bin/spin validate`: gen "validation ok" | -| `qwen36-35b-nvfp4` | Qwen3.6-35B-A3B (MoE) | NVFP4 | B200 | 262144 | ✅ | 13,897,696 tok | 53.02× | 167,612/183,359 MiB | 2026-07-04 | validate OK — after removing `VLLM_USE_FLASHINFER_MOE_FP4` (see note) | -| `gemma4-26b-nvfp4` | Gemma 4 26B-A4B (MoE) | NVFP4 | B200 | 262144 | ✅ | 10,641,503 tok | 40.59× | 165,886/183,359 MiB | 2026-07-04 | validate OK — after removing `VLLM_USE_FLASHINFER_MOE_FP4` | +| `qwen36-35b-nvfp4` | Qwen3.6-35B-A3B (MoE) | NVFP4 | B200 ✅ 2026-07-04; **also 1×RTX PRO 6000 ✅ 2026-09-20** | 262144 | ✅ | 6,269,102 tok (RTX PRO 6000) | 23.91× | 90,283/97,887 MiB | 2026-09-20 | **cc 12.0 ban removed — same stale gate as `gemma4-26b-nvfp4`.** Ran on the shared cu129 pin; vLLM chose the **MARLIN** NvFp4 MoE kernel (its sibling chose `VLLM_CUTLASS`) and both were correct, so the gate was wrong about the family rather than about one kernel. `soak --rounds 3 --concurrency 20 --context 32768`: factual 6/6, both traps declined, **60/60 needles**, 0 wrong/rejected/timed out, p50 17.3 s. **Default lane moves B200 €4.50/h → RTX PRO 6000 €1.65/h list (€1.24 spot)** | +| `qwen38` | Qwen3.8-27B (dense hybrid) | FP8 | **L40S** (its own lane) | 131072 | ✅ | 359,719 tok | 2.74× | 41,023/46,068 MiB | 2026-09-21 | Measured on `GPU-SPOT-8xCPU-64GB-1xL40S` (€0.83/h), the lane a bare `bin/spin up --model qwen38` reaches. FLASHINFER attention; load 27.57 GiB in 72.9 s; init 51.2 s. soak 3×20 @32768: factual 6/6, traps 2/2, **60/60 needles**, 0 wrong; p50/p95 73.2 s/126.2 s. `ceiling`: 128,766 prompt_tokens passes (full deployed context). **Native 262144 also tested** (`-e max_model_len=262144`): KV 372,123, 1.42×, a single 259,512-token request passes in 173.8 s, and 3 concurrent at 260k all return correct answers with **0 rejected** — but p50 391 s, max 586 s. So 262144 is safe for correctness and unusable interactively; 131072 stays the default because it buys a 2.74× multi-user envelope at a 73 s p50. Earlier RTX PRO 6000 figures (1,743,257 tok, 13.30×) are 4.8× more KV and do not transfer | +| `qwen38-nvfp4` | Qwen3.8-27B (dense hybrid) | NVFP4 | **1×RTX PRO 6000** (spot) | 262144 | ✅ | 1,902,460 tok | 7.26× | 88,087/97,887 MiB | 2026-09-20 | Validated live on `GPU-SPOT-16xCPU-80GB-1xRTXPRO6000` (€1.24/h), driver 595.58.03, cc 12.0. FLASHINFER attention backend; CUDA graphs captured PIECEWISE+FULL in 9 s (1.55 GiB); model load 20.29 GiB in 53.8 s; init engine 123.9 s. `validate` returned coherent text — **not** the single-repeated-token signature of the zeroed NVFP4 activation scale (vllm#54189). `soak --rounds 3 --concurrency 20 --context 32768`: factual 6/6, both fabrication traps declined, **60/60 needles** at head/middle/tail, 0 wrong, 0 rejected, 0 timed out; median 32,754 prompt_tokens, p50/p95 45.4 s/68.5 s. `ceiling` ascended 65,429 → 130,803 → **259,512 prompt_tokens, all passing**, restart count unchanged — so the profile's 262144 is a measured ceiling, not a recipe number. Weights came from `bin/spin prefetch` (no download on the GPU box) | +| `qwen38-flash-next` | Qwen3.8-Flash-Next (125B-A6B MoE + 51B N-gram) | FP8 | **4×RTX PRO 6000** (spot) | 262144 | ✅ | 3,640,845 tok | 13.89× | 85,340–85,466/97,887 MiB ×4 | 2026-09-20 | Validated on `GPU-SPOT-64xCPU-320GB-4xRTXPRO6000` (€4.95/h, list 6.60), driver 595.58.03, cc 12.0. TRITON Fp8 MoE; model load 32.72 GiB/rank in ~29 s; init engine 110.7 s. **TP=4 over PCIe with no NVLink came up first try — `NCCL_P2P_DISABLE` was not needed.** soak 3×20 @32768: factual 6/6, **60/60 needles**, 0 wrong/rejected/timed out, p50 20.8 s. `ceiling`: 65,429 → 130,803 → **259,512 prompt_tokens all passing**, so 262144 is measured. **Required a fix to ship: `cap_add: [SYS_PTRACE]`.** Its mandatory `VLLM_PLE_CPU_OFFLOAD=1` passes CUDA tensors over IPC, which calls `pidfd_getfd(2)`; Docker's default seccomp gates that behind CAP_SYS_PTRACE, so engine init died with `RuntimeError: pidfd_getfd: Operation not permitted` ~8 min in, after the 173 GB checkpoint had loaded. Every offline gate passed it — no static check sees a refused syscall | +| `gemma4-26b-nvfp4` | Gemma 4 26B-A4B (MoE) | NVFP4 | B200 ✅ 2026-07-05; **also 1×RTX PRO 6000 ✅ 2026-09-20** | 262144 | ✅ | 4,822,910 tok (RTX PRO 6000) | 18.40× | 88,921/97,887 MiB | 2026-09-20 | **The cc 12.0 ban was stale — removed.** Ran on `GPU-SPOT-16xCPU-80GB-1xRTXPRO6000` (€1.24/h) on the shared **cu129** pin `v0.23.0-cu129-ubuntu2404`, i.e. the very build the profile said returns invalid output. vLLM selected `VLLM_CUTLASS` NvFp4 MoE — the grouped block-scaled GEMM path in question — and the result was correct throughout: `soak --rounds 3 --concurrency 20 --context 32768` gave factual 6/6, both traps declined, **60/60 needles** at head/middle/tail, 0 wrong/rejected/timed out, p50 14.1 s. A short-prompt check alone was deliberately not trusted (vllm#55757 is the case where short passes and long fails); the graded long-context load is what clears it. The cu130 image was not needed. Root cause of the stale gate: flashinfer#2708 closed 2026-04-20 and vllm#54189 closed 2026-08-30, both before this image was built (2026-06-13), but the profile comment was never re-checked. **Default lane moves B200 €4.50/h → RTX PRO 6000 €1.65/h list (€1.24 spot), a 63–72% cut** | | `gemma4-31b-nvfp4` | Gemma 4 31B (dense) | NVFP4 | B200 | 262144 | ✅ | 2,415,691 tok | 9.22× | 166,008/183,359 MiB | 2026-07-04 | validate OK | -| `deepseek-v4-flash` | DeepSeek-V4-Flash (284B-A13B MoE) | NVFP4 | B200 | 262144 | ✅ | 347,227 tok | 1.32× | 170,754/183,359 MiB | 2026-07-04 | **single-B200 TP1 fit CONFIRMED** (157 GB weights); tight — full 256K fits with 1.32×. `requires_review` removed. FP8 build still needs multi-GPU 🚫 | +| `deepseek-v4-flash-vision` | DeepSeek-V4-Flash-Vision-Exp (285B-A13B MoE, multimodal) | FP4+FP8 | **4×B200** (`…4xB200`, TP=4+EP) | 32768 | ⬜ | — | — | — | — | gated by price (€18/h, needs `--allow-expensive`); ~168 GB of weights, ~202 GB VRAM budget before KV. Needs the pinned `vllm/vllm-openai:deepseekv4-flash-vision` image — the stable wheel routes this checkpoint to the text-only class. Published vision runs exist only on GB200 NVL4 (TP4+EP) and MI350X, neither offered here 🚫 | +| `deepseek-v41-flash` | DeepSeek-V4.1-Flash (552B + 196B Engram MoE) | MXFP4+MXFP8 | **4×B200** (`…4xB200`, TP=4) | 1048576 | ⬜ | — | — | — | — | Newly added; gated by price (€18/h, needs `--allow-expensive`). ~511 GB on disk / 476 GiB, recipe floor 614 GB VRAM — one GB200 NVL4 tray at TP4, which 4×B200 matches. Needs a vLLM nightly from 2026-09-10 or later; no release serves `deepseek_v41`. Engram tables alone are 183 GiB 🚫 | +| `deepseek-v4-pro-0813` | DeepSeek-V4-Pro-0813 (1.6T-A49B MoE) | FP4+FP8 | **8×B200** (`…8xB200`, TP=8) | 393216 | ⬜ | — | — | — | — | Newly added; gated by price (€36/h, needs `--allow-expensive`). ~893 GB of weights; the recipe states it does not fit one 768 GB tray and all its recommended deployments are 8-GPU. ~€36/h and a ~1000 GB weights disk. `max_model_len` 393216 is the floor for the "max" reasoning mode 🚫 | +| `k2-horizon-375b` | K2-Horizon-375B-A23B (379B-A27B MoE) | FP8 | **lane: 8×RTX PRO 6000** (`…8xRTXPRO6000`, TP=8) | 131072 (native 524288) | ⬜ | — | — | — | — | Newly added; gated by price (€13.20/h, needs `--allow-expensive`). ~391 GB of weights. `K2HorizonForCausalLM` is in neither v0.23.0 nor v0.29.0, so it pins the same dated nightly as `deepseek-v41-flash`. The recipe's only verified run is BF16 at TP8 on MI355X; this serves the FP8 build at TP8 on 8× RTX PRO 6000 (13.20 €/h vs 18.00 for 4×B200), which is both cheaper and the same TP as the published run. 4× RTX PRO 6000 cannot hold it: 391 GB of weights against 380 GB 🚫 | +| `glm53-flash` | GLM-5.3-Flash (321B-A18B MoE, multimodal) | FP8 | **4×B200** (`…4xB200`, TP=4) | 1048576 | ⬜ | — | — | — | — | Newly added; gated by price (€18/h, needs `--allow-expensive`). ~306 GiB of weights. `Glm5NextForConditionalGeneration` is in neither v0.23.0 nor v0.29.0 — needs the pinned `vllm/vllm-openai:glm53-flash` image. FP8 KV cache is Blackwell-only for this model; Hopper must run BF16 KV. **Kept deliberately alongside the validated `glm53-flash-nvfp4`, which is not a substitute**: the NVFP4 sibling is capped at a *measured* 98304 while this profile carries the recipe's 1048576, and it needs no third-party kernel overlay. Its 1M is unmeasured. **Superseded 2026-09-22:** the prediction that the NVFP4 build's ~100k boundary applies here too no longer holds unqualified — `custom_model` (line below) reached 1,044,002 prompt tokens on the same 45-layer/34-KDA geometry, the same pinned image digest and the same overlay commit, so the vllm#54317 crash is not an unconditional property of the architecture at that length and is worth re-attributing to the NVFP4 build. That is a hypothesis; nobody has run this profile. Anyone paying 18.00 €/h for the long-context lane should still ceiling-search it first (`bin/spin ceiling`), and note this profile now pins `--no-enable-prefix-caching` (vllm#56960 is filed against this exact checkpoint) 🚫 | +| `glm53-flash-nvfp4` | GLM-5.3-Flash (321B-A18B MoE, multimodal) | NVFP4 | **4×RTX PRO 6000** (`…4xRTXPRO6000`, TP=4) | **98304 (capped — native is 1,048,576)** | ✅ | 4,552,849 tok | 4.34× @1M, ~46× at the cap | 85,219/97,887 MiB ×4 | 2026-09-20 | driver 595.58.03, cc 12.0, Server Edition. `Using FLASHINFER_MLA_SPARSE_SM120 attention backend out of potential backends: ['FLASHINFER_MLA_SPARSE_SM120']` — the pinned overlay is the only MLA candidate, so without it the model cannot start. NVFP4 MoE auto-selects **MARLIN**. Engine init 185 s. Graded soak (`bin/spin soak`): **60/60 needles at 3 × 20 × 25,861 tok** (p50 27.4 s), **10/10 at 98,100** (p50 57 s), **10/10 at 65,403** (p50 50 s), **20/20 at 51,670** (p50 27 s). **Context ceiling ~100k per request:** a single 102,400-token request kills the engine (CUDA illegal memory access in KDA linear attention), so `max_model_len` is capped at 98304 and over-long requests get a clean 400. Request length alone decides it: concurrency, prefix caching, KV capacity, memory exhaustion, the overlay and this GPU were each ruled out, and vllm#54317 reports the same frame on 4×B200. Full evidence in `ansible/models/glm53-flash-nvfp4.yml`. 6.60 €/h against 18.00 for the FP8 4×B200 row. | **Re-run 2026-09-20 on a clean disk** (spot, €4.95/h): the pinned overlay rebuilt from source and loaded (`FLASHINFER_MLA_SPARSE_SM120` present), MARLIN MoE, and `init engine` reproduced at **184.94 s vs the 185 s recorded** — the digest+commit pin is deterministic. Three figures came out higher and are NOT overwritten here: KV pool 4,971,913 vs 4,552,849 tok (+9.2%), concurrency 50.58× vs ~46×, VRAM 86,453 vs 85,219 MiB. Same image digest and plugin commit, so most likely a difference in free memory at profiling time; size against the lower figure. The vllm#52225 sustained-load (Xid 13) check is still outstanding — both runs exercised minutes, not hours | +| `glm53` | GLM-5.3 (743B-A39B MoE) | FP8 | **8×B200** (`…8xB200`, TP=8) | 1048576 | ⬜ | — | — | — | — | Newly added; gated by price (€36/h, needs `--allow-expensive`). Same architecture and flags as `glm52`, but the recipe's floor is vLLM 0.28.0, so it pins v0.29.0 rather than the shared v0.23.0. ~756 GB of weights, ~€36/h 🚫 | +| `kimi-k3` | Kimi K3 (93L, 896 experts, multimodal) | compressed-tensors | 8×GB300 — **not available on UpCloud** | 262144 | 🚫 | — | — | — | — | Newly added; refused by preflight (no UpCloud plan fits). ~1561 GB of weights against ~1538 GB on 8×B200, and the recipe's floor is 8×GB300 (2304 GB). Its image is cu130-only and needs an r580+ driver. Every preflight refuses it here. **🚫 is permanent, not pending**: measured 1454 GB of weights (HF file listing, 2026-09-20) against ~1432 GB of VRAM on the largest node sold here (8×B200 = 8×179), and `min_vram_gb: 280` exceeds any single GPU on offer. There is no run to schedule. It also has no `min_plan`, deliberately — `bin/spin up` now refuses a profile with no usable `min_plan` rather than letting it latch onto whatever box is live | | `glm52-nvfp4` | GLM-5.2 (753B MoE) | NVFP4 | **4×B200** (`…4xB200`, TP=4+EP) | 786432 | ✅ | 828,160 tok | 1.05× | 170,786/183,359 MiB ×4 | 2026-07-04 | `bin/spin validate` OK; full soak + quality pass — see **“Soak & output-quality checks”** below. ⚠️ 1.05× = single-user at full ctx. 1M ctx does NOT fit (needs 53.9 GiB KV vs 40.8 free; vLLM ceiling 793,216) | -| `glm52` | GLM-5.2 (753B) | FP8 | 8×B200 node | 1048576 | ⬜ | — | — | — | — | `requires_review`; FP8 ~756 GB needs 8×B200 (36 €/h) — the NVFP4 row above is the validated path at half the price | +| `glm52` | GLM-5.2 (753B) | FP8 | 8×B200 node | 1048576 | ⬜ | — | — | — | — | gated by price (€36/h, needs `--allow-expensive`); FP8 ~704 GB (measured 2026-09-20) needs 8×B200 (36 €/h) — the NVFP4 row above is the validated path at half the price. **Kept deliberately: not a duplicate.** The NVFP4 build serves 786432 context, this one 1048576, and this is the higher-precision weights. Gated on cost, not doubt. It shipped without `validated: false`, so until 2026-09-20 it silently defaulted to claiming a live run; that key is now set | Notes: - All validated L40S rows use `--kv-cache-dtype fp8` (static; **never** `--calculate-kv-scales` on the @@ -35,6 +80,28 @@ Notes: - `qwen36` is also expected to run on H100/B200 with far more KV headroom, but only the **L40S** run is verified. - Each `max_model_len` is capped at the model's **native** context; beyond that needs YaRN/rope_scaling. - **B200 reports 179 GB** usable via nvidia-smi (183,359 MiB) — profile floors use `min_vram_gb: 175`. +- **RTX PRO 6000 (96 GB GDDR7, compute capability 12.0) has its first row: `glm53-flash-nvfp4`, + 2026-09-20.** A 96 GB card reports **95 GB** via nvidia-smi, hence `min_vram_gb: 90` on these + profiles. The card is the *Blackwell Server Edition* — the variant the overlay kernel lists as + verified; upstream reports from Max-Q or Workstation parts do not carry over. It is also the default + lane for `qwen38-flash-next` (4×) and `k2-horizon-375b` (8×), which have no rows yet: they pass + preflight, print an untested warning and serve. Measure them and add the rows. +- **Driver on the UpCloud GPU template: r595 (595.58.03), measured 2026-09-20** on + `GPU-64xCPU-320GB-4xRTXPRO6000` in fi-hel2 (card: *RTX PRO 6000 Blackwell Server Edition*, + 97,887 MiB → `gpu_total_vram_gb` 95, cc 12.0, four per node). That clears the r580+ floor, so + **the official cu130 tags are usable on this account** — `glm53-flash-x86_64-cu130`, + `qwen38-flash-next-x86_64-cu130` and `deepseekv4-flash-vision-x86_64-cu130` all exist in + `vllm/vllm-openai`. It also makes the NVFP4 MoE profiles (`gemma4-26b-nvfp4`, + `qwen36-35b-nvfp4`) worth re-testing on a single card at 1.65 EUR/h against 4.50 on a B200: the + GEMM fault they cite is reported fixed in cu130's FlashInfer, so dense-or-MoE is the only + question left to answer. +- **Why the other large profiles still refuse cc 12.0: sparse attention.** + vllm-project/vllm#55757 reports DeepSeek-style sparse MLA unservable on sm_120, and it fails + *late* — short prompts pass and realistic lengths break, so a quick check proves little. The NVFP4 + MoE GEMM fault is reported fixed on cu130 (flashinfer-ai/flashinfer#2708), and much of that + symptom was vllm-project/vllm#54189: an uninitialised `w13_input_scale` silently multiplying every + expert output by zero. **If a model loads and serves but emits one token repeatedly, suspect that + before the kernels.** - **NVFP4 MoE**: do **not** set `VLLM_USE_FLASHINFER_MOE_FP4=1` on this image — it *forces* the FlashInfer path and raises `NotImplementedError: … no FlashInfer NVFP4 MoE backend supports the configuration`; unset, vLLM auto-selects a working (CUTLASS) backend. Cost: two MoE crash-loop rounds on 2026-07-04. @@ -46,7 +113,8 @@ Notes: gemma4-26b-nvfp4 & gemma4-31b-nvfp4 (B200) live — numbers match the rows above **exactly**, soak 6/6 with fabrication traps declined; qwen36-27b-nvfp4 & qwen36-35b-nvfp4 re-served via llama-swap ("swap ok"). `requires_review: true` was then removed from all non-GLM profiles (qwen36-35b trusted on its 2026-07-04 - row, not re-run). **glm52 FP8 stays `requires_review` by cost choice** (8×B200 ~36 €/h), not doubt. + row, not re-run). **glm52 FP8 stays gated by cost choice** (8×B200 ~36 €/h), not doubt. (The + `requires_review` key itself was retired 2026-09-21; the plan's price gates a deploy now.) Fixed along the way: `ACME_EMAIL` was a placeholder (`…@example.com`) → Let's Encrypt rejects the domain, so Caddy never got a cert and HTTPS silently failed after a full deploy; set a real address + added a deploy **preflight** that rejects placeholder domains. The first B200 create hit UpCloud `state=error` @@ -72,7 +140,8 @@ factual battery with fabrication traps): healthy throughout (no container restarts, VRAM flat at 170.8/179 GiB ×4 — no leak, no crash). - Interpretation: at `max_model_len: 786432` the KV pool fits **1.05 full-length requests**, so vLLM's admission control sheds parallel load. Every *admitted* request returned real content - (availability verified; soak responses were not quality-graded — quality evidence is the battery). + (availability verified; soak responses were not quality-graded at the time — the quality + evidence is the battery. `bin/spin soak` grades them now). - **Recommendation:** the 768 K profile is the *single-user / agentic* configuration. For multi-user serving, deploy with `-e max_model_len=262144` (~3× concurrency by KV math) — **not yet live-verified at 262 K**; a warm re-test costs ~€6–8 (~20 min to serving, weights cached). @@ -93,13 +162,35 @@ see the runbook) · Ansible pre-model ≈ 2 min (image is cached on /data, no pu | `gemma4-31b` / `gemma4-26b` (L40S) | 33/27 GB → **~15 min** (587 s health-wait measured) | ~3–5 min | ~0.3 € | | `qwen36-35b` (H100, 1.79 €/h) | 37.5 GB → **~20 min** (904 s health-wait measured) | ~5 min | ~0.7 € | | `*-nvfp4` dense/MoE 14–31 GB (1×B200, 4.5 €/h) | → **~10–15 min** | ~3–5 min | ~1 € | -| `deepseek-v4-flash` (1×B200, 4.5 €/h) | 157 GB ≈ 20 min + load → **~30 min** | ~8–10 min | ~2.5 € | +| `deepseek-v4-flash-vision` (4×B200, **18 €/h**) | ~168 GB + TP4 init → **not yet measured** | — | — | | `glm52-nvfp4` (4×B200, **18 €/h**) | 447 GB ≈ 60 min + **457 s engine init** (+1 warmup restart) → **~90 min** | **~18–20 min** | **~27 € cold / ~6 € warm** | \* provision→serving→teardown, no usage time. Warm = weights already on `/data` (they persist across teardowns; `/data` is an LRU cache sized by `weights_size_gb`, 150 GB by default — check `du -sh /data/hf-cache/hub/models--*` before assuming). +**Do the download on a different machine.** Every "cold" figure above is dominated by the pull, and +the pull is billed at whatever compute holds the disk — which is not GPU work in any sense. A +`2xCPU-4GB` box costs €0.0446/h against €6.60 for 4×RTX PRO 6000 and €18.00 for 4×B200, so +`bin/spin prefetch ...` collapses that line item by about 150×: + +| Pull | On the GPU box | Via `bin/spin prefetch` | +|---|---|---| +| 173 GB (`qwen38-flash-next`, 4×RTX PRO 6000) | ~23 min ≈ €2.53 | ~23–46 min ≈ €0.02 | +| 354 GB (that plus `glm53-flash-nvfp4`) | ~47 min ≈ €5.17 | ~47–94 min ≈ €0.04 | +| 831 GB (`deepseek-v4-pro-0813`, 8×B200) | ~110 min ≈ €66 | ~110–220 min ≈ €0.10 | + +It also pulls each profile's container image (9–14 GB, currently pulled at GPU rate) onto the same +disk, because Docker's data-root lives there. An UpCloud disk attaches to one server at a time, so +prefetch and a GPU session are strictly sequential — `bin/spin prefetch` refuses while a GPU server +exists. The small box's slower NIC makes the wall-clock longer and the bill smaller; on the +expensive tiers that is always the right trade. + +**Spot plans are 25% off and are plain plan ids** — `--plan GPU-SPOT-64xCPU-320GB-4xRTXPRO6000` +(€4.95/h vs €6.60) works with no code change. A spot box can be reclaimed; with weights already +prefetched onto `/data`, that costs minutes rather than a re-download, which makes validation runs +close to the ideal spot workload. Do not use spot for anything serving traffic. + Session discipline for the expensive tiers: - The health-wait is sized per profile (`health_wait_retries`, ×10 s): GLM 25 min, DeepSeek/35B 20 min, default 15 min. If a play still reports a health-wait failure, **poll `/v1/models` before teardown** — @@ -145,8 +236,66 @@ Session discipline for the expensive tiers: `.../ip_address`). The ephemeral stack needs no recovery — destroy any stray server via the API and `bin/spin up` fresh. -## All gating verified ungated +## Multi-model swap presets + +| Preset | Plan | Entries | Status | Date | Notes | +|---|---|---|---|---|---| +| `rtxpro6000` | `GPU-16xCPU-80GB-1xRTXPRO6000` | qwen36, gemma4-26b, gemma4-31b, qwen36-27b-nvfp4, gemma4-31b-nvfp4 | ✅ | 2026-09-20 | First run of the preset as a whole. All five registered and answered after a cold swap-in of 159–308 s. Graded soak through the swap path (gemma4-31b-nvfp4, 2×6 @16384): factual 6/6, traps 2/2, 12/12 needles, 0 wrong. Rendered config now carries `--tensor-parallel-size`/`--dtype` and a derived `healthCheckTimeout: 900`. **`bin/spin validate` is blind here** — `/v1/models` reports `max_model_len=null` until a backend loads | +| `l40s` | `GPU-8xCPU-64GB-1xL40S` | qwen36, gemma4-26b, gemma4-31b | ⬜ | — | Renders and resolves in CI; never deployed | +| `b200-nvfp4` | `GPU-24xCPU-240GB-1xB200` | qwen36-27b-nvfp4, qwen36-35b-nvfp4, gemma4-26b-nvfp4, gemma4-31b-nvfp4 | ⬜ | — | Renders and resolves in CI; never deployed. Note all four entries now also run on the far cheaper `rtxpro6000` lane | -Per the HF API (`gated:false`), no model needs an `HF_TOKEN`: `google/gemma-4*`, the RedHat FP8 builds, -the `nvidia/*-NVFP4` builds, and Qwen3.6 are all ungated/Apache-2.0. Check a repo with +## Repo, size and image audit — re-verified 2026-09-20 + +Every check below is free and was re-run against the live APIs on 2026-09-20, covering all 21 +profiles (the previous note predated the eleven added on this branch). + +**Gating.** All 21 HF repos return `gated: false`, so no model strictly needs an `HF_TOKEN`. Set +one anyway: anonymous downloads throttle to ~50 KB/s after a few hundred GB in a day, which on a +€18/h box costs far more than the token is worth. Check a repo with `curl https://huggingface.co/api/models/ | jq .gated` (the model-card prose can mislead). + +**Download sizes**, summed from each repo's file listing (`?blobs=true`), not `usedStorage` — +that counts every revision and reads several times too high: + +| Profile | Measured | Profile said | Profile | Measured | Profile said | +|---|---|---|---|---|---| +| qwen38 | 29 GB | ~31 | qwen38-nvfp4 | 22 GB | ~22 | +| qwen36 | 29 GB | ~29 | qwen36-35b | 35 GB | ~37.5 | +| gemma4-26b | 27 GB | ~27 | gemma4-31b | 31 GB | ~33 | +| gemma4-26b-nvfp4 | 18 GB | ~14 | gemma4-31b-nvfp4 | 30 GB | ~21 | +| qwen36-27b-nvfp4 | 20 GB | ~17 | qwen36-35b-nvfp4 | 22 GB | ~19 | +| qwen38-flash-next | 173 GB | ~186 | glm53-flash-nvfp4 | 181 GB | ~195 | +| deepseek-v4-flash-vision | 156 GB | ~168 | glm53-flash | 306 GB | ~306 | +| k2-horizon-375b | 364 GB | ~391 | **glm52-nvfp4** | **433 GB** | **~377 (understated)** | +| deepseek-v41-flash | 475 GB | ~511 | glm52 / glm53 | 704 GB | ~756 | +| deepseek-v4-pro-0813 | 831 GB | ~893 | kimi-k3 | 1454 GB | ~1561 | + +Most profiles were conservative, which is the safe direction. `glm52-nvfp4` was the exception — +it claimed ~377 GB against a real 433 GB pull, and has been corrected. Every profile now carries a +machine-checked `min_disk_gb` (measured + ~40 GB for the container image and Docker's data-root, +which share the disk), and the vLLM role refuses to deploy when `/data` has less free than that. +Before this, an under-sized disk was discovered as "No space left on device" 20–120 minutes into a +pull on a GPU billed by the hour — the one common failure with no machine check. + +**Container images.** All nine pinned tags still resolve on Docker Hub. Four are mutable +(`:qwen38-flash-next`, `:deepseekv4-flash-vision`, `:glm53-flash`, `:kimi-k3`) and should be +pinned by digest as `glm53-flash-nvfp4` already is — that tag moved once already. Note there is +**no cu130 release tag**: vLLM's stable releases (v0.28.0, v0.29.0) are cu129 only, and cu130 +exists solely as model-specific builds (`glm53-flash-x86_64-cu130`, `gemma4-unified-x86_64-cu130`, +`glm52-x86_64-cu130`, `qwen38-x86_64-cu130`, …). Any plan to "move to cu130" is per-profile. + +**Upstream issues the profiles depend on**, re-read on 2026-09-20 — two are stale: + +| Issue | State | Consequence | +|---|---|---| +| flashinfer#2708 (SM120 CUTLASS FP4 GEMM) | **closed 2026-04-20** | The NVFP4-MoE `unsupported_compute_capabilities: [12.0]` gates may be obsolete *with a cu130 image*. Worth a ~€2 retest on 1×RTX PRO 6000 against €4.50/h on B200 | +| vllm#54189 (NVFP4 MoE zeroed activation scale) | **closed 2026-08-30** | Part of what `glm53-flash-nvfp4`'s plugin overlay works around may now be upstream | +| vllm#37554 (`--calculate-kv-scales` corrupts FP8 KV) | closed 2026-03-20 | The "never add this flag" note in `qwen36.yml` is now historical | +| flashinfer#5075 (SM120 NoPE rows) | closed 2026-09-11 | One of the two the overlay waits on has landed | +| vllm#55277 (SM120 NoPE sparse MLA) | open | The other has not — keep the overlay | +| vllm#55757 (sparse MLA unservable on SM120) | open | The cc-12.0 gates on the DeepSeek profiles stand | +| vllm#54317 (GLM-5.3-Flash KDA illegal memory access) | open | Applies to 4×B200 too, so `glm53-flash` FP8 is exposed | +| vllm#52225 (Xid 13, Marlin MoE, RTX PRO 6000) | open | Only a long run surfaces it | +| vllm#41834 (SM12x for DeepSeek V4 Flash) | open | No cheaper lane for `deepseek-v4-flash-vision`; 4×B200 stands | + +Re-check before trusting any of it: `curl -s https://api.github.com/repos/vllm-project/vllm/issues/N | jq '{state,updated_at}'`. diff --git a/requirements.txt b/requirements.txt index 7a9340b..f1d221e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,18 @@ # Python tooling for the control machine (operator side). -# Install into a venv: python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt -ansible>=10,<12 -ansible-lint>=24 -jmespath>=1.0 +# +# REQUIRES PYTHON 3.11, 3.12 OR 3.13. Versions outside that range fail to install or to run: +# * below 3.11 — ansible-core 2.18 (what ansible 11.x pulls in) declares requires_python >=3.11, +# so pip finds no candidate and reports every version it ever published instead of the cause. +# macOS still ships 3.9 as /usr/bin/python3, which is the usual way to land here. +# * 3.14 and above — ansible-lint 26 refuses to import, with +# "Python 3.14 requires ansible-core version >= 2.20.0, and we found 2.18.19". +# Raise the ansible pin past 12 before moving the control machine to 3.14. +# +# Versions are pinned exactly so two machines get the same Ansible. Bump them deliberately and +# re-run the render tests; pinning alone does not protect against the Python floor above. +# +# Install into a venv, naming the interpreter rather than trusting `python3`: +# python3.12 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt +ansible==11.13.0 +ansible-lint==26.8.0 +jmespath==1.1.0 diff --git a/requirements.yml b/requirements.yml index d772b28..897748e 100644 --- a/requirements.yml +++ b/requirements.yml @@ -3,8 +3,11 @@ # No external ROLES are used: the UpCloud "Ubuntu with NVIDIA Drivers and CUDA" # template pre-ships the NVIDIA driver, CUDA, NVIDIA Container Toolkit and Docker, # so our roles only verify those and then configure vLLM + Caddy. +# +# Pinned exactly, for the same reason requirements.txt is: two machines should resolve the same +# collection versions. Bump them deliberately and re-run the render tests. collections: - name: ansible.posix # ansible.posix.mount (fstab) - version: ">=1.5.0" + version: "1.6.2" - name: community.general # community.general.filesystem (mkfs) - version: ">=8.0.0" + version: "10.7.6" diff --git a/terraform/persistent/main.tf b/terraform/persistent/main.tf index 4b1cad3..1d71b47 100644 --- a/terraform/persistent/main.tf +++ b/terraform/persistent/main.tf @@ -13,6 +13,26 @@ resource "upcloud_storage" "weights" { } } +# An optional second disk, for running one model without disturbing what is cached on the primary. +# +# Why this exists: the primary disk is an LRU cache you grow as models get bigger, and UpCloud can +# never shrink a disk. So a one-off experiment on a large checkpoint either forces a permanent +# growth of the primary, or evicts weights someone else is about to use. A separate, disposable +# disk avoids both. Created 2026-09-21 to serve `custom_model` while a 364 GB k2-horizon cache sat +# on the primary. +# +# Deliberately without prevent_destroy, unlike the primary: this one is meant to be thrown away. Set +# weights_alt_size_gb back to 0 (or run the destroy) when the experiment is done, and its standing +# cost goes with it. +resource "upcloud_storage" "weights_alt" { + count = var.weights_alt_size_gb > 0 ? 1 : 0 + + title = "${var.prefix}-weights-alt" + size = var.weights_alt_size_gb + tier = var.weights_tier + zone = var.zone +} + resource "upcloud_floating_ip_address" "vip" { zone = var.zone family = "IPv4" diff --git a/terraform/persistent/outputs.tf b/terraform/persistent/outputs.tf index 0b80c8f..c1965d0 100644 --- a/terraform/persistent/outputs.tf +++ b/terraform/persistent/outputs.tf @@ -21,3 +21,8 @@ output "sslip_host" { output "zone" { value = var.zone } + +output "weights_alt_storage_id" { + description = "UUID of the optional second data disk, or \"\" when weights_alt_size_gb is 0." + value = length(upcloud_storage.weights_alt) > 0 ? upcloud_storage.weights_alt[0].id : "" +} diff --git a/terraform/persistent/variables.tf b/terraform/persistent/variables.tf index fd47d29..a508e71 100644 --- a/terraform/persistent/variables.tf +++ b/terraform/persistent/variables.tf @@ -28,3 +28,14 @@ variable "weights_tier" { type = string default = "maxiops" } + +variable "weights_alt_size_gb" { + description = <<-EOT + Size (GB) of an OPTIONAL second data disk, or 0 for none (the default). Use it to run one model + without growing or evicting the primary cache — point a spin-up at it with WEIGHTS_DISK=alt. + It bills 24/7 exactly like the primary, and unlike the primary it has no prevent_destroy, so + set this back to 0 when the experiment is finished. + EOT + type = number + default = 0 +} diff --git a/terraform/prefetch/.terraform.lock.hcl b/terraform/prefetch/.terraform.lock.hcl new file mode 100644 index 0000000..fbc9994 --- /dev/null +++ b/terraform/prefetch/.terraform.lock.hcl @@ -0,0 +1,72 @@ +# This file is maintained automatically by "tofu init". +# Manual edits may be lost in future updates. + +provider "registry.opentofu.org/hashicorp/http" { + version = "3.6.2" + constraints = "~> 3.4" + hashes = [ + "h1:7+Ewr07oLewLfoe5K7Fhy4T5JUU4y7I7iRGTG2hrzy4=", + "h1:8AvJ6H30J1tgxwEoVKxe0B0Ac9XFn1gwOzidrIc+F5o=", + "h1:AllxtaK2m88aY+QitHui6kik/qJmBakrH5o5W7vOKI0=", + "h1:OimQGm1/9sIkdDCuUb8D5sxDorhektHeCQ337DHDfGk=", + "h1:T5zxkFXQ9To+ZysjCZu6gFqE8G1aTGh58Ol20bCk1Fc=", + "h1:TZlCCHsmKXSI3FyzmSgCU+TarKOHgiOft0wiYh85Y+E=", + "h1:VgyzPdGw1U/KU8gKhefKlIhy5kQYdpoZkpNcGiTgvVs=", + "h1:aF+Ztfy9m0Ll6nFOtC90NxN6WIEKAmfmVFhXAtFF8CI=", + "h1:bgViIoXF7kMXA7xgNikFYfv1D15rregO8I8Sq95sLGQ=", + "h1:d7tCAc1wr1seWpGMBx7tpRyGKOCicvozaHvn5X8ZFfM=", + "h1:dRCrpHH2UrkKs/lCqewyA00L0VfOCETVM14NBhyxHaM=", + "h1:eIqaNIixQTlhmr2GU1fAFH+eVYLQIfeGw7Ec+JT/bEo=", + "h1:hrK+Jzpgivi4Jo356K3LYDoeWBV1bMqKPBEqrEHwgpA=", + "h1:mrpIAC+TK3rQQz2jllAF8o2WLMKM14/WoclLgYKc4WA=", + "h1:zwE/uNBapBDpkKa5I24moawVPTanbG+eKumHRm/qINw=", + "zh:0cf6173a426f65a85e8baf4f9a7808dc94d0e02a82b34bc903555fda1eb45d7a", + "zh:2b5fa2b0487e692014fa86745a2a98d43f34f7aaea044acd3e2cee139627c9f1", + "zh:2c9c18893ce0f34e077559632c88237d172ac6646853bf00c84a5a4ec3b6c241", + "zh:316c848e892764aa10a890cf2455f52b592c6051a92ac8451955ca1d47ca1d2f", + "zh:4300e32ecaa967ca0ed56ca48f8dcd10796e70203bf6b8311e71dc43cd0d2aa1", + "zh:4a0a555d85a31bd257399253b502ad03816ccdba7b8fbbbd1e07a313d04d83fb", + "zh:6278773c31c1605c6d780fc982eacb83944e2807c8d8afe11534253b225c5aa3", + "zh:81b73da8dbcedf54604f131e0acd7ebd36cf673276174b4b4f10af0b0f10de11", + "zh:887beaa6ef6521d3f5e2b1193d458575fb91855f36bc66dd26a10988127a39fb", + "zh:91b2260424c5f8e5968d9e6d5bdbbdb60d61ccc4ff001aa0d637427629349c54", + "zh:a5d9d7032d00a6dd844e8d0d374b5ae89e25f9a6635a90e01c5f65e9dd14a93e", + "zh:bcf8b2aca448774e4701f221b74d762847be55dc312022b4aad66da3b389bf5d", + "zh:d62a3ad37ee1fb943ad0590978f4e7ee0ac064d19554c1c05ec98e9d2bb658f7", + "zh:e517b8b81fe88bf49868923f076515b9157771d1d3a80849cc8a73b9a19039a1", + "zh:fa9e0e39ef3e5ed2e6cf5886a0b569415ca3b89b9289fd6d01824ff11e470618", + ] +} + +provider "registry.opentofu.org/upcloudltd/upcloud" { + version = "5.44.1" + constraints = ">= 5.25.0, < 6.0.0" + hashes = [ + "h1:+hPymcM6McyYclwj9nxxpgI17FcZz3bmVT3zuk00asA=", + "h1:HjLNqUUmprdG6A1BZCRY/9ZBv8GpFZ5q7C3BiPFI3+Q=", + "h1:M15L7AcxBcKvibs6sSI9/Hh4uQ0Wwf65MSxjpxmE6kg=", + "h1:O4MrfksywkKuPi450K/E5QapdbGC8qJb84djx/Au0Xs=", + "h1:Pu+8ZSvAGvcKviBiAyM1rITfvB+6DlOVLa591Z5Rkr0=", + "h1:Sv/8t1cI/yf5Nw82VMpNGZefaM0MUjWWNbY+cSHEY+A=", + "h1:T4JlMj4t5HQC/bBf3RrLY4j8R59Psi/U3yPiLEdTlo4=", + "h1:Tph/ucPbjBLCuFzoqeVod29+karHLsGstuWkVItP6IA=", + "h1:fUf3fusIjKU/NUikgxIaJRZOp6X0eXGBSFc2s3Ere14=", + "h1:h/fERICD1lsG7WUatTHJjHZSc/IpFFdn1M2q/0FFlVQ=", + "h1:ldlFo6tbpjEyYV/YKkVJsJEzhh/Mh+JF2S8YhrQsqI4=", + "h1:mkCVYwbBzj9pJOpDezsPi+ql/Nd2/0dUHKgadK0ONSA=", + "h1:wo6KJhyWNQ9/0H2I6R5nIXqr6HsfBVUXwpklIIRj4aU=", + "zh:10a70d5286715cf9f941a930ad7dadc504b6c2281f99571ae541f86c0afdac4a", + "zh:1609c577a549d34cf1ac9f7cb9e42763b200fda3202689f6038a5e8c836f409d", + "zh:59e9f194bff468fe22036303665aed3188b53efd5b4625feb556642405b7f1cc", + "zh:830ad93554b74b3791060537fe496d515ebceefb5e7d31e0acd786253b82a6d5", + "zh:9768b6b1ab810326716024f5fbf8c3e91531580772ee80007845dc651f0612ac", + "zh:9a139b61aa154925a88d9d40c1a6ea780715e5185e4949239ca84939a102ba42", + "zh:9e7062b51e686261f4d6db25e9c18f1b6dba94bdef809df3bea169d79d5758ca", + "zh:d9b385e8d2b484e7def8dd23ecc6bc46d2b683b8f1a9827a18bc2e6e217293b5", + "zh:d9dc0c03faac7879f04b9cf4ce3ff1e5b2a0f119bf6da9937de43ceebcc71497", + "zh:da6a7ed96b5fab4e74a2e541e8e2392638e1081db09d48a80d6ee7135974d879", + "zh:dd49cfa62d18fd97e0ab561ba4e20706995aef6aeb4b9d387ab63c0a256ff76c", + "zh:dfe231c8e052e6f74ff07ea94438fa4783679608da14d6f3cdf5b8f691e684c7", + "zh:e3692d3fc8461eeab4d72a8163a513b0c6ba74f0530dd1c71012b8b66a16a16b", + ] +} diff --git a/terraform/prefetch/data.tf b/terraform/prefetch/data.tf new file mode 100644 index 0000000..b56be2e --- /dev/null +++ b/terraform/prefetch/data.tf @@ -0,0 +1,33 @@ +# Same allowlist logic as the ephemeral stack: explicit CIDRs > operator_ip > auto-detect. +data "http" "myip" { + count = (length(var.operator_cidrs) == 0 && var.operator_ip == "") ? 1 : 0 + url = "https://api.ipify.org" + retry { + attempts = 3 + } +} + +# Only the weights disk is read from the persistent stack. The floating IP is deliberately NOT +# touched: it carries the TLS cert and the stable hostname, and binding it to a throwaway download +# box would move it off the serving path for no reason. +data "terraform_remote_state" "persistent" { + backend = "local" + config = { + path = var.persistent_state_path != "" ? var.persistent_state_path : "${path.module}/../persistent/terraform.tfstate" + } +} + +locals { + _alt_id = try(data.terraform_remote_state.persistent.outputs.weights_alt_storage_id, "") + weights_id = var.weights_disk == "alt" ? local._alt_id : data.terraform_remote_state.persistent.outputs.weights_storage_id + + _allow_raw = length(var.operator_cidrs) > 0 ? var.operator_cidrs : ( + var.operator_ip != "" ? [var.operator_ip] : ["${trimspace(data.http.myip[0].response_body)}/32"] + ) + operator_cidrs_norm = [for s in local._allow_raw : strcontains(s, "/") ? s : "${s}/32"] + + operator_ranges = [for c in local.operator_cidrs_norm : { + start = cidrhost(c, 0) + end = cidrhost(c, pow(2, 32 - tonumber(split("/", c)[1])) - 1) + }] +} diff --git a/terraform/prefetch/main.tf b/terraform/prefetch/main.tf new file mode 100644 index 0000000..22940a6 --- /dev/null +++ b/terraform/prefetch/main.tf @@ -0,0 +1,64 @@ +# Throwaway CPU box whose only job is to fill the persistent weights disk before a GPU session. +# +# WHY. Weight downloads are the largest single line item in a validation campaign, and they are +# billed at whatever the attached compute costs. Pulling 354 GB at ~126 MB/s is about 47 minutes: +# 5.17 EUR on 4x RTX PRO 6000, or 0.035 EUR here. The GPU box then starts with a warm cache and +# spends its expensive minutes on inference. +# +# A UpCloud storage device attaches to ONE server at a time, so this stack and the ephemeral GPU +# stack are mutually exclusive by construction. bin/spin prefetch refuses to run while a GPU +# server exists, and destroys this box when the pull finishes. +resource "upcloud_server" "prefetch" { + hostname = var.hostname + zone = var.zone + plan = var.plan + metadata = true + firewall = true + + template { + storage = var.os_template + size = var.os_disk_size_gb + filesystem_autoresize = true + } + + login { + keys = var.ssh_public_keys + create_password = false + } + + network_interface { + type = "public" + } + + # The same persistent disk the GPU box mounts at /data, by UUID. An ATTACH, not a clone. + storage_devices { + storage = local.weights_id + } +} + +# SSH only, from the operator allowlist, then drop. No 80/443: nothing is served from here, so +# there is no ACME challenge to answer and no endpoint to expose. +resource "upcloud_firewall_rules" "prefetch" { + server_id = upcloud_server.prefetch.id + + dynamic "firewall_rule" { + for_each = local.operator_ranges + content { + direction = "in" + action = "accept" + protocol = "tcp" + family = "IPv4" + source_address_start = firewall_rule.value.start + source_address_end = firewall_rule.value.end + destination_port_start = "22" + destination_port_end = "22" + comment = "Operator SSH" + } + } + + # No implicit deny on UpCloud: this catch-all MUST be last. + firewall_rule { + direction = "in" + action = "drop" + } +} diff --git a/terraform/prefetch/outputs.tf b/terraform/prefetch/outputs.tf new file mode 100644 index 0000000..1ad5c55 --- /dev/null +++ b/terraform/prefetch/outputs.tf @@ -0,0 +1,12 @@ +output "ssh_host" { + description = "Public IP of the prefetch box (Ansible/SSH target)." + value = upcloud_server.prefetch.network_interface[0].ip_address +} + +output "server_uuid" { + value = upcloud_server.prefetch.id +} + +output "operator_allowlist" { + value = local.operator_cidrs_norm +} diff --git a/terraform/prefetch/providers.tf b/terraform/prefetch/providers.tf new file mode 100644 index 0000000..75eaf91 --- /dev/null +++ b/terraform/prefetch/providers.tf @@ -0,0 +1,2 @@ +# Auth via the UPCLOUD_TOKEN env var (a ucat_ token). Never commit it. +provider "upcloud" {} diff --git a/terraform/prefetch/variables.tf b/terraform/prefetch/variables.tf new file mode 100644 index 0000000..65ae2e0 --- /dev/null +++ b/terraform/prefetch/variables.tf @@ -0,0 +1,84 @@ +variable "zone" { + type = string + default = "fi-hel2" # must match the zone the weights disk lives in +} + +variable "hostname" { + type = string + default = "gpu-spinner-prefetch" +} + +variable "plan" { + description = <<-EOT + Plan for the download box. This is the whole point of the stack: a GPU sits idle during a + weights pull, and on 4x RTX PRO 6000 that idle time costs 6.60 EUR/h against 0.0446 for + 2xCPU-4GB — about 150x. Downloads are network- and disk-bound, not CPU-bound, so a small + plan is not slower in any way that matters. + + RAM is the binding constraint, not CPU. Measured 2026-09-20: on 2xCPU-4GB (3.8 GB usable, no + swap) the `hf` downloader reached ~3.5 GB RSS and the kernel OOM-killed it five times out of + ten profiles — the Xet backend buffers chunks in memory and scales with concurrency, not with + file size. CLOUDNATIVE-2xCPU-8GB is both cheaper (0.0357/h vs 0.0446) and twice the RAM, so it + is the default; the play also adds swap. Prices come from /1.3/price; tests/plans.txt covers + GPU plans only. Its storage_size is 0, which just means no bundled disk — the template block + below supplies one. + EOT + type = string + default = "CLOUDNATIVE-2xCPU-8GB" +} + +variable "os_template" { + description = <<-EOT + OS template TITLE or UUID. A plain Ubuntu image, NOT the GPU (NVIDIA+CUDA) template the + ephemeral stack uses — there is no GPU here and the CUDA image is 20 GB of boot disk this + box never reads. Discover values with: upctl storage list --public --template + EOT + type = string + default = "Ubuntu Server 24.04 LTS (Noble Numbat)" +} + +variable "os_disk_size_gb" { + description = "Boot disk (GB). Nothing large lands here: weights and Docker images both go to the attached /data disk. Destroyed with the server." + type = number + default = 40 +} + +variable "ssh_public_keys" { + description = "SSH public keys installed for root login." + type = list(string) + default = [] +} + +variable "operator_ip" { + description = "Single public IP to allowlist for SSH. Empty = auto-detect via api.ipify.org. Ignored if operator_cidrs is set." + type = string + default = "" +} + +variable "operator_cidrs" { + description = "Allowlisted source IPs/CIDRs for SSH (IPv4 only; a bare IP is treated as /32). Takes priority over operator_ip." + type = list(string) + default = [] +} + +variable "persistent_state_path" { + description = "Override path to the persistent stack's local state file. Empty = anchored to this module dir." + type = string + default = "" +} + +variable "weights_disk" { + description = <<-EOT + Which persistent data disk to attach: "primary" (the shared cache) or "alt" (the optional + second disk, see weights_alt_size_gb in terraform/persistent). bin/spin passes this from + WEIGHTS_DISK. Choosing "alt" when no alt disk exists is an error rather than a silent + fallback — attaching the wrong disk would quietly re-download or evict someone's cache. + EOT + type = string + default = "primary" + + validation { + condition = contains(["primary", "alt"], var.weights_disk) + error_message = "weights_disk must be \"primary\" or \"alt\"." + } +} diff --git a/terraform/prefetch/versions.tf b/terraform/prefetch/versions.tf new file mode 100644 index 0000000..e55100e --- /dev/null +++ b/terraform/prefetch/versions.tf @@ -0,0 +1,14 @@ +terraform { + required_version = ">= 1.7" + + required_providers { + upcloud = { + source = "UpCloudLtd/upcloud" + version = ">= 5.25, < 6.0" + } + http = { + source = "hashicorp/http" + version = "~> 3.4" + } + } +} diff --git a/terraform/providers/upcloud/data.tf b/terraform/providers/upcloud/data.tf index e8b1086..5c37636 100644 --- a/terraform/providers/upcloud/data.tf +++ b/terraform/providers/upcloud/data.tf @@ -20,7 +20,8 @@ data "terraform_remote_state" "persistent" { locals { floating_ip = data.terraform_remote_state.persistent.outputs.floating_ip sslip_host = data.terraform_remote_state.persistent.outputs.sslip_host - weights_id = data.terraform_remote_state.persistent.outputs.weights_storage_id + _alt_id = try(data.terraform_remote_state.persistent.outputs.weights_alt_storage_id, "") + weights_id = var.weights_disk == "alt" ? local._alt_id : data.terraform_remote_state.persistent.outputs.weights_storage_id # Allowlisted sources for SSH(22) + HTTPS(443): explicit CIDRs, else a single # operator_ip, else this machine's auto-detected public IP. Bare IPs -> /32. diff --git a/terraform/providers/upcloud/terraform.tfvars.example b/terraform/providers/upcloud/terraform.tfvars.example index c79ef67..f256a7b 100644 --- a/terraform/providers/upcloud/terraform.tfvars.example +++ b/terraform/providers/upcloud/terraform.tfvars.example @@ -11,6 +11,9 @@ ssh_public_keys = [ # --- Optional overrides --- # plan = "GPU-8xCPU-64GB-1xL40S" # default; smallest tier the current models fit (L4 24GB fits none) +# other tiers: GPU-16xCPU-80GB-1xRTXPRO6000 (96 GB, every FP8 profile + dense NVFP4) +# GPU-12xCPU-240GB-1xH100 · GPU-24xCPU-240GB-1xB200 · GPU-96xCPU-960GB-4xB200 +# full list: https://upcloud.com/docs/products/gpu-servers/configurations/ # operator_ip = "" # empty = auto-detect via api.ipify.org # operator_cidrs = ["198.51.100.0/24", "203.0.113.0/24"] # YOUR egress range(s) for a rotating IP (takes priority over operator_ip) # os_disk_size_gb = 100 diff --git a/terraform/providers/upcloud/variables.tf b/terraform/providers/upcloud/variables.tf index 55d1de4..9f01bfe 100644 --- a/terraform/providers/upcloud/variables.tf +++ b/terraform/providers/upcloud/variables.tf @@ -9,7 +9,7 @@ variable "hostname" { } variable "plan" { - description = "UpCloud GPU plan. Default L40S — the smallest tier that fits the current FP8 models (the L4's 24 GB fits none of them). Use --plan for H100 (GPU-12xCPU-240GB-1xH100) or B200 (GPU-24xCPU-240GB-1xB200). Verify with `upctl zone devices fi-hel2`." + description = "UpCloud GPU plan. bin/spin normally supplies this from the model profile's min_plan (or a swap preset's swap_plan), so this default only applies when nothing else resolves — it stays L40S to match the default model, qwen36. The full allowlist is tests/plans.txt. Identifiers: https://upcloud.com/docs/products/gpu-servers/configurations/" type = string default = "GPU-8xCPU-64GB-1xL40S" } @@ -64,3 +64,19 @@ variable "persistent_state_path" { type = string default = "" } + +variable "weights_disk" { + description = <<-EOT + Which persistent data disk to attach: "primary" (the shared cache) or "alt" (the optional + second disk, see weights_alt_size_gb in terraform/persistent). bin/spin passes this from + WEIGHTS_DISK. Choosing "alt" when no alt disk exists is an error rather than a silent + fallback — attaching the wrong disk would quietly re-download or evict someone's cache. + EOT + type = string + default = "primary" + + validation { + condition = contains(["primary", "alt"], var.weights_disk) + error_message = "weights_disk must be \"primary\" or \"alt\"." + } +} diff --git a/tests/check-cost-gate.sh b/tests/check-cost-gate.sh new file mode 100755 index 0000000..371da81 --- /dev/null +++ b/tests/check-cost-gate.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Unit test for the plan cost gate (bin/lib/plan.sh), which replaced the per-profile +# `requires_review` flag on 2026-09-21. That flag had a regression test; this keeps the coverage. +# +# Pure shell, no network, no cloud: it sources the library and drives plan_price / +# assert_plan_affordable directly. +set -uo pipefail +cd "$(dirname "$0")/.." || exit 1 + +# assert_plan_affordable calls die/warn/log from common.sh. +# common.sh's die EXITS; a stub that merely returns lets execution fall through the guard and the +# caller sees success. Inside $( ) this exits the subshell, so rc=1 reaches the test. +die() { printf 'die: %s\n' "$*"; exit 1; } +warn() { printf 'warn: %s\n' "$*"; } +log() { printf 'log: %s\n' "$*"; } +# shellcheck source=bin/lib/plan.sh +. bin/lib/plan.sh + +fail=0 +check() { # check