Summary
On Windows, llama-server can launch cleanly with a static VRAM reservation that only fits while nothing else uses the GPU. At request time, the dynamic compute buffers push the process over the WDDM device limit and Windows silently migrates pages to shared system memory — no CUDA error, no log line. Throughput then collapses by roughly an order of magnitude, with a symptom signature (GPU util ~100%, memory bandwidth ~2%, CPU idle, clocks/temps/power all normal) that looks exactly like a mysterious software bug.
On my machine this masqueraded for a day as a "draft-MTP + this-GGUF combination-specific prefill bug" before a single-variable re-test exonerated the combination and pinned the real cause on the spill.
Note that TurboQuant's small KV cache is precisely what makes it tempting to size -c right up against the VRAM limit — i.e. this fork's best feature steers users into the danger zone.
Request
- Startup VRAM watermark. After graph/KV/compute-buffer allocation, log a one-line budget: static requirement vs total device memory and the headroom — and emit an explicit
WARNING when headroom is near-zero or negative, with an actionable hint ("reduce -c or KV size; on Windows the driver will silently spill to shared memory otherwise").
- Docs note for Windows/WDDM users: the silent-spill behavior, the symptom signature below, and how to measure per-process shared usage during requests.
- (Optional) a one-shot runtime warning when prompt-eval throughput falls far below a startup-measured reference for the same config.
Environment
- atomic-llama-cpp-turboquant build b10269-1.5.1,
llama-server CUDA build
- Windows 11 Pro (26200), NVIDIA driver 616.64
- RTX 5080 16 GB (16303 MiB); desktop GUI typically occupies ~460–550 MiB of it
- Qwen3.8-27B dense, IQ3_S quant (11.29 GiB), draft-MTP speculative decoding (single slot), turbo4 KV cache,
-fa on, -ngl all, -b 512 -ub 512, --no-mmproj-offload
Cleanest repro (single variable — same binary/GGUF/flags, only -c differs, same evening)
-c |
static VRAM |
prefill (t/s) |
decode (t/s) |
| 115000 |
15757 MiB |
1646 |
60.5 |
| 130000 |
15905 MiB (+148 MiB) |
182.6 |
50.9 |
The extra 148 MiB crosses the point where runtime compute-buffer allocation spills, and prefill falls 9×. In a long real session at -c 130000 the degradation worsens as context grows: prefill 73→29 t/s, decode 18.5→11.2 t/s — while the same probe at -c 115000 reads prefill 1719–1767, decode 33.7–34.8.
Why it's treacherous: it forged a config bug
The collapse initially looked like a "draft-MTP + this-GGUF prefill bug" (~12.5×: ~1756 → ~140–183 t/s). Re-test with the identical model + speculative config + KV at -c 40000: prefill 1611 t/s, decode 61.5 — completely healthy. Every historical "prefill ~180 t/s" figure on this machine turned out to be a spill artifact; the hardware's real prefill is ~1750 t/s.
Invisible to static checks
At startup with -c 130000, per-process shared usage is ~0 and nothing warns. The spill only appears once a request activates the compute buffers: measured GPU Process Memory \ Shared Usage of 418–994 MiB during decode. After reducing -c, shared memory stays at ~196–426 MiB (cold KV tail pages) with zero speed impact — so spill volume alone is not the discriminator, static fit is. That argues for the startup watermark check rather than relying on runtime heuristics.
Symptom signature (worth a line in the docs)
All axes slow simultaneously; GPU util pinned ~100% but memory bandwidth ~2%; CPU idle; temps/power/clocks normal (no throttling); zero errors anywhere in the server log.
Measurement notes (for reproducibility)
- prefill/decode split from
--metrics counters (prompt_seconds_total / tokens_predicted_seconds_total), never wall-clock;
- prompt parts reordered between runs to bust the prompt cache;
- live shared usage via
Get-Counter '\GPU Process Memory(*)\Shared Usage' filtered to the server PID while a request is in flight.
Side observation (probably deserves its own issue)
On the IQ3_S GGUF above, -b/-ub 2048 shows an extreme prefill cliff: 211 t/s (−88%) vs 1742/1763 at 512/1024 — reproducible with no VRAM pressure (verified healthy mode). The same sweep on another GGUF of the same architecture (UD-IQ3_XXS) is flat: 1656/1808/1728, i.e. within noise. Mechanism unknown; happy to split this into its own issue if useful.
Summary
On Windows,
llama-servercan launch cleanly with a static VRAM reservation that only fits while nothing else uses the GPU. At request time, the dynamic compute buffers push the process over the WDDM device limit and Windows silently migrates pages to shared system memory — no CUDA error, no log line. Throughput then collapses by roughly an order of magnitude, with a symptom signature (GPU util ~100%, memory bandwidth ~2%, CPU idle, clocks/temps/power all normal) that looks exactly like a mysterious software bug.On my machine this masqueraded for a day as a "draft-MTP + this-GGUF combination-specific prefill bug" before a single-variable re-test exonerated the combination and pinned the real cause on the spill.
Note that TurboQuant's small KV cache is precisely what makes it tempting to size
-cright up against the VRAM limit — i.e. this fork's best feature steers users into the danger zone.Request
WARNINGwhen headroom is near-zero or negative, with an actionable hint ("reduce-cor KV size; on Windows the driver will silently spill to shared memory otherwise").Environment
llama-serverCUDA build-fa on,-ngl all,-b 512 -ub 512,--no-mmproj-offloadCleanest repro (single variable — same binary/GGUF/flags, only
-cdiffers, same evening)-cThe extra 148 MiB crosses the point where runtime compute-buffer allocation spills, and prefill falls 9×. In a long real session at
-c 130000the degradation worsens as context grows: prefill 73→29 t/s, decode 18.5→11.2 t/s — while the same probe at-c 115000reads prefill 1719–1767, decode 33.7–34.8.Why it's treacherous: it forged a config bug
The collapse initially looked like a "draft-MTP + this-GGUF prefill bug" (~12.5×: ~1756 → ~140–183 t/s). Re-test with the identical model + speculative config + KV at
-c 40000: prefill 1611 t/s, decode 61.5 — completely healthy. Every historical "prefill ~180 t/s" figure on this machine turned out to be a spill artifact; the hardware's real prefill is ~1750 t/s.Invisible to static checks
At startup with
-c 130000, per-process shared usage is ~0 and nothing warns. The spill only appears once a request activates the compute buffers: measuredGPU Process Memory \ Shared Usageof 418–994 MiB during decode. After reducing-c, shared memory stays at ~196–426 MiB (cold KV tail pages) with zero speed impact — so spill volume alone is not the discriminator, static fit is. That argues for the startup watermark check rather than relying on runtime heuristics.Symptom signature (worth a line in the docs)
All axes slow simultaneously; GPU util pinned ~100% but memory bandwidth ~2%; CPU idle; temps/power/clocks normal (no throttling); zero errors anywhere in the server log.
Measurement notes (for reproducibility)
--metricscounters (prompt_seconds_total/tokens_predicted_seconds_total), never wall-clock;Get-Counter '\GPU Process Memory(*)\Shared Usage'filtered to the server PID while a request is in flight.Side observation (probably deserves its own issue)
On the IQ3_S GGUF above,
-b/-ub 2048shows an extreme prefill cliff: 211 t/s (−88%) vs 1742/1763 at 512/1024 — reproducible with no VRAM pressure (verified healthy mode). The same sweep on another GGUF of the same architecture (UD-IQ3_XXS) is flat: 1656/1808/1728, i.e. within noise. Mechanism unknown; happy to split this into its own issue if useful.