prefill-cost

What a prompt costs to prefill when the weights do not fit in the GPU.

Prompt processing is the expensive half of long-context inference on a small card, and it is the half that reads from wherever the experts actually live — VRAM, host RAM, one NVMe drive, or a striped array. This repo collects every prefill measurement the sybil-solutions campaigns have made, normalized to one row per (run, prompt length), each row citing the file it came from. The point is not a leaderboard: it is to show what each storage tier actually buys, and where prefill stops being storage-bound at all.

data/prefill.jsonl is the dataset. data/summary.json counts it. docs/schema.md is the row contract. Numbers below are generated from the dataset by tools/render.py.

Short version

• Four M.2 drives in RAID0 prefill 2.7x faster than one M.2 drive at a 16 GB RAM cap (650 vs 242 tok/s, 8k prompt, GLM-5.3-Flash on one RTX 3090, campaign N129, same day and image), and 1.65x at a 55 GB cap (664 vs 402).
• Prefill buys bandwidth; decode buys drive count. Capping that array to a single drive's 7 GB/s puts prefill back at the single drive's rate (393 vs 402 tok/s at 55 GB, 254 vs 242 at 16 GB) — so prefill's gain is bandwidth, and nothing else. The same capped array still decodes 29-41 % faster than the real single drive (14.34 vs 10.16 tok/s at 55 GB, 4.52 vs 3.50 at 16 GB). Prefill keeps tens of thousands of reads in flight and saturates the link; decode has a few dozen per token and pays per-read latency, where 0.8-1.3 ms beats 4.8-7.0 ms. If you are sizing an array for prefill alone, buy bandwidth; the queue-depth win is a decode win.
• Prefill gets cheaper per token as the prompt grows. On a single 990 PRO with experts in host RAM, DeepSeek-V4.1-Flash goes 589 tok/s at 8k to 813 at 65k, and the marginal cost of the tokens between 8k and 32k is ~866 tok/s. The first chunks pay warm-up; the rest run near the card's own rate. Absolute TTFT still climbs to 342 s for a 261k-token prompt.
• Prefill costs about 8x more on the small box. Per 1,000 prompt tokens: 128 ms on 4x RTX PRO 6000, 1,027 ms on one 3090 with the RAID0 store at 32k (and 1,506 ms at 8k with the same store at a 55 GB cap). A 100k-token prompt is ~13 s of prefill on the big box against ~103 s on the little one — before it generates a token.
• When prefill is already GPU-bound, storage stops mattering. On an Arc Pro B70 whose experts sit in RAM, the same 7 GB/s cap costs nothing measurable (975 vs 1064 tok/s, inside run-to-run spread) because prefill reads only 4-5 GB/s, 16-20 % of the link. A faster array would buy that box nothing.

1. The same prompt, four storage tiers

Campaign N129 (2026-10-08) is the controlled comparison: one host, one image, one set of engine settings, and only the store changed. 4x M.2 array, capped to 1 drive's bandwidth is the same array with docker --device-read-bps at 7.0 GB/s — the arm that separates bandwidth from drive count.

store	RAM cap	prefill tok/s	TTFT s	ms / 1k prompt tokens	decode C1 tok/s	NVMe read GB/s	share of link ceiling	avg read latency ms	stall ms / token
1x M.2 drive	16 GB	242.2	33.82	4128.82	3.50	6.46	87%	7.04	-
4x array, capped to 1 drive's bandwidth	16 GB	254.3	32.21	3932.36	4.52	6.83	98%	1.88	-
4x M.2 array in RAID0	16 GB	649.7	12.61	1539.17	12.79	17.26	66%	0.72	38.4
1x M.2 drive	55 GB	402.4	20.36	2485.09	10.16	6.77	91%	4.84	46.6
4x array, capped to 1 drive's bandwidth	55 GB	393.3	20.83	2542.59	14.34	6.72	96%	1.31	23.5
4x M.2 array in RAID0	55 GB	663.8	12.34	1506.48	17.43	11.18	43%	0.84	10.6

Read the I/O columns next to the speeds, and prefill looks purely bandwidth-limited. The single drive moves 6.5 GB/s of its 7.45 GB/s ceiling (87 %) at 49k IOPS; the array at 16 GB moves 17.3 GB/s of 26 GB/s (66 %) at 148k IOPS. The capped array sits within 5 % of the real single drive on prefill at both RAM caps — 254 vs 242 tok/s at 16 GB, 393 vs 402 at 55 GB — which is what makes it a fair stand-in, and what proves the 4-drive prefill win is the extra bandwidth.

The decode column is where drive count earns more than its bandwidth. At the same 7 GB/s the array decodes 41 % faster than the single drive at 55 GB (14.34 vs 10.16 tok/s) and 29 % faster at 16 GB (4.52 vs 3.50), because decode holds only a few dozen reads in flight per token and so pays the per-read latency: 0.8-1.3 ms on the array against 4.8-7.0 ms on the single drive.

32k prompts: single 990 PRO 338 tok/s, capped array 423, RAID0 957. The gap widens with prompt length because a longer prefill issues more reads.

2. What the next thousand prompt tokens cost

Prefill speed is not a constant, so the useful number is marginal cost. Each block below is one single-run scan at C1; marginal tok/s is the speed of the tokens added over the row above, derived from TTFT.

DeepSeek-V4.1-Flash (EXL3 3.0 bpw on one 3090, experts pinned in host RAM, run D119)

prompt tokens	prefill tok/s	TTFT s	ms / 1k tokens	marginal tok/s	marginal ms / 1k tokens
8,192	589.1	13.9	1697.5	-	-
32,768	774.6	42.3	1291.0	866	1155.2
65,536	813.2	80.6	1229.7	856	1168.5
131,072	791.4	165.6	1263.6	771	1297.6
196,608	781.1	251.7	1280.2	761	1313.3
261,000	764.1	341.6	1308.7	716	1395.8

Qwen3.8-Flash-Next (EXL3 3.05 bpw on one 3090 whose experts live on a Mac mini's SSD over USB4, config C)

prompt tokens	prefill tok/s	TTFT s	ms / 1k tokens	marginal tok/s	marginal ms / 1k tokens
2,565	176.7	14.5	5659.3	-	-
8,018	417.8	19.2	2393.5	1166	857.3
18,330	470.1	39.0	2127.2	521	1920.2
33,842	584.2	57.9	1711.7	819	1220.8
97,811	560.9	175.0	1782.8	546	1830.1
189,481	548.1	346.0	1824.5	536	1865.4

3. What the wider set adds

The controlled experiment above is GLM-only. The rest of the dataset is breadth: 34 datasets across DeepSeek, Qwen, GLM, Nemotron, Welp and LFM checkpoints on everything from one 3090 to 4x RTX PRO 6000, DGX Sparks, Strix Halo, an M-series Mac and a USB4 eGPU. Each bullet below is one dataset's fastest single-stream prefill.

• 7,817 tok/s at 32k tokens on DeepSeek-V4.1-Flash with the Engram tables on local NVMe (deepseek-v4.1-flash-4x-rtx-pro-6000, 4x RTX PRO 6000). That repo's metric includes queueing, so treat it as an upper bound; it still reports 5,845 tok/s at 500k.
• 4,397 tok/s at 32k, and flat from 8k to 64k (4,143-4,397), for Qwen3.8-Flash-Next MXFP4-FP8-GPTQ on 2x R9700 with the experts in host RAM (local-ai-recipe-kit-submissions-2x-r9700-32gb, a community submission to the kit).
• 2,664 tok/s at 106,294 tokens for Qwen3.8-Flash-Next Q2_0 on one RTX 3090 reading a Strata NVMe expert pack (local-ai-registry-gates-rtx-3090-24gb). The IQ3_S pack on the same prompt measures 2,174.
• 2,421 tok/s at ~4k for Qwen3.8-27B EXL3 on one Arc Pro B70 with W8A8 int8 prefill linears, against 1,666 for the same card without them (trellis-serve-xpu-b70-int8 vs trellis-serve-xpu-b70). On a VRAM-resident model, prefill is a kernel problem, not a storage one.
• 1,015 tok/s at 981,841 tokens, 967 s to first token, for GLM-5.3-Flash on 2x DGX Spark (glm53-tensorfold-2x-spark). The only sub-million-token prefill in the set, and the clearest picture of what a million-token prompt costs.
• 70 tok/s at 106k tokens on a Radeon 8060S iGPU (local-ai-registry-gates-radeon-8060s-64gb), against 2,664 for the 3090 above. Same prompt shape, 38x apart.

Every dataset

dataset	family	model	host	storage	prompt tokens	rows	fastest prefill tok/s	at tokens	ms / 1k tokens
deepseek-v4.1-flash-4x-rtx-pro-6000	deepseek	DeepSeek-V4.1-Flash	-	auxiliary tables on disk, not stated	512-500000	89	7816.6	32768	128
local-ai-recipe-kit-submissions-2x-r9700-32gb	qwen	Qwen3.8-Flash-Next	-	all experts in host RAM	8192-65536	16	4396.7	32768	227
deepseek-v4.1-flash-h200	deepseek	DeepSeek-V4.1-Flash	-	all experts in host RAM, auxiliary tables on disk	8192-32980	32	3197.9	32737	313
local-ai-recipe-kit-3090	qwen	Qwen3.8-Flash-Next	omarchy	1x M.2 drive	8192-65536	4	2848.0	32768	351
local-ai-recipe-kit-targets-rtx-3090-24gb	qwen	Qwen3.8-Flash-Next	-	1x M.2 drive	8192-65536	4	2846.0	32768	351
local-ai-recipe-kit-submissions-rtx-3090-24gb	qwen	Qwen3.8-Flash-Next	omarchy	1x M.2 drive	8192-65536	4	2836.9	32768	352
local-ai-registry-gates-rtx-3090-24gb	mixed	Qwen3.8-Flash-Next +3 more	rtx-3090-24gb	1x M.2 drive, all experts in host RAM, everything in VRAM	93437-173372	11	2664.0	106294	375
trellis-serve-xpu-b70-int8	qwen	Qwen3.8-27B	-	everything in VRAM	4096-253952	4	2421.0	4096	413
local-ai-registry-gates-dgx-spark-gb10-128gb	mixed	GLM-5.3-Flash +1 more	dgx-spark-gb10-128gb	one shared memory pool	175457-751923	3	2094.0	751923	478
glm53-tensorfold-3x-spark	glm	GLM-5.3-Flash	three DGX Sparks (TP=3, NCCL, KV_POOL_GIB=27)	one shared memory pool	8212-262169	6	2064.2	32790	484
local-ai-registry-gates-rtx-4080-super-16gb	qwen	Welp-35B-A3B	rtx-4080-super-16gb	everything in VRAM	213570	1	2028.0	213570	493
glm53-tensorfold-2x-spark	glm	GLM-5.3-Flash	two DGX Sparks (TP=2 over ConnectX-7 RoCE)	one shared memory pool	8219-981841	7	1978.9	32790	505
trellis-serve-xpu-b70	qwen	Qwen3.8-27B	-	everything in VRAM	4185-254125	22	1666.0	4200	600
qwen-serve-27b-3090-sglang	qwen	Qwen3.8-27B	-	everything in VRAM	8741-138452	40	1615.8	17397	619
moetier-qwen38fn	qwen	Qwen3.8-Flash-Next	omarchy	4x array, capped to 1 drive's bandwidth, 4x M.2 array in RAID0	8192-32768	55	1193.4	32768	838
qwen-serve-27b-b70-llamacpp	qwen	Qwen3.8-27B	-	everything in VRAM	2500-245000	12	1164.0	2500	859
local-ai-registry-gates-intel-arc-pro-b70-32gb	mixed	Qwen3.8-Flash-Next +1 more	intel-arc-pro-b70-32gb	1x M.2 drive, all experts in host RAM, everything in VRAM	213575-213612	7	1087.0	213612	920
qwen-serve-b70-offload	qwen	Qwen3.8-Flash-Next	omarchy	4x M.2 array in RAID0	942-7406	4	1029.5	7406	971
local-ai-recipe-kit-targets-intel-arc-pro-b70-32gb	qwen	Qwen3.8-Flash-Next	-	1x M.2 drive	8192-65536	4	1027.0	8192	974
local-ai-recipe-kit-b70	qwen	Qwen3.8-Flash-Next	-	1x M.2 drive	8192-65536	4	1022.7	8192	978
trellis-serve-xpu-b70-llamacpp	qwen	Qwen3.8-27B	-	everything in VRAM	7335-58172	2	999.1	7335	1001
glm-flash-lite	glm	GLM-5.3-Flash	omarchy	1x M.2 drive, 4x array, capped to 1 drive's bandwidth, 4x M.2 array in RAID0, all experts in host RAM, not stated	8192-32768	96	973.7	32768	1027
dsv41-flash-offload	deepseek	DeepSeek-V4.1-Flash	-	all experts in host RAM	8192-261000	23	813.2	65536	1230
qwen38-flash-next-3090-usb-offload	qwen	Qwen3.8-Flash-Next	mini	drive on the far side of USB	2565-189481	10	584.2	33842	1712
local-ai-registry-gates-rx-6700-xt-12gb	qwen	Qwen3.5-9B	rx-6700-xt-12gb	everything in VRAM	52583	1	559.0	52583	1789
local-ai-registry-gates-rtx-3060-12gb	qwen	Qwen3.6-35B-A3B +1 more	rtx-3060-12gb	all experts in host RAM, everything in VRAM	106252-106294	2	512.0	106252	1953
framework-research-phase0-strix	qwen	Qwen3.5-122B-A10B-REAP-20 +1 more	Strix Halo Framework Desktop	one shared memory pool	64-131072	33	443.0	512	2257
trellis-serve-xpu-b70-llamacpp-tp2	qwen	Qwen3.8-27B	-	everything in VRAM	7379-58352	2	341.4	7379	2929
framework-research-rpc-hetero	qwen	Qwen3.5-122B-A10B-REAP-20	halo (Strix Halo) + RTX 3090 over Wi-Fi RPC	drive on the far side of USB, one shared memory pool	20-8832	12	320.9	8832	3117
local-ai-registry-gates-apple-m5-pro-48gb	qwen	Qwen3.8-Flash-Next +1 more	apple-m5-pro-48gb	one shared memory pool	25740-166625	3	218.0	25740	4587
local-ai-registry-gates-apple-m5-pro-24gb	qwen	Qwen3.8-27B	apple-m5-pro-24gb	one shared memory pool	166655	1	176.0	166655	5682
local-ai-registry-gates-x86-64-avx2-cpu	other	LFM2.5-2.6B	x86-64-avx2-cpu	all experts in host RAM	5552	1	84.0	5552	11905
local-ai-registry-gates-radeon-8060s-64gb	qwen	Qwen3.8-27B	radeon-8060s-64gb	one shared memory pool	106252	1	70.0	106252	14286
local-ai-registry-gates-apple-m2-max-96gb	qwen	Qwen3.8-27B	apple-m2-max-96gb	one shared memory pool	166667	1	52.0	166667	19231

4. Method

• One row = one measured prefill: prompt_tokens, prefill_tok_s, and where the weights were read from. offload says whether prefill read anything outside VRAM.
• provenance distinguishes machine-read JSON (result-json, registry-run, sweep-json) from numbers read off a Markdown table by hand (transcribed). Hand-transcribed rows are the minority and are marked in the dataset.
• protocol quotes how prefill was measured, because it is what makes rows comparable or not: a median of 3 sweep reps is not the same measurement as one request per length.
• TTFT in the generated tables is the source's own where it states one; otherwise it is prompt_tokens / prefill_tok_s, marked with an asterisk.

5. Caveats

• Compare inside a dataset, not across. Protocols, host CPUs and engine versions differ.
• Session drift. Arms that ran hours apart on this host differ by up to ~17 %. A few-percent gap between different sessions is not a result; the N129 table above is one session per prompt size, and the 8k and 32k blocks ran in different sessions.
• Capped arms are emulations. nvme-capped reproduces one drive's bandwidth on a 4-drive array. It matches the real single drive within 5 % on prefill, but it is not the same hardware, and its read latency (1.3 ms) is nothing like the single drive's (4.8-7.0 ms).
• Aborted and lossy arms are still in the data. The notes field says so; nothing was silently dropped.
• Storage ceilings are the ones the source states for its own array (26.0 GB/s RAID0, 7.45 GB/s single drive). They are not theoretical interface limits.
• The local-ai-registry-gates-* datasets are not sweeps. They are six-gate validation runs; their prefill figure comes from one context-gate prompt with no TTFT instrument, so it is prompt_tokens / whole-request seconds, a lower bound on prefill speed. They are in here because they are the only measurements on a 3060, 4080 Super, 6700 XT, M2 Max and M5 Pro.
• Some dates are commit dates. Rows carrying date_is_commit: true come from repos that state no measurement date; the date is the commit that landed the table, which makes it an upper bound on when the run happened.
• Some prompt lengths are nominal. Transcribed tables label a cell "8K" while the harness sent 8,187 tokens. Where a dataset mixes nominal and measured lengths, prompt_tokens holds what the source states and notes says which kind it is.
• The datasets table's "fastest" column is single-stream only (concurrency 1 or unstated). Aggregate-throughput cells exist in the data (DeepSeek's C2/C4/C6/C8 cells, for one) and are higher; they are not per-stream prefill.

6. Reproduce

git clone https://github.com/sybil-solutions/prefill-cost && cd prefill-cost
for r in moetier glm-flash-lite dsv41-flash-offload qwen38-flash-next-3090-usb-offload; do
  git clone --depth 1 https://github.com/sybil-solutions/$r ../$r
done
python3 tools/extract.py --sources ..   # staging rows, machine-read
python3 tools/merge.py                  # schema check + data/prefill.jsonl
python3 tools/render.py                 # README tables + site/index.html

tools/extract.py covers the repos whose numbers are machine-readable in a stable place. The remaining rows were harvested per family into data/raw/{qwen,deepseek,other}.jsonl under the same schema; tools/merge.py validates and folds all of them together.

Provenance

Every row names its repo, commit and file; data/staging/sources.json carries the sha256 of each file that was machine-read.

repo	commits	files	rows
moetier	2da3ec482d	68	129
deepseek-v4.1-flash-4x-rtx-pro-6000	ce69105cb5	4	89
qwen-serve	730d8101d0	6	56
framework-research	7eb74068e3	2	45
local-ai-recipe-kit	ef9782ce0c	7	36
deepseek-v4.1-flash-h200	8a0104bc9f	5	32
local-ai-registry	c0604cff48	32	32
trellis-serve	a1f72be2ff	3	30
dsv41-flash-offload	08802b6fe0	7	23
glm-flash-lite	6fe077da79	12	22
glm53-flash-tensorfold-4x-spark	16a92c85d6	1	13
qwen38-flash-next-3090-usb-offload	9392e65a95	1	10

License

MIT. The measurements belong to the repos they came from; follow the links.