Weekly TPC-H SF1000 with memory monitoring; fix DataFusion Vortex scan over-decoding

[joseph-isaacs]

Summary

Adds a weekly TPC-H SF=1000 run on NVMe for DuckDB and DataFusion over Parquet and Vortex. At this scale the data (~6B lineitem rows) is larger than the runner's memory, so the run is as much a memory check as a benchmark: it records per-query status, wall time and peak memory, and samples system memory throughout data generation, conversion and every query.

The branch runs found that DataFusion on Vortex decoded far more data at once than on Parquet; this PR fixes that in vortex-datafusion. Results on c8gd.metal-24xl (188 GiB):

Engine	Parquet	Vortex	Peak memory
DuckDB	22/22	22/22	~112 GiB (Q9, Parquet)
DataFusion, before the scan fix	20/22 (Q18, Q21 OOM)	20/22 (Q9, Q18 OOM)	hits 188 GiB
DataFusion, with the scan fix	20/22 (unchanged)	21/22 (Q18 OOM)	Q9 at 173 GiB

DataFusion's hash join keeps its whole build side in memory and cannot spill, so the weekly run also runs DataFusion a second time with sort-merge joins and spilling. That configuration is not validated yet: its first attempt hit DataFusion's default 100 GB spill cap, which is now raised to 500 GiB. Query failures are reported in the job summary and do not fail the job.

Changes

• vortex-datafusion: default the scan concurrency to 1 under DataFusion. A Vortex scan runs concurrency * available_parallelism splits per stream (vortex-layout/src/scan/repeated_scan.rs), and DataFusion opens one stream per partition, one per core by default, so Vortex's default of 4 multiplied with it: ~37k decoded splits in flight on 96 cores. perf showed the memory was decoded output (to_arrow_decimal, take_views). At SF1000 the fix cut Q1 from 121 to 32 GiB and Q5 from 146 to 52 GiB, sped Q5 and Q8 from ~18s to ~12s, and let Q9 complete. vortex.scan_concurrency still overrides it.
• datafusion-bench: opt-in DATAFUSION_BENCH_MEMORY_LIMIT_GIB (a FairSpillPool behind TrackConsumersPool), DATAFUSION_BENCH_SPILL_DIR and DATAFUSION_BENCH_MAX_SPILL_GIB. All are unset by default, so existing benchmarks are unchanged.
• .github/workflows/tpch-sf1000.yml: weekly (Saturday) and workflow_dispatch (engine, scale factor, machine type, queries, iterations, per-query timeout). A build job, then one c8gd.metal-24xl per engine.
  • Parquet is generated with tpchgen-cli 3.0.0 (ZSTD(3), ~1M-row row groups); the in-repo generator writes each table single-threaded, which takes hours for SF1000 lineitem. It uses the same tpchgen-arrow 3.0 schema. data-gen then converts to Vortex.
  • DataFusion runs twice: with the default hash joins, and with prefer_hash_join=false, an 80 GiB memory limit and a 500 GiB spill cap (variant datafusion-spill in the summary).
  • Fails fast without enough free disk for the data plus spill files.
  • Stops systemd-oomd, which killed the whole runner cgroup (and lost the job's artifacts) on an out-of-memory query.
• scripts/bench-memory.py (stdlib only):
  • monitor samples system memory, page cache, swap, memory pressure and the largest process, tagged with the current phase.
  • run-queries runs each query in its own process with a raised OOM score, a timeout, and a watchdog that kills the query when available memory drops below a threshold, so the runner survives. Each query gets a status (ok, oom-killed, resources-exhausted, alloc-failed, timeout, error, skipped), wall time and peak RSS from wait4. --variant labels a differently configured run of the same engine.
  • summarize renders the job summary.

Data generation and conversion take ~2.5h (Vortex conversion of the single-file lineitem dominates); hash-join queries take ~10 min per engine. Results are uploaded as artifacts and not yet ingested into the benchmarks database.

Open follow-ups: DataFusion Q2, Q11 and Q16 still peak 5–7× higher on Vortex than on Parquet with identical plans (suspected: selective filters keeping whole decoded string buffers alive in hash join builds), and Vortex is ~27% slower than Parquet on DuckDB at SF1000 (Q5 and Q10 ~2×).

API Changes

vortex-datafusion: when scan_concurrency is unset, Vortex scans under DataFusion now use a per-core concurrency of 1 instead of the Vortex scan default of 4. Set vortex.scan_concurrency to restore the old behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FDifRZhxcVd5gvt3dsDhu5

[codspeed]

Merging this PR will regress 3 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

• bench_compare_primitive[(10000, 2)]

⚠️ 16 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_scan_filter[int_float]
• test_scan[int_str_float]
• zstd_compress_string
• decompress[u16, (4000, 256)]
• decompress[u32, (1000, 256)]
• test_scan_filter[int_str]
• ilike_contains
• multipolygon_mixed_validity
• take_varbin
• test_scan[int_str]
• …and 6 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

• test_scan_filter_to_arrow_array[int_float]
• test_repeated_scan_filter[int_float]
• test_repeated_scan_filter[int]
• test_scan_filter_to_arrow_array[int]
• test_scan_filter[int]
• test_repeated_scan_scalar_at[int_str_float]
• test_scan_filter[int_float]
• test_repeated_scan_filter[int_str_float]
• test_repeated_scan_scalar_at[int_float]
• test_scan_filter_to_arrow_array[int_str_float]
• …and 30 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 25 improved benchmarks
❌ 3 regressed benchmarks
✅ 2359 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌ ℹ️	Simulation	test_scan_filter_to_arrow_array[int_float]	3.5 ms	4.1 ms	-13.63%
❌ ℹ️	Simulation	test_repeated_scan_filter[int_float]	3.4 ms	3.9 ms	-10.95%
❌ ℹ️	Simulation	test_repeated_scan_filter[int]	2.5 ms	2.7 ms	-10.24%
⚡	WallTime	threshold_neon[8]	2.4 µs	1.3 µs	+79.33%
⚡	WallTime	filter_neon[8]	1,339 ns	921 ns	+45.39%
⚡	WallTime	threshold_neon[80]	4.9 µs	3.4 µs	+44.31%
⚡	WallTime	threshold_neon[64]	3.5 µs	2.5 µs	+38.34%
⚡	WallTime	filter_neon[64]	2.3 µs	1.8 µs	+30.44%
⚡	Simulation	isolated_runs[u64, (161, 944)]	168.2 µs	137.8 µs	+22.06%
⚡	Simulation	embedding_code_runs_expanded_baseline[4096]	1.9 ms	1.6 ms	+20.63%
⚡	Simulation	embedding_code_runs_expanded_baseline[2048]	1,016.3 µs	850.6 µs	+19.48%
⚡	Simulation	embedding_code_runs_expanded_baseline[1024]	548.4 µs	465.2 µs	+17.89%
⚡	Simulation	isolated_runs[u64, (256, 896)]	219.7 µs	189.3 µs	+16.07%
⚡	Simulation	embedding_runs_expanded_baseline[1024]	609 µs	525.7 µs	+15.85%
⚡	Simulation	embedding_runs_expanded_baseline[1536]	913.6 µs	789.1 µs	+15.77%
⚡	Simulation	fragmented_runs_expanded_baseline[1024]	623.2 µs	540.2 µs	+15.36%
⚡	Simulation	embedding_code_runs_expanded_baseline[512]	319.9 µs	278.9 µs	+14.71%
⚡	Simulation	embedding_runs_expanded_baseline[768]	516.1 µs	454.7 µs	+13.51%
⚡	WallTime	bitpack_blocked_compress_avx2	7.6 µs	6.7 µs	+13.26%
⚡	Simulation	fragmented_runs_expanded_baseline[512]	369.9 µs	329 µs	+12.44%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/tpch-sf1000-weekly (5aae915) with develop (6289e58)²}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (bc9e0e6) during the generation of this report, so 6289e58 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩