Repository navigation
Size filtered V2 scan splits by projected bytes and threads - #10800
joseph-isaacs wants to merge 1 commit into
Conversation
|
The Generated by Claude Code |
Merging this PR will degrade performance by 16.49%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
541031c to
90f3f23
Compare
Filtered V2 scans used fixed 1,048,576-row chunks. A DataFusion byte-range partition of a ~1M-row file then became a single chunk, so when the surviving rows sat in one partition, one thread did all the work. Size filtered chunks by the serialized bytes the projection reads per row, cached per projection in ScanCache, and by the partition's rows per thread. Cheap projections over large ranges keep 1M-row chunks. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
90f3f23 to
c888e4f
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.969x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.955x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.045x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
|
Summary
Split out of #10761. This is the scan V2 part of that PR.
Filtered V2 scans used fixed ~1M-row chunks. A DataFusion byte-range partition of a ~1M-row file was therefore a single chunk. When the selected rows all sat in one partition, as they do for PolarSignals time-range filters, one thread did all the work. This was the main reason V2 was 1.79x slower than V1 on PolarSignals.
Changes
TARGET_BYTES_PER_CHUNK. The byte count is computed once per projection and cached inScanCache.filtered_rows_per_chunksits inscan/execution/chunk.rsnext torows_per_chunk, and the two share a bytes-per-row helper.ScanRun::try_newuses it for filtered scans.rstestforfiltered_rows_per_chunkcovering a cheap projection, wide projections, and a partition spread over threads.This branch is rebased onto
developafter #10707.Checks run:
cargo clippy -p vortex-layout --all-targets --all-features -- -D warnings,cargo +nightly-2026-09-10 fmt -p vortex-layout --check, andcargo test -p vortex-layout --lib scan(180 passed). I did not run any benchmarks for this split on its own.🤖 Generated with Claude Code
https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC