Repository navigation
Accept DataFusion dynamic filters in VortexSource - #10375
Draft
joseph-isaacs wants to merge 8 commits into
Draft
joseph-isaacs wants to merge 8 commits into
joseph-isaacs wants to merge 8 commits into
Conversation
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (20)
Powered by Polar Signals Cloud |
Merging this PR will improve performance by 10.84%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | bitpack_blocked_compress_avx2 |
7.6 µs | 6.8 µs | +11.17% |
| ⚡ | WallTime | words_gather_dispatch_neon[65536] |
2.3 µs | 2.1 µs | +10.52% |
| Simulation | bench_compare_primitive[(10000, 2)] |
64.6 µs | < 1 ns | N/A |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/df-dynamic-filter-pushdown (722f10e) with develop (1ee4e3a)2
Footnotes
-
409 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(d451683) during the generation of this report, so 1ee4e3a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.019x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.980x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.978x ➖, 3↑ 1↓)
duckdb / parquet / ns (0.981x ➖, 2↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.971x ➖, 9↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 2↑ 0↓)
duckdb / parquet / ns (1.018x ➖, 1↑ 5↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Convert TopK and hash-join dynamic filters into live `vortex.dynamic` comparisons so the scan prunes zones as the filter tightens. Only `col op literal` bounds are tracked; anything else is relaxed to `true`, so the Vortex filter never drops rows the operator needs. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FL15XgcyJgZ6xkaMo8p5o3
Hash-join dynamic filters are complete by the time the probe side opens a file, so read their bounds once and apply them as static comparisons, but only when the file's min/max statistics suggest they keep at most half of the column's value range. Non-selective join bounds (e.g. TPC-H order and customer keys) otherwise add a row-level filter that costs more than it saves. Filters that are still changing (TopK) keep the live template. TPC-H SF1, DataFusion over Vortex, 30 samples per mode: geomean on/off went from 1.065 to 0.981; q07 0.70x, q21 0.71x. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FL15XgcyJgZ6xkaMo8p5o3
When a file's type differs from the table's, the expression adapter wraps a dynamic filter's columns in casts. The opener then treated the filter as an unpushable static filter and failed the query (ClickBench q23, `ORDER BY "EventTime" LIMIT 10`). Send every dynamic filter down the dynamic path, dropping any that can't be converted, and look through casts to the underlying column. Bounds on a cast column are only applied when the literal converts to the file type and back unchanged, so a lossy cast can never tighten the filter. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FL15XgcyJgZ6xkaMo8p5o3
Multi-column TopK filters have the form `a < v OR (a = v AND b < w)`, which previously yielded no bounds, leaving every row to pay for an always-true template (ClickBench q26 regressed 1.24x). Read bounds recursively: `=` gives both `<=` and `>=`, and an OR keeps, per column and direction, the looser bound when both sides have one, so the example yields `a <= v`. Incomparable values yield no bound. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FL15XgcyJgZ6xkaMo8p5o3
Partitioned hash joins publish `CASE route(key) WHEN i THEN bounds_i ... END`, which previously yielded no bounds, leaving probe-side files with an always-true filter. Bounds are now the hull over a CASE's results, with `false`/`NULL` branches (empty partitions) contributing nothing. When a still-changing filter already has bounds at file open, emit only the comparisons and columns those bounds use rather than the full template. Tidy the opener's conjunct split and share a values helper across the end-to-end tests, adding a hash-join case whose bounds are selective enough to be applied. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FL15XgcyJgZ6xkaMo8p5o3
The cached bounds are read on every evaluation of the Vortex filter but replaced only when the DataFusion filter changes, a few dozen times per query for TopK. Hold them in an ArcSwap instead of a mutex so concurrent reads don't contend; a refresh never replaces a newer generation. With four threads reading the same filter, a lookup drops from ~1.3 us to ~0.3 us. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FL15XgcyJgZ6xkaMo8p5o3
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.831x ➖, 4↑ 1↓)
datafusion / parquet / ns (0.802x ➖, 5↑ 0↓)
duckdb / vortex-file-compressed / ns (0.780x ➖, 4↑ 0↓)
duckdb / parquet / ns (0.810x ➖, 3↑ 0↓)
|
joseph-isaacs
force-pushed
the
ji/df-dynamic-filter-pushdown
branch
from
October 7, 2026 13:12
cb7a5d6 to
7120afd
Compare
joseph-isaacs
added
the
action/bench-sql-extended
github-actions
Bot
removed
the
action/bench-sql-extended
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.911x ➖, 4↑ 2↓)
duckdb / vortex-compact / ns (0.927x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.015x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.018x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.978x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 1↑ 0↓)
duckdb / duckdb / ns (1.014x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 110↑ 91↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.681x ✅, 4↑ 0↓)
datafusion / vortex-compact / ns (0.896x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.321x ✅, 9↑ 0↓)
duckdb / vortex-file-compressed / ns (0.712x ➖, 3↑ 0↓)
duckdb / vortex-compact / ns (0.840x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.541x ✅, 9↑ 0↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.977x ➖, 3↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 4↑ 3↓)
duckdb / parquet / ns (0.985x ➖, 2↑ 1↓)
duckdb / duckdb / ns (1.000x ➖, 1↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.993x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.021x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.981x ➖, 6↑ 1↓)
datafusion / vortex-compact / ns (1.122x ❌, 3↑ 45↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 2↑ 2↓)
duckdb / vortex-compact / ns (1.007x ➖, 2↑ 3↓)
duckdb / parquet / ns (1.005x ➖, 4↑ 4↓)
duckdb / duckdb / ns (1.013x ➖, 4↑ 3↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 1↑ 6↓)
datafusion / vortex-compact / ns (0.810x ➖, 5↑ 1↓)
datafusion / parquet / ns (0.822x ➖, 5↑ 0↓)
duckdb / vortex-file-compressed / ns (0.879x ➖, 4↑ 1↓)
duckdb / vortex-compact / ns (0.883x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.817x ➖, 1↑ 0↓)
|
Contributor
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.914x ➖, 3↑ 2↓)
datafusion / vortex-compact / ns (0.822x ➖, 4↑ 1↓)
datafusion / parquet / ns (0.772x ➖, 9↑ 0↓)
duckdb / vortex-file-compressed / ns (0.863x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.850x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.826x ➖, 2↑ 0↓)
|
Keep small, sparse integer IN lists when min/max bounds miss their pruning. Limit them to 16 keys and 10% estimated range occupancy because repeated comparisons can repeatedly decode compact columns. Avoid the NULL escape when a completed top-level comparison proves that its column cannot be NULL. Preserve it for live filters and nullable OR/CASE bounds. Add focused coverage for nullable ranges and selective, dense, oversized, negated, and disjunctive membership filters. All 36 dynamic-filter tests pass. Paired measurements cover 119 queries in both Vortex formats; the additional changes improve geometric mean latency by 3.8% and 10.4%. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow VortexSource to use DataFusion's TopK and hash-join dynamic filters to prune rows and zones. Live comparisons read generation-cached bounds through ArcSwap; unsupported predicates are relaxed so the reader keeps every row the producing operator may need.
Completed join filters become static comparisons only when file statistics predict useful pruning. Top-level comparisons can reject NULLs, while live filters and nullable OR/CASE bounds retain the NULL escape. Small, sparse positive integer IN lists can also be retained when their min/max bounds are too broad: at most 16 keys and at most 10% estimated range occupancy. The stricter membership threshold accounts for repeated decoding of compact encodings.
Benchmarked all 119 queries with reader dynamic filters after inspecting all 164 plans: 20 TPC-H SF10, 95 TPC-DS SF1, and full partitioned ClickBench IDs 23–26 (zero based). Both Vortex formats have the same inventory. All 119 ordinary-Vortex queries were profiled with Samply using matching ELF build IDs.
Additional reader optimizations versus the original PR head (
7120afde), warm median milliseconds:Across the 119 selected queries, latency-ratio geometric means improve 3.8% for Vortex and 10.4% for compact. Gains are concentrated in TPC-DS (4.9% and 12.7% respectively); TPC-H is effectively unchanged by these additional optimizations. Three further alternating process pairs (21 iterations each) for every selected ClickBench query in both formats found no reproducible regression from the additional changes. The 10% membership rule keeps Q24 at parity in both formats, avoiding repeated PCO decoding. Large hash-lookup predicates and encoding/gather bottlenecks remain; these changes do not make every dynamic-filter query faster. TPC-DS Q10/Q69 improve versus this PR's original head but remain slower than the pre-PR base on this data.
Measurements used local NVMe on ARM64, 16 DataFusion/Tokio/Rayon workers, CPU affinity 16–31, and 11 iterations per query/binary/format. The first invocation is excluded from warm medians; filesystem caches were not dropped. Binary order alternated by query ID. All 476 retained benchmark runs completed successfully with no concurrent compilation detected during the retained attempts.
Validation: 36 focused dynamic-filter tests pass, including TopK updates, casts, joins, NULL semantics, sparse membership, and rejection of costly/unsupported membership forms. The sparse-membership and completed-range NULL behaviors were demonstrated to fail on the original implementation. Formatting used
nightly-2026-09-10;git diff --checkpasses. Full workspace CI was not run. The existing benchmark harness checks expected row counts for TPC-H and ClickBench; TPC-DS result values were not independently compared against another engine.