Repository navigation
Prune in_list expressions - #10617
Merged
Merged
Conversation
robert3005
commented
Oct 19, 2026
robert3005
commented
Contributor
- Make large IN lists cheap to evaluate and prune
- Keep IN lists with null elements on the set lookup path
- Turn integer IN lists covering a range into range checks
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | add_constant_shapes_neon[(32768, PerRowNullableConstant)] |
14.8 µs | 16.6 µs | -10.6% |
| ⚡ | WallTime | mul_u64_nonnull_neon |
39.9 µs | 20.9 µs | +91.4% |
| ⚡ |
Simulation | bench_compare_sliced_dict_primitive[(2000, 10000)] |
13.9 µs | 12.4 µs | +12.04% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/in-list-filters (ae60054) with develop (7216cb5)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
robert3005
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
robert3005
added
changelog/feature
github-actions
Bot
removed
the
action/bench-sql
robert3005
changed the title
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.021x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.902x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.015x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.976x ➖, 3↑ 3↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.047x ➖, 6↑ 17↓)
duckdb / parquet / ns (1.010x ➖, 1↑ 5↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 2↑ 5↓)
duckdb / parquet / ns (1.009x ➖, 1↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.283x ➖, 0↑ 9↓)
datafusion / parquet / ns (1.107x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.120x ➖, 1↑ 6↓)
duckdb / parquet / ns (1.133x ➖, 0↑ 0↓)
|
myrrc
reviewed
Oct 20, 2026
robert3005
changed the title
joseph-isaacs
approved these changes
Oct 20, 2026
robert3005
force-pushed
the
rk/in-list-filters
branch
2 times, most recently
from
October 20, 2026 10:10
3e45a47 to
131f602
Compare
joseph-isaacs
enabled auto-merge (squash)
October 20, 2026 10:18
robert3005
force-pushed
the
rk/in-list-filters
branch
from
October 20, 2026 11:31
e261c34 to
944f178
Compare
Convert a DataFusion IN list of non-null literals into a single Vortex in_list over a constant list, which executes as one prepared-set lookup instead of an OR of equalities that scans the needle once per element. Dynamic hash join filters push IN lists of up to 150 values. Falsify lists of more than 16 elements with the bounds of the list rather than one test per element, keeping the pruning expression small, and evaluate a file-stats pruning expression once per reader instead of once per concurrently starting split. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr (cherry picked from commit 6ba6e07)
Convert IN lists with null elements to a set lookup too, using a nullable list whose SQL null semantics in_list preserves. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr
Add vortex_array::expr::in_list_of, which builds needle IN (constants) as a non-strict BETWEEN when the elements are integers holding every value between their bounds, and as a set lookup over sorted elements otherwise. Join filters push such lists for the keys of small dimension tables, for example the 31 days of a month or all ten warehouses in TPC-DS q39. A range is cheaper to evaluate, prunes with tight bounds, and file statistics can prove it always true. Sorted elements make equal filters deduplicate. Use it for IN lists from DataFusion and from DuckDB table filters and expressions. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr (cherry picked from commit 49b3f58)
Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr
TrustedLenAdapter::next was a manual Option::map once its debug-only checks compile away, which clippy rejects with --release. Check the item after taking it instead. Replace lossy casts in the run-end filter test and fully qualified paths in the DataFusion conversion tests. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr
IN lists now show their elements sorted, integer IN lists that cover a range show as range checks, and some scans list their filters in a different order. No query results change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
The IN list changes do not change q41. Its two filters on i_size and i_units only showed in a different order when the plans were regenerated on macOS. CI on Linux still produces the original order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/in-list-filters
branch
from
October 20, 2026 12:32
944f178 to
ae60054
Compare
robert3005
disabled auto-merge
October 20, 2026 12:56
yiquanzhou
added a commit
to yiquanzhou/vortex
that referenced
this pull request
Oct 20, 2026
…t-pushdown * upstream/develop: Prune in_list expressions (vortex-data#10617) Fix semantic conflict in c++ tests (vortex-data#10638) Signed-off-by: Yiquan Zhou <yiquan.zhou@dash0.com> # Conflicts: # vortex-datafusion/src/convert/exprs/tests.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.