Repository navigation
Speed up PolarSignals scans: spread filtered V2 splits and export primitive dictionaries natively - #10761
Closed
joseph-isaacs wants to merge 6 commits into
Closed
Speed up PolarSignals scans: spread filtered V2 splits and export primitive dictionaries natively#10761joseph-isaacs wants to merge 6 commits into
joseph-isaacs wants to merge 6 commits into
Conversation
joseph-isaacs
added
changelog/performance
Claude
github-actions
Bot
removed
the
action/bench-sql
Merging this PR will regress 1 benchmark4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (7)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.564x ✅, 5↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.971x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.990x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.917x ➖, 1↑ 0↓)
|
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
This was referenced Oct 26, 2026
joseph-isaacs
added a commit
that referenced
this pull request
Oct 26, 2026
BitPacked's CastKernel only took its direct unpack path for widening integer casts. A patch-free bit-packed array holds only values in 0..2^bit_width, so a narrowing cast whose target holds every such value cannot overflow. Take the same unpack path for those casts too, skipping the full-length intermediate buffer and the generic cast's bounds-check scan. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
joseph-isaacs
added a commit
that referenced
this pull request
Oct 26, 2026
Exporting a primitive array as an Arrow dictionary went through arrow_cast, which hashes each value with SipHash and builds an ArrowError per value. On PolarSignals this export was ~90% of scan CPU. Encode primitive, float (by bits), date and timestamp arrays natively, choosing in order: a dense table when the keys span fewer than 65,536 values; the values themselves with sequential codes when a sample is nearly all distinct (Arrow dictionaries may hold duplicates); otherwise an FxHash map sized from the sample. The full min/max scan only runs when the sample's range could fit the dense table. Nulls stay in the codes. Other types, and codes that would overflow, fall back to arrow_cast. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
joseph-isaacs
added a commit
that referenced
this pull request
Oct 26, 2026
BitPacked's CastKernel only took its direct unpack path for widening integer casts. A patch-free bit-packed array holds only values in 0..2^bit_width, so a narrowing cast whose target holds every such value cannot overflow. Take the same unpack path for those casts too, skipping the full-length intermediate buffer and the generic cast's bounds-check scan. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
joseph-isaacs
added a commit
that referenced
this pull request
Oct 26, 2026
ProjectionMorsel wrapped the pieces of a split in one ChunkedArray, which the Arrow export then concatenated, re-merging nested dictionaries. Emit pieces as separate batches in row order, grouping only pieces under 4,096 rows. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
joseph-isaacs
added a commit
that referenced
this pull request
Oct 26, 2026
Filtered V2 scans used fixed 1,048,576-row chunks. A DataFusion byte-range partition of a ~1M-row file then became a single chunk, so when the surviving rows sat in one partition, one thread did all the work. Size filtered chunks by the serialized bytes the projection reads per row, cached per projection in ScanCache, and by the partition's rows per thread. Cheap projections over large ranges keep 1M-row chunks. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
…tion pieces Filtered V2 scans used fixed 1,048,576-row chunks. A DataFusion byte-range partition of a ~1M-row file then became a single chunk, so when the surviving rows sat in one partition (PolarSignals time ranges) one thread did all the work. Size filtered chunks by the serialized bytes the projection reads per row and by the partition's rows per thread, keeping 1M-row chunks for cheap projections over large ranges. ProjectionMorsel also wrapped its pieces in a ChunkedArray, which the Arrow export concatenated, re-merging nested dictionaries. Emit pieces as separate batches, grouping only pieces under 4,096 rows. PolarSignals SF1 on DataFusion, V2 vs V1 geomean: 1.79x -> 0.93x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Exporting a non-dictionary Vortex array to an Arrow dictionary type went through arrow_cast, which hashes each primitive value with SipHash and builds an ArrowError per value. PolarSignals requests its nested `locations` fields as dictionaries, so this was ~90% of scan CPU for both V1 and V2. Encode primitive values directly, hashing their bits with FxHash and keeping nulls in the codes, and fall back to arrow_cast for other types or when the codes overflow. PolarSignals SF1 on DataFusion (V2): q00-q03 ~3.6x faster, total 4.28s -> 1.90s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
…ictionaries Hash-map growth and hashing still dominated exporting PolarSignals' nested dictionary fields. Keys spanning under 65,536 values (line numbers) are now looked up in a dense table. When a sample of the values is nearly all distinct (addresses), the values become the dictionary with sequential codes, reusing their buffer, since Arrow dictionaries may hold duplicates. Other inputs hash into a map sized from the sample. PolarSignals SF1 on DataFusion (V2): q00-q03 another 22-31% faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
A bit-packed array without patches only holds values in 0..2^bit_width, so its unpacked values are already codes into that range. Exporting it to an Arrow dictionary now unpacks straight into the codes type and uses 0..2^bit_width as the values, instead of decoding to the values type and dictionary-encoding the result. Applies up to 16-bit widths when the dictionary is no larger than the array. To unpack into narrower codes, BitPacked's cast kernel now also handles narrowing integer casts when the array has no patches and the target holds every value the bit width allows, skipping the generic bounds check. PolarSignals SF1 on DataFusion (V2): q00-q03 another ~2x faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
…ange The primitive dictionary export scanned every key for its range before choosing an encoding, even for high-cardinality columns such as addresses that can never use the dense table. Sample first, and scan the full range only when the sampled keys fit the dense table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
force-pushed
the
ji/jolly-shannon-f0erx1
branch
from
October 26, 2026 14:46
a47ec03 to
5994f41
Compare
robert3005
pushed a commit
that referenced
this pull request
Oct 26, 2026
## Summary Split out of #10761. This keeps only the `vortex-fastlanes` cast change; the bit-packed Arrow dictionary export from that PR is dropped. `BitPacked`'s `CastKernel` only took its direct unpack path for widening integer casts. Bit-packing rejects negative integers, so a patch-free array packed at a global `bit_width` holds only values in `0..2^bit_width`. A narrowing cast whose target holds every such value cannot overflow. One place this applies is the Arrow export of Vortex dictionaries. `dict_to_dict` exports the codes through `to_arrow_primitive`, which casts them to the requested key type. Dictionary codes are often bit-packed, so a key type narrower than the codes' ptype hits this kernel. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On PolarSignals with DataFusion, scan V2 from #10706 was 1.79x slower than V1 (geomean). This PR fixes that. It also speeds up both V1 and V2 by making the Arrow dictionary export cheaper; that export was ~90% of scan CPU on these queries for both scans. V2 and V1 now perform about the same on PolarSignals.
CI
action/bench-sqlona47ec03againstdevelop(CI's PolarSignals job runs V1):Local runs at SF1 on 4 cores, median of 3 rounds × 5 iterations, sides alternated:
Changes
Filtered V2 splits (
vortex-layout). Filtered scans used fixed ~1M-row chunks. A DataFusion byte-range partition of a ~1M-row file was therefore one chunk, so when the selected rows sat in one partition, a single thread did all the work. Chunks are now sized two ways:ScanCache;Cheap projections over large ranges keep the 1M-row chunks.
No re-concatenation (
vortex-layout).ProjectionMorselemitted its pieces as oneChunkedArray, which the Arrow export then concatenated, re-merging nested dictionaries. Pieces are now separate batches; only pieces under 4,096 rows are grouped.Primitive dictionary export (
vortex-arrow). This replacesarrow_cast, which hashed each value with SipHash and built anArrowErrorper value. The export now uses, in order of preference:The full min/max scan runs only when the sample's range could fit the dense table. Nulls stay in the codes. Other types, and codes that would overflow, fall back to
arrow_cast.Bit-packed arrays exported as dictionaries (
vortex-arrow,vortex-fastlanes). A bit-packed array without patches holds only values in0..2^bit_width, so the unpacked values serve directly as codes into that range. This applies up to 16-bit widths.BitPacked'sCastKernelnow also handles narrowing integer casts when there are no patches and the target type holds every value the bit width allows.Tests cover nulls, float bit-equality, a full
UInt8code space, the overflow fallback, the all-distinct path, bit-packed export, and narrowing bit-packed casts.The CodSpeed micro-benchmarks it flags don't run the code this PR changes; see the comment below.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC