Repository navigation
Dictionary-encode primitive Arrow exports without arrow_cast - #10802
joseph-isaacs wants to merge 1 commit into
Conversation
|
The Generated by Claude Code |
Exporting a primitive array as an Arrow dictionary went through arrow_cast, which hashes each value with SipHash and builds an ArrowError per value. On PolarSignals this export was ~90% of scan CPU. Encode primitive, float (by bits), date and timestamp arrays natively, choosing in order: a dense table when the keys span fewer than 65,536 values; the values themselves with sequential codes when a sample is nearly all distinct (Arrow dictionaries may hold duplicates); otherwise an FxHash map sized from the sample. The full min/max scan only runs when the sample's range could fit the dense table. Nulls stay in the codes. Other types, and codes that would overflow, fall back to arrow_cast. Split out of #10761. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQocaYaDvJHfmorKXnyvJC
794768e to
a6867e9
Compare
Merging this PR will regress 2 benchmarks4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
|
Neither CodSpeed regression on
Generated by Claude Code |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.096x ➖, 0↑ 8↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.056x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 4↑ 4↓)
datafusion / parquet / ns (1.056x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.128x ➖, 0↑ 4↓)
duckdb / parquet / ns (1.088x ➖, 0↑ 3↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.621x ✅, 5↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Summary
Split out of #10761.
Exporting a primitive array as an Arrow dictionary went through
arrow_cast. That cast hashes every value with SipHash and builds anArrowErrorper value. On PolarSignals this export was about 90% of scan CPU, for both V1 and V2. Most of the PolarSignals gains in #10761 (hot geomean 0.564x) came from here.Changes
primitive_to_dictencodes integer, float (by bits), date and timestamp arrays natively. It picks the first of these that applies:The full min/max scan only runs when the sample's range could fit the dense table. Nulls stay in the codes. Other types, and codes that would overflow, fall back to
arrow_cast.New tests cover:
0.0and-0.0, NaN);UInt8code space;Checks run:
cargo clippy -p vortex-arrow --all-targets --all-features -- -D warnings,cargo +nightly-2026-09-10 fmt -p vortex-arrow, andcargo test -p vortex-arrow --lib dictionary.Generated by Claude Code