Skip to content

Canonicalize each chunk once before computing its write statistics - #10657

Closed
robert3005 wants to merge 1 commit into
developfrom
rk/write-canonicalize-once
Closed

robert3005 wants to merge 1 commit into
developfrom
rk/write-canonicalize-once

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

A BtrBlocks-compressed chunk was executed into canonical form once per statistic that needs it (uncompressed size, min/max) and again by the compressor. Compressor plugins can now declare that they canonicalize their input, and the compressing strategy then canonicalizes the chunk once up front. Output files are unchanged.

A BtrBlocks-compressed chunk was executed into canonical form once per
statistic that needs it (uncompressed size, min/max) and again by the
compressor. Compressor plugins can now declare that they canonicalize
their input, and the compressing strategy then canonicalizes the chunk
once up front. Output files are unchanged.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015cWshAaPF4rqz3hLLMo1jY
(cherry picked from commit bb5c5b8)
@robert3005 robert3005 added changelog/fix A bug fix action/bench-compress Run only the compression benchmark on this PR labels Oct 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 20, 2026
@codspeed

codspeed Bot commented Oct 20, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 17 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

ℹ️ 40 benchmarks have no flamegraph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 2405 untouched benchmarks
⏩ 12 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime starts_with_avx512[8, VarBin] 39.6 µs 46 µs -13.85%
⚡ Walltime words_gather_dispatch_neon[65536] 2 µs 1.8 µs +10.36%
⚡ Walltime deferred_bool_avx512[i64, PerRowPerRow] 4 µs 3.7 µs +10.11%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/write-canonicalize-once (b2a4425) with develop (954772d)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done b2a4425 compress-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR b2a4425c vs base ca80583f


vortex / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
name ns (PR / base / %diff)
compress time/Arade 744158443 / 758476855 / -1.9%
compress time/Bimbo 3207172152 / 3267256272 / -1.8%
compress time/CMSprovider 2424398495 / 2403446706 / +0.9%
compress time/Euro2016 253980569 / 252691429 / +0.5%
compress time/Food 304998489 / 308949701 / -1.3%
compress time/HashTags 501588870 / 492784255 / +1.8%
compress time/TPC-H l_comment canonical 565823881 / 566697161 / -0.2%
compress time/TPC-H l_comment chunked 563113810 / 565522917 / -0.4%
compress time/taxi 358154047 / 356815660 / +0.4%
compress time/wide table cols=100 chunks=1 rows=1000 4145234 / 4108837 / +0.9%
compress time/wide table cols=100 chunks=50 rows=1000 4109162 / 4202311 / -2.2%
compress time/wide table cols=1000 chunks=1 rows=1000 40532356 / 40419693 / +0.3%
compress time/wide table cols=1000 chunks=50 rows=1000 40616006 / 40354286 / +0.6%
compress time/wide table cols=10000 chunks=1 rows=1000 466570822 / 473179056 / -1.4%
compress time/wide table cols=10000 chunks=50 rows=1000 464348166 / 463241579 / +0.2%
decompress time/Arade 11927246 / 12128331 / -1.7%
decompress time/Bimbo 54947904 / 56032981 / -1.9%
decompress time/CMSprovider 36218628 / 36562556 / -0.9%
decompress time/Euro2016 11825923 / 11937204 / -0.9%
decompress time/Food 3242644 / 3393308 / -4.4%
decompress time/HashTags 35362392 / 34848841 / +1.5%
decompress time/TPC-H l_comment canonical 13142224 / 13385659 / -1.8%
decompress time/TPC-H l_comment chunked 13321045 / 13521290 / -1.5%
decompress time/taxi 8653928 / 8710808 / -0.7%
decompress time/wide table cols=100 chunks=1 rows=1000 1164180 / 1165206 / -0.1%
decompress time/wide table cols=100 chunks=50 rows=1000 1161636 / 1174642 / -1.1%
decompress time/wide table cols=1000 chunks=1 rows=1000 11485635 / 11451487 / +0.3%
decompress time/wide table cols=1000 chunks=50 rows=1000 11549782 / 11417575 / +1.2%
decompress time/wide table cols=10000 chunks=1 rows=1000 125968486 / 125248976 / +0.6%
decompress time/wide table cols=10000 chunks=50 rows=1000 125669611 / 126264471 / -0.5%
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/Arade 142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo 458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider 420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016 164131252 / 164130260 / +0.0%
vortex-file-compressed size/Food 38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags 196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical 178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked 178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi 49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93277680 / 93277680 / +0.0%
vortex / vortex-file-compressed / ratio (0.998x ➖, 0↑ 0↓)
name ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade 1.01984837 / 1.03762918 / -1.7%
vortex:parquet-zstd ratio compress time/Bimbo 1.65327722 / 1.66397784 / -0.6%
vortex:parquet-zstd ratio compress time/CMSprovider 2.29709939 / 2.27764698 / +0.9%
vortex:parquet-zstd ratio compress time/Euro2016 0.395651435 / 0.392942429 / +0.7%
vortex:parquet-zstd ratio compress time/Food 0.96321563 / 0.974255392 / -1.1%
vortex:parquet-zstd ratio compress time/HashTags 1.48183906 / 1.44551332 / +2.5%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical 0.535864093 / 0.53573663 / +0.0%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked 0.533921256 / 0.535346751 / -0.3%
vortex:parquet-zstd ratio compress time/taxi 1.86461494 / 1.85411428 / +0.6%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000 1.1503454 / 1.14160364 / +0.8%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000 1.21717533 / 1.21445055 / +0.2%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000 1.13364092 / 1.13363136 / +0.0%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000 1.10728432 / 1.12053623 / -1.2%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000 1.23494503 / 1.27090398 / -2.8%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000 1.25842471 / 1.23875195 / +1.6%
vortex:parquet-zstd ratio decompress time/Arade 0.243617987 / 0.247642109 / -1.6%
vortex:parquet-zstd ratio decompress time/Bimbo 0.616479506 / 0.627411268 / -1.7%
vortex:parquet-zstd ratio decompress time/CMSprovider 0.353497671 / 0.356856922 / -0.9%
vortex:parquet-zstd ratio decompress time/Euro2016 0.0788831011 / 0.0796345931 / -0.9%
vortex:parquet-zstd ratio decompress time/Food 0.130215685 / 0.135950826 / -4.2%
vortex:parquet-zstd ratio decompress time/HashTags 0.0847235099 / 0.0832192688 / +1.8%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical 0.159814425 / 0.162621335 / -1.7%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked 0.162342215 / 0.164854575 / -1.5%
vortex:parquet-zstd ratio decompress time/taxi 0.148062239 / 0.14866313 / -0.4%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000 0.577785785 / 0.571148894 / +1.2%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000 0.568546493 / 0.579102665 / -1.8%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000 0.489393405 / 0.487040452 / +0.5%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000 0.497193879 / 0.487603403 / +2.0%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000 0.494562177 / 0.48976214 / +1.0%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000 0.497286224 / 0.498354055 / -0.2%
vortex:parquet-zstd size/Arade 0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo 1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider 1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016 1.33462775 / 1.33461969 / +0.0%
vortex:parquet-zstd size/Food 1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags 1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical 1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked 1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi 0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000 1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000 1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000 1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000 1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000 1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000 1.00040408 / 1.00040408 / -0.0%
vortex / parquet / ns (0.998x ➖, 0↑ 0↓)
name ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade 729675572 / 730971013 / -0.2%
parquet_rs-zstd compress time/Bimbo 1939887704 / 1963521505 / -1.2%
parquet_rs-zstd compress time/CMSprovider 1055417324 / 1055232320 / +0.0%
parquet_rs-zstd compress time/Euro2016 641930109 / 643074940 / -0.2%
parquet_rs-zstd compress time/Food 316646117 / 317113668 / -0.1%
parquet_rs-zstd compress time/HashTags 338490787 / 340906062 / -0.7%
parquet_rs-zstd compress time/TPC-H l_comment canonical 1055909304 / 1057790581 / -0.2%
parquet_rs-zstd compress time/TPC-H l_comment chunked 1054675767 / 1056367515 / -0.2%
parquet_rs-zstd compress time/taxi 192079361 / 192445344 / -0.2%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 3603469 / 3599180 / +0.1%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 3375982 / 3460257 / -2.4%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 35754140 / 35655059 / +0.3%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 36680738 / 36013370 / +1.9%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000 377806956 / 372316921 / +1.5%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000 368991615 / 373958306 / -1.3%
parquet_rs-zstd decompress time/Arade 48958807 / 48975237 / -0.0%
parquet_rs-zstd decompress time/Bimbo 89131761 / 89308216 / -0.2%
parquet_rs-zstd decompress time/CMSprovider 102457897 / 102457186 / +0.0%
parquet_rs-zstd decompress time/Euro2016 149917065 / 149899730 / +0.0%
parquet_rs-zstd decompress time/Food 24902100 / 24959819 / -0.2%
parquet_rs-zstd decompress time/HashTags 417385824 / 418759279 / -0.3%
parquet_rs-zstd decompress time/TPC-H l_comment canonical 82234279 / 82311826 / -0.1%
parquet_rs-zstd decompress time/TPC-H l_comment chunked 82055336 / 82019501 / +0.0%
parquet_rs-zstd decompress time/taxi 58447907 / 58594273 / -0.2%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 2014899 / 2040109 / -1.2%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 2043168 / 2028383 / +0.7%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 23469125 / 23512394 / -0.2%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 23229936 / 23415700 / -0.8%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000 254707076 / 255734296 / -0.4%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000 252710823 / 253362985 / -0.3%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/Arade 258014282 / 258014282 / +0.0%
parquet size/Bimbo 384517292 / 384517292 / +0.0%
parquet size/CMSprovider 376887911 / 376887911 / +0.0%
parquet size/Euro2016 122979049 / 122979049 / +0.0%
parquet size/Food 35699500 / 35699500 / +0.0%
parquet size/HashTags 133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical 158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked 158358238 / 158358238 / +0.0%
parquet size/taxi 55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.984x ➖, 0↑ 0↓)
name ns (PR / base / %diff)
arrow-ipc compress time/Arade 98896367 / 100803876 / -1.9%
arrow-ipc compress time/Bimbo 613710013 / 629971836 / -2.6%
arrow-ipc compress time/CMSprovider 381990730 / 387990979 / -1.5%
arrow-ipc compress time/Euro2016 37679490 / 38797413 / -2.9%
arrow-ipc compress time/Food 33810897 / 34309075 / -1.5%
arrow-ipc compress time/HashTags 78691510 / 79505537 / -1.0%
arrow-ipc compress time/TPC-H l_comment canonical 941842563 / 940479075 / +0.1%
arrow-ipc compress time/TPC-H l_comment chunked 939040470 / 942100512 / -0.3%
arrow-ipc compress time/taxi 74310947 / 75540875 / -1.6%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000 204008 / 202216 / +0.9%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000 203167 / 202143 / +0.5%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000 2254357 / 2270669 / -0.7%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000 2218638 / 2234662 / -0.7%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000 25369968 / 25371376 / -0.0%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000 25408075 / 25718154 / -1.2%
arrow-ipc decompress time/Arade 211195328 / 218420535 / -3.3%
arrow-ipc decompress time/Bimbo 401937936 / 420204662 / -4.3%
arrow-ipc decompress time/CMSprovider 677523073 / 698802732 / -3.0%
arrow-ipc decompress time/Euro2016 83805660 / 85931402 / -2.5%
arrow-ipc decompress time/Food 65915688 / 67604182 / -2.5%
arrow-ipc decompress time/HashTags 160740653 / 165680039 / -3.0%
arrow-ipc decompress time/TPC-H l_comment canonical 625330275 / 637837410 / -2.0%
arrow-ipc decompress time/TPC-H l_comment chunked 625591716 / 639517523 / -2.2%
arrow-ipc decompress time/taxi 50993078 / 51381181 / -0.8%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000 317730 / 323003 / -1.6%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000 317699 / 320920 / -1.0%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000 3510105 / 3529273 / -0.5%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000 3608136 / 3647018 / -1.1%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000 35687867 / 36989093 / -3.5%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000 36605551 / 36894172 / -0.8%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/Arade 691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo 3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider 2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016 360582666 / 360582666 / +0.0%
arrow-ipc size/Food 255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags 704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical 7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked 7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi 482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 125752506 / 125752506 / +0.0%

@robert3005

Copy link
Copy Markdown
Contributor Author

This is really only an improvement if compressor gets non canonical chunks which I think is very rare

@robert3005 robert3005 closed this Oct 21, 2026
@robert3005
robert3005 deleted the rk/write-canonicalize-once branch October 21, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant