compress-bench: write and scan Parquet the way DataFusion does by default by robert3005 · Pull Request #10571 · vortex-data/vortex

robert3005 · 2026-10-16T16:51:22Z

Replace the bespoke Parquet parallelism from #10510 with a port of DataFusion's defaults:

Compress follows ParquetSink: one encoding task per leaf column of the open row group, fed through channels of 2 arrays, with the next row group starting while the previous one is closed and at most 1 finished row group queued for appending. Output leaves the writer buffer every ~1 MB and is joined once at the end.
Decompress follows DataFusion's file scan: the file is split into one byte range per runtime worker (one partition under 1 MiB), each row group goes to the range holding its first page, and each range is one sequential reader in 8,192-row batches with view types.

joseph-isaacs

how much extra time and code to use datafusion?

codspeed · 2026-10-16T16:58:56Z

Merging this PR will improve performance by 12.08%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

25 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

test_scan[int]

test_repeated_scan[int]

test_scan_to_arrow_array[int]

warm[(16, 64)]

test_repeated_scan_scalar_at[int]

test_repeated_scan_filter[int]

test_scan_scalar_at[int]

random_i128[0.01]

test_scan_scalar_at[int_str]

test_scan_filter[int_str]

…and 15 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

test_scan_scalar_at[int_float]

test_repeated_scan_scalar_at[int_float]

test_scan[int]

test_repeated_scan[int]

test_scan_to_arrow_array[int]

test_repeated_scan_scalar_at[int]

test_repeated_scan_filter[int]

test_repeated_scan_scalar_at[int_str_float]

test_scan_scalar_at[int]

test_scan_read_all_to_arrow_array[int]

…and 30 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 2386 untouched benchmarks
🆕 9 new benchmarks
⏩ 12 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡ ℹ️	Simulation	`test_scan_scalar_at[int_float]`	1,085.3 µs	968.3 µs	+12.08%
🆕	Simulation	`prepare_and_compile[1]`	N/A	53.7 µs	N/A
🆕	Simulation	`prepare_and_compile[100]`	N/A	338.3 µs	N/A
🆕	Simulation	`prepare_and_compile[1000]`	N/A	2.8 ms	N/A
🆕	Simulation	`prepare[1]`	N/A	16.2 µs	N/A
🆕	Simulation	`prepare[100]`	N/A	74.6 µs	N/A
🆕	Simulation	`prepare[1000]`	N/A	424.2 µs	N/A
🆕	Simulation	`scan[1]`	N/A	96.9 µs	N/A
🆕	Simulation	`scan[100]`	N/A	1.3 ms	N/A
🆕	Simulation	`scan[1000]`	N/A	12.4 ms	N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing claude/loving-mayer-wp6vev (76637fc) with develop (bc9e0e6)²}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on develop (d3bac42) during the generation of this report, so bc9e0e6 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

github-actions · 2026-10-16T17:06:44Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`87e5ff5`	compress-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-10-16T17:14:14Z

Benchmarks: Compression 📖

Commits: PR 87e5ff59 vs base 6289e58f

vortex / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	761887962 / 779092854 / -2.2%
compress time/Bimbo	3525774499 / 3626152444 / -2.8%
compress time/CMSprovider	2524697237 / 2546383692 / -0.9%
compress time/Euro2016	267568589 / 266507298 / +0.4%
compress time/Food	314207485 / 316413364 / -0.7%
compress time/HashTags	519243516 / 521051326 / -0.3%
compress time/TPC-H l_comment canonical	616477370 / 609102819 / +1.2%
compress time/TPC-H l_comment chunked	614706713 / 609542236 / +0.8%
compress time/taxi	386561150 / 390788845 / -1.1%
compress time/wide table cols=100 chunks=1 rows=1000	4152359 / 4270278 / -2.8%
compress time/wide table cols=100 chunks=50 rows=1000	4044309 / 4181115 / -3.3%
compress time/wide table cols=1000 chunks=1 rows=1000	42652768 / 41806060 / +2.0%
compress time/wide table cols=1000 chunks=50 rows=1000	40178705 / 41955708 / -4.2%
compress time/wide table cols=10000 chunks=1 rows=1000	484318457 / 499430316 / -3.0%
compress time/wide table cols=10000 chunks=50 rows=1000	500424148 / 528768372 / -5.4%
decompress time/Arade	13170589 / 12732716 / +3.4%
decompress time/Bimbo	66187738 / 67411462 / -1.8%
decompress time/CMSprovider	39564791 / 40295319 / -1.8%
decompress time/Euro2016	11965844 / 11826002 / +1.2%
decompress time/Food	3913432 / 3899923 / +0.3%
decompress time/HashTags	57172099 / 54651434 / +4.6%
decompress time/TPC-H l_comment canonical	14606723 / 14828614 / -1.5%
decompress time/TPC-H l_comment chunked	14745858 / 14618918 / +0.9%
decompress time/taxi	9181815 / 9375610 / -2.1%
decompress time/wide table cols=100 chunks=1 rows=1000	1216533 / 1232066 / -1.3%
decompress time/wide table cols=100 chunks=50 rows=1000	1225726 / 1219540 / +0.5%
decompress time/wide table cols=1000 chunks=1 rows=1000	12580360 / 12144577 / +3.6%
decompress time/wide table cols=1000 chunks=50 rows=1000	12451484 / 12147578 / +2.5%
decompress time/wide table cols=10000 chunks=1 rows=1000	139055145 / 131576206 / +5.7%
decompress time/wide table cols=10000 chunks=50 rows=1000	137748471 / 149417706 / -7.8%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164130260 / 164130260 / +0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (0.703x ✅, 15↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	1.02618651 / 3.82997936 / -73.2% 🟢
vortex:parquet-zstd ratio compress time/Bimbo	1.78996075 / 3.62985647 / -50.7% 🟢
vortex:parquet-zstd ratio compress time/CMSprovider	2.34215722 / 5.14502118 / -54.5% 🟢
vortex:parquet-zstd ratio compress time/Euro2016	0.409317952 / 0.831120873 / -50.8% 🟢
vortex:parquet-zstd ratio compress time/Food	0.986989235 / 4.65819603 / -78.8% 🟢
vortex:parquet-zstd ratio compress time/HashTags	1.48358321 / 3.32991323 / -55.4% 🟢
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	0.569363882 / 2.97486918 / -80.9% 🟢
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	0.571021115 / 2.89411398 / -80.3% 🟢
vortex:parquet-zstd ratio compress time/taxi	1.99819335 / 3.91036621 / -48.9% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	1.23075525 / 3.25594896 / -62.2% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	1.19959548 / 3.41342539 / -64.9% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	1.17999656 / 3.07095824 / -61.6% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	1.15240456 / 3.20961913 / -64.1% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	1.2840109 / 3.50146309 / -63.3% 🟢
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	1.30211676 / 3.78795271 / -65.6% 🟢
vortex:parquet-zstd ratio decompress time/Arade	0.268328492 / 0.257173255 / +4.3%
vortex:parquet-zstd ratio decompress time/Bimbo	0.725492582 / 0.737086185 / -1.6%
vortex:parquet-zstd ratio decompress time/CMSprovider	0.382941653 / 0.387263537 / -1.1%
vortex:parquet-zstd ratio decompress time/Euro2016	0.0797442802 / 0.0785439101 / +1.5%
vortex:parquet-zstd ratio decompress time/Food	0.155774546 / 0.155127492 / +0.4%
vortex:parquet-zstd ratio decompress time/HashTags	0.135569478 / 0.13015839 / +4.2%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.176721682 / 0.178777624 / -1.1%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.179049451 / 0.176649858 / +1.4%
vortex:parquet-zstd ratio decompress time/taxi	0.155065319 / 0.158727068 / -2.3%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.589819883 / 0.605004078 / -2.5%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.601387327 / 0.595400021 / +1.0%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.530697751 / 0.512828507 / +3.5%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.529707925 / 0.515478011 / +2.8%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.545437835 / 0.516763661 / +5.5%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.537722373 / 0.593926734 / -9.5%
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33461969 / 1.33461969 / +0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (1.686x ❌, 0↑ 15↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	742445895 / 203419596 / +265.0% 🔴
parquet_rs-zstd compress time/Bimbo	1969749613 / 998979568 / +97.2% 🔴
parquet_rs-zstd compress time/CMSprovider	1077936703 / 494921907 / +117.8% 🔴
parquet_rs-zstd compress time/Euro2016	653693755 / 320660095 / +103.9% 🔴
parquet_rs-zstd compress time/Food	318349455 / 67926159 / +368.7% 🔴
parquet_rs-zstd compress time/HashTags	349992850 / 156475947 / +123.7% 🔴
parquet_rs-zstd compress time/TPC-H l_comment canonical	1082747589 / 204749447 / +428.8% 🔴
parquet_rs-zstd compress time/TPC-H l_comment chunked	1076504348 / 210614454 / +411.1% 🔴
parquet_rs-zstd compress time/taxi	193455328 / 99936636 / +93.6% 🔴
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	3373830 / 1311531 / +157.2% 🔴
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	3371394 / 1224903 / +175.2% 🔴
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	36146519 / 13613360 / +165.5% 🔴
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	34865104 / 13071865 / +166.7% 🔴
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	377191858 / 142634751 / +164.4% 🔴
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	384315880 / 139592126 / +175.3% 🔴
parquet_rs-zstd decompress time/Arade	49083826 / 49510265 / -0.9%
parquet_rs-zstd decompress time/Bimbo	91231447 / 91456689 / -0.2%
parquet_rs-zstd decompress time/CMSprovider	103318066 / 104051415 / -0.7%
parquet_rs-zstd decompress time/Euro2016	150052693 / 150565486 / -0.3%
parquet_rs-zstd decompress time/Food	25122410 / 25140115 / -0.1%
parquet_rs-zstd decompress time/HashTags	421718073 / 419884066 / +0.4%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	82653825 / 82944463 / -0.4%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	82356343 / 82756466 / -0.5%
parquet_rs-zstd decompress time/taxi	59212563 / 59067493 / +0.2%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2062550 / 2036459 / +1.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2038164 / 2048270 / -0.5%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	23705320 / 23681556 / +0.1%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23506320 / 23565657 / -0.3%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	254942243 / 254615825 / +0.1%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	256170243 / 251575990 / +1.8%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (0.991x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	99300608 / 99618158 / -0.3%
arrow-ipc compress time/Bimbo	612429919 / 631947106 / -3.1%
arrow-ipc compress time/CMSprovider	382332159 / 386935260 / -1.2%
arrow-ipc compress time/Euro2016	38257355 / 38779906 / -1.3%
arrow-ipc compress time/Food	33820643 / 34042420 / -0.7%
arrow-ipc compress time/HashTags	78550046 / 79571555 / -1.3%
arrow-ipc compress time/TPC-H l_comment canonical	955481048 / 952517029 / +0.3%
arrow-ipc compress time/TPC-H l_comment chunked	955265596 / 957653642 / -0.2%
arrow-ipc compress time/taxi	74882234 / 75788308 / -1.2%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	198817 / 204018 / -2.5%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	198833 / 206867 / -3.9%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2224894 / 2262411 / -1.7%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2258896 / 2248782 / +0.4%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25327284 / 25523643 / -0.8%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25367095 / 25492500 / -0.5%
arrow-ipc decompress time/Arade	182122440 / 181650628 / +0.3%
arrow-ipc decompress time/Bimbo	403227317 / 423854518 / -4.9%
arrow-ipc decompress time/CMSprovider	583282744 / 582155701 / +0.2%
arrow-ipc decompress time/Euro2016	74771994 / 74853672 / -0.1%
arrow-ipc decompress time/Food	58282509 / 58386052 / -0.2%
arrow-ipc decompress time/HashTags	142741883 / 143508171 / -0.5%
arrow-ipc decompress time/TPC-H l_comment canonical	627711139 / 632894102 / -0.8%
arrow-ipc decompress time/TPC-H l_comment chunked	628140076 / 633069219 / -0.8%
arrow-ipc decompress time/taxi	48448308 / 48457033 / -0.0%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	320518 / 324879 / -1.3%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	319984 / 320798 / -0.3%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3488460 / 3605714 / -3.3%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3691971 / 3507124 / +5.3%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	36279647 / 36762893 / -1.3%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36816764 / 36816405 / +0.0%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

robert3005 · 2026-10-16T23:19:10Z

@joseph-isaacs I prototyped calling DataFusion directly to measure it. On write, a MemTable goes through DataFrame::write_parquet (single file, zstd(1)) into an InMemory object store. On read, read_parquet then execute_stream_partitioned, with one task per partition. target_partitions is set to the runtime's worker count.

Code: about 50 lines for the direct calls, against about 240 non-comment lines for the port in this PR.

Dependencies: compress-bench picks up 49 more crates (the DataFusion stack plus sqlparser, arrow-csv, arrow-json, …). I didn't time a clean build with and without them, so I don't have a build-time number.

Runtime: synthetic 4-column data (i64, i32, f64, short strings), zstd(1), release build, median of 7, on a 4-core box:

rows	threads	write: port / DataFusion	read: port / DataFusion
10k	4	12.8 ms / 24.0 ms	1.1 ms / 1.7 ms
1M	4	214 ms / 187 ms	63 ms / 65 ms
8M	4	1.59–1.92 s / 1.79–1.87 s	171–191 ms / 168–194 ms
8M	1	2.41–2.70 s / 1.99–2.55 s	508–544 ms / 514–534 ms

Large inputs: both perform the same. The 8M write gap flips sides when I swap which one runs first, so it is noise from run order, not a real difference.
Small inputs: DataFusion adds a fixed ~1–10 ms per call for session setup, planning and the object-store round trip. That would show up on the small wide-table datasets.
File contents: with more than one thread, the file DataFusion writes is a little different in size (about 0.02%). I haven't looked into why; my guess is the planner repartitioning the MemTable. This PR's port writes the same file as a sequential ArrowWriter.

Trade-off: calling DataFusion directly is less code and is literally what DataFusion does, which is easier to defend to outside readers. The cost is the extra dependencies and the fixed overhead on tiny datasets. The port keeps compress-bench light and isolates the format cost, but someone has to check that it stays faithful to DataFusion. Happy to switch the PR to the direct calls if you'd rather.

Generated by Claude Code

…ault Replace the bespoke Parquet parallelism from #10510 with a port of DataFusion's defaults: - Compress follows ParquetSink: one encoding task per leaf column of the open row group, fed through channels of 2 arrays, with the next row group starting while the previous one is closed and at most 1 finished row group queued for appending. Output leaves the writer buffer every ~1 MB and is joined once at the end. - Decompress follows DataFusion's file scan: the file is split into one byte range per runtime worker (one partition under 1 MiB), each row group goes to the range holding its first page, and each range is one sequential reader in 8,192-row batches with view types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HGj5uR9atXRsZGJBZwNidZ Signed-off-by: Claude <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HGj5uR9atXRsZGJBZwNidZ Signed-off-by: Claude <noreply@anthropic.com>

robert3005 added changelog/chore A trivial change action/bench-compress Run only the compression benchmark on this PR labels Oct 16, 2026

robert3005 requested a review from joseph-isaacs October 16, 2026 16:51

github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 16, 2026

robert3005 enabled auto-merge (squash) October 16, 2026 16:55

joseph-isaacs reviewed Oct 16, 2026

View reviewed changes

joseph-isaacs pushed a commit that referenced this pull request Oct 16, 2026

Merge #10571 (DataFusion-style Parquet write/scan)

a0a4812

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

claude added 2 commits October 19, 2026 13:02

compress-bench: rustfmt

ec8ac8b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HGj5uR9atXRsZGJBZwNidZ Signed-off-by: Claude <noreply@anthropic.com>

joseph-isaacs approved these changes Oct 19, 2026

View reviewed changes

robert3005 force-pushed the claude/loving-mayer-wp6vev branch from 76637fc to ec8ac8b Compare October 19, 2026 13:05

Navigation Menu

Repository navigation

compress-bench: write and scan Parquet the way DataFusion does by default - #10571

compress-bench: write and scan Parquet the way DataFusion does by default#10571
robert3005 wants to merge 2 commits into
developfrom
claude/loving-mayer-wp6vev

robert3005 commented Oct 16, 2026

Uh oh!

joseph-isaacs left a comment

Uh oh!

codspeed Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

robert3005 commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

robert3005 commented Oct 16, 2026

Uh oh!

joseph-isaacs left a comment

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 12.08%

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Oct 16, 2026

Benchmarks: Compression 📖

Uh oh!

robert3005 commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codspeed Bot commented Oct 16, 2026 •

edited

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading