Repository navigation
Don't pass extension arrays through the compressor unchanged - #10424
robert3005 wants to merge 2 commits into
Conversation
When no scheme compresses a canonical extension array, `choose_and_compress` returns the input itself, and the extension branch kept it whenever its `nbytes` beat the compressed storage. Canonicalizing an extension array only canonicalizes the top level of its storage, so the input can still hold lazy arrays such as `vortex.slice`, which cannot be serialized. A UUID column mixing a dense chunk with a slice of a sparse one hit this on write (#10417). Use the compressed storage whenever no scheme produced a new array. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XzXtnsrnEsBNVa9R2VqKQ
`choose_and_compress` returned its input whenever no scheme beat it, so every caller got an `ArrayRef` that may or may not have been compressed. That is a valid result for leaf arrays, but not for extension arrays, whose canonical form leaves the storage below its top level lazy; the extension branch had to detect the passthrough by pointer identity. Return a `Selection` that says whether anything was compressed, so each caller picks its fallback explicitly and the extension branch falls back to its compressed storage by construction. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XzXtnsrnEsBNVa9R2VqKQ
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.6 µs | -11.45% |
| ⚡ |
Simulation | density_sweep_dense_runs[0.001] |
47.8 µs | 29.7 µs | +60.79% |
| ⚡ |
Simulation | filter_powerlaw_by_mostly_true[250000] |
150.2 µs | 103.7 µs | +44.77% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/compress-explicit-fallback (c3ad942) with develop (731a231)
Footnotes
-
359 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
The compressor indicates whether the array was returned unchanged. Extension scheme canonicalises input and only keeps extension scheme compressed arrays if they actually compressed anything instead of returning input unchanged
fix #10417