Repository navigation
Run-end encode boolean columns via the generic RunEnd encoding - #8467
Draft
joseph-isaacs wants to merge 1 commit into
Draft
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Boolean columns are now run-end compressed through the generic
RunEndencoding rather than a bespoke bool encoding. The generic crate already decodes bool-valued run-end arrays (decompress_bool.rs, and theDType::Boolbranch ofrun_end_canonicalize); the only missing piece was the encode side, which this PR adds.This supersedes the earlier approach in this PR (a standalone
vortex-runend-boolencoding). As discussed on the design,RunEndBoolis really justRunEndwhose values are the implicit alternating pattern[start, !start, …], so it is better expressed by composing the generic encoding than by duplicating its compute surface.Changes
vortex-runend— bool encode path (compress.rs):runend_encode_bool(ArrayView<Bool>, ctx) -> (ends, values)mirrors the existing primitive encoders' validity handling — a new run starts whenever the(value, validity)pair changes, and per-run validity lives in theBoolArrayvalues child.RunEnd::encodenow acceptsBoolin addition toPrimitive.vortex-btrblocks— auto-selection (schemes/bool/runend.rs):BoolRunEndSchemeemits a genericRunEnd{ends, BoolArray values}and cascades both children. Registered inALL_SCHEMES; gated on average run length (word-at-a-time run counting viaset_slices).Design note (future work)
The generic form materializes an
num_runs-length alternating values child that is almost entirely redundant. A doc note onrunend_encode_boolrecords that a future cycling/repeat-pattern array encoding — one that logically yields[start, !start, …]in O(1) space — could stand in as that values child, recovering the single-start-bit compactness of a dedicated bool run-end encoding through composition rather than a separate encoding.Tests / checks
cargo test -p vortex-runend— 78 passed (new bool encode round-trips incl. interior-null cases + existing decode).cargo test -p vortex-btrblocks— 39 passed, incl.test_run_heavy_bool_uses_runend(run-heavy bool selects genericRunEnd) and existing bool/constant tests.cargo clippy -p vortex-runend -p vortex-btrblocksclean;cargo +nightly fmtclean.vortex-duckdbC++ layer (proxy blocks the DuckDB source download); its Rust uses only the unchanged genericRunEndArray.