Skip to content

Run-end encode boolean columns via the generic RunEnd encoding - #8467

Draft
joseph-isaacs wants to merge 1 commit into
developfrom
claude/runend-bool-compression-r148yw
Draft

joseph-isaacs wants to merge 1 commit into
developfrom
claude/runend-bool-compression-r148yw

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Boolean columns are now run-end compressed through the generic RunEnd encoding rather than a bespoke bool encoding. The generic crate already decodes bool-valued run-end arrays (decompress_bool.rs, and the DType::Bool branch of run_end_canonicalize); the only missing piece was the encode side, which this PR adds.

This supersedes the earlier approach in this PR (a standalone vortex-runend-bool encoding). As discussed on the design, RunEndBool is really just RunEnd whose values are the implicit alternating pattern [start, !start, …], so it is better expressed by composing the generic encoding than by duplicating its compute surface.

Changes

  1. vortex-runend — bool encode path (compress.rs):

    • runend_encode_bool(ArrayView<Bool>, ctx) -> (ends, values) mirrors the existing primitive encoders' validity handling — a new run starts whenever the (value, validity) pair changes, and per-run validity lives in the BoolArray values child.
    • RunEnd::encode now accepts Bool in addition to Primitive.
  2. vortex-btrblocks — auto-selection (schemes/bool/runend.rs):

    • New BoolRunEndScheme emits a generic RunEnd{ends, BoolArray values} and cascades both children. Registered in ALL_SCHEMES; gated on average run length (word-at-a-time run counting via set_slices).

Design note (future work)

The generic form materializes an num_runs-length alternating values child that is almost entirely redundant. A doc note on runend_encode_bool records that a future cycling/repeat-pattern array encoding — one that logically yields [start, !start, …] in O(1) space — could stand in as that values child, recovering the single-start-bit compactness of a dedicated bool run-end encoding through composition rather than a separate encoding.

Tests / checks

  • cargo test -p vortex-runend — 78 passed (new bool encode round-trips incl. interior-null cases + existing decode).
  • cargo test -p vortex-btrblocks — 39 passed, incl. test_run_heavy_bool_uses_runend (run-heavy bool selects generic RunEnd) and existing bool/constant tests.
  • cargo clippy -p vortex-runend -p vortex-btrblocks clean; cargo +nightly fmt clean.
  • Not run locally: vortex-duckdb C++ layer (proxy blocks the DuckDB source download); its Rust uses only the unchanged generic RunEndArray.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant