Blocked BitPacked: add fastlanes.bitpacked.v2 serialization

[mhk197]

Tracking Issue: #10167

Summary

Adds a wire format for BitPacked arrays whose blocks each have their own bit width, and makes it the in-memory ID, following fastlanes.for.v2 (#10135) and DecimalByteParts.

ID	Role
fastlanes.bitpacked	Frozen wire format for global-width arrays. Unchanged.
fastlanes.bitpacked.v2	In-memory ID, and the wire format for per-block bit widths.

BitPackedPlugin declares both serialized IDs and picks one per array from bit_widths():

• Global width writes fastlanes.bitpacked exactly as before. Existing files read as before, and default writes stay byte-identical.
• Blocked writes fastlanes.bitpacked.v2.

The plugin moves into bitpacking/plugin/, laid out like FoR's: mod.rs holds the IDs and the dispatch, v1.rs the v1 metadata and serde, v2.rs the new format, and tests.rs the existing tests plus new ones. The v1 code is moved unchanged, except that it writes bitpacked_v1_id() rather than the plugin's ID.

fastlanes.bitpacked.v2 layout

• Buffers: the packed buffer, as in v1.
• Children: v1's children (patch indices, values and chunk offsets, then validity), followed by the block offsets. The block offsets are serialized like any other child, so the compressor can compress them.
• Metadata: a prost message holding offset, the block offsets' unsigned PType, and the patches metadata. There is no bit width, and the block offsets' length, ceil((offset + len) / 1024) + 1, is derived.

Deserialization checks for one buffer and the expected number of children, and validates the offset before deriving the block offsets' length. Construction then checks the block offsets' dtype and length. The decoder (#10202) bounds-checks the boundary values themselves.

BitPackedPatchedPlugin

The experimental-patches plugin declares both IDs as well. It moves a v2 array's interior patches into a Patched wrapper just as it does for v1, rebuilding the inner array with its block offsets instead of panicking on a blocked array.

Editions and compressor

fastlanes.bitpacked.v2 is in no edition. Writers that enforce editions reject it, and readers that predate it report an unknown encoding. Nothing produces blocked arrays yet, so default writes don't change. BitPackingScheme::produced_encodings now declares bitpacked_v1_id(), since BitPacked.id() is the in-memory ID.

[codspeed]

Merging this PR will improve performance by 12.31%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 2188 untouched benchmarks
⏩ 409 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	bitpack_blocked_compress_avx2	7.6 µs	6.8 µs	+12.31%
⚠️	Simulation	bench_compare_primitive[(10000, 2)]	< 1 ns	< 1 ns	N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/bitpacked-v2-wire (fc9ce15) with mk/bitpacked-variable-decode (6be7533)}

Footnotes

1. 409 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩