Blocked FoR: store a global reference scalar as metadata rather than child

[mhk197]

Summary

A FoR array now has either a global reference, stored as a Scalar in FoRData, or blocked references, one per 1024-element chunk, stored in an optional blocked_references child. Previously, a single reference was a ConstantArray child, and constant_reference() recovered it by downcasting that child.

With the reference as metadata, nothing can turn a single-reference array into a different encoding by rewriting its child: casts, optimizer rewrites and compressor passes no longer see a child to rewrite. Kernels read the reference directly instead of downcasting and cloning, and no ConstantArray is allocated for slice, take, filter or try_new.

API

• FoRData { offset, global_reference: Option<Scalar> }. The references slot is renamed to blocked_references: Option<ArrayRef>.

• references() returns the new FoRReferences enum, which replaces constant_reference():

  pub enum FoRReferences<'a> {
      Global(&'a Scalar),
      Blocked(&'a ArrayRef),
  }

• validate requires exactly one of the two, and offset == 0 with a global reference.

• FoR::try_new(encoded, reference) builds a global reference. FoR::try_new_chunked(encoded, references, offset) builds blocked references, except that a ConstantArray of references becomes a global reference.

• FoR::encode_chunked emits a global reference when every chunk has the same minimum. Previously those references stayed a PrimitiveArray, unless the compressor's constant detection, which only keeps a constant when it is smaller, happened to convert them.

[codspeed]

Merging this PR will degrade performance by 31.27%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 10 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• take_varbin
• pcodec_compress_f64
• multipolygon_mixed_validity
• decompress[u16, (4000, 256)]
• decompress[u32, (1000, 256)]
• filter_run_end[len=4096_run=16_density=0.1]
• zstd_compress_string
• ilike_contains
• from_vec_drop_vortex[65536]
• random_i128[0.01]

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 8 regressed benchmarks
✅ 2211 untouched benchmarks
🗄️ 12 archived benchmarks run¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	filter_neon[8]	1.1 µs	2.3 µs	-53.12%
❌	Simulation	density_sweep_dense_runs[0.001]	28.7 µs	46.5 µs	-38.3%
❌	WallTime	threshold_neon[8]	2.8 µs	4.4 µs	-37.04%
❌	WallTime	filter_neon[80]	2.9 µs	4.1 µs	-28.7%
❌	WallTime	filter_neon[64]	2.8 µs	3.9 µs	-27.03%
❌	WallTime	threshold_neon[64]	4.2 µs	5.6 µs	-25.19%
❌	WallTime	threshold_neon[80]	4.6 µs	5.7 µs	-20.39%
❌	WallTime	decode_chunked_neon[i64, 524288]	13.1 µs	14.9 µs	-11.76%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/for-global-reference (28f7192) with develop (b085f29)}

Footnotes

1. 12 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

[joseph-isaacs]

enum ?

[joseph-isaacs]

better error message