[rapour]

Summary

• Tracking Issue: Tracking Issue: Elias-Fano Encoding #10605

This is the first step towards Elias-Fano: the Unary encoding that will hold EF's upper bits. It lives in a new encodings/elias-fano crate, which will also hold the EliasFano parent, as discussed on #10377.

It is deliberately naive. There is no index, no slice kernel and no compute kernel, so the PR is mostly boilerplate, and the benchmarks set the baselines that later work is measured against. It is in no file edition and vortex-file does not register it, so the default writers reject it.

Next up is slicing for Unary, then the naive EliasFano parent with upper and lower children.

Changes

Layout. One BitBuffer. Each value writes its gap from the previous value as unset bits, then one set bit. The first value is stored as base, so the set bit for index i sits at (v[i] - base) + i, and the array takes len + (max - base) bits. For [3, 5, 5, 9], base is 3 and the bits are 1 001 1 00001.

This is the upper-bits array of Vigna's quasi-succinct indices (section 4), counted from base rather than from 0. Unlike rise-rs and vers, there is no sentinel bit before the first value and no guard bit after the last. That way, slicing can trim the bits as they are, and a search by value will need a branch at each end instead.

Read per bucket, these are the same bits as the bucket form of EF's upper bitmap, where each bucket's count is written in negated unary, minus the last bucket's closing zero.

The metadata is two u64s, base and max. The dtype must be a non-nullable unsigned integer. With FoR above EF, as the issue proposes, the upper parts are never negative, so the parent can always give this child an unsigned ptype.

Logical values. A Unary array holds one value per element, not one count per bucket. As EF's upper child it then has the parent's length, like the children of FoR and BitPacked, and an unfused EF decode is just (upper << width) | lower.

Validation. try_new, deserialize and with_buffers check the dtype, that base <= max and max fits the ptype, the bit length, a zero bit offset, that the first and last bits are set, and that exactly len bits are set. The last check is one popcount. Any array that passes encodes some non-decreasing sequence from base to max, so the readers need no per-value checks, and a point read always agrees with a full decode.

Encoding. unary_encode returns None for nullable, signed or non-integer input, for unsorted input, and when the bits would take more than the plain array's len × bit width.

Reads.

• execute walks the set bits once.
• scalar_at finds its set bit with BitBuffer::select, which scans from the first bit.
• Slicing uses the default, which decodes and then slices. A lazy slice can't be serialised until it is executed, so Unary slicing should land before the EF parent.
• Construction sets IsSorted as exact.

Wiring. The crate is a workspace member and dependency, and runs in CodSpeed shard 8.

[codspeed]

Merging this PR will degrade performance by 29.37%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 17 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• chunked_dict_primitive_into_canonical[f64, (1000, 1000, 10)]
• test_scan_filter[int]
• random_i128[0.01]
• test_scan_filter[int_float]
• test_scan_filter[int_str_float]
• test_scan_filter[int_str]
• test_scan[int_str]
• take_varbin
• decompress[u16, (4000, 256)]
• multipolygon_mixed_validity
• …and 7 more

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

• test_scan_filter[int]
• test_repeated_scan_scalar_at[int_float]
• test_scan_filter[int_float]
• test_repeated_scan_scalar_at[int]
• test_repeated_scan_scalar_at[int_str_float]
• test_repeated_scan[int]
• test_scan_filter[int_str_float]
• test_scan_scalar_at[int]
• test_scan[int]
• test_scan_filter[int_str]
• …and 30 more

❌ 3 regressed benchmarks
✅ 2405 untouched benchmarks
🆕 20 new benchmarks
⏩ 12 skipped benchmarks¹
🗄️ 43 archived benchmarks run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	threshold_neon[8]	1.4 µs	2.2 µs	-36.14%
❌	WallTime	filter_neon[64]	1.8 µs	2.6 µs	-31.56%
❌	WallTime	threshold_neon[64]	2.6 µs	3.2 µs	-19.37%
🆕	Simulation	decode[Dense]	N/A	1.2 ms	N/A
🆕	Simulation	decode[Gap]	N/A	1.3 ms	N/A
🆕	Simulation	decode[Runs]	N/A	1.2 ms	N/A
🆕	Simulation	decode[Spread]	N/A	1.3 ms	N/A
🆕	Simulation	encode[Dense]	N/A	1.3 ms	N/A
🆕	Simulation	encode[Gap]	N/A	1.4 ms	N/A
🆕	Simulation	encode[Runs]	N/A	1.2 ms	N/A
🆕	Simulation	encode[Spread]	N/A	1.3 ms	N/A
🆕	Simulation	scalar_at_random[Dense]	N/A	445.1 µs	N/A
🆕	Simulation	scalar_at_random[Gap]	N/A	1.8 ms	N/A
🆕	Simulation	scalar_at_random[Runs]	N/A	286.5 µs	N/A
🆕	Simulation	scalar_at_random[Spread]	N/A	606.4 µs	N/A
🆕	Simulation	scalar_at_sequential[Dense]	N/A	436.2 µs	N/A
🆕	Simulation	scalar_at_sequential[Gap]	N/A	3.1 ms	N/A
🆕	Simulation	scalar_at_sequential[Runs]	N/A	279.2 µs	N/A
🆕	Simulation	scalar_at_sequential[Spread]	N/A	592.5 µs	N/A
🆕	Simulation	slice_decode[Dense]	N/A	1.3 ms	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing rapour:feat/elias-fano-unary-delta (4e57b0f) with develop (ee6da96)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 43 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

[connortsui20]

Ok so I just realized that there are multiple ways that we can imagine the canonical / semantic values of the upper bits.

For example, these two arrays can be thought of as the same:

"delta" encode upper values:   [0, 0, 2, 2, 2]
                                0, 0,  2,  0, 0
->                              1, 1, 001, 1, 1, (0)


                    bucket      00   01   10
bucket counts:                 [2,   0,    3]
->                             110  0    1110 


                                11001110

Notice how these are the same thing!

So I think what we need to decide is what we actually want this logical array to represent, should it be the "delta" encoded upper values (which are not exactly delta but close), or the "bucket counts", by which I mean how many upper bit values fall into each bucket.

I was thinking about the buckets, but I can see how both of these are valid representations of the data. However, in my mind, a general Unary encoding should probably be represented as n 1s followed by a closing 0.

I don't think would change basically anything about the Elias-Fano encoding, but it does mean that this implementation of Unary would have to change. For example, we should not require the inputs to be sorted, as a start.

[rapour]

Thanks, that makes sense. Let's go with bucket counts: Unary becomes a general unsigned encoding where each value v is written as v ones and a closing zero, and the input doesn't need to be sorted. EF's upper child then holds one count per bucket instead of one value per element.