Repository navigation
Poc index layout - #9706
Poc index layout#9706
Conversation
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.8 µs | 7.6 µs | -11.12% |
| ⚡ | WallTime | mul_u64_nonnull_neon |
39.5 µs | 20.9 µs | +88.51% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing polarsignals:poc-index-layout (b0660fd) with develop (e717fab)2
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(f990a5d) during the generation of this report, so e717fab was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
| RowLocator::Rows(rows) => { | ||
| let start = u32::try_from(row_range.start)?; | ||
| let end = u32::try_from(row_range.end)?; | ||
| let mut pos = row_range.start; | ||
| for row in rows.range(start..end) { | ||
| let row = u64::from(row); | ||
| bits.append_n(false, usize::try_from(row - pos)?); | ||
| bits.append_n(true, 1); | ||
| pos = row + 1; | ||
| } | ||
| bits.append_n(false, usize::try_from(row_range.end - pos)?); | ||
| } | ||
| // Broadcast each block's bit across the rows it covers, clipped to `row_range`. | ||
| RowLocator::Blocks { block_len, ids } => { | ||
| let mut row = row_range.start; | ||
| while row < row_range.end { | ||
| let block = row / block_len; | ||
| let block_end = ((block + 1) * block_len).min(row_range.end); | ||
| let hit = ids.contains(u32::try_from(block)?); | ||
| bits.append_n(hit, usize::try_from(block_end - row)?); | ||
| row = block_end; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
This looks quite low. append_n is pretty expensive. Can be optimised later
|
Looking promising |
c5c2830 to
7d08158
Compare
| this.register(&LegacyStats as &dyn LayoutEncoding); | ||
| this.register(&Dict as &dyn LayoutEncoding); | ||
| this.register(&List as &dyn LayoutEncoding); | ||
| this.register(&Indexed as &dyn LayoutEncoding); |
There was a problem hiding this comment.
@robert3005 We added layouts to editions correct so this will be filtered?
There was a problem hiding this comment.
I'm not the expert but looking at the code that appears to be correct.
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
Commenting to remove from stale list. Back from vacation and picking this back up. |
thorfour
added
changelog/feature
Adds an indexed layout to the Vortex layout crate based on vortex-data#9024 Signed-off-by: Thor <thor.hansen@dash0.com>
Intended as an example implementation of a indexed-layout index. Signed-off-by: Thor <thor.hansen@dash0.com>
Signed-off-by: Thor <thor.hansen@dash0.com>
Signed-off-by: Thor <thor.hansen@dash0.com>
vortex-reverse-index was a standalone worked example of the vortex.indexed IndexVTable contract; move it to vortex-layout/examples/reverse_index, following the plain cargo-example convention used by vortex-ffi/examples instead of shipping it as its own workspace crate. vortex-datafusion could no longer dev-depend on it as a library once it became a cargo example, so its DataFusion integration test moves along with it (examples/reverse_index/datafusion_tests.rs), rebuilt against vortex-datafusion's public API and a manually assembled session instead of the vortex facade crate, which vortex-layout cannot depend on. Signed-off-by: "Thor" <thor.hansen@dash0.com> Signed-off-by: Thor <thor.hansen@dash0.com>
IndexedReader::plan_probe returned on the first spec that claimed an expression, regardless of exactness. A Superset claim from an earlier spec could shadow a later, more precise Exact claim, and multiple Superset claims on the same expression never got combined. CachedProbe is now Exact(locator) or Superset(Vec<locator>): plan_probe scans every spec, preferring any Exact claim outright and otherwise collecting all Superset claims. pruning_evaluation intersects every Superset locator (short-circuiting once nothing survives), and filter_evaluation only fast-paths on an Exact claim. Signed-off-by: "Thor" <thor.hansen@dash0.com> Signed-off-by: Thor <thor.hansen@dash0.com>
Signed-off-by: Thor <thor.hansen@dash0.com>
This allows only a portion of an index to be read for a portion of the rows. Signed-off-by: Thor <thor.hansen@dash0.com>
Signed-off-by: Thor <thor.hansen@dash0.com>
Signed-off-by: Thor <thor.hansen@dash0.com>
at build time Updated IndexQueryPlan to use a BoundExpression now. Signed-off-by: Thor <thor.hansen@dash0.com>
options are parsed once per reader, and each chunk is decoded once and shared. Signed-off-by: Thor <thor.hansen@dash0.com>
Added some tests with golden files for serializing/deserializing index layouts. Signed-off-by: Thor <thor.hansen@dash0.com>
3934887 to
cfefdd4
Compare
Signed-off-by: Thor <thor.hansen@dash0.com>
|
Thanks for the contribution! |
Summary
This is a PoC of the Indexed Layout described in the above issue.
Changes
This implements the
IndexVTabletrait that Index implementations can implement to be utilized in an IndexLayout.Adds an example
ReverseIndexthat implements the trait and it's use is demonstrated in a couple unit tests. Additionally adds a datafusion test that validates that fewer bytes are read from a Vortex file when an indexed layout is successfully utilized.