Repository navigation
Add planner-owned selection and bounded ListLayout scans to V2 - #10537
Draft
joseph-isaacs wants to merge 2 commits into
Draft
joseph-isaacs wants to merge 2 commits into
joseph-isaacs wants to merge 2 commits into
Conversation
Merging this PR will degrade performance by 22.54%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | threshold_neon[8] |
1.3 µs | 2.3 µs | -42.28% |
| ❌ | WallTime | threshold_neon[64] |
2.5 µs | 3.3 µs | -23.17% |
| ❌ | WallTime | filter_neon[80] |
2.5 µs | 3.1 µs | -19.79% |
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.6 µs | -11.76% |
| ❌ ℹ️ | Simulation | test_scan_filter[int] |
3 ms | 3.4 ms | -11.18% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/decompression-v2-scan (a8349ec) with develop (f661d4b)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
joseph-isaacs
changed the title
joseph-isaacs
force-pushed
the
ji/decompression-v2-scan
branch
from
October 16, 2026 16:52
6b97c79 to
15f2fed
Compare
joseph-isaacs
changed the base branch from
claude/planner-decode-slice-migration-xppbf1
to
ji/upbeat-volta-adp0xd
October 16, 2026 16:52
joseph-isaacs
force-pushed
the
ji/decompression-v2-scan
branch
from
October 16, 2026 17:00
15f2fed to
02119f6
Compare
joseph-isaacs
added
the
changelog/feature
ChatGPT Codex Connector
joseph-isaacs
added a commit
that referenced
this pull request
Oct 16, 2026
Run the unchanged full 48-case Arrow comparison and 36-case standard suite, including parent and candidate V1 controls for the standard suite. Use the same CI-only wrapper as ccbe15b and baseline 5231b70. Runtime source is 8442d46, tree-equivalent to local 95fda7fc8. The previous run 38972129118 is complete. It confirmed the offset-copy feature-vector improvements but retained one +5.2% Arrow rerun. This run checks the combined offset-copy, generated-offset-statistics, and guarded list-overlap changes. Keep PR #10537 draft; this run is not a merge gate. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Rebase draft PR #10537 onto develop 92a7db0. Apply selections through a composable planner, resolve list offsets before selected elements, and split substantial fixed-width child projections at stored segment boundaries. The projection planner owns subdivision, byte-budget admission, child task execution and ordered collection. Unsupported splits retain the serial path. Keep Planner and Morsel trait methods unchanged. Replace the experimental completion callbacks, WorkFuture and ChildExecutor traits with one PlannerOutput::Await variant; morsels only publish the assembled result. Preserve develop's global local-file coalescing configuration. The 8 KiB experiment caused tiny-lookup regressions and is not part of this version. Preserve masked lazy lowering, bounded dense-list grouping, shared value caches, validated chunk offsets and the fieldless nullable-struct fix. Retain the diagnostic examples and regression benchmark workflow. Backtracking: previous PR head 8442d46; experimental CI head 60fda25; local pre-review checkpoint a253e5727 and reviewed checkpoint e8fd21a4a. The byte budget covers in-flight decoded work, not retained output or encoded/coalesced buffers. Full performance revalidation remains necessary. Validation: 692 layout and 79 scan library tests passed, followed by all 7 await protocol tests including an additional await/IO regression. Scoped all-target/all-feature clippy passed with -D warnings; formatted with nightly-2026-09-10. Focused E2E measurements follow this checkpoint. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
force-pushed
the
ji/decompression-v2-scan
branch
from
October 19, 2026 08:43
8442d46 to
c4fa781
Compare
joseph-isaacs
changed the title
Use the planner-driven V2 ScanBuilder directly for every Vortex random access measurement, including object-store reads. Remove the reader toggle, its configuration plumbing, V1 documentation, and unused reader-cache setup. Remove the experiment-only regression workflow, driver script, and three fixture/selection examples, along with their futures dev dependency. Keep the planner implementation and regression tests. The previous commit c4fa781 preserves the diagnostic harnesses and benchmark context. Inspected the resulting diff and searched for stale switch/example references. No tests, builds, or benchmarks were rerun for this cleanup. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-random-access
github-actions
Bot
removed
the
action/bench-random-access
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.062x ➖, 0↑ 1↓)
random-access / vortex-file-compressed / ns (0.897x ✅, 7↑ 1↓)
random-access / parquet / ns (1.010x ➖, 1↑ 1↓)
random-access / lance / ns (1.000x ➖, 0↑ 0↓)
|
joseph-isaacs
added
action/bench-random-access
github-actions
Bot
removed
action/bench-compress
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Masked scans now apply selection through a planner before projection. ListLayout scans resolve selected offsets and validity before reading elements, and the projection planner can subdivide large fixed-width element ranges into independently scheduled work.
Based on develop
92a7db079. The planner owns subdivision, the shared 32 MiB decoded-work admission budget, child execution and ordered result collection. Small, repeated-segment, oversized-segment and unsupported work retain the serial pipeline.PlannerOutput::Awaitlets the driver suspend a planner while its owned child tasks finish; existingPlanner,Morsel,Source,IoSource, andSegmentSourcemethods are unchanged.Validation before benchmark cleanup (
c4fa7818b): 692 layout and 79 scan library tests passed, followed by all 7 await-protocol tests including one additional regression; scoped all-target/all-feature clippy passed with-D warnings; formatting used nightly-2026-09-10. That release binary passed 200 real-file V1/reference comparisons and five malformed-selection checks.The final cleanup removes the experiment CI workflow, regression script, three diagnostic examples, their dependency, and benchmark reader-switch plumbing. Inspected the diff and searched for stale references; tests, builds and benchmarks were not rerun for this cleanup. The diagnostic scaffolding remains available in parent commit
c4fa7818b.Local benchmarks compare the pre-cleanup head
c4fa7818bwith develop92a7db079, built with the samerelease_debugprofile and default CPU flags on a 32-core ARM host. Random access covers all 18 standard targets at 1 and 30 cores, with two alternating 3-second passes and explicit V1/V2 readers. Compression covers all 15 default CPU datasets at 16 cores, with two alternating passes and five iterations per dataset; reported compression timings use the harness's best-of-five metric.No standard random-access case regressed by more than 0.4% versus develop's V2. All 15 compressed sizes match. The small wide-table compression outlier did not persist: four alternating 30-iteration confirmations measured -0.4% encode / -1.7% decode for that case. Benchmark-only overlays expose V2 random access on develop and a V1 compression reader switch; no runtime baseline changes were made.
The planner refactor does retain an approximately 9–10% regression versus the rebased pre-refactor control for one large nested-list singleton with 1 MiB element chunks. Longer uninstrumented runs measured 0.241 ms before / 0.264 ms after. The cause is changed I/O batching and reduced parallel read overlap: a separate pread probe found 3.08 → 2.53 reads/query, ~3.2 MB combined reads in 14% → 31% of queries, active read concurrency 2.05 → 1.60, and time with reads active 89 → 112 µs/query. The extra ~23 µs accounts for the ~22 µs end-to-end gap. Read volume is effectively unchanged, CPU profiles have the same Arrow-conversion share (~33.6%), and the regression disappears with memory-backed input or one core.
This remains a draft. The 32 MiB estimate covers in-flight decoded work, not retained output or coalesced encoded buffers; the full memory matrix has not been revalidated for this version.