feat(layout): add push-based pipelines and the scan that runs them

[joseph-isaacs]

Adds the pipeline runtime and plan hooks so an in-memory PlanRef can execute over row ranges. Tracking: #10349. Concrete plan operators follow in #10392 and #10393; #10389 is independent.

A pipeline has one Source: Operator, followed by zero or more operators. Source::ready checks inlet/IO readiness and queues segment reads through Cx::request_read, returning Ready or Blocked. Both newly queued and outstanding reads use Blocked(Io); the driver issues queued requests before waiting for bytes. Decoding and transformation stay in non-blocking Operator::compute.

compute accepts Input::{None, Batch, End} and returns Step::{End, More, Final}:

• More(batch) produces a batch and requests another compute call with None, without new input or another source readiness check.
• Final(batch) produces the last batch for this input. A stage waits for its next input; a source checks readiness again.
• End produces no output for the current input. It finishes a source, or finishes a stage when draining upstream's Input::End.

The driver drains downstream continuations first, preserving their state across outlet backpressure. It delivers Input::End once per stage and drains all flush output before closing outlets.

Ports have one writer and reader. Compiler::join_with sets their capacities. The scheduler wakes pipelines when their blocked condition clears, tracks touched inlets in a compact set, and reports changed inlet indices to source readiness. Scan::step returns read requests and output; Scan::deliver supplies bytes tagged with their segment ID. Retiring a pipeline cancels outstanding reads.

Synthetic tests cover input/output order, nested continuations, multi-batch flushing, bounded outlets, batched read requests, and readiness that consumes an inlet before requesting IO.

Physical plans and their optimizer live under vortex_layout::plan::physical. The sibling operator module defines Operator, Source, and the execution context; operators contains concrete implementations; pipeline owns compilation, scheduling, and ports.

Validation: cargo +nightly-2026-09-10 fmt -p vortex-layout; scoped cargo clippy -p vortex-layout --all-targets --all-features -- -D warnings; targeted physical-plan tests (52 passed).

[codspeed]

Merging this PR will degrade performance by 15.74%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 2 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

• bench_compare_primitive[(10000, 2)]
• density_sweep_dense_runs[0.9]

⚠️ 12 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• compress_fsst[(500, 4, 8)]
• random_i128[0.01]
• from_vec_drop_vortex[65536]
• zstd_compress_string
• decompress[u16, (4000, 256)]
• decompress[u32, (1000, 256)]
• multipolygon_mixed_validity
• take_varbin
• pcodec_compress_f64
• filter_run_end[len=4096_run=16_density=0.1]
• …and 2 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 4 regressed benchmarks
✅ 2208 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	decode_primitives[i64, (1000, 32)]	24.7 µs	38.3 µs	-35.49%
❌	WallTime	threshold_neon[8]	2.9 µs	3.8 µs	-22.88%
❌	WallTime	threshold_neon[64]	4.3 µs	4.9 µs	-12.21%
❌	WallTime	threshold_neon[80]	4.7 µs	5.3 µs	-11.65%
⚡	WallTime	words_gather_dispatch_neon[65536]	2.3 µs	2.1 µs	+10.05%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/exec-2-exec-node (b966c27) with develop (e277ccd)}

[codecov]

Codecov Report

❌ Patch coverage is 81.76539% with 157 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.41%. Comparing base (1557fb0) to head (94d7196).
⚠️ Report is 37 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-layout/src/plan/pipeline/scan.rs	86.24%	52 Missing ⚠️
vortex-layout/src/plan/pipeline/compile.rs	64.70%	24 Missing ⚠️
vortex-layout/src/plan/typed.rs	48.83%	22 Missing ⚠️
vortex-layout/src/plan/pipeline/synthetic.rs	88.00%	18 Missing ⚠️
vortex-layout/src/plan/vtable.rs	0.00%	18 Missing ⚠️
vortex-layout/src/plan/pipeline/mod.rs	48.00%	13 Missing ⚠️
vortex-layout/src/plan/pipeline/port.rs	90.90%	9 Missing ⚠️
...ortex-layout/src/plan/pipeline/scheduling_tests.rs	98.75%	1 Missing ⚠️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.