Skip to content

Commit 68ca1b3

Browse files
committed
Add layout scan physical plan model
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
1 parent 12fe696 commit 68ca1b3

15 files changed

Lines changed: 1669 additions & 0 deletions

File tree

‎docs/developer-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ internals/session
2323
internals/async-runtime
2424
internals/vtables
2525
internals/execution
26+
internals/scan-planning
2627
internals/stats-pruning
2728
internals/io
2829
internals/serialization
Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Expression Pushdown in Scan Plans
2+
3+
:::{note}
4+
This is a provisional design, not a description of the current scan planner.
5+
:::
6+
7+
## Expression plan
8+
9+
Add a physical plan that applies an expression to the output of another scan plan:
10+
11+
```rust
12+
pub struct ExpressionPlan {
13+
expression: Expression,
14+
child: PlanRef,
15+
}
16+
```
17+
18+
`ExpressionPlan` inherits its row domain from `child` and derives its output dtype from
19+
`expression`. Any part of the expression that cannot be pushed down remains in this plan and is
20+
evaluated over the child's result.
21+
22+
Scalar functions remain nodes in `Expression`; they do not each become physical scan-plan nodes.
23+
Instead, pushdown behavior is supplied by pluggable kernels registered for a scalar function (or
24+
other expression node) and a concrete scan-plan type.
25+
26+
Composite scan plans expose ordered logical children to this optimizer:
27+
28+
```text
29+
StructPlan.children = [field(0), ..., field(n - 1), validity?]
30+
DictPlan.children = [codes, values]
31+
```
32+
33+
## Pushdown
34+
35+
Optimizing `ExpressionPlan(expression, child)` has three phases.
36+
37+
1. **Annotate dependencies.** Walk the expression and annotate every node with the indices of the
38+
immediate scan-plan children needed to evaluate it. A registered
39+
`(expression node, scan plan) -> [child index]` kernel provides plan-specific dependencies;
40+
ordinary scalar functions can otherwise take the union of their expression children's
41+
annotations. For a struct, `get_item($, "a")` needs field `a` and, when present, the struct
42+
validity child. `is_not_null($)` and `is_null($)` need only the top-level validity child.
43+
44+
2. **Partition and group.** Only a subexpression annotated with exactly one scan-plan child is
45+
eligible for pushdown. Cut maximal eligible subexpressions, group all cuts for the same child,
46+
and build one packed expression for that child. Replace the cuts in the remaining expression
47+
with references to the group results; this remaining expression is the combination expression.
48+
Expressions needing zero or multiple children stay above the current plan. This follows the
49+
grouping model of the existing expression partitioner and ensures each child group is evaluated
50+
once.
51+
52+
3. **Lower into each child.** Rewrite every root reference in a group from the current plan's scope
53+
into the selected child's scope. This uses a second pluggable
54+
`(expression node, scan plan, child index) -> expression` kernel. For a non-nullable struct,
55+
`get_item($, "a")` becomes `$` when lowering into field `a`; targeting another field rejects that
56+
pushdown. When lowering into a struct's validity child, `is_not_null($)` becomes `$` and
57+
`is_null($)` becomes `not($)`. The lowered group is installed as an `ExpressionPlan` over
58+
that child, and pushdown then continues recursively. Every rule must prove equivalence across
59+
the plan boundary; lowering is not a generic replacement of `$`.
60+
61+
The combination expression is retained above the grouped child plans. Missing kernels or failed
62+
lowering leave the affected expression at the current level, preserving the generic execution
63+
fallback.
64+
65+
## Examples
66+
67+
`@name` denotes a reference from the combination expression to a grouped child result.
68+
69+
### Struct without validity
70+
71+
```text
72+
plan = StructPlan.children = [a, b]
73+
expr = (get_item($, "a") + 1) * get_item($, "b")
74+
75+
annotations:
76+
get_item($, "a") + 1 -> {a}
77+
get_item($, "b") -> {b}
78+
expr -> {a, b}
79+
80+
groups before lowering:
81+
@a = get_item($, "a") + 1
82+
@b = get_item($, "b")
83+
84+
groups after lowering:
85+
@a = $ + 1
86+
@b = $
87+
88+
combine = @a * @b
89+
```
90+
91+
The current non-nullable struct shape can therefore install `$ + 1` over child `a`, read child
92+
`b` directly, and evaluate only the multiplication above the grouped results. The `+` needs no
93+
struct-specific rule: it is rebuilt after its `get_item` child is lowered.
94+
95+
### Struct with validity
96+
97+
```text
98+
plan = StructPlan.children = [a, b, validity]
99+
expr = is_not_null($) && (get_item($, "a") > 0)
100+
101+
annotations:
102+
is_not_null($) -> {validity}
103+
get_item($, "a") -> {a, validity}
104+
get_item($, "a") > 0 -> {a, validity}
105+
expr -> {a, validity}
106+
107+
groups before lowering:
108+
@valid = is_not_null($)
109+
110+
groups after lowering:
111+
@valid = $
112+
113+
combine = @valid && (get_item($, "a") > 0)
114+
```
115+
116+
The direct `is_not_null($) -> $` lowering is valid because `$` on the validity child is exactly the
117+
top-level struct validity. The analogous direct lowering of `get_item` is not valid:
118+
119+
```text
120+
get_item(nullable_struct, "a") = mask(a, validity)
121+
122+
invalid lowering:
123+
get_item($, "a") -> $
124+
125+
counterexample:
126+
a = 7, validity = false
127+
get_item(nullable_struct, "a") = null
128+
lowered result = 7
129+
```
130+
131+
The singleton rule prevents this mistake: `get_item($, "a")` is annotated with `{a, validity}` and
132+
therefore remains above the struct. A stronger validity-aware rule may lower a strict function by
133+
factoring out the mask:
134+
135+
```text
136+
(get_item($, "a") + 1)
137+
= mask(a, validity) + 1
138+
= mask(a + 1, validity)
139+
140+
groups after lowering:
141+
@a = $ + 1
142+
@valid = $
143+
144+
combine = mask(@a, @valid)
145+
```
146+
147+
This requires `+` to be strict. It must also be infallible, or execute under `@valid`, so that
148+
evaluating values hidden by the mask cannot introduce a new error. Non-strict functions require
149+
different combination expressions:
150+
151+
```text
152+
is_not_null(get_item($, "a"))
153+
= @valid && is_not_null(@a)
154+
155+
is_null(get_item($, "a"))
156+
= !@valid || is_null(@a)
157+
```
158+
159+
Lowering through validity is therefore a proven factorization into child expressions and a
160+
residual combination expression, not just scope substitution.
161+
162+
### Dictionary
163+
164+
```text
165+
plan = DictPlan.children = [codes, values]
166+
expr = byte_length($)
167+
168+
annotations:
169+
byte_length($) -> {values}
170+
171+
groups before lowering:
172+
@values = byte_length($)
173+
174+
groups after lowering:
175+
@values = byte_length($)
176+
177+
combine = $
178+
179+
result = DictPlan.children = [
180+
codes,
181+
ExpressionPlan(byte_length($), values),
182+
]
183+
```
184+
185+
This is valid for the same strict, infallible, negative-cost functions accepted by the current
186+
dictionary pushdown. The dictionary plan reuses `codes` and applies the function once to
187+
`values`.
188+
189+
## Integration boundary
190+
191+
This initial planner change only constructs, optimizes, and displays physical plan trees. It does
192+
not change layout vtables, layout readers, scan builders, or scan execution. Built-in layouts are
193+
dispatched inside `vortex_layout::plan::new_plan`, keeping the provisional planner isolated while
194+
its plan shapes and optimization rules are developed.
195+
196+
A follow-up can define reader materialization and wire optimized plans into scans. Until then, the
197+
existing reader and scan paths continue unchanged.
198+
199+
## Future work
200+
201+
The same optimizer may eventually support `Plan x Plan -> Plan` transforms. Those
202+
plan-to-plan rewrites are outside the scope of this proposal; this design covers only expression
203+
and scalar-function pushdown through scan plans.

‎vortex-layout/src/lib.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
//! optional filter, optional row range, [`Selection`](vortex_scan::selection::Selection), split
1616
//! strategy, and task concurrency settings, then produces array streams or iterators.
1717
pub mod layouts;
18+
pub mod plan;
1819

1920
pub use children::*;
2021
pub use encoding::*;

0 commit comments

Comments
 (0)