|
| 1 | +# Expression Pushdown in Scan Plans |
| 2 | + |
| 3 | +:::{note} |
| 4 | +This is a provisional design, not a description of the current scan planner. |
| 5 | +::: |
| 6 | + |
| 7 | +## Expression plan |
| 8 | + |
| 9 | +Add a physical plan that applies an expression to the output of another scan plan: |
| 10 | + |
| 11 | +```rust |
| 12 | +pub struct ExpressionPlan { |
| 13 | + expression: Expression, |
| 14 | + child: PlanRef, |
| 15 | +} |
| 16 | +``` |
| 17 | + |
| 18 | +`ExpressionPlan` inherits its row domain from `child` and derives its output dtype from |
| 19 | +`expression`. Any part of the expression that cannot be pushed down remains in this plan and is |
| 20 | +evaluated over the child's result. |
| 21 | + |
| 22 | +Scalar functions remain nodes in `Expression`; they do not each become physical scan-plan nodes. |
| 23 | +Instead, pushdown behavior is supplied by pluggable kernels registered for a scalar function (or |
| 24 | +other expression node) and a concrete scan-plan type. |
| 25 | + |
| 26 | +Composite scan plans expose ordered logical children to this optimizer: |
| 27 | + |
| 28 | +```text |
| 29 | +StructPlan.children = [field(0), ..., field(n - 1), validity?] |
| 30 | +DictPlan.children = [codes, values] |
| 31 | +``` |
| 32 | + |
| 33 | +## Pushdown |
| 34 | + |
| 35 | +Optimizing `ExpressionPlan(expression, child)` has three phases. |
| 36 | + |
| 37 | +1. **Annotate dependencies.** Walk the expression and annotate every node with the indices of the |
| 38 | + immediate scan-plan children needed to evaluate it. A registered |
| 39 | + `(expression node, scan plan) -> [child index]` kernel provides plan-specific dependencies; |
| 40 | + ordinary scalar functions can otherwise take the union of their expression children's |
| 41 | + annotations. For a struct, `get_item($, "a")` needs field `a` and, when present, the struct |
| 42 | + validity child. `is_not_null($)` and `is_null($)` need only the top-level validity child. |
| 43 | + |
| 44 | +2. **Partition and group.** Only a subexpression annotated with exactly one scan-plan child is |
| 45 | + eligible for pushdown. Cut maximal eligible subexpressions, group all cuts for the same child, |
| 46 | + and build one packed expression for that child. Replace the cuts in the remaining expression |
| 47 | + with references to the group results; this remaining expression is the combination expression. |
| 48 | + Expressions needing zero or multiple children stay above the current plan. This follows the |
| 49 | + grouping model of the existing expression partitioner and ensures each child group is evaluated |
| 50 | + once. |
| 51 | + |
| 52 | +3. **Lower into each child.** Rewrite every root reference in a group from the current plan's scope |
| 53 | + into the selected child's scope. This uses a second pluggable |
| 54 | + `(expression node, scan plan, child index) -> expression` kernel. For a non-nullable struct, |
| 55 | + `get_item($, "a")` becomes `$` when lowering into field `a`; targeting another field rejects that |
| 56 | + pushdown. When lowering into a struct's validity child, `is_not_null($)` becomes `$` and |
| 57 | + `is_null($)` becomes `not($)`. The lowered group is installed as an `ExpressionPlan` over |
| 58 | + that child, and pushdown then continues recursively. Every rule must prove equivalence across |
| 59 | + the plan boundary; lowering is not a generic replacement of `$`. |
| 60 | + |
| 61 | +The combination expression is retained above the grouped child plans. Missing kernels or failed |
| 62 | +lowering leave the affected expression at the current level, preserving the generic execution |
| 63 | +fallback. |
| 64 | + |
| 65 | +## Examples |
| 66 | + |
| 67 | +`@name` denotes a reference from the combination expression to a grouped child result. |
| 68 | + |
| 69 | +### Struct without validity |
| 70 | + |
| 71 | +```text |
| 72 | +plan = StructPlan.children = [a, b] |
| 73 | +expr = (get_item($, "a") + 1) * get_item($, "b") |
| 74 | +
|
| 75 | +annotations: |
| 76 | + get_item($, "a") + 1 -> {a} |
| 77 | + get_item($, "b") -> {b} |
| 78 | + expr -> {a, b} |
| 79 | +
|
| 80 | +groups before lowering: |
| 81 | + @a = get_item($, "a") + 1 |
| 82 | + @b = get_item($, "b") |
| 83 | +
|
| 84 | +groups after lowering: |
| 85 | + @a = $ + 1 |
| 86 | + @b = $ |
| 87 | +
|
| 88 | +combine = @a * @b |
| 89 | +``` |
| 90 | + |
| 91 | +The current non-nullable struct shape can therefore install `$ + 1` over child `a`, read child |
| 92 | +`b` directly, and evaluate only the multiplication above the grouped results. The `+` needs no |
| 93 | +struct-specific rule: it is rebuilt after its `get_item` child is lowered. |
| 94 | + |
| 95 | +### Struct with validity |
| 96 | + |
| 97 | +```text |
| 98 | +plan = StructPlan.children = [a, b, validity] |
| 99 | +expr = is_not_null($) && (get_item($, "a") > 0) |
| 100 | +
|
| 101 | +annotations: |
| 102 | + is_not_null($) -> {validity} |
| 103 | + get_item($, "a") -> {a, validity} |
| 104 | + get_item($, "a") > 0 -> {a, validity} |
| 105 | + expr -> {a, validity} |
| 106 | +
|
| 107 | +groups before lowering: |
| 108 | + @valid = is_not_null($) |
| 109 | +
|
| 110 | +groups after lowering: |
| 111 | + @valid = $ |
| 112 | +
|
| 113 | +combine = @valid && (get_item($, "a") > 0) |
| 114 | +``` |
| 115 | + |
| 116 | +The direct `is_not_null($) -> $` lowering is valid because `$` on the validity child is exactly the |
| 117 | +top-level struct validity. The analogous direct lowering of `get_item` is not valid: |
| 118 | + |
| 119 | +```text |
| 120 | +get_item(nullable_struct, "a") = mask(a, validity) |
| 121 | +
|
| 122 | +invalid lowering: |
| 123 | + get_item($, "a") -> $ |
| 124 | +
|
| 125 | +counterexample: |
| 126 | + a = 7, validity = false |
| 127 | + get_item(nullable_struct, "a") = null |
| 128 | + lowered result = 7 |
| 129 | +``` |
| 130 | + |
| 131 | +The singleton rule prevents this mistake: `get_item($, "a")` is annotated with `{a, validity}` and |
| 132 | +therefore remains above the struct. A stronger validity-aware rule may lower a strict function by |
| 133 | +factoring out the mask: |
| 134 | + |
| 135 | +```text |
| 136 | +(get_item($, "a") + 1) |
| 137 | + = mask(a, validity) + 1 |
| 138 | + = mask(a + 1, validity) |
| 139 | +
|
| 140 | +groups after lowering: |
| 141 | + @a = $ + 1 |
| 142 | + @valid = $ |
| 143 | +
|
| 144 | +combine = mask(@a, @valid) |
| 145 | +``` |
| 146 | + |
| 147 | +This requires `+` to be strict. It must also be infallible, or execute under `@valid`, so that |
| 148 | +evaluating values hidden by the mask cannot introduce a new error. Non-strict functions require |
| 149 | +different combination expressions: |
| 150 | + |
| 151 | +```text |
| 152 | +is_not_null(get_item($, "a")) |
| 153 | + = @valid && is_not_null(@a) |
| 154 | +
|
| 155 | +is_null(get_item($, "a")) |
| 156 | + = !@valid || is_null(@a) |
| 157 | +``` |
| 158 | + |
| 159 | +Lowering through validity is therefore a proven factorization into child expressions and a |
| 160 | +residual combination expression, not just scope substitution. |
| 161 | + |
| 162 | +### Dictionary |
| 163 | + |
| 164 | +```text |
| 165 | +plan = DictPlan.children = [codes, values] |
| 166 | +expr = byte_length($) |
| 167 | +
|
| 168 | +annotations: |
| 169 | + byte_length($) -> {values} |
| 170 | +
|
| 171 | +groups before lowering: |
| 172 | + @values = byte_length($) |
| 173 | +
|
| 174 | +groups after lowering: |
| 175 | + @values = byte_length($) |
| 176 | +
|
| 177 | +combine = $ |
| 178 | +
|
| 179 | +result = DictPlan.children = [ |
| 180 | + codes, |
| 181 | + ExpressionPlan(byte_length($), values), |
| 182 | +] |
| 183 | +``` |
| 184 | + |
| 185 | +This is valid for the same strict, infallible, negative-cost functions accepted by the current |
| 186 | +dictionary pushdown. The dictionary plan reuses `codes` and applies the function once to |
| 187 | +`values`. |
| 188 | + |
| 189 | +## Integration boundary |
| 190 | + |
| 191 | +This initial planner change only constructs, optimizes, and displays physical plan trees. It does |
| 192 | +not change layout vtables, layout readers, scan builders, or scan execution. Built-in layouts are |
| 193 | +dispatched inside `vortex_layout::plan::new_plan`, keeping the provisional planner isolated while |
| 194 | +its plan shapes and optimization rules are developed. |
| 195 | + |
| 196 | +A follow-up can define reader materialization and wire optimized plans into scans. Until then, the |
| 197 | +existing reader and scan paths continue unchanged. |
| 198 | + |
| 199 | +## Future work |
| 200 | + |
| 201 | +The same optimizer may eventually support `Plan x Plan -> Plan` transforms. Those |
| 202 | +plan-to-plan rewrites are outside the scope of this proposal; this design covers only expression |
| 203 | +and scalar-function pushdown through scan plans. |
0 commit comments