You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c4c1bbf
Browse filesBrowse the repository at this point in the historyBrowse files
Supersedes #9514.
Optimization now runs only on `BoundExpression`s: callers bind first,
then optimize. This removes the optimizer-owned dtype cache, rules can
read dtypes straight off the tree, and it unblocks lambdas/variables,
which must be typed before optimization.
## Changes
- `BoundExpression::optimize` and `optimize_recursive` replace the
`Expression` versions. Traversal and the fixpoint loop are unchanged.
- `ScalarFnVTable::simplify` now takes a `&BoundExpression`, and
`simplify_untyped` is merged into it. `SimplifyCtx` and `SimplifyCache`
are removed.
- `ExpressionReduceNode` keeps its name and `reduce_expression`
dispatch, but now borrows a bound tree.
- Rules ported: binary, case_when, cast, fill_null, mask, select, zip,
and `find_between`. `GetItem::simplify_untyped` is removed because it
duplicated `GetItem::reduce`. If the rebuilt `Between` fails to
type-check, `find_between` now skips the rewrite instead of producing a
tree that fails to bind.
- Callers change from `expr.optimize_recursive(dtype)?.bind(dtype)` to
`expr.bind(dtype)?.optimize_recursive()`.
- The unbound `expr::transform::partition` is deleted. It was the last
user of the unbound optimizer and has no callers; layouts use
`bound_partition`. #9993 deletes it too, so whichever PR lands second
will have a trivial conflict.
## API changes
- `Expression::optimize(scope)` / `optimize_recursive(scope)` become
`BoundExpression::optimize()` / `optimize_recursive()`.
- `ScalarFnVTable::simplify(options, &BoundExpression)` replaces
`simplify(options, &Expression, &dyn SimplifyCtx)` and
`simplify_untyped`.
- `ExpressionReduceNode::new(&BoundExpression)` drops the scope
argument.
- `partition`, `partition_annotations` and `PartitionedExpr` are
removed.
## Behavior
I checked this with a differential corpus of 30,018 generated
expressions targeting every rule, comparing develop's `optimize + bind`
against the new `bind + optimize` and evaluating before/after on data:
- 28,878 produce identical output.
- 131 panicked on develop (`Mask` simplify called `vortex_expect` on a
null mask literal) and now fail cleanly at bind time.
- 941 have ill-typed inputs that develop accepted only because a rule
removed the ill-typed subtree before binding. They now error.
- 68 differ because of a develop bug. `SimplifyCache` keys dtypes by
`Expression`, and `Scalar` equality ignores nullability, so develop
reused the wrong cached dtype and dropped `fill_null`s it should have
kept.
No well-typed expression regressed.
Existing bugs found, not addressed here:
- The `CASE WHEN`→`fill_null` rewrite is unexecutable for utf8.
- `select([], nullable_struct)` drops the struct's validity.
- `CaseWhen` execution can return a result that is non-nullable when its
dtype is nullable.
- `and`/`or`/`zip` literal rules can change the result's nullability.
## Performance
The optimizer benchmarks bind outside the timed loop and measure only
`optimize_recursive`, the same work develop's benchmarks time. Local
medians:
| case | develop | this PR |
|---|---|---|
| `or_chain` 200 | 17.5 µs | 18.6 µs |
| `balanced_or` 64–256 ids | 6.2–23.0 µs | 6.1–23.5 µs |
| `balanced_or` 1 id | 0.79 µs | 0.94 µs |
| `in_list` | ~0.77 µs | ~0.95 µs |
Small predicates pay a constant ~0.15–0.2 µs in the optimizer, almost
all of it in `find_between`, which now type-checks the `and`/`Between`
nodes it rebuilds (0.56 µs vs 0.34 µs). Across the full `bind +
optimize` pipeline that callers run, the gap is ~0.3–0.45 µs, because
binding also types the comparisons `find_between` later replaces.
Closing it would need an unchecked `BoundExpression` constructor. Open
question.
## Testing
- `cargo nextest run -p vortex-array -p vortex-layout -p vortex-file -p
vortex -p vortex-datafusion -p vortex-duckdb`
- `cargo test --doc -p vortex -p vortex-array`
- `cargo clippy --all-targets --all-features -- -D warnings` on all
changed crates and `fuzz/file_io`
- `cargo check --workspace --all-targets`
Not run: Python tests. `vortex-cuda-ffi` was not type-checked because it
is compiled out on macOS.
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
0 commit comments