Repository navigation
feat(python): fail soft on unsupported Polars predicates in to_polars - #10414
robert3005 wants to merge 2 commits into
Conversation
Merging this PR will degrade performance by 12.33%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.7 µs | -12.33% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/polars-predicate-soft-fail-pfg49n (147344b) with develop (731a231)
Footnotes
-
359 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Previously any Polars predicate that polars_to_vortex could not convert raised NotImplementedError inside _io_source, failing the whole collect(). Instead, decompose the predicate into its top-level AND conjuncts, push the convertible conjuncts into the Vortex scan, and evaluate the residual conjuncts with Polars on each yielded batch. Since Polars does not re-apply a predicate handed to an IO source, both parts are always applied, and a row limit is only pushed into the scan when there is no residual filter (otherwise it is enforced after filtering each batch). Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U7dRrRezdUuF8r67wUyQdf
typos splits "ANDed" into "AN" + "Ded" and flags the latter. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U7dRrRezdUuF8r67wUyQdf
5bac692 to
147344b
Compare
Previously any Polars predicate that polars_to_vortex could not convert raised NotImplementedError inside _io_source, failing the whole collect(). Instead, decompose the predicate into its top-level AND conjuncts, push the convertible conjuncts into the Vortex scan, and evaluate the residual conjuncts with Polars on each yielded batch.
Since Polars does not re-apply a predicate handed to an IO source, both parts are always applied, and a row limit is only pushed into the scan when there is no residual filter (otherwise it is enforced after filtering each batch).