Repository navigation
fix(python): support Polars 2 - #10398
robert3005 wants to merge 1 commit into
Conversation
Polars 2 removed the `rechunk` keyword from the private `DataFrame._from_arrow`, which broke every `VortexFile.to_polars()` scan. Build each batch with the public `pl.DataFrame` constructor instead, which behaves the same on Polars 1.x and 2.x. Bump the dev dependency so the test suite runs against Polars 2. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E92wUoS3LaqWVJmmS7vKVq
Merging this PR will degrade performance by 11.75%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | scalar_subtract_neon |
10.7 µs | 12.2 µs | -12.17% |
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.6 µs | -11.34% |
| Simulation | bench_compare_primitive[(10000, 2)] |
64.2 µs | < 1 ns | N/A | |
| Simulation | density_sweep_dense_runs[0.9] |
< 1 ns | < 1 ns | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/polars-2 (94fb980) with develop (881b867)
Footnotes
-
534 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Summary
Polars 2.0.0 is out, and the
polars>=1.31.0extra has no upper bound, so users already get it. With Polars 2, everyVortexFile.to_polars()scan fails at collect time:to_polars()called the privatepl.DataFrame._from_arrow(batch, rechunk=False), and Polars 2 removed that keyword.Changes
vortex-python/python/vortex/file.py: build each batch, and the trailing empty batch, with the publicpl.DataFrame(record_batch)constructor. It works on both Polars 1.x and 2.x and is typed asDataFrame.pl.from_arrowwould returnDataFrame | Series.polarsdev dependency to>=2.0.0and relock (polars/polars-runtime-321.34.0 → 2.0.0), so CI tests against Polars 2.polars>=1.31.0extra is unchanged.The serialized expression format that
vortex.polars_parses for predicate pushdown is the same in 2.0. All expression-translation and pushdown tests pass unchanged.Checks run
uv run --all-packages pytest vortex-python/teston Polars 2.0.0: 409 passed, 2 skipped, 1 xfailed. Before the fix, 5to_polarstests failed with the error above.uvx ruff check,uvx ruff format --check,uvx ty check vortex-python vortex-python-cuda vortex-ffi/cmake/tests scripts/tests: clean.test_polars_.pyon Polars 1.31.0 (the extra's floor): same 4 failures as ondevelop, so nothing new. They are pre-existing: 1.31 serializes scalars as{"dtype": ..., "value": ...}, whichpolars_to_vortexdoesn't parse. This means the>=1.31.0floor is already too low for predicate pushdown. I've left that for a follow-up rather than widening this PR.🤖 Generated with Claude Code
https://claude.ai/code/session_01E92wUoS3LaqWVJmmS7vKVq
Generated by Claude Code