Repository navigation
fix(python): support Polars datetime literals and timezone expressions - #10183
Conversation
Merging this PR will improve performance by 10.09%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | words_gather_dispatch_neon[65536] |
2.3 µs | 2.1 µs | +10.09% |
| Simulation | bench_compare_primitive[(10000, 2)] |
64.6 µs | < 1 ns | N/A |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing dk/polars-datetime-literals (e5859a0) with develop (1ee4e3a)
Footnotes
-
409 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polars predicate pushdown can serialize unsigned comparison literals as typed scalar variants that the converter rejects. Decode recognized scalar variants through the existing literal type mapping, including UInt8, UInt16, UInt32, and UInt64. Add file-scan regression cases for comparisons on all four unsigned widths. Split from the original combined PR; datetime changes are in #10183 (stacked on #10182), and `is_not_null` changes are in #10184. Validation: 4 targeted regression case(s) passed with this PR’s converter. With only `polars_.py` restored to the base revision and the same tests retained, all 4 failed with Polars `ComputeError` wrapping the unsupported-expression `ValueError` or `NotImplementedError`. The tests compare complete dataframes from Polars alone and a Vortex-backed Polars scan, and check explicit row expectations. No lint or broader suite was run. --------- Signed-off-by: Daniel King <dan@spiraldb.com> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
e2695d2 to
993b177
Compare
6eea2e8 to
1af11fd
Compare
Timestamp expressions need to reinterpret local wall times in another timezone, including daylight-saving transitions. Add native `replace_time_zone` to Vortex's Rust and Python expression APIs, preserving timestamp units and propagating nulls. Support timezone removal, per-row `raise`/`earliest`/`latest`/`null` ambiguity policies, and `raise`/`null` policies for nonexistent times. Serialize options through protobuf and register the function for expression deserialization. Polars expression mapping is in the stacked datetime PR #10183. --------- Signed-off-by: Daniel King <dan@spiraldb.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
1af11fd to
7e0d03e
Compare
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Use datetime.UTC, annotate _time_zone_scan, type the datetime unit mapping for _dtype.timestamp, and build expected frames with pl.DataFrame so ty can see a DataFrame rather than DataFrame | Series. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E92wUoS3LaqWVJmmS7vKVq
7e0d03e to
e5859a0
Compare
Polars datetime literals can serialize as
Scalar.Datetimeor a literal wrapped inReplaceTimeZone, causing Vortex predicate conversion to fail. Decode datetime scalars and map timezone replacement to the native Vortex function provided by the base PR, without evaluating expressions in Polars during conversion.Share datetime decoding across serialization variants, unwrap timezone metadata, and set nullability from whether the value is null. Add file-scan and expression regressions for timestamp units, timezone removal, DST policies, policy columns, and serialization.
This PR is stacked on #10182, which adds native
replace_time_zone.Validation: 1 targeted regression case(s) passed with this PR’s converter. With only
polars_.pyrestored to the base revision and the same tests retained, all 1 failed with PolarsComputeErrorwrapping the unsupported-expressionValueErrororNotImplementedError. The tests compare complete dataframes from Polars alone and a Vortex-backed Polars scan, and check explicit row expectations. The full datetime/timezone selection also passed all 41 cases; comparison literals explicitly match the tested timestamp unit. No lint or broader suite was run.