Skip to content

fix(python): support Polars datetime literals and timezone expressions - #10183

Merged
robert3005 merged 4 commits into
developfrom
dk/polars-datetime-literals
Oct 9, 2026
Merged

robert3005 merged 4 commits into
developfrom
dk/polars-datetime-literals

Conversation

@danking

@danking danking commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Polars datetime literals can serialize as Scalar.Datetime or a literal wrapped in ReplaceTimeZone, causing Vortex predicate conversion to fail. Decode datetime scalars and map timezone replacement to the native Vortex function provided by the base PR, without evaluating expressions in Polars during conversion.

Share datetime decoding across serialization variants, unwrap timezone metadata, and set nullability from whether the value is null. Add file-scan and expression regressions for timestamp units, timezone removal, DST policies, policy columns, and serialization.

This PR is stacked on #10182, which adds native replace_time_zone.

Validation: 1 targeted regression case(s) passed with this PR’s converter. With only polars_.py restored to the base revision and the same tests retained, all 1 failed with Polars ComputeError wrapping the unsupported-expression ValueError or NotImplementedError. The tests compare complete dataframes from Polars alone and a Vortex-backed Polars scan, and check explicit row expectations. The full datetime/timezone selection also passed all 41 cases; comparison literals explicitly match the tested timestamp unit. No lint or broader suite was run.

@codspeed

codspeed Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 10.09%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 2182 untouched benchmarks
⏩ 409 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ WallTime words_gather_dispatch_neon[65536] 2.3 µs 2.1 µs +10.09%
⚠️ Simulation bench_compare_primitive[(10000, 2)] 64.6 µs < 1 ns N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing dk/polars-datetime-literals (e5859a0) with develop (1ee4e3a)

Open in CodSpeed

Footnotes

  1. 409 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

robert3005 pushed a commit that referenced this pull request Oct 8, 2026
Polars predicate pushdown can serialize unsigned comparison literals as
typed scalar variants that the converter rejects. Decode recognized
scalar variants through the existing literal type mapping, including
UInt8, UInt16, UInt32, and UInt64.

Add file-scan regression cases for comparisons on all four unsigned
widths. Split from the original combined PR; datetime changes are in
#10183 (stacked on #10182), and `is_not_null` changes are in #10184.

Validation: 4 targeted regression case(s) passed with this PR’s
converter. With only `polars_.py` restored to the base revision and the
same tests retained, all 4 failed with Polars `ComputeError` wrapping
the unsupported-expression `ValueError` or `NotImplementedError`. The
tests compare complete dataframes from Polars alone and a Vortex-backed
Polars scan, and check explicit row expectations. No lint or broader
suite was run.

---------

Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
@robert3005
robert3005 force-pushed the dk/replace-time-zone branch from e2695d2 to 993b177 Compare October 9, 2026 07:40
@robert3005
robert3005 force-pushed the dk/polars-datetime-literals branch from 6eea2e8 to 1af11fd Compare October 9, 2026 07:41
robert3005 pushed a commit that referenced this pull request Oct 9, 2026
Timestamp expressions need to reinterpret local wall times in another
timezone, including daylight-saving transitions. Add native
`replace_time_zone` to Vortex's Rust and Python expression APIs,
preserving timestamp units and propagating nulls.

Support timezone removal, per-row `raise`/`earliest`/`latest`/`null`
ambiguity policies, and `raise`/`null` policies for nonexistent times.
Serialize options through protobuf and register the function for
expression deserialization. Polars expression mapping is in the stacked
datetime PR #10183.

---------

Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Base automatically changed from dk/replace-time-zone to develop October 9, 2026 13:28
@robert3005
robert3005 force-pushed the dk/polars-datetime-literals branch from 1af11fd to 7e0d03e Compare October 9, 2026 13:37
danking and others added 4 commits October 9, 2026 14:06
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Use datetime.UTC, annotate _time_zone_scan, type the datetime unit
mapping for _dtype.timestamp, and build expected frames with
pl.DataFrame so ty can see a DataFrame rather than DataFrame | Series.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E92wUoS3LaqWVJmmS7vKVq
@robert3005
robert3005 enabled auto-merge (squash) October 9, 2026 14:06
@robert3005
robert3005 force-pushed the dk/polars-datetime-literals branch from 7e0d03e to e5859a0 Compare October 9, 2026 14:08
@robert3005
robert3005 merged commit f8fb744 into develop Oct 9, 2026
87 checks passed
@robert3005
robert3005 deleted the dk/polars-datetime-literals branch October 9, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants