Select.evaluate (ir.py) has a fast-count pushdown for Select(len()) over a Scan(parquet) — it reads the row count from file metadata instead of actually scanning. DataFrameScan.do_evaluate already computes height = pl_df.height for free, and even has a zero-width-schema fast path that returns using only height. But for any non-zero-width schema, it always falls through to DataFrame.from_polars, materializing the whole frame on GPU even when the caller only wants len().
This isn't just a missed optimization, it can crash. polars' Series.new_from_index builds a virtual, unmaterialized series of arbitrary length (no real backing data). Union two of those and do .select(pl.len()), and cudf-polars tries to materialize both branches for real, hitting libcudf's 32-bit size_type limit: OverflowError: ... Number of rows exceeds cuDF's maximum supported row count (cudf::size_type). Under the streaming engine this actually crashes the worker instead of raising cleanly.
Fix would be extending the existing Select._is_len_expr fast path past Scan(parquet) to also recognize DataFrameScan (using pl_df.height), and see through Cache/Union so branch heights can be summed instead of materialized.
Repro is upstream polars' tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657, currently xfailed/skipped in cudf-polars' inject_gpu_engine.py.
Select.evaluate(ir.py) has a fast-count pushdown forSelect(len())over aScan(parquet)— it reads the row count from file metadata instead of actually scanning.DataFrameScan.do_evaluatealready computesheight = pl_df.heightfor free, and even has a zero-width-schema fast path that returns using onlyheight. But for any non-zero-width schema, it always falls through toDataFrame.from_polars, materializing the whole frame on GPU even when the caller only wantslen().This isn't just a missed optimization, it can crash. polars'
Series.new_from_indexbuilds a virtual, unmaterialized series of arbitrary length (no real backing data). Union two of those and do.select(pl.len()), and cudf-polars tries to materialize both branches for real, hitting libcudf's 32-bitsize_typelimit:OverflowError: ... Number of rows exceeds cuDF's maximum supported row count (cudf::size_type). Under the streaming engine this actually crashes the worker instead of raising cleanly.Fix would be extending the existing
Select._is_len_exprfast path pastScan(parquet)to also recognizeDataFrameScan(usingpl_df.height), and see throughCache/Unionso branch heights can be summed instead of materialized.Repro is upstream polars'
tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657, currently xfailed/skipped in cudf-polars'inject_gpu_engine.py.