Skip to content

Select(len()) fast path doesn't cover DataFrameScan #24112

Description

@Matt711

Select.evaluate (ir.py) has a fast-count pushdown for Select(len()) over a Scan(parquet) — it reads the row count from file metadata instead of actually scanning. DataFrameScan.do_evaluate already computes height = pl_df.height for free, and even has a zero-width-schema fast path that returns using only height. But for any non-zero-width schema, it always falls through to DataFrame.from_polars, materializing the whole frame on GPU even when the caller only wants len().

This isn't just a missed optimization, it can crash. polars' Series.new_from_index builds a virtual, unmaterialized series of arbitrary length (no real backing data). Union two of those and do .select(pl.len()), and cudf-polars tries to materialize both branches for real, hitting libcudf's 32-bit size_type limit: OverflowError: ... Number of rows exceeds cuDF's maximum supported row count (cudf::size_type). Under the streaming engine this actually crashes the worker instead of raising cleanly.

Fix would be extending the existing Select._is_len_expr fast path past Scan(parquet) to also recognize DataFrameScan (using pl_df.height), and see through Cache/Union so branch heights can be summed instead of materialized.

Repro is upstream polars' tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657, currently xfailed/skipped in cudf-polars' inject_gpu_engine.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcudf-polarsIssues specific to cudf-polars

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions