Skip to content

Epic: Duckdb projection expression pushdown #8310

Description

@myrrc

We want to push down scalar and aggregate functions that are part of SELECT into vortex.
Example: SELECT strlen(col) doesn't need to decompress the strings.

Subtasks:

Upstreaming:

Additional work:

  • Push down chains of functions f_1(...(f_n(col)) if leaf child is
    BoundColumnRef.
  • Push down multiple expressions for column if we can push all of them i.e.
    allow pushing down (strlen(col), col). This requires a change on projection
    ids passed to vortex. In this example we need to pass projection_ids=[0, 0].

PR with upstreaming aggregation pushdown callback duckdb/duckdb#24697 is declined because Duckdb maintainers feel their plan-time pass is sufficient.

Activity

  1. added
    ext/duckdbRelates to the DuckDB integration
    tracking-issueShared implementation context for work likely to span multiple PRs.
    on Jun 9, 2026
  2. self-assigned this
    on Jun 10, 2026
  3. linked a pull request that will close this issueScalar functions pushdown to Duckdb #8303on Jun 17, 2026
  4. reopened this on Jun 18, 2026
  5. myrrc commented on Jun 18, 2026

    @myrrc
    ContributorAuthor

    This change regressed vortex-compact significantly on the same query. We can't solve this issue now as ZSTD stores lengths with data, so you need to decompress. Implementing a kernel for byte_length which only used decompressed length without constructing a VarBinView didn't yield any performance improvements. We'll accept the regression as there's no way for reader to distinguish vortex vs. vortex-compact

  6. linked a pull request that will close this issuePush down type casts to file readers #22788on Jun 26, 2026
  7. changed the title [-]Tracking Issue: Duckdb projection expression pushdown[/-] [+]Epic: Duckdb projection expression pushdown[/+] on Jun 26, 2026
  8. added
    epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.
    and removed
    tracking-issueShared implementation context for work likely to span multiple PRs.
    on Jun 26, 2026
  9. linked a pull request that will close this issueCast pushdown for duckdb #8620on Jun 29, 2026
  10. linked a pull request that will close this issuePush down scalar functions to file readers #23528on Jun 30, 2026
  11. linked a pull request that will close this issueDon't push down CAST if there's a TRY_CAST #23535on Jun 30, 2026
  12. reopened this on Jul 8, 2026
  13. reopened this on Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.ext/duckdbRelates to the DuckDB integration

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions