Skip to content

feat(python): support row indices in VortexFile.to_arrow - #10373

Merged
joseph-isaacs merged 4 commits into
developfrom
ji/python-arrow-row-indices
Oct 8, 2026
Merged

joseph-isaacs merged 4 commits into
developfrom
ji/python-arrow-row-indices

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

VortexFile.to_arrow cannot select row indices even though scan and prepare support them. Add the indices argument to the Python API and binding, using the same scan builder and strictly increasing, non-null index validation.

Changes

New cases cover empty and sparse selections, nullable strings, requested ordinary-string schemas, scan-equivalent batching, filtering, limits, and invalid indices. The existing restriction on combined filters and limits is preserved. Existing tests remain unchanged.

API Changes

Adds the optional keyword argument indices: vortex.Array | None to VortexFile.to_arrow(). Indices must be strictly increasing and non-null, matching scan() and prepare().

Validation

  • python -m pytest vortex-python/test/test_file_arrow_indices.py vortex-python/test/test_file.py -q — 21 passed.

Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review October 8, 2026 10:27
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Oct 8, 2026
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) October 8, 2026 15:00
DCO Remediation Commit for Joe Isaacs <joe.isaacs@live.co.uk>

I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 0ee1827

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed

codspeed Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

✅ 2187 untouched benchmarks
⏩ 518 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚠️ Simulation density_sweep_dense_runs[0.9] < 1 ns < 1 ns N/A

Comparing ji/python-arrow-row-indices (b11910e) with develop (ceeeff7)2

Open in CodSpeed

Footnotes

  1. 518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (e6668f5) during the generation of this report, so ceeeff7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@robert3005

Copy link
Copy Markdown
Contributor

you're missing types

ANN201 Missing return type annotation for public function `indexed_file`
  --> vortex-python/test/test_file_arrow_indices.py:14:5
   |
13 | @pytest.fixture
14 | def indexed_file(tmp_path: Path):
   |     ^^^^^^^^^^^^
15 |     table = pa.table(
16 |         {
   |
help: Add return type annotation

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs merged commit b96f16b into develop Oct 8, 2026
100 of 105 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/python-arrow-row-indices branch October 8, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants