Repository navigation
feat(python): support row indices in VortexFile.to_arrow - #10373
Conversation
Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
DCO Remediation Commit for Joe Isaacs <joe.isaacs@live.co.uk> I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 0ee1827 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| Simulation | density_sweep_dense_runs[0.9] |
< 1 ns | < 1 ns | N/A |
Comparing ji/python-arrow-row-indices (b11910e) with develop (ceeeff7)2
Footnotes
-
518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(e6668f5) during the generation of this report, so ceeeff7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
|
you're missing types |
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
VortexFile.to_arrow cannot select row indices even though scan and prepare support them. Add the indices argument to the Python API and binding, using the same scan builder and strictly increasing, non-null index validation.
Changes
New cases cover empty and sparse selections, nullable strings, requested ordinary-string schemas, scan-equivalent batching, filtering, limits, and invalid indices. The existing restriction on combined filters and limits is preserved. Existing tests remain unchanged.
API Changes
Adds the optional keyword argument
indices: vortex.Array | NonetoVortexFile.to_arrow(). Indices must be strictly increasing and non-null, matchingscan()andprepare().Validation
python -m pytest vortex-python/test/test_file_arrow_indices.py vortex-python/test/test_file.py -q— 21 passed.