Repository navigation
PERF: interpolate rows with the same NaNs together (GH#48236) - #71963
Draft
jbrockmendel wants to merge 2 commits into
Draft
jbrockmendel wants to merge 2 commits into
jbrockmendel wants to merge 2 commits into
Conversation
For the methods that go through scipy.interpolate.interp1d, slices of a 2-d block whose NaNs are in the same places share x values, so hand them to interp1d together as a 2-d y, in bounded chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jbrockmendel
force-pushed
the
bug-48236
branch
from
November 8, 2026 22:43
6da4b48 to
6d17ea1
Compare
Also drop a test of an input no DataFrame can produce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
I’m skeptical this merits the added complexity |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #48236
interpolatecallsscipy.interpolate.interp1donce per 1-d slice. For the methods that go throughinterp1d(nearest,zero,slinear,quadratic,cubic,polynomial), slices of a 2-d block whose NaNs are in the same places share their x values, so they can go tointerp1dtogether as a 2-dy. Slices are grouped by NaN pattern and processed in chunks of up to 2**18 values, which keeps memory close to the per-slice loop. Results match the per-slice path exactly.This replaces the whole-block approach of GH-66965 and GH-66964 with a much smaller change. It reuses SciPy rather than reimplementing it, so
limit,limit_areaandfill_valuebehave as before.Timings, per-slice loop vs grouped:
nearest,axis=1)nearest/cubicThese cases are unchanged:
linearand the other non-interp1dmethods, longdouble data (SciPy treats 1-d and 2-dydifferently there), list-likefill_value, and unconsolidated frames, since grouping happens within a block.Behavior differences, none affecting results on success:
inplace=True, a different set of columns may already have been filled.bounds_error=Trueon a frame with nothing to fill no longer raises. On main it raised a duplicate-keywordTypeError.ComplexWarningfromfill_value=1j, may be emitted fewer times.AI disclosure: drafted with Claude Code (
claude opus 5.5 (high)), which wrote the change, tests and benchmarks and ran the review loop; I reviewed the result.