Skip to content

PERF: interpolate rows with the same NaNs together (GH#48236) - #71963

Draft
jbrockmendel wants to merge 2 commits into
pandas-dev:mainfrom
jbrockmendel:bug-48236
Draft

jbrockmendel wants to merge 2 commits into
pandas-dev:mainfrom
jbrockmendel:bug-48236

Conversation

@jbrockmendel

Copy link
Copy Markdown
Member

closes #48236

interpolate calls scipy.interpolate.interp1d once per 1-d slice. For the methods that go through interp1d (nearest, zero, slinear, quadratic, cubic, polynomial), slices of a 2-d block whose NaNs are in the same places share their x values, so they can go to interp1d together as a 2-d y. Slices are grouped by NaN pattern and processed in chunks of up to 2**18 values, which keeps memory close to the per-slice loop. Results match the per-slice path exactly.

This replaces the whole-block approach of GH-66965 and GH-66964 with a much smaller change. It reuses SciPy rather than reimplementing it, so limit, limit_area and fill_value behave as before.

Timings, per-slice loop vs grouped:

case speedup
issue repro (nearest, axis=1) 4.0 s -> 66 ms
200x2000, same NaNs, nearest / cubic 41x / 38x
1000x3000, same NaNs 13x
resample upsample, 8k x 500 7x
100000x20, same NaNs 1.5x
NaNs scattered independently 1.0x

These cases are unchanged: linear and the other non-interp1d methods, longdouble data (SciPy treats 1-d and 2-d y differently there), list-like fill_value, and unconsolidated frames, since grouping happens within a block.

Behavior differences, none affecting results on success:

  • If interpolation raises with inplace=True, a different set of columns may already have been filled.
  • bounds_error=True on a frame with nothing to fill no longer raises. On main it raised a duplicate-keyword TypeError.
  • A repeated warning, such as ComplexWarning from fill_value=1j, may be emitted fewer times.

AI disclosure: drafted with Claude Code (claude opus 5.5 (high)), which wrote the change, tests and benchmarks and ran the review loop; I reviewed the result.

@jbrockmendel jbrockmendel added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Performance Memory or execution speed performance labels Nov 8, 2026
For the methods that go through scipy.interpolate.interp1d, slices of a
2-d block whose NaNs are in the same places share x values, so hand them
to interp1d together as a 2-d y, in bounded chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also drop a test of an input no DataFrame can produce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jbrockmendel

Copy link
Copy Markdown
Member Author

I’m skeptical this merits the added complexity

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Performance Memory or execution speed performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENH: faster fill-between interpolation

1 participant