Repository navigation
fix(semantic-layer): avoid truncating cached rollups - #46022
Open
mikebridge wants to merge 2 commits into
Open
mikebridge wants to merge 2 commits into
mikebridge wants to merge 2 commits into
Conversation
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
1 of 6 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #46022 +/- ##
==========================================
+ Coverage 85.00% 85.02% +0.02%
==========================================
Files 3080 3085 +5
Lines 209266 209901 +635
Branches 49296 49297 +1
==========================================
+ Hits 177895 178477 +582
- Misses 27989 28043 +54
+ Partials 3382 3381 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
mikebridge
marked this pull request as ready for review
November 5, 2026 02:09
mikebridge
requested review from
geido,
michael-s-molina,
msyavuz,
rusackas,
sadpandajoe and
villebro
as code owners
November 5, 2026 02:10
![codeant-ai-for-open-source[bot]](https://avatars.githubusercontent.com/u/151810221?s=60&v=4){kind=small}
Comment on lines
+270
to
+273
| frame = _rollup(frame, query) | ||
| if ( | ||
| query.limit is not None | ||
| and len(frame) > (query.offset or 0) + query.limit |
Contributor
There was a problem hiding this comment.
Suggestion: Oversized limited rollups aggregate every group before refusal, then the provider runs too, duplicating work and slowing high-cardinality requests.
Assessment: 🟠 Major · 🔁 Occurrence: Sometimes · 🏷️ Performance
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/semantic_layers/cache_transform.py
**Line:** 270:273
**Comment:**
*Performance: Oversized limited rollups aggregate every group before refusal, then the provider runs too, duplicating work and slowing high-cardinality requests.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
Contributor
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: Superset asks the provider for a limited result when a cached rollup could select different rows.
SUMMARY
Background: Superset's result cache can reuse an unlimited result by rolling it up to the requested grain. It re-sorts the rolled-up rows in pandas before applying the query's limit.
Problem: That local sort can keep different rows than the provider would return, for example when an engine sorts NULLs first, a limit splits tied rows, or string ordering differs.
Changes:
offset + limit; otherwise ask the provider. Unlimited rollups and fitting results remain reusable.semantic_cache.containment.rollup_refused; genuine transformation failures keep their existing counter.This supersedes #45912 and drops that PR's NULL-placement conformance checker. No provider update or cache reset is required.
@aminghadersohi's reviews of #45912 informed this narrower change.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable: backend cache reuse and SDK documentation.
TESTING INSTRUCTIONS
Run the semantic-layer unit and integration suites and
superset-core/tests.The regression cases cache an unlimited result, then request a limited rollup where either a NULL-first provider or a split tie would choose a different page. They verify fallback to the provider, the
rollup_refusedandmisscounters, and a subsequent exact cache hit of the provider's page. Fitting limits, NULL values and ties remain covered.Local validation: 2,268 unit/SDK tests passed with four capability skips; 38 semantic integration tests passed with four dedicated Redis/Sentinel skips; semantic package coverage reached 100%; changed-file hooks passed.
ADDITIONAL INFORMATION
SEMANTIC_LAYERSandSEMANTIC_LAYER_CONTAINMENT_CACHEfor containment reuse