Fork test: PR #1492 - Add allow_nulls option to comparison checks - #1496
Open
mwojtyczka wants to merge 4 commits into
Open
Fork test: PR #1492 - Add allow_nulls option to comparison checks#1496mwojtyczka wants to merge 4 commits into
mwojtyczka wants to merge 4 commits into
Conversation
Comparison checks (is_equal_to, is_not_equal_to, is_not_less_than, is_not_greater_than, is_in_range, is_not_in_range) follow Spark's null-comparison semantics, so null values always pass them. Combining them with a separate is_not_null rule splits the failure metadata across two checks. Add an allow_nulls argument (default True, fully backwards compatible) so null values can fail these checks directly. When allow_nulls=False, the condition also fails on nulls with a dedicated message and an _is_null_or_-prefixed alias, mirroring is_not_null_and_is_in_list. Resolves #1395 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…checks Add test_col_comparison_checks_with_nulls_failing_column_expression, which exercises the allow_nulls=False path with a column expression (a + b). It verifies that nullness is reported against the whole expression when any operand is null, and that non-null out-of-bounds values still fail with the expression rendered in the message. Co-authored-by: Isaac <no-reply@databricks.com>
mwojtyczka
added
the
do-not-merge
mwojtyczka
added
fork-test
do-not-merge
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1496 +/- ##
==========================================
+ Coverage 92.28% 92.44% +0.15%
==========================================
Files 141 141
Lines 13655 13690 +35
Branches 151 151
==========================================
+ Hits 12602 12656 +54
+ Misses 986 967 -19
Partials 67 67
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
✅ 1000/1000 passed, 49 skipped, 6h37m59s total Running from acceptance #5720 |
Contributor
|
✅ 1/1 passed, 25m17s total Running from mcp #469 |
Contributor
|
✅ 195/195 passed, 1 skipped, 6h53m38s total Running from anomaly #1834 |
is_in_range gained an allow_nulls parameter, so build_profiling_rule now enumerates it as a non-column parameter. Unsupplied by the profiler, it freezes to None (its unset default), matching the built-in seeder shape. Update the stale expectation in test_freezes_non_column_parameter_values. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync from fork PR for CI testing.
Original PR: #1492
All tests, including unit and integration tests run on this PR (they are skipped for fork PRs).