Add allow_nulls option to comparison checks - #1492
Merged
mwojtyczka merged 4 commits intoSep 2, 2026
Merged
Conversation
alvaromoureupm
requested review from
mwojtyczka
and removed request for
a team
August 31, 2026 21:11
|
|
Contributor
|
All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits |
1 task
alvaromoureupm
force-pushed
the
feat/allow-nulls-comparison-checks
branch
2 times, most recently
from
August 31, 2026 21:19
156d1f7 to
beeb657
Compare
Comparison checks (is_equal_to, is_not_equal_to, is_not_less_than, is_not_greater_than, is_in_range, is_not_in_range) follow Spark's null-comparison semantics, so null values always pass them. Combining them with a separate is_not_null rule splits the failure metadata across two checks. Add an allow_nulls argument (default True, fully backwards compatible) so null values can fail these checks directly. When allow_nulls=False, the condition also fails on nulls with a dedicated message and an _is_null_or_-prefixed alias, mirroring is_not_null_and_is_in_list. Resolves databrickslabs#1395 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
alvaromoureupm
force-pushed
the
feat/allow-nulls-comparison-checks
branch
from
August 31, 2026 21:20
beeb657 to
17d359d
Compare
mwojtyczka
reviewed
Sep 1, 2026
mwojtyczka
approved these changes
Sep 1, 2026
mwojtyczka
left a comment
mwojtyczka
left a comment
Contributor
There was a problem hiding this comment.
LGTM - yes we can apply the same logic to other checks
mwojtyczka
reviewed
Sep 1, 2026
mwojtyczka
added
the
Approved to Merge
…checks Add test_col_comparison_checks_with_nulls_failing_column_expression, which exercises the allow_nulls=False path with a column expression (a + b). It verifies that nullness is reported against the whole expression when any operand is null, and that non-null out-of-bounds values still fail with the expression rendered in the message. Co-authored-by: Isaac <no-reply@databricks.com>
is_in_range gained an allow_nulls parameter, so build_profiling_rule now enumerates it as a non-column parameter. Unsupplied by the profiler, it freezes to None (its unset default), matching the built-in seeder shape. Update the stale expectation in test_freezes_non_column_parameter_values. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds an
allow_nullsargument (defaultTrue) to the six comparison row-level checks —is_equal_to,is_not_equal_to,is_not_less_than,is_not_greater_than,is_in_range,is_not_in_range— implementing the approach suggested by @ghanse in the linked issue.Because of Spark's null-comparison semantics, null values always pass these checks today; making nulls fail requires pairing each rule with a separate
is_not_nullrule, which splits the failure metadata across two checks. Withallow_nulls=False:Column 'x' value is null), while non-null failures keep their existing messagesis_null_or_prefix (e.g.x_is_null_or_not_equal_to_value), mirroring the existingis_not_null_and_is_in_listnaming conventionThe default behavior is fully backwards compatible: with
allow_nulls=True(default) conditions, messages, and aliases are unchanged. The logic is shared via a single_make_condition_handling_nullshelper.If this direction looks good, I'm happy to follow up extending
allow_nullsto other null-passing checks (e.g. list membership checks) in a separate PR.Linked issues
Resolves #1395
Tests
Ran locally:
make fmt,make lint(mypy + pylint 10.00/10), fulltests/unit/test_row_checks.py+tests/unit/test_check_func_signatures.py(178 passed), and the new integration testtest_col_comparison_checks_with_nulls_failingagainst a Databricks serverless workspace (passed).Documentation and Demos