Row anomaly detection: a second detector, and the end of segment_by - #1489
Open
vb-dbrks wants to merge 58 commits into
Open
Row anomaly detection: a second detector, and the end of segment_by#1489vb-dbrks wants to merge 58 commits into
segment_by#1489vb-dbrks wants to merge 58 commits into
Conversation
`_save_training_record` hand-rolled the `features.feature_metadata` JSON with a literal five-key dict, so `SparkFeatureMetadata.to_json()` was dead on the training path. Any field added to the dataclass would have been dropped silently at persistence, and the model would then score with a different feature set than it trained on — a silent train/score skew rather than a loud failure. Three changes, no behaviour change: - `_save_training_record` now calls `artifacts.feature_metadata.to_json()`, and `to_json` iterates the dataclass fields instead of naming them, so it stays complete by construction. Field order is declaration order, so the emitted JSON is byte-identical for existing models. - `from_json` ignores unknown keys instead of raising. It did `cls(**data)`, so a record written by a newer DQX version made an older reader fail outright; it now falls back to defaults for fields it does not recognise. - New `apply_feature_engineering_from_metadata` collapses the five call sites that each hand-passed `frequency_maps` + `onehot_categories` + `categorical_cardinality_threshold` back out of a `SparkFeatureMetadata` (`core.py` x3, `training_service.py`, `feature_prep.py`). Future metadata fields thread through one function rather than five. Groundwork for #1484.
Four documentation defects found while reading the anomaly module for #1484. Each was verified against the code; no behaviour changes here. - `drift_threshold`'s docstring claimed "default 3.0, None to disable". The signature default is `None`, and `None` disables drift detection outright (`drift.py:227` gates on `is not None`), so the stated default was the exact opposite of the real one. The reference docs already had this right. - Drift was described as watching the "score distribution". It compares the *input feature* distributions against the per-column baseline statistics recorded at training and reports which columns drifted (`compute_drift_score`); it never looks at anomaly scores. Corrected in the check table, the parameter reference, and the guide. Also documented the 1,000-row floor below which drift is skipped as too noisy to judge. - `ensemble_size` was documented as if it always applied. Segmented training passes `allow_ensemble=False`, so it trains exactly one model per segment regardless, which also means `confidence_std` is unavailable for segmented models. Said so on the parameter, in the reference table, and in the guide. - The guide advised "100+ rows per segment" without mentioning that segments below **10** rows are skipped entirely, leaving their rows unscored with no error. Documented the hard threshold alongside the guidance. Groundwork for #1484.
The numbers governing per-segment training were literals spread across the training service. Collecting them in `anomaly/group_config.py` gives each one a name and a reason, and makes the ceiling below reviewable as policy rather than as a magic number. MAX_SEGMENT_MODELS is the cost fuse: a 90-segment run was measured at 70 minutes without completing. MIN_ROWS_TO_TRAIN_SEGMENT and MIN_ROWS_PER_SEGMENT separate "too small to fit at all" from "small enough to warn about", which were previously the same literal used for two different questions.
Five checks, each guarding a failure that is silent rather than loud if it gets through:
the columns must exist; they must not also be feature columns; their types must render
identically in Python and Spark; the list must not be empty when declared; and it must not
duplicate an entry.
The type restriction is the one with teeth. Floating-point and decimal columns are rejected
because Python's `str()` and Spark's `cast("string")` disagree on them, which would produce
one key at training and a different key at scoring — and that mismatch does not raise, it
just misses every baseline lookup and silently falls back to the global baseline.
…ment
Training persists a baseline under a key built in Python; scoring looks it up with a key
built in Spark. A disagreement between the two does not raise — every lookup simply
misses, falls back to the global baseline, and produces a model that appears to work
while conditioning on nothing. That makes this the most expensive invariant in the
feature to get wrong, so both halves and the test that pins them land together.
`_as_spark_string` exists because `str()` is not a drop-in for Spark's `cast("string")`.
Booleans are the case that bites: Spark renders `true`/`false`, Python renders
`True`/`False`. The first real Spark run of this code asserted
`'true\x1f42' == 'True\x1f42'` and failed, which is precisely the class of bug this
contract exists to catch — and a stale unit test had been pinning the wrong behaviour.
Floating-point and decimal baseline columns are rejected rather than shimmed: their two
renderings diverge in ways no small helper can reconcile.
The separator is ASCII unit separator, a control character, so ordinary categorical data
cannot contain it. It is a separator and not an escaping scheme, and the docstring says
so: data that genuinely contains \x1f can still collide. Deliberately not solved by
length-prefixing, which would buy protection against pathological data by widening the
one invariant most expensive to get wrong. A test pins the documented limitation so
nobody mistakes it for collision-proofing.
Closes #1484. A value can be perfectly ordinary for the table and badly wrong for its own group. Row anomaly detection had no scalable way to express that: the only grouping mechanism was `segment_by`, which trains one model per group, so conditioning cost one model per group and a discovered 90-way grouping became an hours-long run. `baseline_by` adds, for each numeric metric, its deviation from that group's baseline: rel = signed_log(value) - signed_log(group_median(value)) on a **single** model, whatever the group count. Baselines are computed once at training, persisted in the feature metadata, and broadcast-joined at scoring. The log-ratio form is stable when a baseline is near zero and symmetric for halving versus doubling. The raw metric is kept alongside, so a globally absurd value stays detectable even where it is ordinary for its group. Severity is calibrated per baseline group as well, since a score that is extreme for one group is unremarkable for another, with the global calibration as the fallback for groups that have none. A row whose group was never seen at training is reported rather than scored. The alternatives are both silent and both wrong: a high-cardinality frequency map returns 0.0 for an unseen key, which reads as "perfectly normal", and a one-hot encoding leaves every indicator at zero, which reads as extreme. `is_new_baseline` and `new_baseline_key` name the case instead. This widens the `_dq_info` struct, so appending to an existing results table needs `mergeSchema`. Three properties worth calling out, because they are what makes this safe to enable: * **Baseline columns are never features.** They are the basis of comparison, not a metric being compared, so they are excluded before the sklearn pipeline and the inferred MLflow signature see anything. `prepare_training_features` projects to the feature list explicitly; the key column rides through as a passthrough for the calibration stage, which runs downstream of where the raw group columns are dropped. * **`segment_by` keeps its exact behaviour.** `_resolve_grouping` clears `baseline_by` on that path. Leaving both set would compute relative features *inside* each segment, where the key is constant because the frame is already filtered, while `compute_config_hash` is built from `segment_by` alone and would not change — an identical hash with a different feature list, which is a silent train/score hazard. * **Models trained before this change score identically.** Every new metadata field defaults to empty, so the relative transform returns immediately and `engineered_feature_names` is byte-identical to what it was. There is deliberately no strategy selector and no heterogeneity gate. `baseline_by` declared means relative features, always. The evidence for that choice, including the measurement that removing the gate costs nothing, is in `benchmarks/anomaly_conditioning/`. Also folded in here, because `config.py` and `training_service.py` carry it in the same hunks: **the segment model ceiling now errors rather than warns.** Above the ceiling the previous behaviour was to log a warning and continue, which meant a 90-segment run started training 90 models and was measured at 70 minutes without completing. A warning that precedes an hour of unusable work is not a warning. It now raises, and the message names the two ways forward — segment more coarsely, or raise `max_segment_models` explicitly — because an error that only says no is a worse experience than the warning it replaces. `max_group_models` is renamed `max_segment_models`. Its only consumer is the legacy `segment_by` path, the only path that trains N models, so it is a cost fuse on that path rather than a general tuning knob. The name now says which mechanism it bounds. `AnomalyParams` cannot import from the anomaly package — `anomaly/__init__.py` raises without the anomaly extra — so the default is a literal, kept in step with `MAX_SEGMENT_MODELS` by a unit test rather than by hope.
Predates this feature — `git log -- tests/perf/test_anomaly_benchmark.py` last touches it in #1129/#990 — and is separable from it. It lands here rather than as its own PR because the anomaly benchmark is what this branch changes the behaviour of. `generate_md_report.py` has always contained a fully written "Anomaly Benchmarks" section with ROC-AUC, precision, recall, F1 and precision@N columns. It has never rendered. The cause was one missing line: neither benchmark carried `@pytest.mark.benchmark(group="anomaly_synthetic")`, so pytest-benchmark recorded `group: null` and the report, which filters on exact equality with `anomaly_synthetic`, found nothing. `baseline.json` held 213 benchmarks and zero anomaly entries. Also fixed, all found while making the section actually appear: * The two benchmarks would have listed **twice** once the marker worked — once in the main results table, which is built from every benchmark, and once in their own section. * Five rounds of a Databricks-backed train-and-register cycle ran per nightly, twice over since the job invokes pytest again for the comparison step. Converted to `benchmark.pedantic(rounds=1)`, which takes its round count explicitly; verified against `--benchmark-min-rounds=5` and confirmed to record one round. * A module-global `_TRAINED_MODEL` with `needs_training` fallbacks made the score benchmark order-dependent and able to train up to three times. Each test now trains its own model as unmeasured setup. * The score benchmark read `_dq_info`, which does not exist on the frame `has_no_row_anomalies` returns — that column is assembled a layer up by `DQEngine`. So the test failed and recorded none of the quality numbers this section exists to publish. It now reads the struct column the check actually returns. Quality is published as *indicative and first-observed only*, and labelled as such. The nightly's baseline merge keeps the existing entry on conflict, so once a benchmark exists its `extra_info` is frozen at first observation and never refreshed — publishing quality that way publishes a fossil. Quality regressions are gated by assertions in `tests/integration_anomaly/` instead. The anomaly module is marked `pytest.mark.anomaly` and excluded from the timing gate. `--benchmark-compare-fail=mean:25%` is global and cannot be scoped per test, and these benchmarks are dominated by MLflow and Unity Catalog control-plane latency, so they would trip it on variance alone. They still produce a baseline; only the comparison skips them.
Reimplements the transform in numpy and fits sklearn's IsolationForest on raw columns and on raw-plus-relative columns, so the claim behind #1484 is checked in the unit suite in under five seconds with no Spark session, no workspace and no MLflow. Measured: PR-AUC 0.0028 to 0.6962 on a contextual collapse, against a 0.0026 base rate -- so the pooled model is at chance, not merely worse. The reverse case is asserted not to regress: an anomaly already extreme against every group stays at 1.0000 either way. This guards the *mechanism*, not the pipeline, and the docstring says so. The pipeline is covered by tests/integration_anomaly/, which needs a real session.
No Spark path in this feature had ever been executed before these tests existed, which was by far its largest outstanding risk: the baseline-key agreement, the relative transform, the broadcast joins, per-baseline severity and unseen-baseline marking were verified only by inspection and by unit tests that never start a session. Running them found real defects that inspection had not: * Training any grouped model raised `UNRESOLVED_COLUMN`. Feature engineering drops the raw group columns so they cannot reach the sklearn pipeline or the inferred MLflow signature, but training-time severity calibration runs downstream of that, on the scored engineered frame, and rebuilt the key from columns that were no longer there. Fixed by computing the key once and reading it thereafter, via `with_baseline_key`. * The Python and Spark halves of the baseline key disagreed on booleans. * A discovered grouping trained one conditioned model where the test still expected one model per segment — the intended behaviour change, asserted the old way. `test_anomaly_quality.py` measures detection quality rather than shape: it trains a conditioned and an unconditioned model on identical data and asserts conditioning wins by a margin, beats a random and a max-abs-z baseline, and does not concentrate false positives in one group. Metric discipline lives in `quality_metrics.py`, which computes no point-adjusted F1 and says why. Two measurement flaws in those tests, both fixed here and both worth recording because they would have made the assertions meaningless rather than wrong: * Average precision over a single planted positive is a coin flip, so the fixture now plants an incident across several groups and days. * A per-group false-positive rate over four rows can only be 0, 0.25, 0.5, 0.75 or 1, so the maximum across ninety groups reached 1.0 by chance and measured the fixture rather than the model. The fixture now emits enough control days for the rate to exist, the metric ignores groups too small to express one, and the test asserts the statistic was actually measured instead of silently NaN. The severity assertion also matched on country alone, where one country spans fifteen groups and only one collapsed, so `first()` returned an arbitrary sibling — passing at 95.5 on one run and failing at 82.3 on the next with identical data. It now matches the whole key, built with the production function so the assertion depends on Python/Spark agreement too.
`benchmarks/anomaly_conditioning/` compares three ways of relating a model to a group — pooled, baseline-relative, and one model per group — across 1,395 configurations: a synthetic two-factor sweep plus the Server Machine Dataset and NSL-KDD, 15 seeds each, paired by seed and tested with Wilcoxon signed-rank. Run manually rather than nightly: it downloads third-party data and yields a correlation rather than a pass/fail. Datasets are fetched at run time and cached, never vendored, which is what keeps the licensing position simple. SMAP and MSL are excluded — "(c) Original Authors", no permissive licence. ADBench is unusable here despite being the obvious choice: it ships pre-processed numeric matrices, so the categorical identity this experiment measures is already gone. The results, in `docs/dqx/docs/reference/anomaly_detection_quality.mdx`: * Contextual anomalies: baseline-relative beats pooled by a median **+0.0734** PR-AUC. * Globally extreme anomalies: **+0.0000**, and the test does not reject (p = 0.20). Not "costs little" — costs nothing. At low heterogeneity every group median approaches the global median, so the relative feature degenerates into a monotone transform of the raw metric: a near-duplicate of an informative column rather than noise. * Real datasets: **-0.0010** at p = 0.003. Negligible in size, real in sign, and reported rather than buried — SMD and NSL-KDD anomalies are largely globally extreme or sequence-dependent, so the extra feature dilutes slightly without adding signal. * Baseline-relative beats one-model-per-group in **every** mechanism, on real data included, while training one model instead of twelve. That asymmetry — about +0.07 where conditioning applies, about -0.001 where it does not — is the actual argument for auto-discovery enabling it rather than requiring opt-in. **On the heterogeneity gate.** The decision rules were fixed before the first run: no gate if the worst delta below eta-squared 0.10 stayed above -0.01, gate needed if any such cell reached -0.02. The pre-registered rule fired. It is a minimum over single cells, which makes it maximally sensitive to estimator variance, and it fired on one seed of `nslkdd/protocol_type` whose other seeds were +0.1003 and +0.0989. Rather than rewrite the criterion to get the preferred answer, the rule is left in place and reported, with the per-grouping median published beside it and the disagreement explained in the code. Raising the seed count fivefold moved no grouping's median below zero. No grouping is systematically harmed at low heterogeneity, so the gate had nothing to gate on. Eta-squared is not useless: it correlates with the size of the benefit (Spearman rho +0.597, bootstrap CI +0.523 to +0.668, still +0.276 after controlling for anomaly mechanism). But it predicts how much you gain, never whether you lose, and a gate needs to identify harm. It cost a full Spark aggregation per training run to answer a question that never changed the decision. `scipy` joins the existing mypy `ignore_missing_imports` list, alongside pandas, sklearn, shap and mlflow — the harness needs Wilcoxon and Spearman, and hand-rolling statistical tests invites subtler errors than the missing stubs do.
The guide gains a "Baseline conditioning" section built around the one mechanism there is: what `baseline_by` compares against, that it costs one model however many groups you have, that baseline columns are the basis of comparison rather than metrics to be compared, and what happens to a group that was never trained on. The measured numbers are quoted with a link to `anomaly_detection_quality.mdx` for the full sweep and, more importantly, for what they do not mean — DQX scores rows independently, so its figures on time-series benchmarks are a different task, not a worse implementation. `segment_by` is documented as legacy with the reason rather than a bare recommendation. On the Server Machine Dataset per-entity models were the worst of three configurations, and one entity produced 15,963 false positives across 28,392 normal rows. Each per-group model calibrates its own contamination on its own rows, so a group containing nothing unusual still has its most-unusual few percent scored as extreme. Four breaking changes, of which the second is the one existing users will notice: * Above the segment ceiling, training now raises instead of warning. * **An auto-discovered grouping now trains one conditioned model instead of N per-group models**, so those runs produce different scores. * A row whose group was never seen at training returns a null score and is reported via `is_new_baseline` rather than being scored 0.0. * The `_dq_info` struct is wider, so appending to an existing results table needs `mergeSchema`.
1,395 cells: the synthetic two-factor sweep plus the Server Machine Dataset and NSL-KDD, 15 seeds each, stamped with the commit they were produced from so the numbers are reproducible rather than merely asserted. Committed as its own change because the results are an observation, not code. Re-running the harness at the same seeds reproduces this file, which is how the determinism claim in the README is checked; the figures quoted in docs/dqx/docs/reference/anomaly_detection_quality.mdx are read directly from here.
Ten ADBench datasets (BSD-2-Clause, redistributing the ODDS / UCI / Kaggle collections), chosen to span the axes that actually change a detector's behaviour: 1.8k to 285k rows, 9 to 100 features, and base rates from 0.17% (the Kaggle credit-card set) to 40%. These answer a different question from the rest of the harness. SMD and NSL-KDD exist to test whether conditioning on a group helps; these have no grouping at all, so they characterise the ungrouped path -- which this branch does not change -- and serve as its regression baseline. Reported per dataset against its own random floor and its own max-abs-z baseline, never pooled into a headline: PR-AUC moves with the base rate, so 0.19 at a 0.17% base rate (a 99x lift) and 0.48 at 40% are not comparable numbers. An earlier draft of this harness rejected ADBench outright because its preprocessing discards categorical column identity. That is disqualifying for the grouping question but not for plain tabular data, so the exclusion was too broad.
…ted one Auto-discovery routed a discovered grouping to `baseline_by` but kept the selection policy that only ever made sense for `segment_by`: take a single column, the lowest-cardinality candidate, requiring 100 rows per group. Both rules exist because a per-segment model has to train a forest on each group. Under baseline conditioning there is one model however many groups there are, and a group only has to yield a median. Measured on a dataset grouped by country x event_type x product -- 90 groups available: discovered baseline_by [product] -> [product, event_type, country] groups 3 -> 90 training.columns country, event_count, event_type -> event_count engineered features 13 (11 one-hot dummies) -> 2 PR-AUC, contextual 0.1302 -> 0.5703 PR-AUC, global 0.2702 -> 1.0000 The feature-list change falls out of the grouping fix rather than being a separate decision: the profiler already drops the chosen grouping from the feature columns, so recognising country and event_type as dimensions stops them being one-hot encoded as metrics. Eleven dummies alongside a single real metric were diluting the isolation signal, which is why the global scenario was scoring 0.2702 where explicit columns scored 0.7200. `select_baseline_columns` is public so its policy can be unit-tested directly rather than through a Spark session; `MIN_ROWS_PER_BASELINE_GROUP`, `MAX_BASELINE_GROUPS` and `MAX_BASELINE_COLUMN_CARDINALITY` sit beside the segmented thresholds with the reason they differ. The legacy segmented policy is untouched. Also closes a latent hole on the explicit-columns path: a discovered baseline column that the caller had named as a feature is now excluded, rather than producing the feature-and-baseline overlap that `validate_baseline_columns` rejects.
The page carried only `AvailableSinceVersion`, and the lifecycle reference is explicit that "a feature with no status badge is generally available" — so for three releases the docs promised semantic versioning and backward-compatibility guarantees for a feature that was Experimental and is now Beta. DQX Studio is tagged experimental, actions and the MCP server beta; this page was simply missed. The distinction is not cosmetic. It is what makes the breaking changes in the following commits legitimate: Beta states that "the API may still change in backward-incompatible ways between releases", where GA would not.
The hash was f(columns, segment_by). Retraining under the same model_name with a different baseline_by therefore produced an identical hash, so the single thing this hash exists to catch -- same name, different configuration -- was blind to the grouping, even though the grouping changes the feature list and the persisted baselines. baseline_by now joins the inputs, which changes the hash of every configuration including ungrouped ones, since the key is present either way. A model registered before this fails the configuration check in score_global_model and must be retrained. That break is the point rather than a side effect. Silently scoring a model whose persisted hash no longer describes its configuration is worse than refusing to, and the error now names retraining as the remedy and says which version moved the goalposts. Row anomaly detection was Experimental through 0.16.0 -- "API, behavior, and on-disk or table formats may change ... without notice or a migration path" -- so no compatibility was owed here. At scoring, baseline_by is read back from the persisted feature metadata rather than taken from the caller: it is a property of the trained model, not a scoring argument. That makes the recomputed hash match for anything this version trained, and mismatch for anything older, which is exactly the intended boundary.
RobustScaler sat in front of the IsolationForest and could not have changed a single prediction. It is an affine per-feature transform, and Isolation Forest splits on per-feature thresholds drawn uniformly between each feature's min and max, so the induced partitions are identical with or without it. Measured rather than argued, across five ADBench datasets at five seeds each -- PR-AUC with the scaler and without it agreed to four decimal places every time: covertype 0.0572 / 0.0572 shuttle 0.9789 / 0.9789 mnist 0.2740 / 0.2740 fraud 0.1926 / 0.1926 cardio 0.5766 / 0.5766 What it did cost: a fit and a transform on every training run, a transform on every scoring pass, and bulk inside every pickled artifact. On every path, grouped or not. The single-step Pipeline stays. named_steps["model"] is how the SHAP explainer reaches the tree model, keeping it avoids churning eight -> Pipeline annotations across core.py and ensemble_training.py, and it leaves a slot for a transform that genuinely does something. One real hazard fixed alongside: the second SHAP call site indexed named_steps["scaler"] directly, so it would have raised KeyError on any model trained by this version. It now looks the step up, which also keeps working for older models that do carry a scaler.
baseline_by lived only inside the features.feature_metadata JSON blob. That is where scoring reads it, and functionally it was enough -- but it made the first question worth asking when detection looks wrong unanswerable in SQL. That cost real time this cycle. Diagnosing why zero-config conditioning was not engaging needed a script to pull a model record, parse the JSON, and print baseline_by and the size of baseline_medians, just to tell "conditioning never engaged" apart from "conditioning engaged on a 3-group basis where 90 were available". Both look identical from the outside: one model, is_global_model true. It now sits in the segmentation struct beside training.columns, so: SELECT identity.model_name, training.columns, segmentation.baseline_by FROM <registry> WHERE identity.status = 'active' answers it. The metadata blob stays the source of truth for scoring; this is a projection for humans, and the duplication is noted where the field is declared. The registry write gains mergeSchema. Without it a table created by an earlier DQX would reject the wider struct, which matters more than usual right now: the configuration-hash change tells users to retrain, retraining writes to their existing registry table, and a remedy that fails is not a remedy.
The harness fitted IsolationForest(n_estimators=100, contamination="auto") while DQX ships IsolationForestConfig(num_trees=200) with contamination taken from expected_anomaly_rate (0.02 by default). So every absolute figure it published described a lighter model than the product. contamination is the harmless half: it only shifts the predict/offset_ threshold and cannot reorder score_samples, so PR-AUC was unaffected by that difference. Tree count is not neutral -- a held-out measurement at 200 trees put fraud at 0.2607 where the 100-tree fit-and-score run had reported 0.1926. Paired comparisons were never affected, since the configuration was held constant on both sides of each comparison. It is the absolute per-dataset column that was understated. Found while mapping the end-to-end heuristic flow, which is the sort of thing that exercise is for.
Isolation Forest loses to a four-line z-score on covertype and mnist under every hyperparameter tried, so the gap is inductive bias rather than tuning. The obvious remedy is to put a cheap complementary scorer in the ensemble. This commits the measurement that says not to, so the idea is not re-proposed from scratch and re-derived by whoever notices the same gap next. Held-out splits (70/30), 5 seeds, the shipped forest configuration, each member calibrated to percentiles against its own training distribution: mean-of-percentiles wins 5/10 median -0.0028 worst -0.1409 (thyroid) max-of-percentiles wins 3/10 median -0.0253 worst -0.0690 It helps exactly where predicted (covertype +0.0295, mnist +0.0724) and hurts where the forest is genuinely stronger (thyroid, fraud, shuttle, spambase, satellite). A coin flip on average with a 0.14 worst case is not a default. Two things the script records that are easy to miss when re-litigating this: * Naive blending is not even implementable as-is. DQX averages ensemble member scores raw and the scales are incomparable -- the forest sits around 0.4-0.7, max-abs-z is unbounded -- so a real version has to persist each member's training-score distribution. Ranking inside the scoring UDF would be wrong, since the UDF sees a pandas batch and ranks would depend on partitioning. * Choosing the better detector per dataset requires labels, and DQX is unsupervised. So the honest options are to expose the choice or document the limitation; guessing is not one.
Two dev pages, both written because this cycle found its defects by accident and that is not a repeatable way to work. **anomaly_heuristic_map** — every decision made between train(df) and a flagged row, as a mermaid flowchart, with the governing default at each stage and the three questions that isolate most failures: what was chosen (stages 1-2, which decide silently and held both defects found while drawing this), did conditioning actually engage and on what, and is the comparison even valid given that a 30% sample drawn with .sample is partition-dependent under Spark Connect. It also records the measured weak points rather than only the design. Drawing it immediately surfaced that the harness modelled a 100-tree forest where DQX ships 200, which is the kind of thing the exercise is for. **anomaly_compatibility** — every affordance that exists only to keep old behaviour alive, marked so `git grep "COMPAT(anomaly-v1)"` finds them, with the registry-schema item called out because it touches persisted data rather than code. It deliberately separates that debt from permanent runtime fallbacks -- unseen-group and missing-quantile handling look identical in a diff and must not be deleted alongside.
The sweep in this page measured mechanisms against each other; it never measured this build against the one users are running. That is the question that actually matters, and the answer now leads the page: v0.16.0 and this build, through the real Spark/MLflow/Unity Catalog path, over the same Delta tables. contextual, zero config 0.1885 -> 0.5703 contextual, explicit columns 0.0376 -> 0.5703 global, zero config 0.3743 -> 1.0000 global, explicit columns 1.0000 -> 1.0000 Better on three, identical on the fourth, worse on none. The 0.0376 -> 0.5703 row is the cleanest statement of what conditioning buys, since v0.16.0 has no conditioning mechanism at all; the two exact 1.0000s are the regression check, on the path this work does not touch. The page also records how the measurement went wrong the first time, because it will catch anyone who repeats it: training samples 30% of rows by default via .sample, which under Spark Connect depends on partition ordering, so an uncontrolled run had the two builds training on different rows and showed the new build losing three of four cells. With sample_fraction=1.0 every cell reproduces to four decimals. A single-point comparison of this pipeline is not evidence.
Regenerating the sweep with the corrected 200-tree harness reversed a published claim, so it is
corrected here rather than left standing.
baseline-relative minus pooled, real datasets
was (100 trees): -0.0010 p = 3.05e-03 "marginally worse"
now (200 trees): +0.0025 p = 1.75e-02 marginally better
The page keeps the old number and says why it moved: the harness fitted a lighter forest than DQX
ships, which understated it on exactly the datasets where the forest does the work. The contextual
(+0.0742) and global (+0.0000) rows barely moved, because a paired comparison holds configuration
constant on both sides — it was the real-data row, where the two mechanisms are closest, that a
fidelity gap could flip. The guide and changelog carried the same figure and are updated too.
Also adds the plain-tabular section the sweep has been producing but the page never showed: ten
ADBench benchmarks spanning 1.8k to 30k rows, 6 to 100 features, and base rates from 0.17% to 40%.
DQX beats the random floor on all ten and max-abs-z on eight, with fraud at 115x the floor.
The two it loses get their own subsection rather than a footnote. covertype (0.0534 vs 0.1122) and
mnist (0.2766 vs 0.3367) are Isolation Forest's inductive bias: it splits one random feature at a
time, so a single extreme value among few dimensions is diluted and in 100 dimensions the split
rarely lands on the informative axis. Tuning does not recover it and the obvious knob trades shuttle
and cardio away for fraud. Publishing the losses with the diagnosis is more use to a reader than a
table of wins, and anyone with ADBench reproduces covertype in three lines regardless.
Supersedes the two partial results files from earlier runs.
The page had accumulated material that existed to convince us the work was an improvement, not to help anyone use the feature: a build-versus-build comparison against v0.16.0, the story of getting that measurement wrong the first time, a note about which harness revision reported which number, and the design rationale for a heterogeneity gate users never saw. That belongs in the changelog, the dev docs and the PR, and it is already in those places. 265 lines down to 151. What a reader actually needs is now the shape of the page: whether anomaly detection suits their data, whether to give it a grouping, and what it is bad at. The conditioning tables are reframed from "mechanism A versus mechanism B" to "your anomalies are like this, so do that", and the underperformance section ends with what to do instead -- a range check or an outlier rule serves single-column extremes better, and both can run together. Kept, because they are the reader's protection rather than our defence: rows are scored independently so time-series comparisons are a category error, PR-AUC moves with the base rate so the columns are not comparable, no point-adjusted F1, and every figure measures separability rather than generalisation.
The detection-quality numbers sat in a page nobody regenerated, so they would go stale the first
time anyone changed the model. They now refresh with the timing benchmarks and ride the same PR.
Not routed through pytest-benchmark, for a specific reason. The nightly merges baseline.json with
if ($old[.] != null) then $old[.] else $new[.] end
so an entry that already exists keeps its previous value entirely -- which is why no existing
benchmark mean has changed across the last four baseline PRs. Quality published through
extra_info would be frozen at first observation and republished as fresh for ever. emit_docs.py
writes the numbers into the page directly instead.
Generated tables sit between HTML comment markers, so regeneration replaces the numbers and
leaves the prose. That is what makes this page different from benchmarks.mdx, which
generate_md_report.py rewrites wholesale and where hand-written guidance therefore cannot
survive -- the reason the two are separate pages rather than one.
Scoped for CI at 5 seeds rather than 15, and marked continue-on-error: the real datasets are
~250MB fetched from GitHub on a cold cache, and a third-party download failing should not fail
the timing benchmarks that already ran. The trade to be aware of is that the nightly now depends
on an external host it did not before.
emit_docs warns rather than stays quiet when handed a results file with no tabular data, since a
stale table that looks current is worse than an obvious gap.
SHAP contributions are keyed by *engineered* feature name, and redaction filters those keys by exact match against the caller's redact_columns. Baseline conditioning adds a derived feature per metric, so redacting `amount` did not stop `amount_rel_baseline` -- a signed log-ratio of the same column -- from being embedded in a prompt and sent to an external serving endpoint. Naming a column sensitive has to mean everything computed from it is sensitive, so the redaction set now expands to cover the derived feature. Expansion is by exact suffix rather than prefix match, because prefix matching would also swallow `amount_paid` -- a different column the caller never named -- and quietly narrow what an explanation can discuss. The suffix is now a named constant in transformers.py, where the feature is created, rather than an inline f-string, so the two cannot drift apart. Known gap, documented at the function and not closed here: one-hot and frequency-encoded features are still not covered, because their names cannot be reconstructed from the source column alone -- `country` becomes `country_C3`, `country_DE`, one per observed value. Closing that needs the feature metadata at prompt-construction time, which the explainer does not currently receive. That hole predates this branch; this commit closes the one this branch opened. redaction_set is public so its policy can be unit-tested directly rather than through a private.
…line_by grouping Row anomaly detection was Experimental through 0.16.0 and owed no compatibility. This removes the per-group model path (segment_by / max_segment_models) entirely, leaving a single pooled baseline_by model that measured better on detection, reliability and cost across all four benchmark regimes. Source removals: - config.AnomalyParams.segment_by / max_segment_models - training_service: segmented dispatch, segment resolution and validation - scoring_run: score_segmented, load_segment_models, score_single_segment, and the now-orphaned _split_max_groups_budget / _warn_if_max_groups_below_segments helpers - scoring_orchestrator segmented fallback; scoring_strategies.score_segmented - model_discovery segment-record selection; drift.check_segment_drift - SegmentationConfig -> GroupingConfig; persisted `segmentation` struct -> `grouping` - _dq_info.anomaly.segment field (permanently null once segmentation is gone) - group_config segmented thresholds (MAX_SEGMENT_MODELS, MIN_ROWS_PER_SEGMENT, ...) - segment_utils: canonicalize_segment_values / build_segment_name / build_segment_filter Models trained on a prior build must be retrained: the config hash changed and the registry struct was renamed. mergeSchema is retained on the registry write so the retrain writes cleanly to an existing registry table. Tests: delete test_anomaly_segments, test_anomaly_segment_naming, test_anomaly_scoring_run; update autodiscovery, registry, drift and apply-checks tests to the baseline_by policy (baseline columns are grouping, not features). Docs: CHANGELOG BREAKING entries; anomaly_compatibility.mdx converted to a migration guide; anomaly_heuristic_map.mdx segmented branch removed; user guide reworked around group-aware detection with a breaking-change/migration section; Experimental -> Beta. The LLM explainer's segmentation vocabulary is intentionally left for the next commit, which rewrites it with the baseline_by replacement in hand rather than blanking it. Co-authored-by: Isaac <no-reply@databricks.com>
Feature engineering expands each source column into engineered features by a fixed set of naming conventions (one-hot, frequency, null indicator, boolean, datetime cyclicals, numeric identity, baseline-relative). Nothing inverted those conventions, so two things were impossible: redacting every feature derived from a redacted column, and showing a reader a raw key like `event_count_rel_baseline` as a human phrase. New anomaly/feature_naming.py, pure functions over SparkFeatureMetadata: - source_column(name, metadata) -> str | None - human_label(name, metadata) -> str (falls back to the name; never raises, never hides a driver) - engineered_from(source, metadata) -> frozenset[str] (defined via source_column, so forward and reverse can never disagree) Resolution order is deliberate: one-hot first (matched against onehot_categories, since a value may contain an underscore or collide with a fixed suffix), then fixed suffixes accepted only when the remainder is a real source column, then numeric identity. Unit-tested across every convention, including a category value that collides with a suffix and a numeric column named like one. Unused until the next two commits wire it into redaction and prompt rendering. Co-authored-by: Isaac <no-reply@databricks.com>
…column Redaction filters LLM-prompt contribution keys by exact match, and those keys are engineered feature names. A prior fix covered the baseline-relative feature (`amount` -> `amount_rel_baseline`) by reconstructing that one name from the column, but one-hot and frequency features could not be reconstructed from the column alone (`country` -> `country_US`, `country_DE`, `country_freq`, `country_is_null`), so they still reached an external serving endpoint after the user redacted `country`. Thread the model's feature metadata to the point of redaction and enumerate the derived features exactly: - ExplanationContext gains an optional `feature_metadata` field; `from_scoring_config` takes it and `scoring_run` passes the `parsed_metadata` already in scope at the call site. - `redaction_set(redact_columns, metadata)` now unions `engineered_from(column, metadata)` per redacted column, covering one-hot, frequency, null-indicator, baseline-relative and identity. This replaces the suffix-only expansion. With no metadata (a caller who built the context directly), it falls back to the baseline-relative feature only -- best effort, unchanged. Redaction still matches engineered keys, upstream of any human rendering, so the hole stays shut when 2c adds human labels. Tests: metadata-driven redaction over one-hot + frequency + null-indicator + baseline-relative; the existing metadata-less tests are unchanged and still pass. Co-authored-by: Isaac <no-reply@databricks.com>
…ines Contributions were shown to users and to the LLM as raw engineered names — a reader saw `event_count_rel_baseline (74%)` in `_dq_info` and in the prompt — and the prompt still spoke the removed language of segments. - feature_contributions in the prompt now renders human labels via the reverse map, after the redaction filter: `amount vs its group baseline (82%), quantity (11%)`. A driver whose label cannot be resolved falls back to its raw name, so nothing is ever hidden. The label lookup is a SQL map literal with escaped keys/values (column names are user-derived). - The ai_explanation struct gains a `top_drivers` string (the same human-labelled sentence) for display. `top_features` stays engineered names for grouping/tooling, and the top-level `contributions` map keeps engineered keys so redaction and downstream tooling still match. - Prompt vocabulary moves from segments to baselines: the dead `segment` field (always empty since the segment path was removed) becomes `baseline_grouping`, the columns forming each row's baseline group — the material fact for a contextual anomaly, and a per-run constant with no PII. Few-shot examples and instructions reframed accordingly. The dead `_format_segment` helper and its tests are removed; the committed prompt snapshot is regenerated. Billable ai_query surface: the fixed prompt header grows ~322 chars (~+80 est. tokens) per call from the richer baseline vocabulary and label guidance, plus a few chars per group for longer human labels. Tests: unit coverage for the baseline-grouping string and the human-label map (identity omitted, derived features labelled, empty without metadata); the AI-explanation integration test asserts top_drivers is populated and carries the top features with weights. Co-authored-by: Isaac <no-reply@databricks.com>
…subtract it
The previous commit concluded trend was not cheaply fixable after two fixes failed. That was one fix
short. A third works, nearly perfectly, and this records it so the limitation reads as "not yet built"
rather than "cannot be done".
## Why the first two failed points straight at it
An elapsed-time *feature* fails because the model must learn the slope from data covering only the
training range, then meet values outside it. A time-bucket `baseline_by` fails because a median **lookup
table** has no entry for a future bucket. A *fitted* trend has neither problem: it is a function of time,
so it extrapolates to any future t, and the model never sees time at all -- it sees the residual, which is
stationary by construction.
Which makes it the same shape as the `_rel_baseline` feature this PR already ships: observed value minus
its expected level. Only the source of the expected level changes, from a per-group median to a fitted
line. Same append-at-tail rule, same persistence path through `SparkFeatureMetadata`, whose from_json is
already unknown-key tolerant, so slope/intercept/reference-epoch are additive.
## Measured
trend over window raw detrended (correct answer ~2%)
10% 19.6% 2.8%
40% 80.2% 2.8%
100% 95.8% 2.8%
200% 99.2% 2.8%
Flat at 2.8% however steep the trend. And it does not blunt real detection: on a batch carrying a 3x
spike in 5% of rows both reach 100% recall, but the raw feature emits **96.4%** false positives against
the detrended feature's **2.1%**.
## Four things a design has to answer, all measured rather than guessed
**Extrapolation horizon.** Accuracy decays with distance past the training window -- 2.8% at the boundary,
3.4% one window out, 6.4% five out, 89.6% twenty-five out. So it needs a horizon cap and a warning past
it, the same shape of contract as `is_new_baseline`.
**Regime change.** When the trend itself changes, 70-85% of rows flag. Arguably correct, since growth
stalling *is* an anomaly, but reporting a table-level event row by row is not useful. Worth noting
detrending is the only one that notices at all: when growth *reverses*, the raw feature flags just 6.0%,
because falling values look like a return to trained levels, while detrended flags 85.2%.
**Functional form.** Exponential growth fitted with a straight line barely helps -- 95.8% against a raw
99.8%. Business metrics compound, so a log-scale fit is needed and the form becomes a choice, not a
default.
**API cost.** It needs a time column, and the current design deliberately requires none: `"timeseries"`
models cross-metric correlation, not time. This would add the first temporal parameter to the public
surface, which is a decision about the shape of the API rather than an implementation detail.
## Not implemented here, and why that is a scope call rather than a verdict
This PR already carries a breaking removal, baseline conditioning, a second detector behind `profile`, the
attribution rework, the invariant pins, docs and demos. A temporal transform brings a new public
parameter, a new persisted field, an extrapolation contract, a functional-form choice and its own failure
modes -- and it is a new capability, not a repair to anything this PR changed.
The user-facing guidance is unaffected either way: model a quantity that does not trend, and where the
level matters, retrain on a schedule. That advice costs nothing and remains correct whether or not the
transform is built.
No `src/` change. The docs are untouched: they describe what ships, and what ships has no trend handling.
Co-authored-by: Isaac <no-reply@databricks.com>
A `type="warning"` admonition on the quality reference page, plus a bulleted "What neither profile covers" section in the guide, gave the limitations more prominence than they earn. Most readers do not need them, and a warning block reads as an alarm about the feature rather than as guidance — the wrong weight for "here are the shapes of problem this is not for". Nothing factual is dropped or softened. The same content is now one FAQ entry, *Are there anomalies DQX will not find?*, in the section a reader reaches when they have exactly that question. Trend, non-calendar cycles, forecasting, single-metric series and already-labelled data are all still there, each with what to do instead. Discoverable without shouting. The reference page keeps two sentences pointing at the guide, since a reader of the quality page may want the caveats and should not have to guess where they went. Its `Admonition` import went with the block -- nothing else on that page used it. **"What is handled automatically" stays**, and is now the only section of its kind in that part of the guide. It is a positive statement -- calendar seasonality, group context, categorical and null handling -- and it is the correction from two commits ago, where the docs had wrongly claimed seasonality was *not* modelled. Keeping it while the limitations move is the right asymmetry: what DQX does needs a section, what it does not do needs an answer to a question. One dead cross-reference fixed as a consequence: the "Trend" bullet linked to `#what-neither-profile-covers` from the reference page, an anchor that no longer exists. Verified there are no remaining references to it. Verified: `make docs-build` SUCCESS after a full cache clear, 156 documents, no broken links; the FAQ entry renders and the warning block is absent from the built quality page. Co-authored-by: Isaac <no-reply@databricks.com>
…scribing correlation breaks
The AI explanation was **wrong** for the correlation-aware detector. Observed on a live workspace, on rows
whose every metric sat inside its healthy range:
"Abnormal coolant flow and bearing temperature may signal impending equipment failure"
"Abnormal motor and spindle readings may signal equipment stress"
"Multiple sensor deviations suggest equipment malfunction or calibration drift"
action: "Inspect coolant system and bearing sensors"
The same scoring run printed the ranges that contradict it:
motor_current healthy 10.2-25.4 during incident 14.1-22.4 (inside)
coolant_flow healthy 14.0-36.0 during incident 16.6-30.2 (inside)
No metric was abnormal. Every value was ordinary and the *relationship between them* had broken -- which
is the entire reason this detector exists. So the explanation asserted something the data does not
support, and its action sent an operator to inspect a sensor that reads fine. That is worse than no
explanation: a false lead is paid for twice, once in wasted time and once in trust.
## Cause
The prompt never said which detector produced the numbers. It carries feature_contributions, group_size,
severity_range, confidence, baseline_grouping, threshold and drift_summary -- and a per-feature importance
from a correlation-aware detector is *identical in shape* to one from a tree. Given no way to tell them
apart, the model used the only reading it knew:
* tree / tabular -- a high contribution means this feature's own value was unusual (true)
* correlation-aware -- a high contribution means this metric left its usual relationship
with the others, and its value may sit mid-range (was not conveyed)
Same class of error as the signed attribution decomposition rejected earlier in this work: a number that
is technically derived being rendered to a user as a claim the data does not support. Caught by reading
the generated text rather than by checking it was non-null -- the previous verification confirmed 41 of 41
narratives were present, which was true and insufficient.
## Fix
A new `attribution_basis` field, placed **first** in the prompt so the semantics are read before the
numbers, carrying a per-algorithm sentence rather than the raw algorithm name -- the model needs to know
what a contribution *means*, not an implementation label it would only guess about. The correlation-aware
text also carries the explicit prohibition on the observed failure: do not call an individual metric
abnormal unless the contributions are concentrated in one.
`record.identity.algorithm` was already available at the call site, beside the `is_ensemble` threaded
through the same way, so this needed no new plumbing. `ExplanationContext.algorithm` is additive and
defaults to None; matching is by prefix so `IsolationForest_Ensemble_3` resolves, and an unknown algorithm
falls back to the value-based reading -- the conservative direction, since claiming an extreme value where
a relationship broke understates the finding, whereas the reverse invents a relationship claim.
## Verified by re-running and reading the output, both ways
Correlation-aware, after:
"26 rows show broken relationships between coolant_flow and bearing_temp (42% and 32%), which no
longer align as expected with each other or other metrics."
impact: "Decoupled coolant and temperature signals may mask equipment stress..."
action: "Verify sensor calibration and expected correlations between coolant_flow and bearing_temp."
action elsewhere: "Investigate why throughput and spindle_load no longer correlate as expected."
No metric is called abnormal, and the action points at the relationship, which is where the problem is.
Tabular, unchanged and still correct -- the value framing is preserved exactly where it is true:
"37 rows are flagged primarily by item_count (29%) and amount (26%), both unusual relative to their
merchant_category baseline, with weekend timing contributing (11%)."
The committed prompt snapshot test did its job and failed on the first run; regenerated per its own
docstring, and the diff is one added line. Four unit tests cover the distinction, the prefix match for
ensembles, and the fallback. Gates: unit 2554 passed, mypy clean, pylint 10.00/10.
Co-authored-by: Isaac <no-reply@databricks.com>
The existing demo teaches the API: generate a frame, train, score, inspect. That is the right shape for a
reference demo and the wrong shape for teaching `profile`, because the whole point of that argument is
that **the user's data decides it** -- so a demo that does not start from a recognisable problem cannot
teach the choice. One per profile, each named for its domain rather than the DQX feature it exercises.
## `dqx_demo_anomaly_tabular_transactions.py` -- "The transaction that passed every rule"
Card transactions across twelve merchant categories, each with its own typical basket. Opens by applying
the rules a payments team would already have -- amount in range, item count in range, amount not null --
and showing they catch **0 of 31** injected rows, because every individual value is ordinary and only the
combination is not. Then the model catches 22 of 31 with no thresholds specified.
Also gives `baseline_by` its first demo: the same amount is unremarkable for electronics and absurd for
coffee, so the notebook trains a second model with `baseline_by=[]` and compares. Measured 22 against 18,
and the takeaway says so plainly rather than overselling it -- some injected rows (a GBP 900 coffee) are
extreme enough to stand out against the whole table too, and conditioning earns its keep on the ones that
are not. An earlier draft of that bullet claimed more than the run supports; corrected against the number.
## `dqx_demo_anomaly_timeseries_fleet.py` -- "The machine where every gauge read normal"
Eight machine metrics driven by two latent factors, so they move together the way telemetry does. The
incident permutes three of them among a block of rows, which preserves each metric's own distribution
exactly -- the same values, reordered -- so only the joint behaviour changes.
The notebook then **verifies its own premise before modelling**, printing each metric's healthy range
against its range during the incident and showing every incident reading falls inside. Without that, "no
threshold could catch this" is an assertion; with it, it is demonstrated. Both profiles are then trained on
identical data:
profile incident rows caught total flagged false alarms
tabular 0 of 60 71 71
timeseries 47 of 60 147 100
A fixed severity threshold is a percentile of the *training* score distribution, so the two need not flag
the same number of rows -- which leaves the fair question of whether the second one simply alerted more. So
a following cell repeats the comparison at an equal budget: rank by severity, take the same N from each,
count. That is how the DQX benchmarks compare detectors and it closes the only real hole in the argument.
Both demos are synthetic and self-contained (SMD is real telemetry and not shippable), quote the measured
SMD figures without claiming state of the art, and state where DQX is weak. Registered in `demos.mdx`.
## Two bugs found only by running them
**`.cache()` raises on serverless** -- `NOT_SUPPORTED_WITH_SERVERLESS: PERSIST TABLE is not supported`.
Both demos died on their first data cell. Serverless is what a reader will reach for, so this was not a
missed optimisation but a crash on cell one.
**Removing the cache then emptied the AI-explanation display**, which is the more interesting failure. The
scored frame became a lazy plan, and AI explanations call an LLM through `ai_query` *inside* that plan, so
every action re-invoked the model -- roughly eight times across the fleet demo's cells, at eight times the
cost, with a later re-execution returning nulls and blanking the display. Fixed by writing the scored
frame to a Delta table once and reading it back, which is also the pattern a real pipeline should use.
That diagnosis is what surfaced the misleading-narrative defect fixed in the previous commit.
Verified by executing both on a live workspace and reading every cell's output, not the exit code: 34 and
19 cells, zero errors, no empty tables, and the numbers quoted above are the ones the run printed.
Co-authored-by: Isaac <no-reply@databricks.com>
…hensive demo
Found while extracting the presentation style of `dqx_row_anomaly_detection_demo.py` to apply elsewhere.
Three defects in its closing cells, all pre-existing:
**Two 404 links.** Both omit the `/docs/` path segment:
https://databrickslabs.github.io/dqx/guide/row_anomaly_detection
https://databrickslabs.github.io/dqx/reference/quality_checks#has_no_row_anomalies
Confirmed against the built site rather than assumed: `docs/dqx/build/docs/guide/…` exists and
`docs/dqx/build/guide` does not. The anchor was wrong too — the reference page's heading renders as
`#row-anomaly-detection`, not `#has_no_row_anomalies`.
**A parameter that has never existed.** "Add group conditioning (`group_by` for training)" — the argument
is `baseline_by`. Not even a stale rename: `group_by` was never accepted, so a reader following that line
gets a TypeError. (`segment_by` was the old name, and it was removed earlier in this PR.)
Nothing verifies demo links, so these could only be caught by looking. Checked the rest of `demos/` for
the same malformed link shape and found none.
Co-authored-by: Isaac <no-reply@databricks.com>
… both in e2e
The two domain demos ran correctly but read like engineering write-ups: 19 cells each with a 76-line and a
55-line wall of code, and both built around A/B comparisons of DQX's own options — training two models and
racing them. That was the right instinct for verification and the wrong shape for a demo, which should say
what the tool is for and show it working.
## Restructured against the house benchmark
`dqx_row_anomaly_detection_demo.py` was the reference (its style, extracted: emoji-led H1, `---` rules,
`## Section N:`, DBTITLE on every code cell, emoji print narrative, print-before-display, f-string numeric
tables, and a Summary / Resources / You're Ready close). Worth noting the other demos deliberately use no
emoji and no rules — where they conflicted, the anomaly demo won as the closest sibling.
largest code cell 76 → 21 and 55 → 20 lines
cells 19 → 27 and 19 → 24
code cells > 25 lines 3 → 0 and 2 → 0
## Comparisons removed, per the instruction to state rather than compare
The fleet demo no longer trains both profiles or runs the equal-budget cell; a markdown table says which
profile is for which data and cites the measured SMD figures (79% against 33%) as documented fact. The
transactions demo no longer trains a pooled model to argue for `baseline_by`. Removing these deleted the
three largest cells and about a third of the code in each.
Kept: the rules-catch-nothing cell (rules versus ML is the value proposition, not a comparison of DQX
options — the old demo taught it too) and the fleet range-proof cell, without which "no threshold could
catch this" is an assertion rather than a demonstration.
## Unity Catalog, not hand-rolled write-then-read
Scoring now goes through `apply_checks_and_save_in_table(input_config=…, output_config=…)` — name the
input table, name the output table — instead of `apply_checks(...).write.saveAsTable(...)` followed by
`spark.table(...)`. That is the shape UC users expect, and it still gives the materialisation the AI
explanations need, since `ai_query` runs inside the scoring plan and every action on a lazy result would
re-invoke the LLM. Also stopped re-reading frames already held in a variable.
## Old demo retired
`dqx_row_anomaly_detection_demo.py` is deleted, with its `demos.mdx` entry, its published workspace copy,
and its e2e test. The e2e test is replaced by a parametrised `test_run_dqx_anomaly_demo` over both new
notebooks, following the shape of every other test in that file and keeping the 45-minute wait for the
same reason: these train a model and score with contributions and AI explanations on by default.
## Three defects that only reading cell output could find
**The registry display was showing stale, contradictory data.** Re-running accumulated rows, so the
"registered model" cell printed 8 rows for transactions and 4 for fleet — and the transactions rows showed
`columns=['amount','item_count','transaction_time']`, contradicting the notebook text that had just
explained the timestamp is excluded. Fixed by dropping the registry during setup, which is what the old
demo already did. Now one row, correct configuration.
**A cell rendered nothing at all** — the fleet correlation `display()`, the only `display()` of a pure
aggregate. Replaced with an f-string table, which is better style anyway and lets each pair carry its own
explanation.
**The model was being fed noise.** `transaction_time` was in `columns` while this generator draws hours
uniformly over 90 uniform days, so seven of eleven engineered features carried no signal — diluting the two
that did and manufacturing "unusual timing" alerts. Measured at threshold 98: 23/30 caught with the column,
30/30 without. Making the timestamps realistic did *not* help (20/30) — the model then spends budget on
genuine timing outliers. Dropping it wins at every threshold. The column stays in the table, out of the
model, and the notebook now teaches the rule: feed a column only if it relates to the anomalies you want,
and note that auto-discovery would have included it.
## Threshold semantics, which are the most misread part of the feature
`threshold=95` means "flag the top 5% by *training* severity" — roughly 75 alerts on 1,500 rows before any
anomaly exists. It is an alert budget, not a confidence score, and that puts a hard ceiling on precision:
ask for 75 alerts when 33 rows are bad and 44% is the best anyone could do. The tuning table now prints
that ceiling beside the observed value, because precision read against 100% looks like failure and read
against the budget looks like what it is. On the verified run the model sits *at* the ceiling with 100%
recall at both threshold 90 and 95.
One thing checked and deliberately not changed: the injected rows were suspected of being partly
borderline, since injection kind 0 keeps whichever category was drawn. Measured over 24 of them, none were
— the mildest sat 9.2x from its category's normal item count, the rest 14x–42x. The fixture is sound.
Verified by executing both on a live workspace and reading every cell: 27 and 24 cells, zero errors, no
empty outputs, registry showing one correct row, contributions naming only meaningful features
(`amount 60.4, amount_rel_baseline 32.1` on an £893 coffee), and the fleet narratives still describing
broken relationships rather than abnormal metrics. Recall at the default threshold varies run to run
(76% then 100% observed) — expected, since training samples 30% of rows and the tabular path ensembles
three differently-seeded forests.
Co-authored-by: Isaac <no-reply@databricks.com>
The slide had to land one thing: there is an anomaly the forest cannot isolate, which is why a second
detector exists. Six mockup rounds got there; two of the wrong turns are worth recording because both were
found by looking at the rendered result rather than by reasoning about it.
**Colour must not vary.** Early versions scattered ripeness filters so the belt would not look monotonous.
Colour is the most salient channel on a screen, so the eye nominates the green fruit — and when the gate
then flags an ordinary yellow one the verdict looks arbitrary. It also broke the argument being made: "each
value is normal, only the pair is impossible" needs the culprit to be genuinely indistinguishable, which it
cannot be if its neighbours all look wildly different. So the crate is one ripeness. What varies visibly is
length, one of the two numbers the check reads. Weight — the number that is actually wrong — is invisible,
which is precisely why a check is needed and the eye is not enough.
**No instrument panel.** A readout grew to six rows, collided with its own nameplate, and turned the
argument into something read rather than seen. One pill now carries the verdict; the reasoning lives in the
slide body, where there is room for it.
What ships: a conveyor with a scanner gantry, fruit of varying length at one ripeness, and a pill reading
`19cm · 38g pair ✗`. On a failure the whole line halts — belt, slats and rollers all stop — and the culprit
drifts *up* off the belt, because being hollow is the one thing about it that is visible. The packing line
also continues the deck's existing factory story: trucks deliver, DQM watches the trucks, DQX inspects what
comes off them one row at a time. Left-to-right motion supplies the sequence feel honestly, as throughput
rather than trend, which matters because this profile does not model time.
Three CSS bugs found along the way, all of the same shape — changing how something was positioned or
animated without rechecking what its units referred to:
* `bvPopIn` animates `transform: scale()`, which silently replaced the `translate(-50%,-50%)` a plotted
element relied on for centring, leaving every point half a glyph off its coordinate.
* Rebuilding the strip with `innerHTML` each tick meant every element was new, so there was no previous
position to animate from and the transition never fired. Riders are now created once and only moved.
* `translateX(105%)` is 105% of *the element*, not the track. With a 27px glyph that is 28px of travel, so
all eight riders piled up at the left edge. Positions are pixels now; the `-50%` stays only for centring.
Verified in the minified output rather than the source, because the CSS minifier renames keyframes: the
slats animation resolves to `@keyframes n{to{transform:translateX(-15px)}}` and the paused-state override
lands on the same class. Full cache clear before building, since an incremental docs build has served stale
CSS from this file before. `make docs-build` SUCCESS, 156 documents, no broken links; `tsc --noEmit` clean
apart from the four pre-existing `FeatureTags` errors.
Co-authored-by: Isaac <no-reply@databricks.com>
The FAQ said that with only one metric `profile="timeseries"` "has nothing to work with". The first half of that is right and the conclusion is wrong, because DQX derives seven cyclical features from a datetime column — so one metric plus a timestamp is not one feature, it is eight, and the metric can be judged against where it sits in the week. Measured on a series that is busy on weekdays and quiet at weekends, then given a weekday-sized value landing on a Saturday: the anomalous rows score a median 177.4 against 3.2 for ordinary weekend rows, a 55x separation, while the value's own z-score is 0.92 — nowhere near extreme. So the contextual case genuinely works, and the earlier wording talked a user out of it. Verified separately that with a *bare* single feature the detector does reduce to a z-score, matching (x-mu)^2/sigma^2 to within 0.05% — the residual being the ridge floor on the covariance. So the advice to reach for a range check or `has_no_outliers` is correct for that case, and now stated as being about that case rather than about single metrics in general. Same failure as the seasonality claim corrected earlier in this PR, from the same cause: having established that the profile does not model *time*, I over-corrected into implying the calendar features do not count. They do. The boundary that actually holds is contextual versus sequential — "unusual for a Tuesday morning" is in reach, "unusual compared to five minutes ago" is not, because there are no lag features and no memory of the previous row. Kept deliberately non-technical, per review: no mention of covariance or z-scores in the FAQ text, just what to compare against and why. `make docs-build` SUCCESS, 156 documents, no broken links — which also confirms the new `has_no_outliers` reference resolves. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1489 +/- ##
==========================================
- Coverage 92.62% 92.22% -0.40%
==========================================
Files 141 144 +3
Lines 13579 13833 +254
Branches 151 151
==========================================
+ Hits 12578 12758 +180
- Misses 932 1006 +74
Partials 69 69
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Naming `columns` keeps the whole-table comparison, as it did before `baseline_by` existed. Discovery still runs, but only to name the grouping it found so the caller can opt in. Reverting the coupling change: the defect was the silence, not the pooling. An explicit column list is a decision, and a discovered grouping is data-dependent, so applying one makes the feature set shift when a cardinality crosses a threshold between retrains, moving every score and drifting a calibrated alert budget. The advisory is silent when the data supports no grouping, which is what keeps it worth reading, and `suggest_baseline_columns` scans only what the grouping decision needs rather than computing a full profile and discarding the numeric half. Also removes the anomaly migration guide and heuristic map. The migration steps now sit in the user guide beside the feature; the heuristic map was a second copy of the control flow that had to be updated alongside it.
The release notes are theirs to write and are generated at release time; a feature branch editing them just creates a conflict for whoever cuts the release. Everything these entries said is in the PR description and the user guide.
Removes the separate anomaly_conditioning harness and its dedicated quality page. Detection quality now rides in the benchmarks report alongside DQX core's timings, fed by tests/perf/test_anomaly_benchmark.py. Two perf tests are added for the features this branch introduces: one scores profile="timeseries" on correlated metrics whose relationship breaks, and one scores baseline_by on a contextual collapse. Both fixtures are generated in-repo from a fixed seed, so nothing is downloaded and no third-party dataset is redistributed, and the published numbers carry no licence conditions. The anomaly table gains fixture, row-count, feature-count and anomaly-rate columns, because the rows are now different problems and comparing quality across them would otherwise look meaningful. The nightly step that regenerated the removed page goes with it. The harness is kept outside the repo for future reference rather than carried here: it downloads ~250MB of third-party data and committed 27k lines of results for numbers the repo cannot verify on its own.
Contributor
|
✅ 955/955 passed, 49 skipped, 7h4m36s total Running from acceptance #5685 |
Contributor
|
✅ 1/1 passed, 24m34s total Running from mcp #434 |
Contributor
|
✅ 208/208 passed, 1 flaky, 1 skipped, 7h20m2s total Flaky tests:
Running from anomaly #1799 |
ghanse
added
the
under-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row anomaly detection: a second detector, and the end of
segment_byCloses #1484 and #1490.
TL;DR
profilearg ontrain().profile="timeseries"uses Mahalanobis distance instead of Isolation Forest, for data where metrics move together. Default stays"tabular", so nothing changes unless you ask for it.baseline_byarg. Judges each row against its own group's baseline instead of the whole table, on a single pooled model.segment_byandAnomalyParams.max_segment_modelsare removed.baseline_byreplaces them but is not a rename, scores will differ.segmentationis renamed togrouping, and_dq_info[].anomalygets two new fields. Both writes usemergeSchema.What's new
profile="timeseries"profile"tabular"(default)"timeseries"Isolation Forest splits on one feature at a time. That makes it good on tabular data and close to blind when the anomaly is a broken relationship between two readings that are both in range. There's no single feature to split on.
Measured on the Server Machine Dataset (28 machines, 38 metrics, real telemetry with labelled incidents), inside an alert budget of 1% of rows:
"tabular""timeseries"Same features, same budget, same pipeline. Only the estimator changed.
There is deliberately no
"auto". You can't verify the choice without labels, and silently swapping the estimator would move every score people have already set thresholds against. The resolved profile is logged on every run.baseline_byThe same value can be fine in one group and wrong in another. Before this, every row was compared against the whole table, so that kind of anomaly was invisible. One group's volume dropping 80% behind a flat daily total scored 45.1, which is the 45th percentile. No threshold recovers that.
baseline_byadds, per metric, its deviation from that metric's median inside the row's group, as a signed log ratio. One pooled model, so cost doesn't grow with the group count.The same collapse now scores above 95. PR-AUC on a contextual anomaly goes from 0.0376 to 0.5703, and an anomaly that was already globally extreme stays at 1.0000, so conditioning costs nothing when there's nothing to gain.
Behaviour changes if you're already using the default profile
These all apply to
profile="tabular"too:Rows whose group wasn't in training now return a null score with
is_new_baseline = true, instead of 0.0. 0.0 was the most normal-looking value in the table, which is the wrong thing to say when you actually mean "couldn't judge this".New advisory warning at training time. Passing
columnskeeps the whole-table comparison, same as before, but if the data looks grouped you now get told so instead of it passing in silence:It only fires when there's something to act on, so it stays worth reading. Deliberately advice and not action: your explicit column list is a decision, and a discovered grouping is data-dependent, so applying one would make the feature set shift under you when a cardinality changes and move every score. That gives the three
baseline_byvalues distinct jobs:Noneis "nudge me if it matters",[]is "decided, whole table, be quiet", a list is "condition on this".AI explanations now say which detector produced the contributions. A per-feature importance means something different coming from a tree than from a correlation-aware detector, and without that the narrative called a broken relationship an abnormal metric value, which sends someone to a healthy sensor.
tests/resources/ai_query_prompt_header.txtis regenerated to match.profilethreads throughAnomalyConfigand the anomaly workflow, so it works from run-config YAML, the CLI and scheduled retraining, not just Python.Why Mahalanobis
Picked by measurement, not preference. A bake-off across featuriser, estimator and scope was run on the Server Machine Dataset before settling on it. That harness is not in this PR: it downloads ~250MB of third-party data, so it lives outside the repo and the numbers below are stated as measurements rather than as something CI reproduces. What the repo does carry is
tests/perf/test_anomaly_benchmark.py, which measures detection quality on synthetic fixtures generated in-repo and publishes it in the benchmarks report.The metric is event recall at a fixed alert budget, not PR-AUC. SMD's 3,732 anomalous rows sit in 39 incidents of median length 6 and max length 1041, so point-wise PR-AUC mostly measures whether you found the one huge incident. The best-PR-AUC config in the sweep covers 2 of 39. No point-adjusted F1 anywhere either, since Kim et al. (AAAI 2022) showed random scores hit state of the art under it. Published SMD numbers around 0.80 F1 aren't a comparable target and this PR doesn't claim state of the art.
Then, on the shortlist:
abs(), so a feature that reduced the distance would have been shown to the LLM as a driver. There's a test asserting that.ensemble_sizeis ignored andconfidence_stdisn't available for it, both logged rather than dropped quietly.Two things tried and dropped, with numbers recorded so nobody re-argues them:
Breaking changes
segment_byis removed. It trained one model per group.baseline_byis the only grouping mechanism now, and it isn't a rename:segment_bypartitioned into N models,baseline_byjudges each metric against its own group's baseline on a single model. The scores differ.Why it's gone rather than kept alongside: on SMD, per-group models were the worst of the three configs measured (PR-AUC 0.1416, against 0.1499 for plain pooling and 0.1536 with conditioning), and they fail in a way the average hides. One entity produced 15,963 false positives across 28,392 normal rows, a 56% false alarm rate, because each per-group model calibrates its threshold on its own rows. Cost is linear in group count too: 90 groups took roughly 88 minutes, against 0.17s for one conditioned model.
Also removed or changed:
AnomalyParams.max_segment_modelsis no longer accepted.segmentationstruct is nowgrouping(baseline_by,sklearn_version,config_hash). The write usesmergeSchema, so retraining into an existing registry table adds the renamed column in place._dq_info[].anomaly.segmentis gone. It carried a per-segment identity and was always null._dq_info[].anomalygainsis_new_baselineandnew_baseline_key. Named-field queries keep working, but the struct is wider, so appending to a table that already has_dq_infoneedsmergeSchema.compute_config_hashnow includesbaseline_by, so a stale hash raises at scoring time instead of silently scoring against a different feature list.Row anomaly detection is Beta and its formats were allowed to change without a migration path, so I'd rather take the break now than carry it. The migration steps are in the user guide under "Upgrading and breaking changes".
Docs
threshold=95flags the top 5% by training severity, which caps precision, and the tuning table now prints that cap next to the observed value.drift_thresholdwarns you about trend (it doesn't, the score saturates below 1.8 against a documented threshold of 3.0).segment_byormax_segment_modelsas accepted args.Demos
dqx_row_anomaly_detection_demo.pyis replaced by two notebooks built around a domain problem, both covered by a new parametrised e2e test:dqx_demo_anomaly_tabular_transactions.py: card transactions that pass every rule but are jointly implausible. Opens by applying the rules a payments team would already have and showing they catch 0 of 31. Also the first demo ofbaseline_by.dqx_demo_anomaly_timeseries_fleet.py: machine telemetry where every metric stays inside its safe band while the relationship breaks. It checks its own premise first, printing each metric's healthy range against its range during the incident, so "no threshold could catch this" is shown rather than asserted.Both ran end to end on a live workspace with every cell's output read. Two things to know before editing them:
.cache()raises on serverless (PERSIST TABLE is not supported), and AI explanations call an LLM inside the scoring plan, so the scored result has to be written to a table once instead of re-actioned lazily.Tests
test_anomaly_isolation_forest_inertness.py: 16 tests pinning the default path, includingscore_samplesover a fixed grid against a committed reference array. Sabotage-checked by deliberately breakingcore.pyand confirming the right assertions fired.test_anomaly_mahalanobis_detector.py: 13 tests covering the sklearn outlier contract, the leave-one-out identity torel=1e-6, constant features,n < prefusal, and the signed-decomposition rejection.test_anomaly_timeseries_profile.py: one integration test for the two things only a workspace can settle, that MLflow round-trips a DQX-defined estimator class and that signature inference accepts it.test_anomaly_train_and_score.py,test_anomaly_quality.py,test_anomaly_explainability.pyandtest_anomaly_ensemble.py.Unit 2554 passed, mypy clean over 350 files, pylint 10.00/10,
make docs-buildclean.Still open
Two follow-ups are scoped and measured but not in here: detrending as a feature transform (measured flat 2.8% false flags, against raw's 19.6% to 99.2%), and a trend-aware drift statistic. Both are new capabilities rather than fixes, and the second one touches existing behaviour. The FAQ says what to do in the meantime.
make anomaly -n 4hasn't been re-run since the detector landed, so this is a draft until it's green.This pull request and its description were written by Isaac.