feat: Relative Effects and MDE for Ratio Metrics by luizhsuperti · Pull Request #264 · david26694/cluster-experiments

luizhsuperti · 2026-03-20T15:55:39Z

feat: Relative Effects and MDE for Ratio Metrics

Summary

This PR adds proper relative (percent) lift estimation and minimum detectable effect (MDE) calculation for ratio metrics using an outer delta method, instead of relying on naive scaling by the control mean.

Problem

For ratio metrics such as conversion rate or revenue per user:

Naive relative lift (absolute effect divided by control mean) underestimates uncertainty because the denominator is treated as fixed.
Naive relative MDE from scaling absolute MDE is only an approximation and can be inaccurate when denominator variance matters.

What Changed

Added DeltaMethodLiftTransformer with:
- fit(mean_diff, var_abs, ctrl_mean, ctrl_var)
- lift_and_se(...) static helper for relative lift and SE
- relative_mde(alpha, power, ctrl_mean, ctrl_var, treat_var) static helper solving the quadratic power equation
Extended DeltaMethodAnalysis with relative_effect (default false):
- When true, uses DeltaMethodLiftTransformer internally to return relative effect and proper SE
- Works with and without covariates (including CUPED flows)
Updated PowerAnalysis config validation to allow relative_effect with DeltaMethodAnalysis
Added documentation notebook showing:
- naive vs proper relative lift
- naive vs proper relative MDE
- integration with NormalPowerAnalysis and AnalysisPlan
Added tests for:
- transformer math
- DeltaMethodAnalysis integration
- config behavior
- end-to-end AnalysisPlan and power workflows

Impact

Users can now estimate relative effects for ratio metrics with uncertainty propagation that is statistically consistent with delta-method assumptions, and compute a principled relative MDE via the quadratic formulation.

Notes

This keeps backward compatibility while introducing the new transformer API and helper functions for direct use in custom workflows.

codecov-commenter · 2026-03-20T16:04:16Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.29630% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.18%. Comparing base (b0a737a) to head (e67d729).

Files with missing lines	Patch %	Lines
cluster_experiments/power_analysis.py	94.50%	5 Missing ⚠️
cluster_experiments/relative_lift_transformer.py	97.05%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #264    +/-   ##
========================================
  Coverage   95.17%   95.18%            
========================================
  Files          18       18            
  Lines        2094     2220   +126     
========================================
+ Hits         1993     2113   +120     
- Misses        101      107     +6

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

david26694 · 2026-03-21T07:55:59Z

+            transformer = DeltaMethodLiftTransformer(self.treatment_col)
+            transformer.fit(
+                mean_diff=mean_diff,
+                var_abs=treat_var + ctrl_var,


perhaps you can pass std_error instead of var,and you have it in this variable: standard_error

david26694 · 2026-03-21T07:57:20Z

+            "percent_lift": self._relative_lift_value,
+            "_se_relative_lift": self._se_relative_lift,
+            "pvalue": self.pvalues[self.treatment_col],
+            "conf_int": self.conf_int(0.05).loc[self.treatment_col],


shouldnt we use alpha in here?

…variance; update related tests - Changed the `DeltaMethodLiftTransformer` to accept standard error in the `fit` method instead of variance. - Updated the `experiment_analysis.py` to reflect this change in the transformer usage. - Refactored `relative_lift_transformer.py` to introduce a base class for lift transformers, consolidating shared functionality. - Modified the Jupyter notebook `relative_delta.ipynb` to align with the new transformer behavior and improve clarity in output formatting. - Added tests to ensure parity between OLS and DeltaMethodAnalysis results for relative effects.

david26694 · 2026-04-28T15:27:37Z

+
+    # Point estimates come from different estimators (unweighted vs weighted by
+    # scale) so a 5% relative tolerance is appropriate
+    assert ols_point == pytest.approx(delta_point, rel=0.05)


just curious. do you think we can get it closer?

hmm did not think of that. maybe some simulations to check how close, or rather will check in some other sources

david26694 · 2026-04-28T15:56:02Z

-            "pvalue": self.pvalues[self.treatment_col],
-            "conf_int": self.conf_int(0.05).loc[self.treatment_col],
-        }
+        Solves A*m^2 + B*m + C = 0 for the smallest positive m satisfying


this is very new to me, could you add some reference or small proof?

oki! I'll add references in the .py, but I'll add a more thorough view in a notebook

david26694 · 2026-04-28T16:11:06Z

+            "percent_lift": self._relative_lift_value,
+            "_se_relative_lift": self._se_relative_lift,
+            "pvalue": self.pvalues[self.treatment_col],
+            "conf_int": self.conf_int(0.05).loc[self.treatment_col],


let's change this 0.05 to alpha

david26694 · 2026-04-28T16:14:12Z

+    )
+
+    assert transformer.bse["treatment"] >= se_naive
+    assert transformer.bse["treatment"] == pytest.approx(se_naive, rel=0.15)


this also looks big

david26694

.

david26694 · 2026-04-30T15:14:31Z

+        "cluster_experiments.relative_lift_transformer.stats.norm.ppf", mock_ppf
+    )
+
+    with pytest.raises(ValueError, match="invalid power equation"):


in which scenario do we not get an MDE?

I'm adding scenarios hashtags comments, but be more specific on the types of errors where the MDE fails (ie, high variance, imcompatible variance with asked alpha,etc)

david26694 · 2026-05-01T08:30:54Z

+        return relative_lift, float(np.sqrt(var_relative))
+
+    @staticmethod
+    def relative_mde(


after having a look I like this, but I think at some point it better bolngs in the power analysis class. Maybe we can merge this changing to internal method but later we move to NormalPowerAnalysis and use it to calculate the relative mde of ols

Suggested change

def relative_mde(

def _relative_mde(

but I don't see this method being called anyway, is this correct?

david26694 · 2026-05-04T11:02:55Z

+        return relative_lift, float(np.sqrt(var_relative))
+
+    @staticmethod
+    def _relative_mde(


imo this is a responsibility of the PowerAnalysis class, I would move it in there

david26694 · 2026-05-06T09:20:43Z

        standard_error = np.sqrt(treat_var + ctrl_var)

+        if self.relative_effect:
+            transformer = DeltaMethodLiftTransformer(self.treatment_col)


a bit confused in here, where is this method used? does this handle covariates?

Used by [_get_mean_standard_error] (p-value/SE path) and analysis_standard_error_with_stats(power path).
For Covariates: yes. It calls [_get_group_statistics] which builds [thetas_dict] from [self.covariates] and applies CUPED variance reduction inside [_get_group_mean_and_variance] So relative effects respect covariates.

luizhsuperti · 2026-07-25T20:26:45Z

Hey @david26694 !
I’ve pushed the review-response refactor for the relative MDE work.

Summary of the main changes:

Moved the quadratic relative-MDE calculation out of DeltaMethodLiftTransformer and into NormalPowerAnalysis, so the power-analysis layer now owns MDE calculation.
Wired the quadratic relative-MDE path into the actual MDE entry points: mde(), mde_power_line(), mde_time_line(), and mde_rolling_time_line().
Added StandardErrorResult to carry the regular standard error plus the optional ratio-metric stats needed for the quadratic calculation: ctrl_mean, ctrl_var, and treat_var.
Added ExperimentAnalysis.get_standard_error_with_stats() so analyses can provide enriched SE information when needed while existing analyses fall back to the normal linear MDE path.
Updated DeltaMethodAnalysis(relative_effect=True) to return the relative SE plus the control/treatment variance stats needed by NormalPowerAnalysis.
Kept relative OLS / clustered OLS on the existing linear MDE path, since those analyses do not currently expose the ratio-metric group stats needed for the quadratic relative MDE.
Removed the old _relative_mde helper from DeltaMethodLiftTransformer.
Added/updated tests for the moved solver, alpha usage, one-sided hypotheses, degenerate/no-solution cases, quadratic-vs-linear behavior, and the covariate/CUPED path.
Updated the relative-delta notebook with the derivation, references, and architecture explanation.

One decision I’d like your opinion on before finalizing: run_average_standard_error is a non-underscore method, and this refactor changes its yielded value from (float, int) to (StandardErrorResult, int). because the method now exposes StandardErrorResult, should we export StandardErrorResult from cluster_experiments.__init__?

luizhsuperti · 2026-07-31T12:34:58Z

hey @david26694 , to not clog the PR too much, as a follow up I propose the following notebook explaining the %MDE for ratio metrics (with Claude)

Steps

Create docs/relative_ratio_mde.ipynb — 12 short cells mirroring relative.ipynb's rhythm:
- Intro: "relative.ipynb covers OLS; this is the MDE of a relative effect on a ratio metric" + 3-step outline.
- Customer-level ratio dataframe (binomial target / scale, no treatment column).
- Two NormalPowerAnalysis.from_dict configs ("analysis": "delta", relative_effect True/False) — same config-dict idiom as relative.ipynb.
- Relative MDE (direct percent lift) vs absolute MDE, then the naive mde_abs / ctrl_mean comparison with the one-line "ignores baseline variance" explanation.
- Closing pointer to relative_delta.ipynb for the derivation.
Register the notebook in mkdocs.yml nav, mirroring how the other relative notebooks are listed (parallel with step 3).
Execute end-to-end and persist outputs; open the follow-up PR referencing feat: Relative Effects and MDE for Ratio Metrics #264.

Relevant files

docs/relative_ratio_mde.ipynb — new (follow-up branch only)
relative.ipynb — structural template
mkdocs.yml — nav entry

Verification

jupyter nbconvert --to notebook --execute --inplace docs/relative_ratio_mde.ipynb runs clean; proper relative MDE ≥ naive division.
mkdocs build (or existing docs CI) passes with the new nav entry.

add-backbone-of-calculations

654d39c

luizhsuperti added 3 commits March 20, 2026 18:08

delete backward compatiblility

041a284

Update relative delta

286ac10

add a test

3628494

david26694 requested changes Mar 21, 2026

View reviewed changes

luizhsuperti requested a review from david26694 April 28, 2026 10:13

Update relative_delta.ipynb

8eb5170

david26694 requested changes Apr 28, 2026

View reviewed changes

david26694 reviewed Apr 28, 2026

View reviewed changes

Add_small_and_big_effects_ses

e863186

david26694 requested changes May 1, 2026

View reviewed changes

fix: address david review comments on DeltaMethodLiftTransformer

90e484d

david26694 reviewed May 4, 2026

View reviewed changes

david26694 reviewed May 6, 2026

View reviewed changes

refactor exp. analysis and and lift transformer.

2f98606

luizhsuperti requested a review from david26694 July 25, 2026 20:27

luizhsuperti added 5 commits July 27, 2026 15:10

adjust minor bugs

9d211ec

fix test

7cc4da2

Merge branch 'main' into MDE-new-ratio-metrics

bf24376

add-close-form-mde-relative-eff-outer-delta

f00ddc1

Update relative_delta.ipynb

e67d729

Conversation

luizhsuperti commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!