test: tighten schema-v2 warm P95 budgets (Fixes #511)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: karthiknadig

docs/QUALITY_SNAPSHOTS.md

@@ -17,16 +17,20 @@ A metric blocks when it exceeds both its absolute and relative budget:
 | Server startup P50 | 5 ms / 100% | 10 ms / 50% | 100 ms / 50% |
 | Server startup P95 | 50 ms / 200% | 50 ms / 100% | 750 ms / 100% |
 | Full refresh P50 | 25 ms / 30% | 50 ms / 30% | 100 ms / 50% |
-| Full refresh P95 | 1,000 ms / 100% | 5,000 ms / 100% | 1,000 ms / 50% |
+| Full refresh P95 | 50 ms / 50% | 250 ms / 100% | 300 ms / 100% |
 | Time to first environment P50 | 20 ms / 100% | 25 ms / 50% | 150 ms / 50% |
-| Time to first environment P95 | 250 ms / 100% | 500 ms / 100% | 750 ms / 100% |
+| Time to first environment P95 | 25 ms / 100% | 100 ms / 100% | 250 ms / 100% |
 | Cold refresh P50 | 100 ms / 50% | 150 ms / 50% | 250 ms / 50% |
 
-Each cell is `absolute / relative`. The Linux and Windows budgets plus the macOS P50 budgets reflect observed GitHub-hosted runner variance from 11 consecutive main-branch baselines. Tighten them when a noisy path is fixed rather than normalizing a known regression into the baseline.
+Each cell is `absolute / relative`. The warm P50 and server-startup budgets reflect observed GitHub-hosted runner variance from 11 consecutive main-branch baselines. Tighten them when a noisy path is fixed rather than normalizing a known regression into the baseline.
 
-The macOS P95 budget recalibration is tracked by issue #507 and follows PR #506's fix for issue #504. It uses three unchanged-content pull-request runs and the exact merged baseline at `f0c62d9`; the resulting absolute headroom is four to six times the observed post-fix run-to-run range.
+The macOS server-startup P95 budget recalibration is tracked by issue #507 and follows PR #506's fix for issue #504. It uses three unchanged-content pull-request runs and the exact merged baseline at `f0c62d9`; the resulting absolute headroom is four to six times the observed post-fix run-to-run range.
 
-Schema v2 records `full_refresh` and `time_to_first_env` from the warm member of each pair and adds cold refresh/time-to-first distributions. While the exact base still uses schema v1, cold P50 is checked against explicit absolute ceilings of 500ms on Linux, 750ms on Windows, and 1,000ms on macOS. Once both snapshots use schema v2, the table's dual budgets apply.
+The warm refresh and warm time-to-first P95 budgets were recalibrated in issue #511 after PR #510 separated cold and warm samples. Three unchanged-code PR runs plus the exact schema-v2 baseline at `ad7ca14` retain at least 2.5 times the observed absolute run-to-run range.
+
+Schema v2 records `full_refresh` and `time_to_first_env` from the warm member of each pair and adds cold refresh/time-to-first distributions. During its one-time rollout, comparisons against a schema-v1 base checked cold P50 against explicit absolute ceilings of 500ms on Linux, 750ms on Windows, and 1,000ms on macOS. Schema-v2-to-v2 comparisons use the table's dual budgets.
+
+The cold P50 budgets were calibrated in issue #509 using two unchanged-head all-platform runs and the final pull-request validation.
 
 The dual budget avoids failing on tiny percentage changes while still blocking material latency regressions. Warm tail metrics remain mandatory; cold P95 remains diagnostic because a single host event can dominate it, while cold P50 blocks delays that affect the independent cold iterations consistently.
 

scripts/quality_snapshot.py

@@ -84,25 +84,25 @@ def regressed(self) -> bool:
         RegressionBudget(5, 100),
         RegressionBudget(50, 200),
         RegressionBudget(25, 30),
-        RegressionBudget(1_000, 100),
+        RegressionBudget(50, 50),
         RegressionBudget(20, 100),
-        RegressionBudget(250, 100),
+        RegressionBudget(25, 100),
     ),
     'windows': (
         RegressionBudget(10, 50),
         RegressionBudget(50, 100),
         RegressionBudget(50, 30),
-        RegressionBudget(5_000, 100),
+        RegressionBudget(250, 100),
         RegressionBudget(25, 50),
-        RegressionBudget(500, 100),
+        RegressionBudget(100, 100),
     ),
     'macos': (
         RegressionBudget(100, 50),
         RegressionBudget(750, 100),
         RegressionBudget(100, 50),
-        RegressionBudget(1_000, 50),
+        RegressionBudget(300, 100),
         RegressionBudget(150, 50),
-        RegressionBudget(750, 100),
+        RegressionBudget(250, 100),
     ),
 }
 PERFORMANCE_METRICS_SCHEMA_VERSION = 2

scripts/tests/test_quality_snapshot.py

@@ -158,6 +158,53 @@ def test_newer_performance_schema_is_invalid(self):
                 'Windows',
             )
 
+    def test_schema_v2_warm_p95_variance_passes_on_all_platforms(self):
+        cases = (
+            ('Linux', 60, 69, 16, 16),
+            ('Windows', 109, 206, 24, 47),
+            ('macOS', 160, 270, 133, 228),
+        )
+        for platform, base_refresh, current_refresh, base_first, current_first in cases:
+            with self.subTest(platform=platform):
+                baseline = performance_snapshot(
+                    schema_version=2,
+                    refresh_p95=base_refresh,
+                    first_p95=base_first,
+                )
+                current = performance_snapshot(
+                    schema_version=2,
+                    refresh_p95=current_refresh,
+                    first_p95=current_first,
+                )
+
+                _, failures = compare_performance(current, baseline, platform)
+
+                self.assertEqual(failures, [])
+
+    def test_schema_v2_warm_p95_budgets_reject_multi_second_regressions(self):
+        cases = (
+            ('Linux', 60, 16),
+            ('Windows', 109, 24),
+            ('macOS', 160, 133),
+        )
+        for platform, base_refresh, base_first in cases:
+            with self.subTest(platform=platform):
+                baseline = performance_snapshot(
+                    schema_version=2,
+                    refresh_p95=base_refresh,
+                    first_p95=base_first,
+                )
+                current = performance_snapshot(
+                    schema_version=2,
+                    refresh_p95=2_000,
+                    first_p95=1_000,
+                )
+
+                _, failures = compare_performance(current, baseline, platform)
+
+                self.assertTrue(any('Full refresh P95' in failure for failure in failures))
+                self.assertTrue(any('Time to first environment P95' in failure for failure in failures))
+
     def test_post_fix_macos_tail_variance_passes(self):
         baseline = performance_snapshot(startup_p95=621, refresh_p95=1_343, first_p95=649)
         current = performance_snapshot(startup_p95=691, refresh_p95=1_435, first_p95=745)
@@ -168,7 +215,7 @@ def test_post_fix_macos_tail_variance_passes(self):
 
     def test_tightened_macos_tail_budgets_reject_multi_second_regressions(self):
         baseline = performance_snapshot(startup_p95=621, refresh_p95=1_343, first_p95=649)
-        current = performance_snapshot(startup_p95=1_500, refresh_p95=2_500, first_p95=1_500)
+        current = performance_snapshot(startup_p95=1_500, refresh_p95=3_000, first_p95=1_500)
 
         _, failures = compare_performance(current, baseline, 'macOS')