Recalibrate warm P95 budgets after schema-v2 baselines

Problem

PR #510 changes the existing full_refresh and time_to_first_env distributions from one cold + nine warm samples to ten independently paired warm samples. The current P95 budgets were calibrated against the old mixed distribution and will become too permissive once schema-v2 baselines land.

Three schema-v2 hosted runs report:

Platform	Warm refresh P95	Warm time-to-first P95	Current absolute budgets
Linux	60 / 58 / 69ms	15 / 16 / 15ms	1,000ms / 250ms
Windows	171 / 143 / 206ms	38 / 28 / 47ms	5,000ms / 500ms
macOS	198 / 270 / 171ms	166 / 228 / 136ms	1,000ms / 750ms

After #510, those old limits could allow multi-second warm-cache drift even though cold performance is gated separately.

Scope

After #510 merges and exact schema-v2 main baselines exist:

inspect repeated warm refresh/time-to-first baseline variance on all platforms;
tighten warm P95 dual budgets with measured hosted-runner headroom;
keep P50, server-startup, and cold-refresh budgets unchanged unless independent evidence requires adjustment;
update comparator tests and docs/QUALITY_SNAPSHOTS.md;
retain the schema-v1 migration behavior only where still required.

Acceptance criteria

Repeated unchanged-head schema-v2 runs stay green.
Synthetic multi-second warm refresh and warm time-to-first regressions fail on every platform.
Cold P50 gating remains independent and unchanged.
Missing metrics, inventory mismatches, and malformed snapshots still fail closed.
Full CI, coverage, performance, and CodeQL workflows pass.

Depends on #510.

Exact merged schema-v2 baseline run 31516981874 passed at ad7ca14. Combined with the three unchanged-code PR runs:

Platform	Warm refresh P95 range	Warm time-to-first P95 range
Linux	58-69ms	15-16ms
Windows	109-206ms	24-47ms
macOS	160-270ms	133-228ms

Proposed dual budgets:

Platform	Warm refresh P95	Warm time-to-first P95
Linux	50ms / 50%	25ms / 100%
Windows	250ms / 100%	100ms / 100%
macOS	300ms / 100%	250ms / 100%

Against the merged baseline these block above roughly 110ms/41ms on Linux, 359ms/124ms on Windows, and 460ms/383ms on macOS. Absolute headroom is at least 2.5x each observed four-run range while reducing old effective ceilings by 48-95%.

Cold P50, P50, and server-startup budgets remain unchanged. Tests will prove each observed worst run passes and multi-second warm regressions fail on every platform.

Final quality conclusion for PR #512 at 135cb20:

Exact-head Copilot review: 3/3 files, 0 comments, 0 unresolved threads.
Full CI, lint, CodeQL, Linux/Windows coverage, and all performance jobs pass.
Linux warm refresh/time-to-first P95: 53/12ms vs 60/16ms baseline.
Windows rerun warm P95: 122/23ms vs 109/24ms baseline.
macOS warm P95: 150/111ms vs 160/133ms baseline.
All artifacts contain 10 samples, matching environment/manager inventories, schema v2, and empty interpreter-timeout maps.
Line/function coverage is unchanged on Linux and Windows.
Comparator suite: 33/33 passed; exact schema-v2 baselines pass and synthetic multi-second warm regressions fail on every platform.
The first Windows attempt failed only the unchanged warm P50 gate (164ms vs 105ms baseline); the isolated rerun passed at 117ms. That calibration issue is tracked separately in Recalibrate Windows warm P50 budget for schema-v2 variance #513 rather than widening test: tighten schema-v2 warm P95 budgets (Fixes #511) #512's P95 scope.

test: tighten schema-v2 warm P95 budgets (Fixes #511) (#512)

Navigation Menu

Repository navigation

Recalibrate warm P95 budgets after schema-v2 baselines #511

Problem

Scope

Acceptance criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Recalibrate warm P95 budgets after schema-v2 baselines #511

Description

Problem

Scope

Acceptance criteria

Activity

karthiknadig commented on Aug 11, 2026

karthiknadig commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions