fix(#459): the AAA gate counted "could not measure" as "passed" — now it measures - #779
Merged
Merged
Conversation
… it measures
axe returns a PASS for any element whose ratio it could not compute, with
`contrastRatio: null`. Those land in `passes`, so a gate asserting only on
`violations` treats an unanswered question as a verified one. Measured: 193 such
nodes across six routes — one in five of everything reported as passing.
The spec already DETECTED this and printed a warning. A printed number can grow
for months unread, so it closed nothing.
EVERY unmeasured node was text on a gradient — one cause, so one technique.
A gradient interpolates between ADJACENT stops, so luminance along the ramp
always lies between the two bounding it; checking every declared stop covers the
extremes without rasterising. Resolving the colours needs canvas readback:
getComputedStyle hands back oklch()/oklab() verbatim and this codebase is
entirely oklch, so parsing as RGB yields nonsense.
WHAT IT FOUND, all previously green:
4.21:1 /pricing "Select" — the buy button, 11.84px/600 needing 7:1
6.07:1 /sign-in "Create an account"
6.8:1 /sign-in "Forgot password?"
1.66:1 /blog/seo — text-base-content and .badge overriding .alert-warning's
own content colour: dark text on dark amber
All fixed here so the gate lands green rather than red-on-arrival. The pricing
gradient's dark stop moved 78% -> 96%, solved numerically (96% is the FIRST
passing value at 7.11:1) rather than eyeballed; the consequence, stated in the
CSS, is that a visible downward darkening and AAA cannot both hold with this
token pair.
FOUR OF MY OWN BUGS, each caught by verifying rather than by review:
1. A regex layer-splitter `/,(?![^()]*\))/` also split inside `rgba(0, 0, 0, 0)`,
shredding a chevron gradient. The transparent stop landed in one fragment and
the opaque arrow colour in another, so the arrow read as a surface: fg and bg
both resolved to base-content and scored 1:1 on selects that are perfectly
legible. Replaced with paren-depth counting.
2. Treating `"none, none"` as a gradient sent 30 nodes down the stop-parsing path
and reported them unresolvable.
3. The allowlist regex `^button\.btn\.` missed `a.btn.btn-primary` — the gate
caught its own too-narrow entry, which is the mechanism working.
4. Every local run before the root build measured the 404 PAGE. BASE_URL with a
basePath plus an absolute goto resolves to a path the dev server does not
serve, so all 47 routes reported an identical 31 passes. Mutation M1 "passed
on broken code" because of it. This is the #391 trap; verification now
requires DISABLE_BASE_PATH=true AND an empty NEXT_PUBLIC_BASE_PATH, because
next.config.ts:15 lets the env var win over the detected config.
VERIFIED BY MUTATION, against a real root build served at root:
pricing gradient back to 78% -> RED at 4.21:1 on all three SKUs
restored to 96% -> green
canvas readback -> rgb() parsing -> 8 failed
allowlist entry removed -> that node fails
visibility filter disabled -> NO CHANGE (see below)
The visibility filter is documented as NOT load-bearing, because the mutation
says so. It was added for a 1:1 select that turned out to be bug (1); the
surface-layer filter fixes that independently. Kept for the principle, credited
with nothing.
Vendor chrome is excluded by name with a reason, not by threshold: Leaflet's
attribution control measures 4.94:1 and we neither set nor can change its
colours. Scoped to the ELEMENT so the rest of /map stays measured — the spec's
route-level EXCLUDED map already takes this position for Cesium.
Full sweep: 81 passed, 0 failed (47 routes x 2 themes, admin excluded locally —
those seed against the shared Supabase and CI covers them).
pnpm test 4669/4669 · type-check · lint clean
Closes #459
Closes #778
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #459
Closes #778
The defect
axe returns a pass for any element whose ratio it could not compute, with
contrastRatio: null. Those land inpasses, so a gate asserting only onviolationstreats an unanswered question as a verified one — the one direction a probe must never
round. 193 such nodes across six routes: one in five of everything reported as passing.
The spec already detected this and printed a warning. A printed number can grow for months
unread, so it closed nothing.
The technique
Every unmeasured node was text on a gradient — one cause, so one technique.
A gradient interpolates between adjacent stops, so luminance along the ramp always lies
between the two bounding it. Checking every declared stop covers the extremes; no rasterising
needed. Resolving the colours requires canvas readback —
getComputedStylehands backoklch()/oklab()verbatim and this codebase is entirely oklch, so parsing as RGB yieldsnonsense.
What it found, all previously green
/pricing"Select" — the buy button (11.84px/600)/sign-in"Create an account"/sign-in"Forgot password?"/blog/seo—text-base-contentand.badgeoverriding.alert-warning's own content colour: dark text on dark amberAll fixed here so the gate lands green, not red-on-arrival.
The pricing gradient's dark stop moved 78% → 96%, solved numerically rather than
eyeballed — 96% is the first passing value, at 7.11:1. The CSS records the honest
consequence: with a light
--copperand a dark--on-copper, a visible downwarddarkening and AAA cannot both hold.
Four of the bugs were mine, and verification caught all four
/,(?![^()]*\))/also split insidergba(0, 0, 0, 0),shredding a chevron gradient. The transparent stop landed in one fragment and the opaque
arrow colour in another, so the arrow read as a surface: fg and bg both resolved to
base-contentand scored a perfect 1:1 on selects that are perfectly legible(confirmed by screenshot). Replaced with paren-depth counting.
"none, none"treated as a gradient — 30 nodes wrongly reported unresolvable.^button\.btn\.misseda.btn.btn-primary— the gate caught itsown too-narrow entry, which is the mechanism working.
BASE_URLwith abasePath plus an absolute
gotoresolves to a path the dev server does not serve, so all47 routes reported an identical 31 passes, and mutation M1 "passed on broken code" because
of it. This is the trap that got The 'all form inputs have labels' gate never excludes hidden inputs: its filter({hasNot}) has no effect #391 retracted. Verification requires
DISABLE_BASE_PATH=trueand an emptyNEXT_PUBLIC_BASE_PATH, becausenext.config.ts:15lets the env var win over the detected config.Verified by mutation, against a real root build served at root
rgb()parsingThe visibility filter is documented as NOT load-bearing, because the mutation says so. It
was added for a 1:1 select that turned out to be bug (1); the surface-layer filter fixes that
independently. Kept for the principle, credited with nothing.
Vendor chrome: excluded by name, never by threshold
Leaflet's attribution control measures 4.94:1 and we neither set nor can change its colours.
Excluded as a named entry with a written reason, scoped to the element so the rest of
/mapstays measured — the spec's route-levelEXCLUDEDmap already takes this position forCesium's error panel.
Same for
UNRESOLVABLE_ALLOWLIST: asserted as a set, not a count, so a new unmeasurableelement fails even while the total holds steady.
Verification
/adminexcluded locally becauseit seeds against the shared Supabase — CI covers it)
pnpm test4669/4669 · type-check · lint cleanNote for reviewers: this gate runs in the E2E lane, not the
accessibilitycheck — thatjob runs pa11y, which ignores both contrast rules entirely.
🤖 Generated with Claude Code