The conformance harness gates each RP plan on a single aggregate — expect(successfulLogins, greaterThan(0)) across all modules (packages/oidc/example/integration_test/shared_e2e.dart, ~lines 770-790). RP plans deliberately mix positive and negative modules: a negative module such as oidcc-client-test-missing-athash must end with no user. So the gate only proves that some module logged in, and a negative module that wrongly succeeds stays green as long as any positive module in the same plan still passes.
This is how the implicit-flow at_hash gap went unnoticed: in #447's CI (run 37212794378), the implicit plan's oidcc-client-test-missing-athash logged Login successful: tokens=[access, id] on linux and windows, and the job passed.
Possible fixes
- Per-module polarity table: map each module name to its expected outcome (user returned / rejected) and assert per module. Needs a table kept current across the Basic / Hybrid / Implicit / Config / Form Post / Logout plans and suite releases.
- Suite verdict: after each module, poll
getTestStatus / api/runner/{id} (already in conformance/api.dart) until the module is terminal and assert its own result. This mirrors certification, but the harness's comment (~715-726) records that reading status['status'] / status['result'] returned null for every module, so the payload shape for this suite deployment still has to be confirmed from real CI output. Also open: how long the suite takes to finalize a verdict after the redirect, and whether WARNING / REVIEW count as acceptable.
Either way, a negative module that produces a user should fail the plan.
🤖 Generated with Claude Code
https://claude.ai/code/session_0126i8ymDVXbifp6LrbCiyQL
The conformance harness gates each RP plan on a single aggregate —
expect(successfulLogins, greaterThan(0))across all modules (packages/oidc/example/integration_test/shared_e2e.dart, ~lines 770-790). RP plans deliberately mix positive and negative modules: a negative module such asoidcc-client-test-missing-athashmust end with no user. So the gate only proves that some module logged in, and a negative module that wrongly succeeds stays green as long as any positive module in the same plan still passes.This is how the implicit-flow
at_hashgap went unnoticed: in #447's CI (run 37212794378), the implicit plan'soidcc-client-test-missing-athashloggedLogin successful: tokens=[access, id]on linux and windows, and the job passed.Possible fixes
getTestStatus/api/runner/{id}(already inconformance/api.dart) until the module is terminal and assert its own result. This mirrors certification, but the harness's comment (~715-726) records that readingstatus['status']/status['result']returned null for every module, so the payload shape for this suite deployment still has to be confirmed from real CI output. Also open: how long the suite takes to finalize a verdict after the redirect, and whetherWARNING/REVIEWcount as acceptable.Either way, a negative module that produces a user should fail the plan.
🤖 Generated with Claude Code
https://claude.ai/code/session_0126i8ymDVXbifp6LrbCiyQL