Problem
gaze quality --analyzer snake-eyes calls the analyze protocol method to fetch side effects but does not call classify_signals, even when the external analyzer advertises classify_signals: true in its initialize response.
Because ComputeContractCoverage in internal/quality/coverage.go treats unclassified effects as contractual (conservative assumption, line 47–48), every side effect — including CallbackInvocation (constructor calls) and ContainerMutation (internal list/dict mutations) — lands in the contract coverage denominator. This inflates the gap count and deflates coverage scores.
Evidence
Running gaze quality --analyzer snake-eyes --language python . against the snake-eyes repo itself:
- 326 tests analyzed, 28% average contract coverage
- 729
CallbackInvocation gaps — almost all are "Ambiguous call to ClassName" (constructors), most of which would likely classify as Incidental
- 277
ContainerMutation gaps — internal state mutations, also likely Incidental
- 915
ReturnValue gaps — many of which would remain Contractual, but the denominator is bloated by the unclassified Incidental effects above
Snake-eyes correctly advertises classify_signals: true in its initialize response and the classify_signals method works correctly when called directly.
Root Cause
ExternalSideEffectAnalyzer in internal/adapter/sideeffect.go calls analyze but never calls classify_signals. The AllResults() method returns effects without classification labels. When these reach ComputeContractCoverage, line 47 (if e.Classification != nil && e.Classification.Label == taxonomy.Incidental) never triggers because Classification is always nil.
Proposed Fix
In the gaze quality external analyzer path (runQualityWithExternalAnalyzer in cmd/gaze/main.go), after fetching analysis results via providers.SideEffects.AllResults():
- Check
providers.Capabilities.ClassifySignals
- If true, call the
classify_signals protocol method for each function's effects
- Attach the returned classification labels to the
taxonomy.SideEffect objects before passing them to BuildQualityFromMappings
This follows the same conditional pattern used for test_mapping (line ~30 of session.go:Initialize).
Impact
HIGH — with classification, the ~28% average contract coverage on snake-eyes should rise to an estimated 60–70% by correctly excluding Incidental effects from the denominator. Without this fix, external analyzer quality reports are misleadingly pessimistic.
Related
Problem
gaze quality --analyzer snake-eyescalls theanalyzeprotocol method to fetch side effects but does not callclassify_signals, even when the external analyzer advertisesclassify_signals: truein itsinitializeresponse.Because
ComputeContractCoverageininternal/quality/coverage.gotreats unclassified effects as contractual (conservative assumption, line 47–48), every side effect — includingCallbackInvocation(constructor calls) andContainerMutation(internal list/dict mutations) — lands in the contract coverage denominator. This inflates the gap count and deflates coverage scores.Evidence
Running
gaze quality --analyzer snake-eyes --language python .against the snake-eyes repo itself:CallbackInvocationgaps — almost all are "Ambiguous call toClassName" (constructors), most of which would likely classify as IncidentalContainerMutationgaps — internal state mutations, also likely IncidentalReturnValuegaps — many of which would remain Contractual, but the denominator is bloated by the unclassified Incidental effects aboveSnake-eyes correctly advertises
classify_signals: truein itsinitializeresponse and theclassify_signalsmethod works correctly when called directly.Root Cause
ExternalSideEffectAnalyzerininternal/adapter/sideeffect.gocallsanalyzebut never callsclassify_signals. TheAllResults()method returns effects without classification labels. When these reachComputeContractCoverage, line 47 (if e.Classification != nil && e.Classification.Label == taxonomy.Incidental) never triggers becauseClassificationis alwaysnil.Proposed Fix
In the
gaze qualityexternal analyzer path (runQualityWithExternalAnalyzerincmd/gaze/main.go), after fetching analysis results viaproviders.SideEffects.AllResults():providers.Capabilities.ClassifySignalsclassify_signalsprotocol method for each function's effectstaxonomy.SideEffectobjects before passing them toBuildQualityFromMappingsThis follows the same conditional pattern used for
test_mapping(line ~30 ofsession.go:Initialize).Impact
HIGH — with classification, the ~28% average contract coverage on snake-eyes should rise to an estimated 60–70% by correctly excluding Incidental effects from the denominator. Without this fix, external analyzer quality reports are misleadingly pessimistic.
Related
--analyzertogaze quality)classify_signalsmethod