Repository navigation
Rework AOI benchmark scoring around coverage and hallucination - #34
Merged
Merged
Conversation
Replaces the false-positive penalty (which contradicted the AOI scanner's recall-biased contract) with a three-way classifier (covered / aligned / hallucinated) plus an avg-line-offset precision metric. Adds TestAOIBenchmarkFixturesValid as a CI sanity test that validates every GT line range against the fixture diff; the audit that came out of running it tightened 18 of the original GT entries to bug-precise single lines and surfaced 5 new entries from a manual review of one model's previously-uncredited output. Provider work: - New opencode provider (internal/ai/opencode.go) routes calls through the local opencode CLI, with 11 new model entries registered (3 native + 7 copilot + GPT-5.5). - ReportedCostUSD added to TokenUsage so providers with native cost reporting bypass the per-1M-token estimate. - Shadow pricing for claude-code models lets subscription-billed runs compare honestly against metered API providers. Tuning defaults from the empirical 3-rep sweep: - gemini-3.1-flash-lite temperature 0.1 -> 0.3 (greedy fell into a low-coverage attractor 40% of the time at this scale). - claude-opus-4-7 marked AOI: true and defaults to --effort xhigh (best mean coverage, tightest variance, third the wall-clock of --effort max). See docs/articles/evaluating-aoi-pre-filters.md for the methodology writeup and the variance findings.
…ctation CI feedback on the previous commit: - internal/ai/opencode.go was not gofmt-clean. - TestLoadModels_EmbeddedDefaults hardcoded Temperature: 0.1 for all models; gemini-3.1-flash-lite is now 0.3. Made the expected value per-row so the assertion matches each model's actual default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
covered/aligned/hallucinated) plus anavg_line_offsetprecision metric.TestAOIBenchmarkFixturesValidas a CI-friendly sanity test that validates every ground-truth line range against the actual fixture diffs. Tightened 18 of the original GT entries to bug-precise single lines and surfaced 5 new entries via manual review of one model's previously-uncredited output.opencodeprovider (internal/ai/opencode.go) routes calls through the local opencode CLI; 11 new model entries registered. AddedReportedCostUSDtoTokenUsageso providers with native cost reporting bypass per-1M-token estimates. Shadow pricing forclaude-codemodels lets subscription-billed runs compare honestly against metered API providers.Context
Tuning defaults from the empirical 3-rep variance sweep:
gemini-3.1-flash-litetemperature 0.1 → 0.3. Greedy decoding fell into a low-coverage attractor about 40% of the time on this fixture; 0.3 stays in the higher-coverage band consistently.claude-opus-4-7is now markedAOI: trueand defaults to--effort xhighwhen invoked through claude-code. Acrosslow | medium | high | xhigh | max, xhigh gives the highest mean coverage (91%), tightest variance (4-point range across 3 reps), and roughly a third of the wall-clock and output tokens ofmax.The methodology writeup, the variance findings (reasoning-class models swing 8 to 24 coverage points across reps of identical prompts; cheap deterministic models swing zero), and the two prompt gaps no model surfaces as AOIs (lockfile mismatch, UTF-8 byte-slice) are documented in
docs/articles/evaluating-aoi-pre-filters.md.