[andreujuanc]

Summary

Five related threads on this branch:

1. New ai-slop category (internal/ai/prompts/categories/ai-slop.md)

Covers AI-generated code smells the existing categories don't catch cleanly: bug-narration comments, commit-message prose in code, vacuous comments, defensive noise, single-use abstractions, performative naming, ceremonial error wrapping. Category count goes 22 → 23.

2. Type-safe state.Category boundary

Introduces state.Category as the validated category type. UnmarshalJSON is the single validation point — downstream code can rely on every Category value being either a known slug or the zero value. The old scattered validators (filterValidCategories, normaliseCategory, the out-of-taxonomy log in validateAOIs) are removed.

Deep-review parsing splits into two stages:

• parseDeepReviewRaw → rawDeepReviewResult with Category as raw string
• convertRawToTyped → promotes via state.ParseCategory after the corrector runs

Between the two stages, verifyAndCorrectCategory loops up to 2 round trips on the same chat thread, asking the model to fix off-list categories or withdraw findings. A new applyTypoFixes pre-pass repairs edit-distance-1 slugs (e.g. correctnes → correctness) with zero LLM calls (levenshtein + cached nearestSlug). RetryTransient wraps the corrector call so 5xx blips don't silently lose findings.

rawDeepFinding embeds state.DeepFinding (shadowing Category as string) so adding fields to DeepFinding doesn't require updates in three places. ParseDeepReviewResult now returns []DroppedFinding so callers can see how many findings were lost to off-list categories instead of relying on log scraping.

AreaOfInterest.Category is typed too; AOIScanResult.UnmarshalJSON parses AOIs one at a time so a single off-list category drops that AOI alone instead of failing the whole file's batch.

The follow-up refactor commit (50c48fa) tightens the corrector internals: bundles corrector params into a correctorContext struct, embeds DeepReviewResult instead of duplicating fields, and skips re-listing allowed categories on attempt 2+ to save tokens.

3. Reflected-XSS prompt shape (33339b2)

The input-validation category listed only client-side XSS shapes (innerHTML, v-html, template engines). Server-rendered XSS where user input is interpolated directly into the response body — fmt.Fprintf, ResponseWriter.Write, framework equivalents — wasn't documented, so models had no shape-match for that pattern. One-line addition.

4. Benchmark infrastructure overhaul (b26f846, 7feab45)

• Archive layout: ~/.config/prr/benchmarks/<name>-benchmark-<datetime>.json so AOI and deep-review runs coexist. SaveBenchmarkResults / LoadBenchmarkResults are name-keyed.
• Scoring: replace first-match-wins greedy matcher with independent per-GT assertion in both suites. The old logic silently under-reported recall when ground-truth entries overlapped — an over-wide handler.go GT was swallowing matches for narrower XSS / SQLi GTs.
• Config independence: both benchmarks stop reading tuning from ~/.config/prr/models.json. Tuning is now fixed by env-overridable benchmark constants (PRR_BENCH_* for AOI, PRR_DEEP_* for deep review). Stop auto-creating models.json on first read.
• Deep-review parity: cross-product fan-out (temperature × thinking budget) matching AOI; skip thinking variants for providers that decide internally (github-copilot, claude-code); persist dated archives (previously only logged to test output); schema extended with severity_correct, severity_total, tool_calls.
• Pricing: provider-qualified model IDs for EstimateCost. Bare-ID lookup preferred priced entries, so subscription-billed claude-code runs were being charged at API rates.
• Model defaults: gemini-3.1-flash-lite fast thinking 2048 → 0 (no recall benefit, +1s latency); gemini-3.5-flash fast thinking 1536 → 1024 (avoids the thinking=0 latency quirk; same recall). Refreshed aoi-benchmark.json with post-matcher-fix numbers.
• Tracking: every benchmark archive now records HEAD short SHA, whether the working tree was dirty, and an optional free-form tag from PRR_BENCH_TAG. Makes archives unambiguously traceable to the prompt/code state that produced them.

5. Prompt rework (9e24b02, a1aff88)

Two research-driven prompt changes shipped after per-change bench gating. Two more were planned, evaluated, and dropped (one reverted as net-regressive, one skipped on architectural grounds). Full per-change tables in plans/benchmark-results-prompt-rework-2026-05-22.md.

• Path-feasibility dismissal in recheck (9e24b02). Inspired by LLM4PFA (arXiv:2506.10322). The adversarial recheck pass now walks the existing trace field from suspect → boundary and dismisses with rationale infeasible-path only when a specific hop can be shown to contradict the bug's required inputs. Touches recheck.md + recheck_dismiss.md.
• Sources / sinks / sanitizers on security AOIs (a1aff88). Inspired by IRIS (arXiv:2405.17238). For the six security-shaped categories (input-validation, external-io, authorization, authentication, cryptography, web-security), the AOI scanner emits short identifier-shaped lists describing the taint shape. An empty sanitizers list is the signal of a real bug. Reviewer prompts (individual + grouped) require evidence to trace at least one source→sink end-to-end. Scoped to security categories to avoid the +52% AOI latency hit a wider scope produced.

Prompt-rework deltas

Benched on gemini-3.1-flash-lite (AOI, 4 reps baseline / 2 reps per change) and claude-opus-4-7 via Claude Code (deep review, 2–4 reps per change). Medians:

	Baseline	After Change 1 (recheck)	After Change 1 + 3 (HEAD)
Deep-review recall	100%	100%	100%
Deep-review must-find	100%	100%	100%
Deep-review FP	1	0.5	0.5
Deep-review SevAcc	50%	69%	72% (+22pp, peaks 75–77%)
Deep-review latency	516s	550s	534s
AOI must-find	95%	95%	100%
AOI latency	4.6s	4.6s	5.0s (+10%)

Two deep-review reps hit 0 FP — a first on this fixture. One rep tied the all-time SevAcc peak at 77%.

Why the other two changes didn't land

• AOI verification questions (reverted before commit). +52% AOI latency and regressed Change 1's SevAcc 69% → 60% — the per-AOI question-list bookkeeping ate the reviewer's attention budget for severity reasoning. The latency-on-AOI gate (+15%) caught it before the SevAcc cost became obvious.
• Cross-pass agreement filter (skipped). The Multi-Review (arXiv:2509.01494) approach assumes N independent passes per AOI; prr's ParallelReviews knob is concurrency, not redundancy. The existing deep review → adversarial recheck pipeline is already the heterogeneous-judge pattern Multi-Review prescribes, and Change 1 strengthened it. Documented for follow-up.

Context

Before this branch, "category is one of the allowed slugs" was checked in at least three places with subtly different behaviour (drop vs. coerce vs. log). The corrector loop made that worse — the LLM was being asked to fix categories that downstream code had already rewritten or dropped. Centralizing validation in Category.UnmarshalJSON is what makes the corrector well-defined: there is exactly one moment where "unknown slug" can be observed, and the corrector owns it. The benchmark overhaul gave us a sharp enough signal to land the prompt-rework changes with per-change gates instead of vibes — the FP/SevAcc deltas above would have been invisible under the old greedy matcher.