docs: implement changelog generation audit findings (#7522)#7667
Open
docs: implement changelog generation audit findings (#7522)#7667
Conversation
Address all 6 findings from the changelog generation audit: 1. Dual PR number extraction - use last PR number as canonical reference 2. Enforce PR links on every changelog entry in Step 5 validation 3. Cross-release deduplication check in Step 4 (PR + issue numbers) 4. Document alpha/beta feature gating in exclusion rules 5. Default borderline user-facing changes to included 6. Post-generation cross-reference validation against commit range Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a Go tool that retroactively audits past changelog releases against the updated generation rules, producing a side-by-side comparison report. The tool checks for: - F1: Dual PR number extraction (use last as canonical) - F2: Missing PR links on changelog entries - F2b: Issue links used instead of PR links - F3/F3b: Cross-release and intra-release duplicate entries - F4: Alpha/beta feature gating verification - F5: Borderline excluded commits that should be included - F6: Phantom entries (PRs not in commit range) - F6b: Link text/URL number mismatches Results across 20 releases (1.22.2-1.23.15): - 9 errors (all missing PR links in 1.23.9, 1.23.13) - 18 warnings (phantoms, borderline exclusions, duplicates) - 2 info (alpha/beta gating checks) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enhance the changelog audit tool to generate GitHub-renderable diff blocks showing exactly how each release section would change under the new rules. Each corrected line includes annotations (e.g., ← F2: add missing PR link) so reviewers can see the rule justification at a glance. Changes: - Add correctRelease() engine that mechanically applies all 6 finding rules - Add renderUnifiedDiff() that produces unified diff with annotations - Finding struct now carries LineNumber, OldPR, NewPR for precise corrections - Report regenerated with diff blocks for all 20 audited releases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jongio
requested review from
danieljurek,
rajeshkamal5050,
tg-msft,
vhvb1989 and
wbreza
as code owners
📋 Milestone: April 2026This work is tracked for April 2026. The team will review it soon! |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the changelog-generation skill documentation to incorporate the audit findings from #7522 (rules/validation gates around PR number selection, deduplication, alpha/beta gating, and inclusion defaults) and adds a standalone Go-based audit tool under .github/skills/changelog-generation/audit/ to retroactively check recent releases and emit a Markdown report.
Changes:
- Extend
SKILL.mdandreferences/pr-processing.mdwith new rules/validation steps for deduplication, PR-link validation, alpha/beta gating, and “borderline include” guidance. - Add a Go audit tool (
main.go+go.mod) with docs and a generatedreport.mdexample output. - Add minimal repo hygiene for the tool (
.gitignorefor Windows executables).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
.github/skills/changelog-generation/SKILL.md |
Adds cross-release dedup + PR-link validation + commit-range cross-check steps to the skill workflow. |
.github/skills/changelog-generation/references/pr-processing.md |
Adds dual-PR-number extraction guidance and expands exclusion/include heuristics (alpha/beta + borderline defaults). |
.github/skills/changelog-generation/audit/main.go |
Implements the audit engine, report generator, and “side-by-side diff” rendering. |
.github/skills/changelog-generation/audit/README.md |
Documents running the audit tool and explains checks/flags. |
.github/skills/changelog-generation/audit/report.md |
Adds a generated sample audit report for reviewers. |
.github/skills/changelog-generation/audit/go.mod |
Introduces a standalone Go module for the audit tool. |
.github/skills/changelog-generation/audit/.gitignore |
Ignores Windows .exe outputs in the audit tool folder. |
.github/skills/changelog-generation/references/pr-processing.md
Outdated
Show resolved
Hide resolved
.github/skills/changelog-generation/references/pr-processing.md
Outdated
Show resolved
Hide resolved
Replace monolithic report.md with per-release comparison files: - published/<version>.md: verbatim changelog extract from CHANGELOG.md - corrected/<version>.md: deterministic corrections only (F1, F2b, F3, F6, F6b) - findings.md: clean findings report without embedded diffs Key improvements: - No more fake [[#???]] placeholders — only apply corrections when the right answer is provable (e.g., F1 PR swap, F6 phantom removal) - Per-release files are small and diffable with standard tools - Findings use indented code blocks to avoid backtick nesting issues - 5 of 20 releases have corrections; 15 are identical Reviewers: diff -ru published/ corrected/ to see all corrections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Reorder PR extraction rules: check dual PR numbers first (before single-match squash merge) to avoid ambiguity when multiple (#NNNN) patterns are present - Add warning log when changelog entries contain multiple PR links (only first is tracked; affects F3/F6 accuracy for older releases) - Regenerate findings.md with updated commit range display Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the 6 audit findings from #7522, which audited 10 releases (1.23.5–1.23.14) and identified systematic issues in changelog generation.
What Changed
Skill Rule Updates
pr-processing.md — 3 new rules:
SKILL.md — 3 new validation steps:
[[#NNN]](url)PR link before outputBacktesting Audit Tool
A standalone Go tool at
.github/skills/changelog-generation/audit/that retroactively audits the last 20 releases against all 6 rules and generates a detailed report with:The diff blocks render with GitHub's syntax highlighting (red/green coloring) and include rule annotations (e.g.,
← F2: add missing PR link) so reviewers can see the justification for each change at a glance.Key Findings from 20-Release Audit
How to Review
pr-processing.mdandSKILL.mddiffs for the 6 new rulesaudit/report.mdand scroll to any release's Side-by-Side Diff section to see how the rules would have changed that release's changelogaudit/main.gofor the backtesting logicHow to Run the Audit Tool
Closes #7522