Skip to content

Feed the model diffs and file stats, not just commit subjects #7

Description

@tabkram

Why

parseCommits in src/git.ts:52 gives the model only the commit subjects. On most repositories those are fix stuff and wip, and the model is asked to write a customer-facing note from them.

This is the largest quality gap in the tool, and where the closest competitors are ahead: @entro314labs/ai-changelog-generator and github/copilot-release-notes both read diffs. There is no diff access anywhere in this codebase.

Prerequisite

getChangelog returns string[] and parseCommits hardcodes hash: "" — the hash is thrown away, so there is nothing to hang a diff on. Commits must travel as objects first. No user-visible change, but it touches src/git.ts, src/generator.ts, src/types.ts and the tests.

Side effect: --changelog and --changelog-file produce hashless commits, so they are never enriched. Degradation for free.

Scope

  1. File stats — --numstat: which files changed and by how much. Cheap, high signal. One git log --numstat <from>..<to> for the whole range, not one process per commit.
  2. Small-commit diffs — full diff under a line threshold, where the code is the summary.
  3. Budget — degrade instead of blowing up a large release.

Config

git:
  maxCommits: 200

  # What the model receives for each commit.
  #
  # auto (default): messages + files everywhere, diff wherever the message is
  #                 poor (wip, fix stuff…), within diffBudget.
  #
  # Or an explicit list:
  #   messages — the commit subject
  #   files    — files touched, with +/- line counts
  #   diffs    — full diff of commits under the threshold
  commitDetail: auto

  diffBudget: 30000

A list rather than a single level so it can subtract: on a repo where every subject is wip, [files, diffs] drops the noise and keeps the code.

auto spends the budget on the commits whose message says least — feat(auth): add SSO login already explains itself. An explicit list is reproducible to the token, which the #6 fixtures need. Ties break on hash.

CLI: --commit-detail files,diffs.

This also removes git.commitFormat — declared at src/types.ts:90, read nowhere. Backward compatible, since Zod strips unknown keys.

Safety

  • Diffs are untrusted input: same labelled blocks and neutralizeDelimiters as the changelog (source files can contain =====).
  • Redact hunks against SECRET_CONTENT_PATTERNS in src/context.ts — a diff adding an API key would otherwise ship it to the LLM.
  • Skip lockfiles, dist/, minified and binary files.

Acceptance criteria

  • Commits carry their hash end to end; existing tests pass unchanged
  • commitDetail and diffBudget documented in docs/configuration.md, defaulting to auto
  • commitFormat removed; a config still containing it loads without error
  • Diffs labelled as untrusted and redacted against SECRET_CONTENT_PATTERNS
  • Budget degradation is deterministic and reported
  • Add an eval harness for release-note quality #6 fixtures show measurably better output on the messy-history fixtures
  • --dry-run shows what was included, the estimated cost, and what was dropped

Files

src/git.ts, new src/git-context.ts, src/generator.ts, src/prompts/builder.ts, src/types.ts, src/bin/cli.ts, docs/configuration.md

Relates to

The mechanism — hash plumbing, collection, budget, redaction, --dry-run — is unit-testable and does not depend on #6. Only the tuning does: the small-commit threshold, whether auto beats an explicit list, how stats read best in the prompt. Land the mechanism, tune once the harness exists.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:gitGit reading and parsingarea:llmModel calls, prompts, cachingeffort:MA few daystier:qualityImproves the quality of generated notes

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions