Skip to content

Add mutation testing (mutmut) for test-quality enforcement - #7

Closed
MSAdministrator wants to merge 1 commit into
mainfrom
feature/mutation-testing
Closed

Add mutation testing (mutmut) for test-quality enforcement#7
MSAdministrator wants to merge 1 commit into
mainfrom
feature/mutation-testing

Conversation

@MSAdministrator

Copy link
Copy Markdown
Owner

Summary

Adds mutation testing to measure test quality, not just coverage: does a test actually fail when a bug is introduced? Coverage says a line ran; mutation says a bug on that line would be caught.

What's included

  • [tool.mutmut] config — mutates src/domain_profiler, skips __main__.py/logger.py (plumbing with no test-assertable behavior), disables coverage plugins during the per-mutant forked runs (large speedup).
  • Makefile targets: mutation (whole package), mutation-module MODULE=<name> (fast scoped run), mutation-survivors (the test-gap work list), plus mutation-results / mutation-html / mutation-clean.
  • mutmut dev dependency; mutmut artifacts gitignored.

Demo result (proves it works)

Scoped run on caa.py: 197 mutants, 126 killed, 71 survived (~64%). The survivors are real, actionable gaps — e.g. all 41 _query_caa mutants survived because the CAA tests mock _query_caa entirely, so its rdata-decoding/exception logic is never exercised by a real test. Exactly the signal this is meant to surface.

Workflow

The full sufficiency → necessity process (kill survivors to prove tests are sufficient; then remove redundant "slop" tests, gated on a mutation re-run that proves the score held) is encoded in a reusable mutation-testing skill with three mandatory safety gates (per-mutant granularity, post-deletion re-run, coverage intersection). This PR is the per-project wiring; killing the 71 caa.py survivors is follow-up work.

  • make test: 236 passed.

🤖 Generated with Claude Code

Sets up mutation testing to measure whether tests would *catch* bugs,
not just execute code. Config in [tool.mutmut]: mutates src/domain_profiler,
skips CLI/logger plumbing, disables coverage during per-mutant runs.

Makefile targets:
- mutation            whole-package run (slow)
- mutation-module     scope to one module, e.g. MODULE=caa
- mutation-survivors  list surviving mutants (the test-gap work list)
- mutation-results / mutation-html / mutation-clean

Adds mutmut to the dev group; gitignores mutmut artifacts.

The full sufficiency->necessity workflow (kill survivors, then remove
redundant tests gated on a mutation re-run) lives in the reusable
mutation-testing skill.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MSAdministrator

Copy link
Copy Markdown
Owner Author

Closing — we want to run the mutation-testing skill as an assessment, not commit the tooling into this project. The skill lives in ~/.claude/skills (and security-skills); running it here is a local exercise.

@MSAdministrator
MSAdministrator deleted the feature/mutation-testing branch July 2, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant