Add mutation testing (mutmut) for test-quality enforcement - #7
Closed
MSAdministrator wants to merge 1 commit into
Closed
Add mutation testing (mutmut) for test-quality enforcement#7MSAdministrator wants to merge 1 commit into
MSAdministrator wants to merge 1 commit into
Conversation
Sets up mutation testing to measure whether tests would *catch* bugs, not just execute code. Config in [tool.mutmut]: mutates src/domain_profiler, skips CLI/logger plumbing, disables coverage during per-mutant runs. Makefile targets: - mutation whole-package run (slow) - mutation-module scope to one module, e.g. MODULE=caa - mutation-survivors list surviving mutants (the test-gap work list) - mutation-results / mutation-html / mutation-clean Adds mutmut to the dev group; gitignores mutmut artifacts. The full sufficiency->necessity workflow (kill survivors, then remove redundant tests gated on a mutation re-run) lives in the reusable mutation-testing skill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Closing — we want to run the mutation-testing skill as an assessment, not commit the tooling into this project. The skill lives in ~/.claude/skills (and security-skills); running it here is a local exercise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds mutation testing to measure test quality, not just coverage: does a test actually fail when a bug is introduced? Coverage says a line ran; mutation says a bug on that line would be caught.
What's included
[tool.mutmut]config — mutatessrc/domain_profiler, skips__main__.py/logger.py(plumbing with no test-assertable behavior), disables coverage plugins during the per-mutant forked runs (large speedup).mutation(whole package),mutation-module MODULE=<name>(fast scoped run),mutation-survivors(the test-gap work list), plusmutation-results/mutation-html/mutation-clean.mutmutdev dependency; mutmut artifacts gitignored.Demo result (proves it works)
Scoped run on
caa.py: 197 mutants, 126 killed, 71 survived (~64%). The survivors are real, actionable gaps — e.g. all 41_query_caamutants survived because the CAA tests mock_query_caaentirely, so its rdata-decoding/exception logic is never exercised by a real test. Exactly the signal this is meant to surface.Workflow
The full sufficiency → necessity process (kill survivors to prove tests are sufficient; then remove redundant "slop" tests, gated on a mutation re-run that proves the score held) is encoded in a reusable
mutation-testingskill with three mandatory safety gates (per-mutant granularity, post-deletion re-run, coverage intersection). This PR is the per-project wiring; killing the 71caa.pysurvivors is follow-up work.make test: 236 passed.🤖 Generated with Claude Code