Identify files with custom rules before the built-in rules

[ebursztein]

Part of #1537.

Stacked on #1529: the first commit (d96c7422, "Veto ML output when rules have no false negatives") is #1529 itself. Once #1529 merges, this branch is rebased and that commit disappears.

Two fixes to #1529, at the bottom of the stack

They are separate commits so they can move into #1529 or land here, whichever is easier.

• Fix the GPU tests for the rules veto (#1529): rust/lib/src/tests/gpu.rs only compiles on macOS or with CUDA, so CI didn't see that it still used Features.0 and the old FileType::convert signature. It now passes the options with rules off, since the reference outputs come from the model alone. The ignored GPU qualification test passes on an Apple GPU.
• Veto only when the rules ran (#1529): features extracted with use_rules = false recorded an empty list of matching rules. A session with rules on then vetoed every prediction of a content type whose rules have no false negatives. Through the public API (extract_file with rules off, then identify_features on a default session), real WAV, PSD, Parquet and XLSX samples from tests_data/basic came out unknown. The C API, which takes extraction options separately, hits this directly. Features now record None when no rules ran. In this PR, they record which rule sets ran.

Custom rules

Options::custom_rules: Option<Rules>. Rules is an Arc, so Options stays cheap to clone. Extraction (FeaturesOrRuled::extract_*) applies the rules, so custom rules work in any pipeline built on the low-level API, not only in our front ends.

Order (each step only if enabled):

1. custom rules;
2. built-in rules (use_rules);
3. the model (use_model).

When each step decides:

• Custom rules identify a file when the ones that match agree on one content type.
• Built-in rules identify it when no custom rule matched and the ones that match agree.
• Otherwise the model decides.

Veto: custom rules use #1529's mechanism. A content type whose enforced custom rules are all class = "full" (fn_rate = 0) vetoes a model prediction of it when none of them matched; partial doesn't veto. The veto follows the rule sets that ran at extraction, which Features records, not the options of the session that runs the model.

Shared with the built-in rules:

• the same YARA subset and metadata validation (magika-rules);
• the zip and PE facts;
• one shared read of a zip archive's tail.

Labels must be Magika content types, and a file must enforce at least one rule.

Commit	What
Name the line and column of YARA syntax errors	magika-rules reported the parser's debug output with byte spans (Span(13..13)). It now reports line 2, column 9: expecting \condition`, found end of file`.
Identify with custom rules before the built-in rules	magika::Rules (compile, from_files, content_types), Options::custom_rules, Builder::with_custom_rules. from_files validates each file alone, so an error names its file and line.
Add --rules-file and --rules-check to the CLI	--rules-file PATH (repeatable, hidden and experimental like --rules) and --rules-check, which validates the files and prints their content types. Adds tests_data/rules/custom.yar, shared by the CLI, Python and C tests: it enforces the PNG signature, which the built-in rules leave to the model.
Add custom rules to the Python Magika class	Magika(rules=...) and Magika(rules_files=[...]). Invalid rules raise ValueError.
Add custom rules to the C library	magika_rules_new / magika_rules_free, an opaque MagikaRules, MagikaOptions.custom_rules, and MAGIKA_STATUS_INVALID_RULES with the message written to a caller buffer. The options keep their own reference. The header is edited by hand, as cbindgen would write it.
Document custom rules in the rules README

Numbers

Apple M-series, release build. Extraction runs over the 101 in-memory files of tests_data/basic, best of 20 runs.

	Time
Compile 10 custom rules	0.56 ms
Compile 100 custom rules	5.5 ms
Extract, before this PR (fde2d0ab)	3.9 µs/file
Extract, no custom rules	2.3 µs/file
Extract, 10 custom rules	2.7 µs/file
Extract, 100 custom rules	5.7 µs/file

Unset custom rules cost nothing measurable; the two runs without them differ only by noise on a loaded machine.

Testing

• Library:
  • custom rules come before built-in ones, also with built-in rules off;
  • a full custom rule vetoes the model, and a partial one doesn't;
  • features extracted without rules are never vetoed;
  • a custom zip-facts rule on the 290 KB docx sample, which needs the tail read;
  • errors for an unknown label, bad syntax (with line and column), bad metadata (with the rule), no enforced rule, and an unreadable file (with its path);
  • clones share the compiled rules.
• CLI test.sh: --rules-check output; PNG is unknown with --rules=only and png once the custom file is given, also with --rules=off; a broken file is reported with its line.
• Python: the custom file through identify_path, identify_paths and identify_bytes; the veto (RULES_VETO); the errors. The suite has 40 passing tests, and ruff and mypy are clean.
• C test.sh: the same PNG check through the C API, freeing the rules before use, plus invalid rules with and without an error buffer. It passes under ASan and UBSan with gcc and clang, static and shared.
• Rust: rust/test.sh passes on stable and nightly (48 test suites). Its final sync check only flags model.probe.f32le, which regenerates differently on this Mac, as it does on main. rust/changelog.sh passes.
• Dataset gate: 19,360 validated hits and 0 false positives. Custom rules are off by default, so nothing changes there.

🤖 Generated with Claude Code

[coveralls]

Coverage Report for CI Build 37950811830

Coverage increased (+0.005%) to 96.841%

Details

• Coverage increased (+0.005%) from the base build.
• Patch coverage: No coverable lines changed in this PR.
• 1 coverage regression across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
python/magika.py	1	98.94%

---

Coverage Stats

Relevant Lines:	728
Covered Lines:	705
Line Coverage:	96.84%
Coverage Strength:	0.97 hits per line

---

💛 - Coveralls