Magika 2.0: custom rules, and one fast pipeline for the CLI, Python and C #1537

This issue tracks the work that puts Magika's rules and speed in the library, where every front end gets them, instead of in the CLI alone. Each PR below is self-contained and says Part of this issue.

Design

• The low-level library API stays the real API:

  • Builder/Runtime;
  • FeaturesOrRuled::extract_file / extract_content;
  • Session::identify_features_batch.

  Integrators build their own pipelines on it.

• Custom rules live in Options (Options::custom_rules, a cheaply cloned Rules), so extraction applies them in any pipeline.

  • Order: custom rules, then built-in rules, then the model, each when enabled.
  • Veto: custom rules veto the model the same way built-in rules do since Veto ML output when rules have no false negatives #1529. A content type whose enforced rules are all class = "full" is vetoed when none of them matched.

• Shared orchestration for our front ends lives in magika::pipeline, behind a pipeline feature that is off by default, and uses the public API only.

  • Engine: identifies from any thread, on the CPU at once and on the GPU once it's ready.
  • Pipeline: parallel walk, read, batch, infer and reorder.
  • Front ends: the CLI, Python and C use these helpers instead of their own copies.

PRs in progress

• Veto ML output when rules have no false negatives #1529
• Identify files with custom rules before the built-in rules #1530
• Share the CLI's fast pipeline with Python and C through the library #1531
• Prepare CPU plans beyond the smallest on first use #1532
• python: accept raw binary streams in identify_stream #1533
• python: reject a bare str or bytes in identify_paths (#1472) #1476
• python: propagate the real error for unseekable streams in identify_stream (#1473) #1475

Merged earlier, from splitting #1447

Closed

Next

• Misdetection sweep, once Veto ML output when rules have no false negatives #1529 and Identify files with custom rules before the built-in rules #1530 land: rules now cover docx, pptx, xlsb, pgp, heif, avif and gguf, so [Misdetection] DOCX file misdetected as spreadsheetml (XLSX) or application/zip #1326, [Misdetection] pptx file misdetected as xlsb #1259, [Misdetection] ASCII-armored GPG key file misdetected as PEM certificate #1338, GPG key type #1339, [Misdetection] HEIF Image file misdetected as MP4 media file #1439, [Misdetection] AVIF image file misdetected as MP4 video file #1233 and gguf files aren't recognized #1430 are worth rechecking with validated samples.
• Contributor sample PRs for Add more "basic" tests samples to cover supported content types #662 (Add basic test samples for cpp, go, shell, java, sql, and xml content types #1417, test: add basic samples for gif pickle powershell and ttf #1394, Add basic test samples for 10 missing content types (fixes #662) #1393, test: enable Windows python suite #1372, feat(tests): add sample test files for Bash, Go, Java, SAS, and XML #1367, Add Test Cases for Various File Types #1366, Add multiple sample files to test data #1346, test: add basic properties and env samples (#662) #1307): checking each sample with a format validator and importing the proven ones.
• Startup: CLI start pays about 1.5 ms for linking Metal even when it never uses the GPU #1493 (about 1.5 ms of Metal linking at startup) stays open.

Rules: preparation and packaging

• Authoring: authors edit rust/rules/rulesets/{full,partial,notworking}/*.yar.
• Generation: rust/sync.sh regenerates the committed src/bundled.rs and the rule labels, which rust/gen checks against the knowledge base YAML.
• Freshness: a test fails if bundled.rs is stale.
• Shipping: the rules are compiled into the CLI, the Python extension and the C library, so no rule file is read at runtime.
• Custom rules: they use the same validation, and are compiled once when the options are built.

Testing

• In every PR: rust/test.sh (library, CLI, rules, C, generator), rust/changelog.sh, and the Python suite against the PyO3 build.
• Rules PRs: the rules dataset gate (0 false positives).
• Performance: front-end changes report interleaved before/after timings for 1 to 3,000 files.

🤖 Generated with Claude Code