Share the CLI's fast pipeline with Python and C through the library

[ebursztein]

Part of #1537.

Stacked on #1530 (custom rules), itself on #1529: review only the last 4 commits, from Add Engine and Pipeline helpers behind the pipeline feature. Once those merge, this branch is rebased.

Why

The CLI had its own fast orchestration: parallel reading, batching, and identifying on the CPU while the GPU is prepared. Python and C didn't share it. Python's Magika() waited about 95 ms for the GPU before its first answer, and its identify_paths read every file on one thread.

This PR moves that orchestration into the library as two helpers behind a pipeline feature that is off by default. The real API doesn't change: FeaturesOrRuled::extract_* and Session::identify_features_batch are still the way to write a pipeline specialized to where Magika is embedded. Both helpers are written against that public API only, so they also serve as examples of it.

The helpers (magika::pipeline)

• Engine:
  • Engine::new(builder) prepares the CPU runtime on a background thread, and then (unless Backend::Cpu) the GPU runtime, so it returns at once.
  • The GPU starts only once the CPU is ready, because preparing Metal competes with the CPU plans a short run waits for. Starting both at once cost about 0.5 ms on a single file.
  • Each thread identifies through an EngineSession (identify_file, identify_content, identify_features_batch), which holds one session per backend, spawned on first use. Full batches (8 or more) go to the GPU once it's ready, or always when it's required.
  • Options are given per call, so one engine serves any options.
  • This is the CLI's backends.rs, moved.
• Pipeline:
  • Pipeline::new(engine, options, PipelineOptions { recursive, stdin, batch_size, threads, readers }).
  • identify_paths(paths) returns (path, Result<FileType>) in walk order. An outer error means the pipeline itself failed; dropping the iterator stops the run.
  • Inside are the CLI's walk (cycle detection, sorted entries, read_dir types), readers, batcher, inference workers and reordering. Two thirds of the workers start while the GPU is prepared, and the rest only if it fails.
• Small additions:
  • DirectoryCycle is a typed error, replacing a string comparison.
  • Builder::backend() returns the selected backend.
  • The _trace feature keeps the CLI's per-stage report.

Front ends

Commit	Change
Identify paths in the CLI with the library pipeline	The CLI keeps its flags, standard input and printing. backends.rs, the stages and the trace code go (+39 / −640 lines). The CLI's test.sh passes as is. One deliberate change: with --backend=gpu, a short run now waits for the GPU instead of quietly identifying on the CPU.
Identify in Python with the library pipeline	One Engine per process, an EngineSession per thread, and identify_paths on Pipeline. Magika() waits for the CPU runtime only, so a model failure is still reported at construction and a forked process doesn't inherit a preparation in progress. The Python API and results are unchanged.
Identify paths in parallel from the C library	magika_engine_new / magika_engine_free and magika_engine_identify_paths, with a result and a status per path. Existing functions are unchanged. The header is edited by hand, as cbindgen would write it.

Numbers

Apple M-series, release builds, evaluation dataset files, interleaved with the code before this PR (#1530).

CLI (median of 10 runs). The gate was "no more than 5% slower at any size".

Files	Before	After
1	11.9 ms	11.9 ms
10	18.1 ms	18.1 ms
100	40.5 ms	40.6 ms
1,000	140.4 ms	140.0 ms
3,000	260.4 ms	259.8 ms

Python (median of 9 cold processes, including import magika):

	Before	After
Magika() then one identify_path	120.7 ms	28.2 ms
identify_paths, 100 files	144.5 ms	58.0 ms
identify_paths, 3,000 files	549.0 ms	272.9 ms

Review fixes

A separate review of this branch (design, safety, concurrency) checked that:

• early drop doesn't deadlock;
• the FFI pointer handling is sound;
• the hand-edited header is byte-identical to cbindgen's output.

It found four issues, fixed here:

• Silent truncation: a panicking pipeline thread used to end the run with fewer results and no error. The iterator now joins the threads and yields an error. The C function also checks that it got one result per path before writing the statuses.
• Panicked GPU preparation: it now counts as a failed GPU, so the remaining CPU workers start.
• Python construction: Magika() waits for the CPU runtime.
• Documentation: the --backend=gpu change is in the changelog, the arrays' state on error is documented, and freeing an engine notes that its background preparation may still be running.

Testing

• Engine, with an injected GPU that is still preparing or has failed: the CPU identifies meanwhile; a failed GPU is an error only when required; an engine is shared by 8 threads. On macOS, full batches move to a real GPU.
• Pipeline:
  • a recursive run over tests_data/basic gives the same label as Session::identify_file for every file, in sorted walk order;
  • a missing path yields its error without stopping the run;
  • a directory cycle yields DirectoryCycle;
  • a FIFO is Unsupported, and a directory is Directory without recursion;
  • dropping the iterator early stops the run;
  • invalid options are rejected.
• Library test.sh: tests with --features=pipeline, and checks and lints with --features=_trace. The default build doesn't compile the module.
• CLI test.sh: unchanged and passing, covering the three rules modes, custom rules, symlinks, missing files, non-UTF-8 names, directory cycles and broken pipes.
• Python: 41 tests, including the slow one, and run_quick_test_magika_module.py pass; ruff and mypy are clean.
• C test.sh: passes under ASan and UBSan with gcc and clang, static and shared, including the engine with a missing path.
• Rust: rust/test.sh passes on stable and nightly (52 test suites). The final sync check only flags model.probe.f32le, which regenerates differently on this Mac, as on main. rust/changelog.sh passes.

🤖 Generated with Claude Code

[coveralls]

Coverage Report for CI Build 37947211612

Coverage increased (+0.005%) to 96.841%

Details

• Coverage increased (+0.005%) from the base build.
• Patch coverage: No coverable lines changed in this PR.
• 1 coverage regression across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
python/magika.py	1	98.94%

---

Coverage Stats

Relevant Lines:	728
Covered Lines:	705
Line Coverage:	96.84%
Coverage Strength:	0.97 hits per line

---

💛 - Coveralls