[mjsuhonos]

This PR brings some new functionality to the optimize CLI tool:

1. New class: OptimizeEvaluator(): enables per-worker evaluation for parallel optimization. run_optimize() now uses this class to run optimization metrics in parallel (as specified by the existing -j parameter).

2. Pareto front calculation: the optimize output remains identical, but the summary now provides a list of Pareto points (by ascending Recall) rather than "Best Precision" and "Best Recall"; "Best F1" is kept. This provides a choice of operational points depending on the shape of the Precision-Recall curve.

3. Results file output: -r --results-file parameter added to mirror the implementation in hyperopt. Output of the file is a TSV that contains the same rows as the optimize stdout, with Pareto points in an additional column.

The results file is especially useful for cannif because it allows generating charts like these:

Some of the code was provided by GLM-5.3 (Mistral Hosted) but every line of code was reviewed by myself.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.64%. Comparing base (2852115) to head (e47c969).
⚠️ Report is 34 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #967      +/-   ##
==========================================
+ Coverage   99.63%   99.64%   +0.01%     
==========================================
  Files         103      104       +1     
  Lines        8307     8549     +242     
==========================================
+ Hits         8277     8519     +242     
  Misses         30       30              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

Static review found no blocking issues in parallel evaluation, Pareto selection, or results export; project tests were not executed.

No open issues 🎉

What changed in this PR

This PR extends Annif’s optimize CLI with parallel metric evaluation and richer output for choosing limit and threshold settings.

Changes:

• Uses per-worker evaluators to run parameter evaluations with the existing --jobs option.
• Reports a recall-sorted Pareto front while retaining the best-F1 summary.
• Adds TSV results export with Pareto membership, plus CLI tests for export and multiprocessing.

File	Description
tests/​test_cli.py	Updates summary assertions and adds export, error-handling, and multiprocessing tests.
annif/​parallel.py	Adds OptimizeEvaluator for per-worker parameter evaluation.
annif/​cli.py	Integrates parallel evaluation, Pareto reporting, and TSV export.
annif/​cli_util.py	Adds Pareto-front selection with deduplication and recall ordering.

_{Review effort: Balanced}

[juhoinkinen]

Just reporting about my tests for the performance optimization, that I did by running

/usr/bin/time -v annif optimize yso-ensemble-en ../FintoAI-data-YSO/corpora/fulltext-test/en/**/ -j X

on our server (yso-ensemble-en project is the simple ensemble version of the one used in FintoAI at the moment, and the dataset (1865 documents) is the one we use to test the YSO models).

With -j 6

Before

Best  F1 score (doc avg): 0.4825	Limit: 9	Threshold: 0.10

User time (seconds): 1350.00
System time (seconds): 8.95
Percent of CPU this job got: 444%
Elapsed (wall clock) time (h:mm:ss or m:ss): 5:05.53
Maximum resident set size (kbytes): 6810004

After

Best  F1 score (doc avg): 0.4825	Limit: 9	Threshold: 0.1000

User time (seconds): 1326.41
System time (seconds): 8.73
Percent of CPU this job got: 547%
Elapsed (wall clock) time (h:mm:ss or m:ss): 4:03.82
Maximum resident set size (kbytes): 6811272

A second runs produced very similar results.

So the optimization cut some 20% of the wall time! An did not affect the results or the memory consumption.

With -j 1

Before

Best  F1 score (doc avg): 0.4825	Limit: 9	Threshold: 0.10

User time (seconds): 1350.17
System time (seconds): 6.62
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 22:39.65
Maximum resident set size (kbytes): 7729196

After

Best  F1 score (doc avg): 0.4825	Limit: 9	Threshold: 0.1000

User time (seconds): 1350.99
System time (seconds): 8.17
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 22:41.53
Maximum resident set size (kbytes): 7755996

With one job the PR changes have no effect as expected.

[juhoinkinen]

Here are some plots of the optimize command file output (the dataset is the same as for the performance evaluation above). Credit goes to Qwen 3.8. Jupyter notebook of the plots as a GH gist here.

[juhoinkinen]

I think this could be merged 👍

The stdout output after this PR is like this:

Limit	Thresh.	Prec.	Rec.	F1
1	0.00	0.7512	0.1338	0.2141
1	0.05	0.7496	0.1336	0.2138
1	0.10	0.7453	0.1329	0.2127
[...]
15	0.90	0.0000	0.0000	0.0000
15	0.95	0.0000	0.0000	0.0000

Pareto front: best precision at each recall level
Limit	Thresh.	Prec.	Rec.	F1
1	0.00	0.7512	0.1338	0.2141
2	0.00	0.7029	0.2397	0.3328
3	0.10	0.6557	0.3160	0.3966
[...]
14	0.00	0.3404	0.6917	0.4285
15	0.00	0.3254	0.7050	0.4187

Best  F1 score (doc avg): 0.4825	Limit: 9	Threshold: 0.1000
Documents evaluated:	1865

The results file (TSV) output is like this:

Limit   Threshold       Precision (doc avg)     Recall (doc avg)        F1 score (doc avg)      Pareto front
1       0       0.751206        0.133784        0.214122        1
1       0.05    0.749598        0.133593        0.213785        0
1       0.1     0.745308        0.132949        0.212698        0
1       0.15    0.735657        0.131129        0.209767        0
1       0.2     0.720643        0.127940        0.204853        0
1       0.25    0.699196        0.122567        0.197261        0
[...]

I tried to think if the outputs could be simplified somehow (it's already quite a long output to stdout, we could later think about having an option to silent the limit-threshold loop output), but could not see a good way for it.

Overall, the only alternative that comes to mind is to have an option to that would append the Pareto-front column to the stdout output, so that the results file output would not be needed, but with the existing results file output in the hyperopt command that makes no sense.

[juhoinkinen]

Formatting threshold using :g for the results file is different than for stdout output. The threshold values vary by two digits, so it could be more naturaly to use :.2f.

[mjsuhonos]

Good catch -- I've used :.4f because that's the standard in eg. optimize and it allows rounding to 3 places.

[juhoinkinen]

The Pareto-front optimization functionality would benefit from documentation, probably in Achieving good results page.

Do you have something in mind that you could write, @mjsuhonos?

[osma]

I'm a bit confused - shall we merge this now that Juho has approved it or is there still something to be done? @juhoinkinen @mjsuhonos

[mjsuhonos]

Yes, @juhoinkinen’s comments from yesterday are worth a commit, I think. Will do shortly.

[osma]

Thanks again @mjsuhonos for the fixes! I'll leave it to @juhoinkinen to merge this.

[osma]

@mjsuhonos It looks like your latest commit broke the test_optimize_resultsfile test. It expects a different number of digits, I think.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud