amyloid finder: add experimental downscale-once path

[hilaolu]

This PR adds an opt-in downscale_once image-preparation backend to the amyloid finder. Instead of rotating the full-resolution micrograph and Fourier-cropping every sampled orientation, the new backend Fourier-crops the micrograph once and rotates the smaller image for every orientation.

The legacy backend remains the default. The new backend is explicitly experimental because changing the order of discrete rotation and downscaling changes the FOM and PSI maps. It should not be used as a numerically equivalent replacement until its biological output has been validated on a broader dataset.

This work is separate from #1351. That PR uses a three-angle ring buffer to reduce memory while preserving bit-identical results. This PR explores a different performance/quality tradeoff: it reduces image-preparation work, but does not preserve legacy numerical results.

Rationale

The legacy preparation path performs the following work for each of the K = nr_psi / 2 independently rotated orientations:

legacy = K * [rotate(A) + FFT(A) + crop(A -> B) + inverse_FFT(B)]

where:

• A is the full-resolution padded image;
• B is the Fourier-cropped image;
• the other half of the orientations is produced by the existing 90-degree reuse;
• K = 18 for 36 PSI orientations at the benchmark settings.

The experimental backend instead performs:

downscale_once = FFT(A) + crop(A -> B) + inverse_FFT(B) + K * rotate(B)

For the benchmark pixel sizes, the linear scale factor is

s = 0.83 / 2.1 = 0.3952

and the approximate two-dimensional pixel-area fraction is

s^2 = (0.83 / 2.1)^2 = 0.1562.

Each low-resolution rotation therefore touches approximately 15.6% as many pixels, an 84.4% reduction. The number of full-resolution Fourier transforms and crops falls from 18 to 1, a 94.4% reduction.

The 2.1 A/pixel downscaled image has a 4.2 A Nyquist limit, which includes the configured 4.65-4.85 A signal band and reaches the edge of the 4.2-4.4 A non-signal band. In continuous mathematics, isotropic low-pass filtering and rotation can commute. In this implementation they are discrete operations with interpolation, finite support, and boundary handling, so their order is not numerically interchangeable. That difference is visible in the validation below.

Interface and safety

The new expert option is:

--image_prepare_backend legacy|downscale_once

legacy remains the default. Selecting downscale_once prints a warning that the interpolation order changes and that FOM/PSI quality must be validated before production use. Unknown backend names are rejected.

Benchmark method

The benchmark used:

• one motion-corrected, fiber-rich EMPIAR-12870-derived micrograph;
• input dimensions 5760 x 4092, float32, at 0.83 A/pixel;
• --down_angpix 2.1;
• 36 PSI orientations from --psi_step 5.0;
• --shift_step 5;
• tracing disabled with --skip_tracing;
• one MPI rank and eight OpenMP threads;
• OMP_PLACES=cores and OMP_PROC_BIND=close;
• a 54 GiB cgroup memory limit, with job swap disabled;
• process-tree CPU/RSS samples every 0.1 seconds plus cgroup-v2 memory.peak;
• AMD Zen 2 32C/64T @ 3.1GHz, 128 MiB L3 cache.
• 32GiB DDR4 2666MHz x 2, dual channel memory, which suggest to be bottleneck of the system

Benchmark results: 1 MPI x 8 threads, 1 micrograph

Metric	Legacy	Downscale once	Change
Wall time	66.04 s	37.19 s	43.68% lower; 1.78x faster
Implied sequential throughput	54.51 micrographs/hour	96.79 micrographs/hour	77.56% higher
User + system CPU time	396.33 CPU-s	175.55 CPU-s	55.71% lower; 2.26x less CPU time
Sampled mean CPU use	603.69%	472.59%	21.72% lower
Sampled peak CPU use	810.47%	805.92%	approximately equal
Cgroup peak memory	10.697 GiB	10.192 GiB	4.72% lower
Swap	0	0	unchanged

The coarse internal progress timings localise most of the gain to image preparation:

Phase	Legacy	Downscale once
Rotate/downscale orientations	39-40 s	2-3 s
Search coordinates	19 s	19 s

The coordinate-scoring phase is intentionally unchanged by this PR.

Weak scaling: 2-4 MPI ranks x 8 threads

Additional single-run benchmarks tested node-level weak scaling with two, three, and four MPI ranks. Each rank was bound to eight cores and processed one micrograph, so both the rank count and the number of micrographs increased together. All input micrographs had the same file size and came from the same dataset.

MPI x threads	Micrographs	Legacy wall time	Downscale-once wall time	Downscale-once speedup	Legacy throughput	Downscale-once throughput
2 x 8	2	73.17 s	40.30 s	1.82x	98.40 micrographs/hour	178.65 micrographs/hour
3 x 8	3	86.13 s	42.89 s	2.01x	125.40 micrographs/hour	251.78 micrographs/hour
4 x 8	4	104.89 s	45.27 s	2.32x	137.29 micrographs/hour	318.09 micrographs/hour

Using the 2-rank result as the weak-scaling baseline, legacy efficiency falls to 84.96% at three ranks and 69.76% at four ranks. downscale_once retains 93.96% and 89.03%, respectively. From two to four ranks, the legacy wall time grows by 43.3% for the same work per rank, while the downscale_once wall time grows by only 12.3%.

On this node, the legacy preparation path therefore does not scale well beyond two MPI ranks, whereas downscale_once continues to scale. The growing gap is consistent with the legacy path hitting shared memory-bandwidth pressure: every rank repeatedly rotates and Fourier-transforms a full-resolution padded image, so additional ranks increase traffic to shared memory. downscale_once removes most of that full-resolution work and shows much less cross-rank contention. Hardware memory-bandwidth counters were not collected, so bandwidth saturation is an inference from the scaling behavior rather than a direct measurement.

Peak memory grew approximately linearly with rank count and was nearly identical between backends: 20.73/20.38 GiB at two ranks, 31.01/30.41 GiB at three ranks, and 40.80/40.86 GiB at four ranks for legacy/downscale_once. Every run completed the expected FOM and PSI maps with exit status 0, zero swap, and no memory-limit or OOM events. The legacy scaling loss is therefore not explained by exhausting the memory-capacity limit. Aggregate user and system CPU time was 59.86%, 64.85%, and 68.00% lower with downscale_once at two, three, and four ranks.

It is worth to note that the reported 10x throughput improvement in #1351 requires a joint effort of 4 optimizations in place, including MPI proc scaling. Downscale-once only doesn't yield such result.

[hilaolu]

Here are the neural trace of legacy, downscale-once, and manual pick(shipped with EMPIAR data), of three selected micrographs in EMPIAR-12870 dataset. 40A pass is applied on background.
Although the numerical doesn't preserved due to limit of computer arithmetic, the result quite agree on the three columns.