amyloid finder: Stream amyloid FOM scoring through a three-angle ring buffer

[hilaolu]

This PR reduces peak memory usage in amyloid FOM calculation by streaming orientation scores through a three-angle ring buffer. The implementation preserves the legacy numerical results while reducing measured peak RSS per MPI rank by 81.54% (5.42×).

How it works

The legacy implementation retains rotated images and signal/non-signal score maps for all 36 PSI orientations before combining them.

The new implementation:

1. Calculates the signal and non-signal maps for one PSI orientation at a time.
2. Keeps only the previous, current, and next orientations in a three-slot ring buffer.
3. Uses those three maps to preserve circular neighbour-PSI exclusion.
4. Updates the running sums, maxima, best PSI, and neighbour contributions.
5. Reuses the oldest slot for the next orientation.

This changes the dominant score-map storage from O(nr_psi) to O(3), plus bounded scratch space.

Memory and numerical validation

Both versions processed the same three fiber-rich EMPIAR-12870 micrographs using three MPI ranks and four OpenMP threads per rank. Numerical preservation was verified at two levels:

• The focused synthetic regression test passed all 516 assertions against an implementation of the legacy all-PSI algorithm.
• All 881,790 values across six real-data FOM and PSI maps were bit-identical.
• Every numeric value was exactly equal, including skewness and kurtosis.

Runtime tradeoff

The current streaming implementation increases wall time by 87.22%. However, this does not represent a proportional increase in CPU work:

• user CPU time increased by approximately 16%;
• 3 micrographs is just for benchmark memory footprint and correctness, not for wall time bench;
• wall time increased mainly because rotation and downscaling now have less parallel overlap;
• no disk spilling or rereading was observed: the optimized run had zero major page faults and zero filesystem inputs.

The current implementation intentionally prioritizes bounded memory use. Follow-up optimizations are expected to recover throughput without restoring the original memory footprint.

[biochem-fan]

@hilaolu Thank you very much. Could you also put the images you attached to your email? You can just paste them in this Markdown editor.

[scheres]

I am looking into this one. Memory usage is decreased, but at the expense of more CPU. Why is this desirable? Memory usages wasn't that bad, no? What size micrographs and what machine made it necessary to buy memory reduction with compute increase?

[hilaolu]

@hilaolu Thank you very much. Could you also put the images you attached to your email? You can just paste them in this Markdown editor.

This patch preserve the numerical and the result is bit identical. I will post images for patch where numerical is not preserved.

I will file another PR which target CPU runtime optimization where numerical is not preserved , after this PR is decided.

[hilaolu]

I am looking into this one. Memory usage is decreased, but at the expense of more CPU. Why is this desirable? Memory usages wasn't that bad, no? What size micrographs and what machine made it necessary to buy memory reduction with compute increase?

I will attach more benchmark data later, include fine grade cpu time, memory bandwidth and memory footprint. IMO, the point is you may get the most of limited memory size/memory bandwidth of under a limited system, by cutting down memory footprint and pursuit a higher concurrency.

[hilaolu]

Hi, @scheres, thank you for raising this concern, which is fair. The additional measurements clarify the distinction between the current ring-buffer PR and the separate follow-up optimizations.

Dataset and system configuration

The benchmark uses motion-corrected micrographs derived from EMPIAR-12870, a dataset of K3 superresolutioned patient derived PHF. The inputs to the amyloid finder are binned 5760 x 4092 float32 images at 0.83 A/pixel,
evaluated over 36 PSI orientations with tracing disabled.

The benchmark machine has:

• 32C/64T AMD Zen 2@3.1GHz;
• 128 MiB L3 cache;
• 32 GiB DDR4 2666MHz x 2, dual Channel RAM;
• 54 GiB cgroup limit for each benchmark job;
• job swap disabled.

Current PR: ring buffer only

The current PR contains only the bit-identical ring buffer; it does not contain downscale-once and further optimization. (Downscale-once, which improve throughput further). For one MPI rank and four threads, the median results were:

Implementation	Wall/micrograph	Peak memory
Legacy	100.3 s	10.35 GiB
Ring buffer	420.6 s	1.90 GiB

Thus, the ring-buffer uses 5.44x less memory but is 4.19x slower. I therefore agree that this PR alone should not be presented as a runtime optimization.

The memory reduction becomes more significant for larger images or denser orientation sampling. At four threads:

Geometry and orientations	Legacy	Ring buffer
5760 x 4092, 36 orientations	10.35 GiB	1.90 GiB
5760 x 4092, 72 orientations	19.55 GiB	1.90 GiB
8640 x 6138, 36 orientations	22.72 GiB	4.17 GiB
8640 x 6138, 72 orientations	42.94 GiB	4.17 GiB

The ring-buffer memory remains bounded in orientation count. All tested FOM and PSI values, skewness, and kurtosis were bit-identical.

Multi-MPI scaling of the ring-buffer

Multi-MPI does improve the ring-buffer's own throughput:

Ring-buffer shape	Throughput	Aggregate memory	MPI scaling efficiency
1 MPI x 1 thread	6.74/hour	1.90 GiB	100%
2 MPI x 1 thread	13.14/hour	3.76 GiB	97.4%
4 MPI x 1 thread	25.39/hour	7.42 GiB	94.2%
8 MPI x 1 thread	47.80/hour	14.74 GiB	88.6%
16 MPI x 1 thread	84.94/hour	29.36 GiB	78.7%
16 MPI x 2 threads	95.10/hour	29.64 GiB	75.6% relative to 1 x 2

Therefore, reducing the per-rank working set creates useful capacity for higher micrograph-level MPI concurrency. However, it does not yet overcome the ring buffer's compute cost. In a preliminary 32-core comparison, ring-buffer 16 x 2 achieved 95.1 micrographs/hour, while legacy 1 x 32 achieved 108.4 micrographs/hour. These runs used different manifest lengths, so I regard this as preliminary, but it does not show a throughput win for the current PR.

The ring buffer is therefore primarily a memory-capacity optimization. Its current unfused stages still perform repeated rotation, downscaling, and load-process-store cycles, which prevents effective thread scaling.

Separate complete-stack experiment

I also tested a separate experimental revision containing:

• the ring buffer;
• sliding narrow-band scoring;
• paired-angle rotation reuse;
• downscale-once/rotate-once.

These results do not describe the code in the current PR. Downscale once is fast while it isn't bit identical to legacy. To explain what downscale-once actually does, I opened a separated PR.

Single-rank results

Each configuration below processed the same four real micrographs sequentially and was run once:

Implementation	Shape	Wall for 4 micrographs	CPU time	Mean CPU	Peak CPU	Peak memory
Legacy	1 x 1	1328.7 s	1320.3 CPU-s	99.4%	109.6%	10.113 GiB
Complete stack	1 x 1	84.5 s	83.8 CPU-s	99.1%	109.5%	1.354 GiB
Legacy	1 x 2	694.1 s	1336.6 CPU-s	192.6%	209.2%	10.199 GiB
Complete stack	1 x 2	72.7 s	85.3 CPU-s	117.4%	198.4%	1.360 GiB
Legacy	1 x 8	251.3 s	1487.7 CPU-s	592.2%	813.5%	10.720 GiB
Complete stack	1 x 8	66.0 s	117.1 CPU-s	177.5%	535.6%	1.400 GiB

At equal shapes, the complete stack was 15.73x faster at 1 x 1, 9.55x faster at 1 x 2, and 3.81x faster at 1 x 8.

The optimized 1 x 2 run briefly reached approximately 200% CPU, and 1 x 8 peaked at approximately 536%, but their whole-run means were only 117% and 178%. These highly parallel phases are transient rather than sustained.

Whole-node results

The final matched comparison used the same 32 micrographs and three alternating repetitions:

Implementation	Shape	Median wall	Throughput	CPU time	Peak memory
Legacy	4 MPI x 8 threads	693.5 s	166.1/hour	17,258 CPU-s	41.27 GiB
Complete stack	16 MPI x 2 threads	68.0 s	1,694.7/hour	1,159 CPU-s	20.53 GiB

The complete stack therefore achieved:

• 10.20x higher whole-node throughput;
• 90.20% lower wall time;
• 14.89x lower total CPU time;
• 50.24% lower aggregate peak memory;
• four times as many concurrent micrographs.

All six final jobs completed all 32 FOM and all 32 PSI maps, with zero job swap and no cgroup memory-limit, OOM, or OOM-kill events.

Diminishing returns from parallelism

The complete-stack batch32 shape measurements were:

Shape	Wall	Throughput	CPU time	Peak memory
8 MPI x 4 threads	88.6 s	1,300.5/hour	887 CPU-s	10.41 GiB
16 MPI x 2 threads	68.0 s	1,694.7/hour	1,159 CPU-s	20.53 GiB
32 MPI x 1 thread	80.5 s	1,430.3/hour	2,446 CPU-s	40.61 GiB
8 MPI x 8 threads	96.5 s	1,193.9/hour	1,244 CPU-s	10.61 GiB
16 MPI x 8 threads	81.2 s	1,418.3/hour	1,849 CPU-s	21.08 GiB

More parallelism was not always beneficial. The 16 x 2 shape was faster than 32 x 1 and the unbound 16 x 8 shape. At fixed eight-rank concurrency, 8 x 8 was 8.9% slower than 8 x 4 despite consuming approximately 1.40x more CPU time.

This behavior is consistent with memory/cache bandwidth pressure and thread overhead. I have not collected reliable hardware bandwidth counters, so I cannot claim that DRAM-bandwidth saturation has been proven.

My conclusion is that the current ring-buffer is a bit-identical, bounded-memory foundation. It enables higher MPI concurrency, but does not improve node throughput by itself. The performance advantage becomes visible only under practical parallelism applies.

The 10.20x result belongs to the complete stack and cannot be attributed to the ring buffer alone. Downscale-once/rotate-once also changes floating-point operation order and is not bit-identical.

[hilaolu]

It turns out that the ring-buf is not so good and I'd use another fix target both runtime and memory footprint. Sorry though.