linalg/mmm: take the chunking slack only where the cache absorbs it

[czoli1976]

Follow-up to #2534: the a53 inceptionv3 · pass_mt regression looks like the
chunk count, and this gates that count on whether the machine has a cache that
can absorb the re-reads it costs.

Why a53 and not the rest of the fleet

The a53 number is identical to three digits across the 09:12 and 11:02 runs on
#2534 — 1.12 s → 1.26 s both times. The L2-sharers commit only does anything when
l2_sharers > 1, so it was a no-op there. If cache_info().l2 == 0 on that
runner, inner_tier_pays short-circuits, blk and blk_outer are both
usize::MAX, and run_blocked is a plain naive walk — which leaves chunk_grid
as the only thing that changed on that device.

Replaying old vs merged over InceptionV3's 94 threaded convs at nth = 4,
arm64simd_mmm_f32_12x8_a53 (m = Cout, k = Cin·kh·kw, n = Hout·Wout):

    m     k     n |  panels  |   old  #  |   new   #  |  oldMB  newMB    ×
  384  4032    64 |  32x8    |  4x1   4  |  16x1  16  |   10.3   22.7  2.20
  384  2592   289 |  32x37   |  1x4   4  |   4x4  16  |   18.9   27.9  1.48
  320  2048    64 |  27x8    |  4x1   4  |  14x1  14  |    4.7   10.0  2.11
  192  1344   289 |  16x37   |  1x4   4  |   3x5  15  |    5.7    9.8  1.73
   96   864  1225 |   8x154  |  1x4   4  |  1x16  16  |    5.6    9.5  1.72

packed-operand traffic per inference: 458 MB → 707 MB (1.54×), +248 MB

+248 MB at 1.5–2.5 GB/s of effective DRAM bandwidth is +100 to +165 ms against a
measured +140 ms.

The cause is two rules meeting rather than the sqrt shaping. The old
nchunks < 4·nth was a re-plan trigger, and what it re-planned to was nth.
On grids below the old 16-panel cap — nearly all of InceptionV3 — the merged code
therefore asks for 4× the chunks the old code chose, and on a part with a 512 KB
cluster L2 and no L3 each extra chunk is a full re-read of a packed operand from
DRAM. That is also why a53 and not a55: a55 sits in a DynamIQ cluster with a DSU
L3 behind it, so the same re-reads hit cache.

The change

chunks_per_thread() gates CHUNKS_PER_THREAD on the detected last-level cache —
falling back to L2 where no LLC is exposed, which is the case on Apple — and takes
one chunk per thread below 2 MB. TRACT_MMM_CHUNKS_PER_THREAD overrides it,
resolved once like the cache probe it reads, so the fleet can sweep 1/2/4 without a
rebuild.

Above 2 MB of last-level cache the chunk grid is bit-for-bit what it is today, so
this cannot move the m1-max, i9 or Orin numbers. Below it, the grid returns to the
old chunk count:

default (cpt=4)                      forced cpt=1 (what an a53 gets)
384x4032x64:  16x1 = 16 ch, 22.7 MB   4x1 = 4 ch, 10.3 MB
384x2592x289:  4x4 = 16 ch, 27.9 MB   2x2 = 4 ch, 14.0 MB
192x1344x289:  3x5 = 15 ch,  9.8 MB   1x4 = 4 ch,  5.7 MB
  total 79.9 MB                         total 40.2 MB

It covers the a53 whichever way the cache probe goes there: if L2 is undetected,
cache_info().l2 == 0 drives the count to 1 anyway.

Tests

cargo test -p tract-linalg --features multithread-mm passes 4420 tests across all
targets. harness/nnef-test-cases/parallel-matmul passes at --approx exact for 2,
3 and 8 threads, run both on the default path and under
TRACT_MMM_CHUNKS_PER_THREAD=1 — the second is the one that matters, since forcing
the constant is the only way to exercise the a53 chunk counts on Apple hardware.

cargo test --workspace gives 29 suites clean plus test-f16's own 4747, with one
failure — nnef_f16::onnx::node::v1_19_1::test_averagepool_2d_ceil_last_window_starts_on_pad
— which fails identically on a6c4243c6 with the diff reverted, so it is
pre-existing and unrelated. test-tflite, tract-proxy, tract-proxy-sys and
test-cuda could not build or run on this host (no OpenSSL, no FFI dylib, no CUDA
driver) and were excluded. Nothing outside linalg can see this change in any case:
it is entirely inside #[cfg(feature = "multithread-mm")], which is off by default.

What I could not check

No wall clock. On a 12 MB-L2 M1 Pro chunks_per_thread() resolves to 4, so this
diff is inert on every machine I have; the traffic numbers above are arithmetic
over canonical InceptionV3 conv geometry, not measurements. A bench run is the
thing that would settle it, and inceptionv3 · pass_mt on cortex-a53 is the metric
to watch.

CHUNK_SLACK_LLC_BYTES = 2 MB is a calibration guess, chosen to put Orin's 2 MB L3
on the slack side and A7/A9/A53 off it. The env knob is there so a sweep can answer
where that boundary belongs rather than a rebuild per value — and if the constant
survives its bench run it should become a proper declare_knob!.

🍍

[czoli1976]

@kali maybe will save A53