feat: stream a model's weights uncached and NZ-ordered

docs/en/dev/language/05-cache-policy.md

@@ -8,10 +8,12 @@ The policy is a *contract the author states*, never a hint the compiler infers.
 It is therefore written explicitly, at one of two granularities, and is carried
 unchanged from the DSL to codegen.
 
-> **Requires PTOAS >= v0.61** (`PTOAS_VERSION` in `toolchain/versions.env`). A
-> `BYPASS` declaration becomes a `cache_policy` attribute on `pto.tload`, which
-> the assembler lowers to pto-isa's own L2 hint. See
-> [What codegen emits](#what-codegen-emits).
+> **Requires PTOAS >= v0.64** (`PTOAS_VERSION` in `toolchain/versions.env`), and
+> **has an effect on a2a3 only**. A `BYPASS` declaration becomes a
+> `cache_policy` attribute on `pto.tload` plus an `offset` operand carrying that
+> device's no-cache alias distance, which the assembler adds to that one load's
+> source address. See [What codegen emits](#what-codegen-emits) and
+> [Architectures](#architectures).
 
 ## Two surfaces
 
@@ -196,37 +198,101 @@ with pl.at(level=pl.Level.CORE_GROUP, name_hint="mm"):
 
 ## What codegen emits
 
-A `BYPASS` read becomes one attribute on the emitted load — there is no extra
-operation, no second tensor view, and no architecture-specific address alias:
+A `BYPASS` read becomes one attribute on the emitted load and one operand — the
+distance from the tensor's address to the uncached alias of the same bytes:
 
 ```mlir
-pto.tload ins(%b__ssa_v0_pview : !pto.partition_tensor_view<256x256xf32>)
-          outs(%b__ssa_v0_mat  : !pto.tile_buf<loc=mat, ...>)
-          {cache_policy = #pto.load_cache_policy<l2_bypass>}
+func.func @main(%arg0: !pto.ptr<f32>, %arg1: !pto.ptr<f32>,
+                %__pypto_l2_cache_offset: i64) {
+  ...
+  pto.tload ins(%b__ssa_v0_pview : !pto.partition_tensor_view<256x256xf32>)
+            outs(%b__ssa_v0_mat  : !pto.tile_buf<loc=mat, ...>)
+            {cache_policy = #pto.load_cache_policy<l2_bypass>}
+            offset = %__pypto_l2_cache_offset : i64
 ```
 
-PTOAS >= v0.61 lowers that to pto-isa's own L2 hint, which is the whole
-difference in the generated CCE:
+The attribute alone declares the policy and moves no address: PTOAS applies the
+offset only to a load that also declared `l2_bypass`, and a declaration without
+one compiles to an ordinary cached load. Both together are what bypasses L2 —
+which is why the offset, not the attribute, is what makes the feature real, and
+why [only a2a3 has one](#architectures).
+
+### Where the offset comes from
+
+A2/A3 maps every GM page twice — once cached, once not — and a load issued
+against the uncached alias does not allocate in L2. The distance between the two
+mappings is a per-device value only the driver knows (`rtGetL2CacheOffset`), so
+it cannot be a constant in the compiler: one box answers `0x80000000000` where a
+pto-isa comment names `0x100000000000`.
+
+simpler queries it once per Worker and carries it to every core's
+`GlobalContext`, where an incore kernel reads it with
+`get_l2_cache_offset(args)` (simpler PR #2323). The generated kernel wrapper
+reads it **once at entry** — the value cannot change during a dispatch, so a
+per-load read would go back through GM for a constant — and forwards it through
+the synthetic `%__pypto_l2_cache_offset` parameter:
+
+```cpp
+extern "C" __aicore__ void kernel_entry(__gm__ int64_t* args) {
+    // ... tensor unpacking ...
+    uint64_t __pypto_l2_cache_offset = get_l2_cache_offset(args);
+    main(a__ssa_v0, b__ssa_v0, out__ssa_v0, __pypto_l2_cache_offset);
+}
+```
+
+PTOAS >= v0.64 turns the pair into address arithmetic on a *copy* of the source
+descriptor, so other loads of the same tensor keep the cached address, and then
+issues an ordinary `TLOAD`:
 
 ```diff
 -  TLOAD(v45, v50);
-+  TLOAD<pto::TLoadL2Hint::NotAllocKeep>(v45, v50);
++  __gm__ uint8_t* v51 = reinterpret_cast<__gm__ uint8_t*>(PTOAS__GLOBAL_TENSOR_DATA(v50));
++  __gm__ float* v52 = (__gm__ float*) (v51 + v5);
++  GlobalTensor<float, ...> v53(nullptr);
++  v53 = v50;
++  TASSIGN(v53, v52);
++  TLOAD(v45, v53);
 ```
 
-Three properties of the emit are worth stating, because each one is asserted in
-`tests/ut/codegen/test_cache_policy_codegen.py`:
+Zero is a valid answer, not a failure: an a2a3 device that exposes no alias
+reports zero, `addr + 0` is the ordinary address, and the declaration costs
+bandwidth rather than correctness. The simulator is that case by construction.
+That is a *device* answering zero — distinct from an architecture that has no
+alias at all, which never gets an offset operand to begin with.
+
+Five properties of the emit are worth stating, because each one is asserted in
+`tests/ut/codegen/test_cache_policy_codegen.py` (the argument order, in
+`tests/ut/codegen/test_prefetch_codegen.py`):
 
 | Property | Why |
 | -------- | --- |
-| `CachePolicy.DEFAULT` emits **nothing** | A kernel that states no policy keeps the PTO form it had before this existed, so the attribute is the only difference between two otherwise identical kernels |
+| `CachePolicy.DEFAULT` emits **nothing** | A kernel that states no policy keeps the PTO form it had before this existed, so the attribute, the operand and the parameter are the only difference between two otherwise identical kernels |
 | The attribute is emitted **per load**, not per tensor | It is a property of the instruction; a hint on only the first of two loads would leave the second one allocating in L2 (the superseded `[CacheBypassUnsupported]` diagnostic was deliberately once-per-tensor — the opposite granularity) |
-| It joins the MX `layout` in **one** attribute dict, after it | PTOAS takes all present attributes in a single dict; keeping `layout` first leaves an MX load that declares no policy byte-identical |
+| The offset parameter is appended **once per kernel** | One runtime value serves every bypassing load, read once at entry |
+| It joins the MX `layout` in **one** attribute dict, after it, and the operand follows the dict | PTOAS takes all present attributes in a single dict; keeping `layout` first leaves an MX load that declares no policy byte-identical |
+| a5 emits the attribute and **no** offset | There is no second mapping to address (see [Architectures](#architectures)), and no accessor to read a distance from |
+
+### Architectures
+
+The double mapping is an a2a3 property, and so is everything built on it:
+
+| Target | What a `BYPASS` declaration does today |
+| ------ | -------------------------------------- |
+| a2a3 device | The load is issued against the uncached alias, `addr + get_l2_cache_offset(args)`. This is the case the feature exists for |
+| a2a3 device with no alias | The driver reports `0`, `addr + 0` is the ordinary address, and the read is cached — correct, one optimisation short. The simulator is this case by construction |
+| a5 | Codegen emits the attribute and no offset, and PTOAS v0.64 lowers a bare attribute to an ordinary `TLOAD`. The declaration is accepted and **does nothing** |
+
+A5 is not a missing offset waiting to be supplied: it has no second mapping to
+address. pto-isa carries an L2 hint on A5's `TLOAD` as an instruction operand
+instead, which is where a future A5 path would go — it is not wired to
+`cache_policy` in v0.64, so nothing in the emitted CCE distinguishes a declared
+load from an undeclared one there.
 
 ### Older assemblers
 
-The emit itself is unconditional — `cache_policy` is a v0.61 addition, and an
-older assembler would fail it at the `pto.tload` verifier. That never happens
-in practice: before the first `.pto` is assembled, codegen runs `ptoas --version`
+The emit itself is unconditional — `offset` is a v0.64 addition, and an older
+assembler would fail it at the `pto.tload` verifier. That never happens in
+practice: before the first `.pto` is assembled, codegen runs `ptoas --version`
 and rejects any assembler older than the `PTOAS_VERSION` this repo pins, with
 an error that names both versions.
 
@@ -251,6 +317,9 @@ an error that names both versions.
 | Lowering | `src/ir/transforms/convert_tensor_to_tile_ops_pass.cpp` |
 | Printer | `src/ir/transforms/python_printer.cpp` (`PrintScopeCachePolicyStmts`) |
 | Codegen | `src/backend/common/pto_ops_memory.cpp` (`MakeTileLoadCodegenPTO`) |
+| Synthetic param | `src/codegen/pto/pto_codegen.cpp` (`MemRefCollectorVisitor::UsesL2BypassLoad`, signature emission) |
+| Kernel wrapper | `python/pypto/backend/pto_backend.py` (`_uses_l2_cache_offset`, `_generate_kernel_wrapper`) |
+| Runtime accessor | `runtime/src/a2a3/runtime/*/common/intrinsic.h` (`get_l2_cache_offset`), `runtime/docs/l2-cache-bypass.md` |
 
 ## See Also
 

docs/en/dev/passes/11-convert_tensor_to_tile_ops.md

@@ -70,7 +70,7 @@ def kernel(
 ```
 
 The reduction receives a `tile.load(x, ...)` result. The final write remains
-`tensor.write(x, ...)`, emitted as a GM `pto.store_scalar`; it does not become a
+`tensor.write(x, ...)`, emitted as a GM `pto.store`; it does not become a
 `tile.write` into the reduction's input tile. The same rule applies to
 `pld.DistributedTensor` parameters, scalar reads, and returned GM aliases.
 
@@ -555,7 +555,7 @@ The pass materializes the loop directly:
 rows = tensor.dim(indices, last_axis)                  # runtime gathered-row count
 acc  = tile.create([max_indices, size], target_memory=space)   # static on-chip buffer
 for i in [0, rows):                                    # ForStmt, iter_arg = acc
-    idx   = tensor.read(indices, [i])                  # scalar GM read (pto.load_scalar)
+    idx   = tensor.read(indices, [i])                  # scalar GM read (pto.load)
     phys  = block_table[idx // block_size] * block_size + idx % block_size   # scalar
     acc   = tile.gather_row(acc, src, [i, 0], [phys, col_off], [1, size])    # GM->on-chip
     yield acc

docs/en/dev/passes/15-block_nz_tensor_views.md

@@ -127,11 +127,17 @@ the enclosing function in one read-only sweep:
 | ------- | --------------------------- | ------------ |
 | `AssignStmt` (`n0 = nb * 256`) | one factor of the product is a multiple | both factors are non-negative |
 | `ForStmt` (`for k0 in pl.pipeline(512, 4096, 512)`) | `start` and `step` are both multiples | `start` and `step` are both non-negative |
+| `ForStmt` with a symbolic start (`for ob in pl.range(core, TILES, CORES)`) | both recurse into `start` and `step` | both recurse into `start` and `step` |
+| `Min` / `Max` (`min(800 - o0, 256)`) | both operands are multiples | `Min`: both operands; `Max`: either one |
 | `tile.get_block_idx` / `tile.get_block_num` | — | a lane number is never negative |
+| `FloorMod` / `FloorDiv` by a positive constant (`(blk % 2) * 512`) | the other factor carries it | both recurse: the dividend must be non-negative too |
 | `ConstInt` | the value is a multiple | the value is `>= 0` |
 
 Sums and products compose from those; a difference proves divisibility but never
-its sign, so it is refused. **Both** columns must hold — see [Why the sign is
+its sign, so it is refused. Division by a positive constant is proven from its
+dividend rather than from the operation's name: `FloorMod` lowers to
+`arith.remsi` and `FloorDiv` to `arith.divsi`, which truncate toward zero, so
+a negative dividend yields a negative remainder. **Both** columns must hold — see [Why the sign is
 proven too](#why-the-sign-is-proven-too). The grouped-matmul weight path that
 motivated the feature therefore compiles:
 
@@ -242,11 +248,18 @@ diagnostic naming the fix — an NZ tensor must never be silently mis-addressed.
 | symbolic trailing slice offset, sign not provable | rejected — a negative offset is clamped, not caught, at the partition view |
 | logical rank 2 | blocked to `[1, C/c0, R/16, 16, c0]` — batch materialised |
 | logical rank 3 | blocked to `[B, C/c0, R/16, 16, c0]` — leading axis is the batch |
+| logical rank > 3 | blocked with every leading axis folded into the batch (see below) |
 | logical rank < 2 | rejected — the trailing pair is the fractal plane |
-| logical rank > 3 | rejected — one batch slot cannot hold two leading axes (see below) |
+| dynamic leading extent, rank > 3 | rejected — the fold needs static extents to multiply |
 | `target_memory != Mat` (or absent) | rejected — NZ→NZ is the cube operand path |
-| consumer other than `tile.load` | rejected — NZ is read-only here |
+| `tensor.slice` narrowing the leading axes only | blocked like the load that follows it (see below) |
+| `tensor.slice` windowing the trailing `[R, C]` pair | rejected — the window is not contiguous |
+| `tensor.reshape` flattening the whole tensor to `[N]` | kept as written — see [Flattening an NZ tensor](#flattening-an-nz-tensor) |
+| `tensor.reshape` to any other shape | rejected — it reinterprets coordinates the blocked form does not carry |
+| consumer other than `tile.load` / `tensor.slice` / a whole-tensor flatten | rejected — NZ is read-only here |
 | explicit stride or partial `valid_shape` | rejected |
+| dynamic `valid_shape[-2]` provably a multiple of 16 (a ragged last tile) | blocked — the row-fractal count becomes `FloorDiv(rows, 16)` |
+| dynamic `valid_shape[-2]` that may end inside a fractal | rejected — a partial fractal has no blocked form |
 | distributed tensor | rejected — `remote_load` has no NZ blocking |
 | `tensor.view` / `tensor.reinterpret_view` of NZ | rejected at op construction |
 | GM row gap above 65535 blocks, on a multi-column-block load | rejected — **temporary**, see [GM row gap](#gm-row-gap-a-temporary-guard) |
@@ -294,7 +307,9 @@ a narrowed `valid_shape` loads **fewer** row fractals and leaves a **larger**
 gap than `shapes` alone would suggest. A `[65552, 64]` INT8 weight read with
 `shapes=[32, 64]` and `valid_shape=[16, 64]` emits
 `partition_tensor_view<1x2x1x16x32>` and so a gap of 65536, not the 65520
-`shapes` implies.
+`shapes` implies. When the loaded row extent is dynamic (a ragged last tile),
+the check takes the worst case, nothing loaded, so a load that passes fits
+for every run-time width.
 
 **A single-column-block load is exempt.** `TLoadGm2L1Nz2nz` passes the load's
 column-block extent as `nBurst`, and the DMA applies `gmGap` only when stepping
@@ -305,20 +320,61 @@ load at `gShape1 = 2` corrupts 1837/4096 elements.
 
 [hw-native-sys/pto-isa#317]: https://github.com/hw-native-sys/pto-isa/issues/317
 
-### Why logical rank 4+ is rejected
+### Leading axes fold into the one batch slot
 
 pto-isa's NZ `GlobalTensor` has exactly **one** batch slot, so a logical
-`[G, E, N, K]` weight would have to fold its two leading axes into it. That fold
-is sound on the *shape* — a dense row-major tensor's leading strides collapse
-exactly, `G*E` with stride `C*R` — but not on the *offsets*: a slice `w[g, e, ...]`
-would need the coordinate re-associated into `g*E + e`, which is precisely the
-arithmetic `BlockNzOffsets` refuses to invent (see [Symbolic trailing
-offsets](#symbolic-trailing-offsets) for why re-association is unsound in
-general). Rejecting names the restriction at the annotation; the alternative is a
-view PTOAS refuses while naming SSA the user never wrote.
-
-Reshape to `[B, R, C]` before the NZ annotation, or annotate the tensor as
-`pl.ND`.
+`[G, E, N, K]` weight folds its two leading axes into it: the blocked batch is
+`G*E`, with stride `C*R`. The fold is exact because those axes are dense and
+row-major — it removes only the strides it multiplies back in — and an offset
+folds the same way, `[g, e, 0, 0]` addressing batch `g*E + e`
+(`FoldNzLeadingOffsets`).
+
+That is *not* the re-association [the trailing offsets
+refuse](#why-the-offset-is-divided-not-re-associated): nothing is divided and
+nothing is assumed about alignment, so the fold is exact for every coordinate,
+not only aligned ones. It is the same arithmetic the ND path performs at address
+computation, written once into the coordinate instead.
+
+The extents being folded must be static — a dynamic one cannot be multiplied
+into the batch — which the diagnostic names.
+
+A rank-4 parameter is what a multi-card entry declares (`[RANKS, E, R, C]`,
+sliced per rank before dispatch), so the fold is what lets a distributed program
+carry NZ weights at all.
+
+### Slicing an NZ tensor
+
+A layer- or rank-stacked weight reaches its kernel through `tensor.slice`, so
+the slice blocks like the `tile.load` that follows it: the shapes and offsets
+become rank-5, and a rank-reducing scalar index (`w[r]`) needs no `drop_dims`
+afterwards because the fold already collapsed every leading axis.
+
+Only the **leading** axes may be narrowed. A window inside the trailing `[R, C]`
+pair is rejected: in NZ order one layer's rows sit inside *every* fractal column
+block, so `[layer*R, 0]` selects `C/c0` disjoint runs, and the blocked view has
+no stride of its own to describe them — `MaterializeTensorStrides` derives a
+row-major one from the blocked shape. Annotate the stacked axis as a leading
+axis (`[LAYERS, R, C]`) instead of stacking rows.
+
+### Flattening an NZ tensor
+
+A rank-1 view of *every* element is layout-invariant: the blocked form permutes
+the index space, not the memory, so both spellings walk the same contiguous GM
+range in the same order. Such a `tensor.reshape` is therefore kept exactly as
+written — no coordinate rewrite, and the result is ND, which is what
+`prefetch.async_prefetch` wants of its source. Without it, annotating a weight
+`pl.NZ` would silently cost it its SDMA L2 warm.
+
+Any other target shape does reinterpret coordinates — `[256, 512] -> [128,
+1024]` pairs rows in logical row-major order, and in the blocked form those
+elements are scattered across fractal blocks — so it is rejected rather than
+addressed as if it were ND.
+
+An NZ argument also arrives at the orchestration entry in its *logical* shape:
+the caller allocates the weight that way, and only the compiled parameter is
+blocked. The entry restates it in blocked terms once (a metadata-only reshape,
+same elements in the same order), so every `Tensor::view` derived from it clamps
+against the rank it is written in.
 
 Sub-byte dtypes (INT4 / UINT4 / FP4 / HF4 / BOOL) are rejected as a **PyPTO
 milestone-1 scope limit, not a hardware one** — pto-isa's NZ machinery does

docs/en/dev/passes/99-verifier.md

@@ -366,7 +366,7 @@ buffer lifetime verifiers remain separate obligations.
 | 110 | `DISTRIBUTED_WINDOW_IDENTITY_MISMATCH` | Distributed tensors refer to different window buffers |
 | 111 | `TILE_VIEW_MISMATCH` | Effective TileView metadata mismatch |
 | 112 | `BUFFER_DESCRIPTOR_MISMATCH` | Buffer descriptor or multi-buffer slot count mismatch |
-| 113 | `TENSOR_LAYOUT_MISMATCH` | Call/Submit argument layout differs from the callee parameter's (DN exempt — a parameter cannot declare it) |
+| 113 | `TENSOR_LAYOUT_MISMATCH` | Call/Submit argument layout differs from the callee parameter's (DN exempt — a parameter cannot declare it; an ND argument of a `device=` dispatch exempt — the device program's parameter is the claim) |
 
 ### NoNestedCall
 

docs/en/dev/ptoas-op-status.md

@@ -69,8 +69,8 @@ for lowering/compiler plumbing, plus other dialects such as VPTO, VMI, and SIMT.
 | pto.make_prefetch_async_context | pto::PrefetchAsyncContext | internal | ✅ | — | — | — | — | validated as part of async-prefetch integration |
 | pto.get_prefetch_async_session | .session | internal | ✅ | — | — | — | — | validated as part of async-prefetch integration |
 | pto.tstore | TSTORE | tile | ✅ | ✅ | ❌ | ✅ | — |  |
-| pto.load_scalar | direct pointer load | tensor | ✅ | ❌ | ✅ | ✅ | — | emitted by `tensor.read` |
-| pto.store_scalar | direct pointer store | tensor | ✅ | ❌ | ✅ | ✅ | — | emitted by `tensor.write` |
+| pto.load | direct pointer load | tensor | ✅ | ❌ | ✅ | ✅ | — | emitted by `tensor.read` |
+| pto.store | direct pointer store | tensor | ✅ | ❌ | ✅ | ✅ | — | emitted by `tensor.write` |
 | pto.tmov | TMOV / TMOV_FP | tile | ✅ | ✅ | ❌ | ✅ | — |  |
 | pto.ttrans | TTRANS | tile+tensor | ✅ | ✅ | ✅ | ✅ | — |  |
 | **Matrix Computation (12)** |  |  |  |  |  |  |  |  |