[Bug] a2a3 onboard: predicated clobber intermittently does not dispatch when its gate is 1 (max_diff is exactly POISON-SENTINEL)

[ChaoWao]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

TestPredicatedDispatch::PredicateTrueDispatches intermittently fails its golden comparison on st-onboard-a2a3, and the failing value identifies the mechanism exactly:

AssertionError: TestPredicatedDispatch::PredicateTrueDispatches:
  Golden mismatch on 'x': max_diff=957.0, rtol=0, atol=0

957.0 == 999.0 - 42.0 == POISON - SENTINEL. In this case the gate is 1, so the
clobber task (X = 999.0) is supposed to dispatch on top of the producer's
42.0. x[0] held 42.0, so the producer ran and the gated clobber did
not dispatch — the predicate behaved as if the gate were 0.

This is not a stale or uninitialised buffer: INIT_VAL is -1.0, and 42.0 is
the legitimate producer output, so the graph ran and only the predicated task
was missing. The comparison is exact (rtol=0, atol=0), so the diff is not
numerical drift either.

It is intermittent. Observed once, at inv=11 in an xdist worker (gw3), so
the case had already passed ten invocations in that same process before failing.

Not reproducible on demand. Re-running the same CI job on the same commit
passed. Locally, on real a2a3 silicon with the same binaries and the same
job-level watchdog env the workflow sets
(SIMPLER_SCHEDULER_TIMEOUT_MS=2000, SIMPLER_OP_EXECUTE_TIMEOUT_US=3000000,
SIMPLER_STREAM_SYNC_TIMEOUT_MS=4000), three consecutive runs passed.

Steps to Reproduce

task-submit --device auto --device-num 1 --run \
  ".venv/bin/python -m pytest \
     tests/st/a2a3/tensormap_and_ringbuffer/predicated_dispatch/test_predicated_dispatch.py \
     --platform a2a3 --device \$TASK_DEVICE --exclude-level 4 \
     --require-pto-isa --pto-session-timeout 600 --manual exclude"

Intermittent; one observed occurrence. It surfaced inside the full
st-onboard-a2a3 sweep, under xdist, after ten passing invocations of the same
case in the same worker process — so repeated invocation in one process, rather
than a single cold run, is the shape worth trying.

Expected Behavior

With gate[0] == 1, the predicated clobber dispatches and x[0] == y[0] == 999.0.

Actual Behavior

x[0] == 42.0 — the producer's value. The predicated clobber did not run, so
the predicate resolved false against a gate of 1, or the task was dropped
between predicate evaluation and dispatch.

Git Commit ID

cfb6c92

CANN Version

CI self-hosted a2a3 runner — as installed there

Driver Version

CI self-hosted a2a3 runner — as installed there

Host Platform

Linux (aarch64)

Additional Context

Failing job: https://github.com/hw-native-sys/simpler/actions/runs/34843921364/job/103975978255
(green on re-run of the same commit)

Context that is probably not the cause, recorded so it is not re-derived.
The same job also failed three L3 async cases with 507018, and a sibling a5
runner was undergoing a card soft restart in the same window. But a 507018 is
a reaped or timed-out op, whereas this is a completed run with a wrong
result — a different class of failure, which is why it is filed separately
rather than folded into the watchdog story. A destabilised device pool is a
plausible contributor and cannot be ruled out from one occurrence.

Why this one deserves a look despite being a single occurrence: the whole
point of this scene test is that a predicate gates a dispatch, and the observed
value is precisely "the gated task did not run". If the predicate path can drop
a dispatch it should have made, the same hazard is silent in production — a
skipped task leaves a plausible-looking earlier value rather than an error.
PredicateFalseSkips passing tells us nothing here: skipping is its expected
outcome, so it cannot detect an over-eager skip.

Related: #995

[ChaoWao]

Root-cause candidate: the predicate read has no cache invalidate

Found by reading the dispatch path. Strong hypothesis, not yet proven by experiment — see the last section for what would prove it.

DispatchPredicate::pass() reads the predicate element straight out of GM with a bare memcpy (src/a2a3/runtime/tensormap_and_ringbuffer/runtime/submit_types.h:67):

bool pass() const {
    if (op == PredicateOp::NONE) return true;
    int64_t v = 0;
    __builtin_memcpy(&v, reinterpret_cast<const void *>(addr), elem_size);
    ...

No cache_invalidate_range. The comment above it justifies the read like this:

the predicate address' producer has completed and the value read is current

That argues ordering — the producer task finished before this task became ready. It does not establish visibility: on a2a3 the AICPU reading GM that an AICore wrote needs an explicit invalidate, and this runtime's own scheduler does exactly that everywhere else it reads an AICore-written word:

site	what it reads	invalidates first
scheduler/scheduler.h:1488	async-wait completion counter	cache_invalidate_range(counter_line, …)
backend/sdma/sdma_completion_scheduler.h:35	SDMA completion record	cache_invalidate_range(…, CHIP_ALIGN_SIZE)
submit_types.h:67	dispatch predicate	no

This predicts the observed value exactly

generate_args fills the gate with INIT_VAL, and for the gate that is torch.full((16,), -1, dtype=torch.int32) — so the pre-producer content of that line is -1.

A stale read therefore yields -1, and the predicate is gate[0] > 0:

-1 > 0  ->  false  ->  clobber skipped  ->  x[0] stays 42.0 (SENTINEL)
expected 999.0 (POISON);  999.0 - 42.0 = 957.0

which is the reported max_diff=957.0, to the bit, with rtol=0, atol=0.

It also predicts the three things that were otherwise unexplained

Why it is intermittent. The fault needs that line to be resident in the AICPU's cache at the moment of the read. Usually it is not, the read reaches HBM, and the fresh value is seen. Cache residency is exactly the kind of thing that varies with timing and pressure — and this failed at inv=11, after ten clean invocations in the same process.

Why the failure is one-directional. A stale read returns the line's earlier content, which here is the pre-producer value, so the predicate is biased toward false — toward skipping. Every failure of this mechanism is an over-eager skip; it can never cause an over-eager dispatch.

Why PredicateFalseSkips can never catch it. Skipping is that case's expected outcome, so it is blind to an over-eager skip by construction. I noted that blindness in the report above as a caveat; the mechanism explains why it is structural rather than incidental.

Why hardware only. invalidate_range_impl is a real cache op onboard (src/common/platform/onboard/aicpu/cache_ops.cpp:20) and an empty function on sim (src/common/platform/sim/aicpu/cache_ops.cpp:17). So the hazard cannot exist on a2a3sim, matching the observation that this only ever appeared on a2a3 onboard.

Proposed fix

Invalidate the predicate's line before the read, matching the idiom the same scheduler already uses for AICore-written counters. It costs one invalidate on a path only predicated tasks take, and predicated dispatch is the one feature whose correctness depends on that read.

What would prove it

Adding the invalidate and observing no further failures proves little on its own — the failure is roughly one in many, so absence is weak evidence. A convincing test constructs a case that reliably leaves the predicate's line resident and stale in the AICPU cache before the read, shows the wrong result deterministically, and shows the invalidate removing it. I have not built that yet.

[ChaoWao]

Withdrawing the missing-invalidate root-cause candidate — the experiment refutes it

I proposed above that DispatchPredicate::pass() reads the gate without a
cache_invalidate_range and can therefore see a stale line. I ran that on real
a2a3 silicon. It does not reproduce, and the reasoning behind it was wrong.
Retracting it so nobody spends time on the fix I suggested.

The experiment

The hypothesis needs the predicate's line to be resident in the AICPU cache with
its pre-producer content at the moment of the read. So force exactly that, and
measure, instead of waiting for a 1-in-many failure. Per trial, in the test's
own orchestration kernel (AICPU side):

1. read gate[0] — the line is now provably resident, and the value read is the
   positive control for that
2. submit an AICore task that writes a value different from what step 1 saw
3. read gate[0] back through get_tensor_data(), which waits for the producer
   and then does the same bare memcpy pass() does

A read returning step 1's value is a stale read. 64 trials per case, both cases:

EXP2233 case=1 trials=64 stale_hits=0 first_warm=-1 first_got=7
EXP2233 case=2 trials=64 stale_hits=0 first_warm=-1 first_got=7

128/128 fresh. first_warm=-1 is the positive control firing: the line was
resident holding the host-DMA'd -1, the AICore then wrote 7, and the next
AICPU bare read returned 7. A tighter window than the real dispatch path, and
it never went stale.

Why the original analysis was wrong

Three things I should have checked before posting it, each of which sinks it on
its own:

1. The two sites I cited as "the runtime invalidates everywhere else" are
   spin-polls, not post-handshake reads. poll_sdma_post_done_record()
   (sdma_completion_scheduler.h:31) and the async-wait counter loop
   (scheduler.h:1488) re-read one address while the AICPU spins, before any
   completion has been signalled — without an invalidate the first read caches
   the line and the loop never observes the update. pass() reads after the
   producer is known complete. Same instruction, categorically different
   requirement; my table lumped them together and that was the error.

2. The sanctioned orchestration read does the same bare memcpy.
   get_tensor_data() (runtime_core.cpp:216) waits via
   wait_for_tensor_ready() and then memcpys, with no invalidate anywhere in
   either function. If the invalidate were required, every workload reading a
   device-produced tensor from orchestration would be broken, not just this
   predicate.

3. The producer already flushes. kernels/aic/kernel_write_gate.cpp ends
   with dcci(&out[0], SINGLE_CACHE_LINE, CACHELINE_OUT) — the AICore cleans its
   own store. Coherence here is the producer's job, and on a2a3
   OUT_OF_ORDER_STORE_BARRIER() is deliberately a no-op with the comment
   "no-op on real hardware (dcci handles cache coherency)"
   (src/a2a3/platform/onboard/aicore/inner_kernel.h:41). So the design says
   dcci, not consumer-side invalidation, and the experiment agrees.

The 957.0 == 999.0 - 42.0 arithmetic still says the clobber did not dispatch —
that part of the original report stands. What does not stand is my explanation of
why, and the one-line fix I proposed would have been a no-op against the real
cause.

What this rules out, and what it does not

Ruled out: a resident stale line at the predicate read, under a warm-then-write
sequence tighter than the real one. Also ruled out is my framing that the
runtime's invalidate idiom was applied inconsistently — it is applied exactly
where polling requires it.

Not ruled out: a rare window in which the AICPU observes the producer's
completion before the producer's flush has landed. The kernel's output dcci
is followed by write_reg(RegId::COND, MAKE_FIN_VALUE(task_id))
(aicore_executor.cpp:243) with no intervening barrier, because the design
holds dcci to be sufficient. Whether it is sufficient relative to the SPR
write immediately after it is a pto-isa / hardware-semantics question I cannot
settle by reading this repository, and it is the only ordering left on the path
between "gate written" and "predicate read". Note my experiment cannot see this
one either: get_tensor_data() reaches the value through the scheduler's
completion bookkeeping, a longer path than the scheduler's own dispatch-point
read, so it gives any in-flight flush far more time to land than the real window
does.

Flagging that as the remaining lead rather than the answer. It needs someone who
owns the AICore completion contract, and it is a claim about hardware ordering,
not about this runtime's code being internally inconsistent — which is what I
wrongly claimed the first time.

[ChaoWao]

Reproduced locally for the first time, and the barrier hypothesis is null

Follow-up to the retraction above. Three results, one of which is the useful one.

1. This bug reproduces on this box, at about 1%

Across 89 runs of predicated_dispatch on real a2a3 today it fired once,
with the reported signature exactly:

TestPredicatedDispatch::PredicateTrueDispatches: Golden mismatch on 'x': max_diff=957.0, rtol=0, atol=0

That is the first occurrence outside CI. It matters mainly because it makes the
bug chaseable: 89 runs cost about 20 minutes wall clock under a
task-submit device lock, so a 200-run instrumented batch is a perfectly
reasonable next step rather than a wait-for-CI proposition.

2. A producer-visibility failure is sufficient to produce this exact signature

Negative control: delete the output dcci from kernels/aic/kernel_write_gate.cpp
so the gate write is never flushed. Result, deterministically on the first run:

TestPredicatedDispatch::PredicateTrueDispatches: Golden mismatch on 'x': max_diff=957.0, rtol=0, atol=0

Byte-identical — same case, same tensor, same value — and PredicateFalseSkips
still passes, because a stale gate read is biased toward false and skipping
is that case's expected outcome. So the failure class is now pinned: the gate
value is not what the predicate read, and the one-directional bias in the
original report is explained rather than assumed. It is not "the task was
dropped" and not "the predicate address was wrong".

3. The missing dsb(DSB_DDR) is not shown to be the cause

Worth writing down since the code observation looks compelling and is a dead end
so far.

The observation: pto-isa documents a2a3 visibility as a pair — "visibility
on A2/A3 requires an explicit dsb(DSB_DDR) + dcci pair around the store: AICORE
caches are not coherent between cores" (pto/npu/a2a3/grid_intrinsic.hpp:447),
and pto/comm/a2a3/TNotify.hpp implements that pair. This runtime does too when
it publishes completion data — defer_flush_range(...), dsb(DSB_DDR),
pipe_barrier(PIPE_ALL) (runtime/async_kernel_api.h:83). But the per-task
output path does not: the scene-test kernels flush without any dsb, and
execute_task() ends with OUT_OF_ORDER_STORE_BARRIER()
(aicore/aicore_executor.cpp:43) which is ((void)0) on a2a3 onboard —
i.e. there is a barrier slot in exactly the right place, immediately before
write_reg(RegId::COND, MAKE_FIN_VALUE(task_id)), and it is compiled away.

The test: define OUT_OF_ORDER_STORE_BARRIER() as dsb(DSB_DDR) on a2a3
onboard, rebuild, and A/B it.

arm	runs	failures
stock (((void)0))	40	0
dsb(DSB_DDR)	40	0

Null. The one occurrence in §1 landed on the dsb build, before those 40 —
so my first read of it was that the barrier had made things worse. Forty clean
runs on that same build says otherwise: 1/44 versus 0/45 is not a difference, and
attributing that single failure to the barrier was wrong. Both directions of that
claim are unsupported; the barrier neither causes nor fixes this.

So the code-level inconsistency in the per-task flush path is real as an
observation and remains unproven as a mechanism. Anyone tempted by it should know
it has been A/B'd at n=40 per arm and came out flat.

What I would do next

The rate is now known, so instrument rather than guess. On every predicate
evaluation log the call site and the value read — there are four pass() call
sites in the a2a3 TMR scheduler (scheduler.h:547, 1108, 1149, and
push_ready_routed reached from the orchestrator at orchestrator.cpp:1247/1265)
— then run ~200 iterations to catch one or two occurrences. That distinguishes
the two remaining candidates directly:

• the read returned a stale gate (value -1, the pre-producer content), or
• the predicate was evaluated at a site where the producer had not completed.

Both produce 957.0; only the log distinguishes them. I have not run that batch.

[ChaoWao]

Root cause found, with a reliable reproducer and a validated fix

Superseding both of my earlier comments. The first one (missing consumer-side
cache_invalidate_range) and the retraction of it were both wrong, for the same
reason: I was reading bare/fresh pairs from an experiment that could not tell
"stale cache line" apart from "value had not arrived yet", because the invalidate
itself costs time. The measurement below removes that confound.

A reproducer that fires every run

The graph, the kernels and the golden are unchanged. The only change is that
the true-predicate case is replicated in CASES so one process performs 33
Worker.run() invocations instead of 2:

CASES = [ {...PredicateFalseSkips, "params": {"case": 1}} ] + [
    {"name": f"PredicateTrueDispatches{i:02d}", ..., "params": {"case": 2}}
    for i in range(32)
]

That turns roughly 1-in-489 into 20 red runs out of 20. The axis matters and
it is the one the original report was already pointing at: inv=11. It is per
invocation, not per predicate evaluation — I confirmed the latter separately by
running 32 predicate evaluations inside a single invocation, which gave 0
failures in 6400 evaluations where a per-evaluation hazard predicted ~22.

The mechanism, measured directly

Instrumented pass() so that on a false result it re-reads the element with no
cache maintenance whatsoever, spinning until the value changes or 50000 reads
elapse, and only then invalidates once:

EXP2233 site=2 v0=-1 spins=71    v1=1 v2=1 target=0   <- the failure
EXP2233 site=2 v0=0  spins=50000 v1=0 v2=0 target=0   <- case 1, control

• v0 = -1 — the dispatch-point read returned the pre-producer value.
• spins = 71 — 71 plain re-reads later the value was 1, with no dcci, no
  invalidate, nothing. Across the residual failures the range was 35–679.
• v2 == v1 always — the invalidate contributes nothing.
• The control line is the negative control: case 1's gate is legitimately 0, never
  changes, and the loop runs its full 50000. So the loop does spin, and it does
  observe changes when they happen.

So this is not cache staleness. The AICPU can see memory fine; the predicate
is simply evaluated before the producer's write is there. The task was routed as
ready — fanin satisfied, i.e. producer completed — while its data had not landed.

That is consistent with how completion travels versus how data travels: the FIN
is write_reg(RegId::COND, MAKE_FIN_VALUE(task_id)), an SPR write
(aicore/aicore_executor.cpp:243), while the kernel's output goes to HBM. The
register path can reach the AICPU ahead of the memory path.

Four arms, 640 invocations each

arm	red runs
baseline	20 / 20
consumer-side cache_invalidate_range in pass()	10 / 20
producer-side dsb(DSB_DDR) before FIN (OUT_OF_ORDER_STORE_BARRIER())	7 / 20
consumer-side bounded re-read in pass()	0 / 20

The middle two are informative rather than fixes. The invalidate "helps" only
because dc civac; dsb sy; isb burns enough time for the write to land — which is
exactly the confound that misled me. The producer barrier is a real and
statistically solid effect (20/20 → 7/20, Fisher p ≈ 2e-5) and it is the right
direction, but it does not close the window, which is what makes me think the
SPR path genuinely outruns the data rather than merely racing an unflushed line.

The bounded re-read closes it (20/20 → 0/20, p ≈ 1e-11). What I tested was the
blunt form — on a false comparison, re-read up to 4096 times before concluding
false:

for (int32_t attempt = 0; attempt < 4096; attempt++) {
    int64_t v = 0;
    __builtin_memcpy(&v, reinterpret_cast<const void *>(addr), elem_size);
    __asm__ __volatile__("" ::: "memory");
    ... sign-extend, compare ...
    if (ok) return true;
}
return false;

What still needs deciding, and what I have not measured

I am not proposing to ship that loop as-is, for two reasons.

1. A legitimately-false predicate pays the whole bound. PredicateFalseSkips
   is a real use of the feature and it would spin 4096 reads every time. I did
   not measure that cost, and sizing the bound by the observed 35–679 window is
   fitting a constant to one workload on one box.
2. The defect is arguably one level up. Every AICPU read of AICore output
   after completion has this exposure, not just the predicate — get_tensor_data()
   (runtime/runtime_core.cpp:216) waits on the same completion-derived signal and
   then does the same bare memcpy. It is latent there only because the
   orchestration path is slower. A fix that makes readiness imply visibility would
   cover both; a spin inside pass() covers one caller.

A spin is allowed on the dispatch path by .claude/rules/codestyle.md §5 (only
sleeping and yielding are banned), so the blunt form is not disqualified — it is
just the narrower of the two options.

Happy to turn either shape into a PR. The reproducer above is worth keeping as a
regression test in some reduced form; 33 invocations adds roughly 3 s to the case,
and I have not measured how few invocations still catch it reliably.

[ChaoWao]

Root cause, stated plainly

The evidence is in the comment above; this is the one-line version, plus the one
thing that comment left out.

The dispatch-point predicate read assumes that a task being ready implies its
producer's output is readable. On a2a3 that implication does not hold.

The assumption is written down, in the comment over PredicateOp
(src/a2a3/runtime/tensormap_and_ringbuffer/runtime/submit_types.h:46-49, and
identically in the a5 TMR and src/common/host_build_graph copies):

// at the dispatch point — the task is ready (fanin satisfied), so the predicate
// address' producer has completed and the value read is current, ...

The first clause is true — readiness does mean the producer reported FIN. The
second clause is false: completion is not visibility. Any fix has to change
that comment in the same commit, which is why it belongs in this report.

The failure then follows mechanically:

FIN observed -> task routed ready -> pass() reads gate -> gets the pre-producer -1
-> -1 > 0 is false -> the clobber is retired inline instead of dispatched
-> x[0] stays 42.0 (expected 999.0) -> max_diff = 957.0

Proven by measurement

• The read returns the pre-producer value (v0 = -1 against a gate that
  should be 1).
• The value arrives 35–679 plain re-reads later, with no dcci, no
  invalidate and no barrier anywhere in the loop.
• Therefore not cache staleness: the post-invalidate read v2 never differed
  from the plain re-read v1.
• "Gate not visible at the read" is sufficient for this exact signature —
  deleting the producer's dcci reproduces byte-identical max_diff=957.0
  deterministically.
• A bounded consumer-side re-read closes it: 20/20 red → 0/20.

Inferred, not measured

Why the completion signal can overtake the data. FIN is
write_reg(RegId::COND, MAKE_FIN_VALUE(task_id)) — an SPR register write
(aicore/aicore_executor.cpp:243) — while the kernel's output is a memory write,
so the two travel different paths and the register one can land first. That is
the explanation which fits, and the producer-side dsb(DSB_DDR) result supports
it (it reduces 20/20 to 7/20 but cannot close the window, i.e. waiting for your
own flush is not enough). I did not instrument the hardware paths. What is
measured is the fact — completion observed before data visible; the path
asymmetry is my reading of why that is possible.

Scope

Not predicate-specific. get_tensor_data()
(runtime/runtime_core.cpp:216) waits on the same completion-derived signal via
wait_for_tensor_ready() and then does the same bare memcpy. It carries the
same exposure and is latent only because the orchestration path is slower. So
"readiness does not imply visibility" is the defect; the predicate is just where
it became observable, because there a stale read silently changes control flow
instead of producing a wrong number.

[ChaoWao]

Another observation, same exact signature, on a2a3 silicon under a task-submit device lock:

AssertionError: SceneTest case failed: TestPredicatedDispatch::PredicateTrueDispatches:
  Golden mismatch on 'x': max_diff=957.0, rtol=0, atol=0

957.0 == POISON - SENTINEL again, so the gate was 1 and the clobber did not dispatch — the same mechanism this issue describes, not drift and not a stale buffer.

Context that may help bound the rate:

• Seen in a full pytest examples tests/st --platform a2a3 sweep (the L2 tensormap_and_ringbuffer group, 61 cases), 1 failed / 60 passed. Two earlier full sweeps of the same corpus on the same box, same day, were green — so roughly 1 in 3 full sweeps here.
• The same case then passed 10/10 when run in isolation in a fresh process each time (pytest tests/st/a2a3/tensormap_and_ringbuffer/predicated_dispatch, one device, sequential). Consistent with this issue's note that the failure landed at inv=11 after ten clean invocations in the same process: isolation alone does not reproduce it.
• The box was heavily loaded by unrelated concurrent NPU jobs on other dies at the time, so this observation cannot separate "load-sensitive" from "invocation-count-sensitive".

No fix attempted; recording the data point rather than re-deriving the mechanism.

[ChaoWao]

Root cause, remeasured — two defects, one on each side

This supersedes my earlier comments on this issue. The numbers I posted
before came from a harness whose round request and round ACK shared one
counter, which let the producer publish round r+1 the moment it saw the
ACK for r — before the consumer had requested that round. Splitting them
into round_request and round_ack changed the measured magnitudes
materially, so treat the earlier figures as withdrawn. The tool is now in
tree (#2317); the counter split is a follow-up to it.

What survived the fix is the shape, and it is reproducible across three
independent runs of 50 rounds per cell.

The defects

side	defect	fix
producer (AICore kernel)	signals FIN without waiting for its output flush to land	wait after the flush (dsb(DSB_DDR))
consumer (AICPU scheduler)	after observing the completion signal, reads GM with nothing ordering the two reads	ldar the flag, or dmb ld before the data load

Both are required. Neither alone helps.

Measured, three runs, older count out of 50 per cell

Producer waits for its flush (P1):

consumer arm	run a	run b	run c
C0 bare load	3	23	27
C1 dsb sy	0	0	0
C2 invalidate payload line	0	0	0
C3 dsb sy + invalidate	0	0	0
C4 delay ≈ invalidate cost	6	20	29
C5 dmb ld	0	0	0
C6 ldar on the flag	0	0	0
C7 delay ≈ dsb sy cost	10	18	20

Producer does not wait for its flush (P0) — every arm fails:

	C0	C1	C2	C3	C4	C5	C6	C7
run a	49	41	5	1	44	50	50	50
run b	49	48	46	47	50	50	50	50
run c	49	49	41	41	48	50	50	50

Across all three runs: 1600 rounds each, zero newer outcomes and zero
wait timeouts, so the frozen handoff held for every round — a first load
below the round the FIN carried cannot be the producer having advanced.

What this establishes

It is ordering, not latency. Both delay arms spend the measured cost of
the operation they stand in for — C4 the invalidate, C7 the dsb sy — and
both fail in all three runs, while the operations themselves are 0/50. So
the barriers are not buying time.

dmb ld is enough, and so is ldar on the flag. dsb sy is not
needed; neither is cache maintenance. That matters because this sits on the
dispatch path.

Read-side ordering cannot rescue a producer that has not flushed. Under
P0 the ordering arms are the worst performers (C5/C6 at 50/50), which is
what you would expect: ordering a read against a write that has not left the
producer yet changes nothing.

Why the consumer side is a defect and not a hardware bug

The wait loop and the data read are two loads to different addresses with
only a branch between them. On ARM a control dependency does not order
load→load — this is the read side of the standard message-passing case, and
the prescribed remedies are exactly an acquire load on the flag or a
dmb ld between the two. The COND register being Device-nGnRE does not
help: the nR attribute orders Device accesses against each other (which is
why polling N cores costs ~95 ns × N), not against Normal cacheable
accesses.

This also explains why the runtime's existing pollers never hit it: the SDMA
completion record and the async-wait counter poll a value that is the
data, so there is no second address to order against. Predicated dispatch is
the one place where the flag lives in MMIO and the data in GM.

Where the fix goes

Read side: pass() (runtime/submit_types.h) is the C0 shape. But
ldar on the scheduler's COND poll is the better placement — one change
covers every consumer of a completion signal rather than only predicates,
and it adds nothing to the data-read path.

Write side: the AICore kernels publish with store → dcci → FIN and no
wait. The scene-test kernels all have this shape.

Withdrawn and not established

• "Cache invalidation partially rescues P0" — withdrawn. Run a alone
  showed C2/C3 at 5/1, and runs b and c put them at 41–47. Not reproducible.
• The prefetch anomaly I reported earlier (deliberately warming the
  payload line made P0 pass) — withdrawn. It was an artifact of the shared
  counter; with request and ACK separated, the prefetch arms behave the same
  as the rest.
• Which microarchitectural mechanism produces the reordering —
  speculative issue past the branch, or a load satisfied from a stale source
  — is not distinguished by this experiment. All candidates are covered by
  the same fix, so the fix does not depend on knowing.
• No AICore-side instruction-level evidence. llvm-objdump does not
  support elf64-hiipu and ccec rejects -S, so the producer arms rest on
  source plus the P0/P1 behavioural split.
• Not yet verified in the runtime itself. The 0/50 figures are from the
  standalone tool. The 20-runs-of-33-invocations reproducer for this issue is
  the next step for that.

[ChaoWao]

Fixed on the write side — PR #2324

Measured on the real path, not just in the standalone tool. The predicated-
dispatch case replicated to 33 invocations per process (the inv= axis this
report already pointed at), 20 runs per arm on a2a3 silicon:

arm	red runs
upstream/main	18 / 20, every one max_diff=957.0
with the flush wait	0 / 20
repeat	0 / 20
on the exact shipped binary	0 / 20

Full a2a3 tensormap_and_ringbuffer suite passes (44 L2 + 4 L3).

The cause

A kernel's output dcci issues a write-back and does not wait for one.
execute_task() returns, the task writes FIN, and a consumer that observes FIN
can read the output before the flush has landed. OUT_OF_ORDER_STORE_BARRIER()
already sits in exactly the right place — between the kernel's return and the
FIN write — and was a no-op on onboard, on the premise that dcci alone carries
coherency. It now waits.

Correcting my previous comment

I said the consumer side needed an ordering barrier too. That part does not
apply to this runtime. The scheduler's completion poll already follows its
COND read with rmb() (dsb ld), with a comment naming the exact
Device-to-Normal reorder hazard, and that is the only path that observes
completion — the other COND read only builds a stall diagnostic string. So the
read side was already ordered, and more strongly than the dmb ld my
measurements showed sufficient. My earlier framing was drawn from the standalone
tool, where the consumer deliberately reads bare; the runtime does not.

The tool's evidence still stands as corroboration in the other direction: with a
producer that does not wait for its flush, every read-side variant fails
41–50 of 50, including ones that add ordering or cache maintenance. No read-side
treatment can rescue data that has not been flushed. That is what makes the write
side the whole fix here.

Two gaps in the PR, stated there as well

• a5 is unvalidated. The macro was identically a no-op on a5, so the sibling
  moves in the same commit, but this box is a2a3 only. a5 has review and a build
  behind it, nothing more.
• The per-task cost is unmeasured. One dsb per task completion, on the
  dispatch tail. tools/benchmark_rounds.sh cannot run here — its corpus names
  example paths that no longer exist, so all nine cases fail before doing any
  work, on main as well. If the cost matters, the narrower shape is to wait
  only when the completing task has a consumer that reads its output.

Incidentally

build_runtimes does not treat platform/onboard/aicore/inner_kernel.h as a
dependency — editing it and rebuilding leaves aicore_kernel.o byte-identical.
That silently invalidated one of my earlier A/B runs on this issue (both arms ran
the same binary and the barrier looked ineffective). rm -rf build/cache/<arch>/onboard is required. Worth knowing for anyone measuring a
platform-header change.

[ChaoWao]

Closing note: this ships without a regression test, by decision rather than oversight.

The reproducer that made it near-certain is the predicated_dispatch case replicated to 33 invocations per process, which turns roughly 1-in-489 into 18/20. Adding that to the suite costs about 11 s of CI on every run, and the contract it guards is now enforced one level down — OUT_OF_ORDER_STORE_BARRIER() waits for the flush for every kernel, not just this one, so a per-case test would only re-check the platform macro.

tools/cann-examples/aicore-fin-ordering covers the contract directly if it ever needs re-checking: its P0 arm is exactly "producer does not wait for its flush", and it reports 41–50 failures out of 50 in that arm. It is developer tooling, not CI, so this is a barrier someone has to run deliberately — recorded here so that is a known gap rather than an assumed one.