[Bug] a5sim tensormap_and_ringbuffer chip_swimlane suite segfaults in the L2 native wait, serially, ~1 in 6

[ChaoWao]

Component

Host Runtime

Runtime Variant

tensormap_and_ringbuffer

Description

The five a5sim chip_swimlane cases, run as one serial session with the
swimlane and dep_gen channels on, segfault at a low rate inside the L2 native
wait. Observed once in six runs of the same command.

Fatal Python error: Segmentation fault

  File "python/simpler/worker.py", line 11585 in _wait_run_handle
  File "python/simpler/worker.py", line 4354  in wait
  File "python/simpler/worker.py", line 11227 in run
  File "simpler_setup/scene_test.py", line 1925 in _run_and_validate_l2

worker.py:11585 is return chip_run.wait(...), so the fault is inside the
native wait beneath the binding.

This is not #2121, though it is the closest relative. That issue had the same
symptom and the same deepest frame, and its root cause was AsyncWaitList
reaching the AICPU with a residual count because hbg's forked async_wait.h
had lost reset_for_reuse(). The fix (#2124) restored it for hbg and noted the
tensormap_and_ringbuffer twin already had it — and a5 tmr does:
src/a5/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp:258
calls sched->async_wait_list.reset_for_reuse() from SchedulerState::reset_for_reuse.
So this crash has a different cause.

It also differs from #2121 in three ways that narrow the search:

	#2121 (fixed)	this
platform	a2a3sim	a5sim
runtime	host_build_graph	tensormap_and_ringbuffer
trigger	needed -n 4	reproduces serially, -p no:xdist

Not #2182 either: that one is a2a3 onboard, faults during forked-worker
init rather than in the wait, and needs a full-suite run.

The two failing case names do rhyme — #2121's was
TestSpmdSyncStartEarlyDispatchHbg and this one is
TestSyncStartEarlyLocalOwner — so the sync-start early-dispatch machinery is
worth looking at first even though the known defect in it is handled on this side.

Steps to Reproduce

python -m pytest tests/st/a5/tensormap_and_ringbuffer/dfx/chip_swimlane/ \
    --platform a5sim --enable-chip-swimlane --enable-dep-gen \
    --device 0-15 -p no:xdist --pto-session-timeout 600 --manual include -q

Observed 1 segfault in 6 runs of exactly this command. The failing case was the
fifth, TestSyncStartEarlyLocalOwner, which passes in isolation (20/20), so
the trigger is order- or timing-dependent within the session rather than
specific to that case.

Note the channel flags: a bare sweep enables no DFX channel and does not
exercise this path at all.

Expected Behavior

Five passes.

Actual Behavior

rc=139, Fatal Python error: Segmentation fault, no diagnosis left behind —
the process dies before any teardown or artifact export.

Git Commit ID

0cc51ad

NPU Kind

N/A (simulation only)

Host Platform

Linux (aarch64)

[doraemonmj]

One candidate can be narrowed, but the current evidence is not enough to mark the crash root cause as confirmed.

The reported commit predates #2145 and the reproducer enables the resident chip-swimlane/dep-gen collectors. Before #2145, runtime replenishment could mutate dev_to_host_ and block_ranges_ while drain shards performed lock-free lookups; container rehash/reallocation made that a real native data race, and #2145 added the missing reader/writer synchronization. Such a background native crash can terminate the process while Python's main thread is blocked in chip_run.wait(), so the Python frame alone does not rule it out.

However, there is no core dump, sanitizer report, or faulting native-thread stack tying this occurrence to the mapping lookup. It could also be a different lifetime error below wait. The clean way to resolve this issue is to rerun the serial reproducer on a head containing #2145: no recurrence over a meaningful repetition count would support closing it as fixed; any recurrence should retain a native backtrace/core and would disprove that attribution.

[ChaoWao]

Still live on current main, and it can abort as well as segfault.

Probed 20 consecutive serial runs of the suite on 8414356a1:

=== 20 runs on 8414356a1: segfaults=0 other-failures=1
run 6   FAIL(rc=134)

rc=134 is SIGABRT, not SIGSEGV — but the same deepest frame as the occurrence in the issue body:

Fatal Python error: Aborted
Thread 0x0000ffff8e12b940 (most recent call first):
  python/simpler/worker.py, line 11585 in _wait_run_handle
  python/simpler/worker.py, line 4354 in wait
  python/simpler/worker.py, line 11227 in run
  simpler_setup/scene_test.py, line 2089 in _run_and_validate_l2
  tests/st/a5/tensormap_and_ringbuffer/dfx/chip_swimlane/test_chip_swimlane.py, line 180 in test_run

Two things this adds:

• Not fixed by Fix: preserve device profiling across captures and repeated runs #2225. I had suggested that link and have since withdrawn it — see the correction on [Bug] a5 onboard chip_swimlane: single_scheduler_idle intermittently publishes zero records, so no chip_swimlane_records.json is written #2220. This probe is independent confirmation: the failure reproduces on a base that contains Fix: preserve device profiling across captures and repeated runs #2225.
• The fault is an abort, not only a segfault. A raw SIGSEGV says little beyond "bad pointer", whereas SIGABRT usually carries a reason — a failed assertion, an uncaught C++ exception reaching std::terminate, or glibc detecting heap corruption. None of those messages appeared in this log, which is consistent with pytest's capture swallowing the C++ stderr that precedes the abort.

Chasing that message is the obvious next step and is cheap, since the rate is roughly 1 in 20 and each run is ~20 s: re-run the same suite with pytest capture disabled so anything libstdc++ or glibc prints before aborting survives. I have that running now and will post whatever it yields.

Rate note: the issue body estimated roughly 1 in 6; this probe saw 1 in 20. Same family, and 20 runs is far too few to call the rate changed.

[ChaoWao]

The uncaptured probe did not reproduce the abort

I said above I would run this with pytest capture off to recover the C++ stderr
that precedes an abort() / std::terminate, and post whatever it yielded.
Result: 0 of 25 runs aborted, so there is no abort message to report.

Same commit (8414356a1), same command as the earlier probe with -q replaced
by -s:

python -m pytest tests/st/a5/tensormap_and_ringbuffer/dfx/chip_swimlane/ \
    --platform a5sim --device 0-15 -p no:xdist --pto-session-timeout 600 \
    --require-pto-isa --manual include --enable-chip-swimlane --enable-dep-gen -s

probe	runs	this bug	other failures
earlier, -q	20	1 (rc=134, abort in _wait_run_handle)	0
this one, -s	25	0	1 (unrelated — see below)

I am not reading that as evidence the bug is gone or as evidence that -s
suppresses it. At roughly 1-in-20, a clean run of 25 is weak: if the rate is
unchanged, P(0 hits in 25) is about 28%, so this outcome is entirely ordinary
under the hypothesis that nothing changed. It is a failed capture attempt, not a
result.

What it does mean practically: recovering the abort message needs either a much
longer batch or a way to capture the message that does not depend on catching the
failure live. The second is cheaper — running under catchsegv/coredumpctl, or
just making sure a core file is kept and reading the C++ frames out of it, gets
the diagnostic from the next natural occurrence instead of requiring me to sit
on top of one. Worth noting that the earlier hit did dump core
(Aborted (core dumped)) and that core was not retained.

The one failure in this batch is a different bug

Run 19 failed with rc=1, not an abort, and its cause is unrelated: the dep_gen
collector collected zero of the 5 records the device wrote, so deps.json was
never emitted and a post-processing tool then failed on the missing DAG. Filed
separately as #2246 so it does not get absorbed into this report.

Recording it here only because it is the kind of thing that could otherwise be
miscounted as an instance of this issue in a future probe: this bug is an abort
or segfault inside the native wait, while that one is an ordinary red test with a
CalledProcessError. They are distinguishable by exit status (134 vs 1).