[Bug] a5sim dep_gen intermittently collects zero of the device's records, so deps.json is dropped

[ChaoWao]

Platform

a5sim (Ascend 950 simulation)

Runtime Variant

tensormap_and_ringbuffer

Description

The dep_gen collector intermittently collects none of the records the device
wrote, so reconcile_counters() correctly refuses to emit deps.json and the
run loses its whole task DAG.

Observed once in 25 serial runs of the a5sim chip_swimlane suite on
8414356a1 (case TestChipSwimlane_aicpu_threads_2):

[WARN] reconcile_counters: [dep_gen_collector.cpp:260] dep_gen reconcile:
  record count mismatch (collected=0 + dropped=0 != device_total=5 + overflow=0,
  silent_loss=5)

This is a total loss, not a partial one, and not a capacity problem:

counter	value	what it rules out
collected	0	append_buffer_records() never ran — on_buffer_collected never fired for any buffer
dropped	0	device did not drop (free_queue / ready_queue were fine)
overflow	0	no submit expanded past its slot
device_total	5	the device did record all 5 submits

No dep_gen reconcile: un-flushed buffer error accompanied it, so
current_buf_ptr was clear — the device completed its hand-off. So the five
records were published by the device and collected by nobody.

Hypothesis (not yet proven)

The loss sits between the device publishing a full buffer and the host's
collector shard appending it — a ready-queue hand-off or shutdown race, rather
than anything on the device side.

There is a matching asymmetry in the reconcile path: it inspects
current_buf_ptr for a buffer the device never flushed
(dep_gen_collector.cpp:227-241), but never re-checks the ready queue for a
buffer that was flushed and never collected. So the second case is visible only
as the arithmetic mismatch above and is never recovered, where the first one is
at least named precisely.

I have not confirmed the mechanism — stop() is documented as "drain queues,
join threads" (dep_gen_collector.h:31), so if that drain is sound the loss must
be elsewhere. Treating it as a hypothesis, not a finding.

Steps to Reproduce

for i in $(seq 1 25); do
  python -m pytest tests/st/a5/tensormap_and_ringbuffer/dfx/chip_swimlane/ \
      --platform a5sim --device 0-15 -p no:xdist --pto-session-timeout 600 \
      --require-pto-isa --manual include --enable-chip-swimlane --enable-dep-gen \
      > "run${i}.log" 2>&1
  grep -l "record count mismatch" "run${i}.log"
done

Roughly 1 in 25. Grep for record count mismatch rather than relying on the
exit status — see the diagnosability note below for why the failure that reaches
pytest names something else entirely.

Expected Behavior

Either every record the device wrote is collected and deps.json is emitted, or
the run reports the loss as the thing that went wrong.

Actual Behavior

deps.json is absent, and the test fails with an unrelated-looking error from a
post-processing tool:

subprocess.CalledProcessError: Command '[... '-m',
  'simpler_setup.tools.sched_overhead_analysis',
  '--chip-swimlane-records-json', '.../chip_swimlane_records.json']'
  returned non-zero exit status 1

with the tool's own stderr, which points the reader at their command line rather
than at a data loss:

Error: scheduler-overhead analysis needs the task DAG (deps.json). Capture it in
a SEPARATE run with --enable-dep-gen (not co-run with --enable-chip-swimlane),
then pass --deps-json.

Secondary defect — the guard cannot skip. _swimlane_validate.py:161-164
only decides whether to pass --deps-json:

deps_sibling = Path(perf).parent / "deps.json"
if deps_sibling.exists():
    sched_cmd += ["--deps-json", str(deps_sibling)]
result = subprocess.run(sched_cmd, check=True, ...)

but sched_overhead_analysis requires the DAG unconditionally, so the
not exists() branch is dead-ended: omitting the flag guarantees exit 1. The
harness has the right shape for this one line above
(scene_test.py:1424 logs deps.json not produced; skipping deps_viewer);
the swimlane smoke does not.

Note that failing here is arguably correct — a real data loss should be red.
The defect is that it is red for the wrong stated reason, and the fix is to
assert on the missing deps.json naming the reconcile loss, not to skip.

Git Commit ID

8414356

CANN Version

9.0.0

Driver Version

N/A (simulation)

Host Platform

Linux (aarch64)

Additional Context

Related, and deliberately not filed as duplicates. There is now a family of
intermittent all-or-nothing DFX collector losses on a5, but each of these has a
different collector or artifact and I have not established a shared mechanism
for any pair of them:

• [Bug] a5 onboard chip_swimlane: single_scheduler_idle intermittently publishes zero records, so no chip_swimlane_records.json is written #2220 — a5 onboard: the chip_swimlane collector publishes zero records,
  so chip_swimlane_records.json is not written. Same all-or-nothing shape, a
  different collector, and on hardware.
• Flaky: a5 sim chip_swimlane test_run asserts 'Fanout edges mismatch: printed=0, oracle=6' #2191 — a5 sim: deps.json held 6 edges and reconcile passed; the loss was
  in what reached chip_swimlane_records.json. That is the opposite side of the
  same pipeline from this report, where dep_gen itself collected nothing.
• [Bug] a5 HBG AICore scheduler profiling intermittently publishes incomplete task timing, so chip_swimlane level 2 falls back to the aicpu producer #2181 — a5 HBG: AICore scheduler profiling intermittently publishes incomplete
  task timing.

Whether these share a hand-off/shutdown race in the common
ProfilerBase/BufferPoolManager machinery is the obvious question, but it is a
question, not a claim. Found while running a 25x probe for #2206 (that probe
captured no instance of #2206's abort).