CI: partition the a2a3 scene corpus and isolate the SDMA fault case

[ChaoWao]

What this changes

The a2a3 scene-test lane ran every sdma-marked test in one task-submit job
on two devices, and one of those three tests is a fault-injection case. It now
runs in a job of its own, and the lane's three selections are made a partition.

Step	Selection	Devices
Run pytest scene tests (a2a3)	examples tests/st -m 'not sdma and not sdma_fault'	$DEVICE_NUM
SDMA pytest (a2a3)	examples tests/st -m 'sdma and not sdma_fault'	2
SDMA fault recovery pytest (a2a3)	examples tests/st -m sdma_fault	1

A new sdma_fault marker carries the selection, registered in conftest.py
beside sdma.

Why the fault case needs a job to itself

Per #1425, provisioning
the PTO-ISA async-SDMA workspace creates 48 device-only STARS streams that sit
in the device fault domain, and once any AICore task has poisoned that card,
tearing those streams down blocks ~306 s — whether the teardown is
rtDeviceReset or an explicit aclrtDestroyStream. Destroying them first does
not help; the time just moves.

So the cost falls on whichever process still holds those streams when the
fault happens, not on a process that later faults on a previously-provisioned
device. The sdma_fault case is
test_sdma_worker_aicore_fault_teardown_is_bounded: it provisions SDMA itself
and faults an AICore on purpose, asserting that its own close stays bounded
(@pytest.mark.timeout(90)). It is #1425's regression test, not a victim of
the demos.

What sharing a job costs is therefore the other direction: the two SDMA demos
are also sdma-marked, so their Workers hold 48 streams each, and the existing
marker rule — such a test never shares an L2 Worker, and sorts after every
ordinary test — does not make those Workers gone by the time the fault case
poisons a card. Giving the fault case a session containing nothing else is what
guarantees no other stream holder is alive at that moment.

Why the selections are a partition

Two markers give four combinations, and each must reach exactly one step:

sdma	sdma_fault	runs in
no	no	ordinary sweep
yes	no	SDMA demo task
no	yes	isolated fault task
yes	yes	isolated fault task

All three steps name the same corpus (examples tests/st), so nothing falls
between them by living under a path one of them does not walk.

The shape matters because the first commit here was not a partition: the fault
case was excluded by name from the demo task but included only by path in
the isolated one, and the sweep excluded sdma alone. That held only while
every sdma_fault case also carried sdma and sat under tests/st, and
nothing enforced either. A sdma_fault case under examples/ would have run
in no task; one marked sdma_fault without sdma would have run inside the
ordinary sweep on shared devices, which is the single outcome this arrangement
exists to prevent. Both failure directions were silent —
.claude/rules/ci-change-detection.md asks every axis to be written in the
same shape for exactly this reason.

Today there is one sdma_fault case, it carries both markers, and it lives in
tests/st, so neither hole was live. The partition is so that adding the next
case cannot open them.

As it stands

Three tests carry sdma; one of them also carries sdma_fault. Every step
therefore collects at least one test, so none hits pytest's exit-5-on-empty,
and the fault case runs exactly once rather than in two steps.

Device counts match what each step needs: the fault case is
@pytest.mark.device_count(1) and asks for one; the demos keep two. The
narrower request is also easier to place in the shared queue.

Docs

docs/ci.md describes the partition and carries the #1425 mechanism above,
replacing a sentence that said "an AICore fault on a device that has
provisioned SDMA costs minutes instead of milliseconds". The
local-reproduction recipes move with the sweep's new expression —
docs/testing.md, docs/user/reference/cli.md, the a2a3 507899
troubleshooting page, and the testing, test-all-device and
test-runtime-device skills. A recipe that still deselects sdma alone no
longer reproduces what CI runs.

Provenance

The first commit is Crane-Liu's, cherry-picked out of #2447 where it arrived
alongside that PR's Qwen qualification work. It is unrelated to Qwen and
touches CI implementation plus shared test infrastructure (conftest.py),
which per ci-change-detection.md must run the full matrix and be verified by
observing what actually ran — evidence a PR reviewed for decode numerics will
not produce. #2447 reverts it, so the two do not both land it.

Verification, and where it stops

Static: the workflow parses as YAML and yields the three selections tabulated
above; the four-combination partition was checked exhaustively rather than by
reading the expressions; the marker is registered; markdownlint-cli2 and
ruff are clean over every changed file; and a tree-wide grep finds no
remaining -m "not sdma" recipe.

pytest --collect-only was not run: this worktree has no built
_task_interface, so conftest.py cannot import.

What CI has to confirm, because nothing else can: that the ordinary sweep
and the demo task each report the tests they should and not the fault case,
and that SDMA fault recovery pytest (a2a3) reports exactly that one case. A
selection expression that silently matches nothing, or everything, is a green
check either way — so the job logs are the evidence, not the check marks.

Effectiveness is not measured here. No before/after job timing exists for
the split. The previous shared step took two devices, so the demos' Workers may
have held their streams on a card the fault case never poisoned, in which case
the old arrangement was already adequate and this change is preventive. The
cost side is real and worth stating: the lane now makes a second task-submit
submission, each with its own --timeout 1800 queue-wait budget.

No hardware was taken and no task-submit lock was acquired. #1425's ~306 s is
quoted from that issue, not measured here.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 31 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 2a2b938c-21a4-41d7-a1f5-ee805a5a42a8

📥 Commits

Reviewing files that changed from the base of the PR and between 92a14e4 and 6a93d43.

📒 Files selected for processing (10)

• .claude/skills/test-all-device/SKILL.md
• .claude/skills/test-runtime-device/SKILL.md
• .claude/skills/testing/SKILL.md
• .github/workflows/_st-npu-a2a3.yml
• conftest.py
• docs/ci.md
• docs/testing.md
• docs/troubleshooting/a2a3-507899-aicpu-shared-so-fault.md
• docs/user/reference/cli.md
• tests/st/aicore_op_timeout/test_aicore_op_timeout.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}