[Code Health] a5 die-affinity core assignment: prerequisite questions the design depends on

[ChaoZheng109]

Category

Robustness (potential edge-case failure)

Component

AICPU Scheduler

Description

This is an umbrella issue. It tracks the questions that must be answered before the a5 "AICPU scheduler threads claim AICore clusters by die affinity" design can be finalized. Each question is independently verifiable; answers go in the comments. The design is settled only once every question has a conclusion.

The design goal in one line: each scheduler thread should prefer AICore clusters on its own die, and only go cross-die once its quota is full. That needs two facts at once:

• (a) which AICPU die each scheduler thread lands on
• (b) which AICore die each AICore cluster belongs to

Question 1 is about (a).

---

Question 1 — phy_cpu_id semantics under PG are undefined, so the cluster_id / die_id derivation can be silently wrong

Status: awaiting hardware verification

Fact 1: the driver does not report cluster/die — the runtime derives them

CPU_TOPO gives four fields per logical CPU (aicpu_topology_probe.cpp:58-64):

struct DsmiSingleCpu {
    uint64_t cpu_mask;        // SMT sibling mask
    uint8_t  cpu_id;
    uint8_t  is_share;
    uint8_t  phy_cpu_id;
    uint8_t  hyperthread_id;
};

No cluster_id, no die_id. Both are pure arithmetic inferences (live path :233-234, JSON fallback path :1158-1159):

entry.cluster_id = entry.phy_cpu_id / 2;   // 2 physical CPUs per cluster
entry.die_id     = entry.phy_cpu_id / 4;   // 2 clusters per die

The internal design draft "A5 AICPU PG 场景拓扑与选核策略设计" (2026-07-30), §2.2, says the runtime obtains surviving-cluster and die distribution "through driver topology interfaces such as DSTI". The current code does not call DSTI — it infers everything from phy_cpu_id.

Fact 2: FG/PG classification rests entirely on those two inferences

classify_aicpu_scenario() (:570-580) does exactly two things: count distinct cluster_id values (4 → FG, 3 → PG1, 2 → PG2), then check that the die_id set is {0,1} and that the per-die cluster counts match the template.

The problem: nobody has confirmed what phy_cpu_id means after a cluster is masked off

• Semantics A — true physical numbering is preserved; faulted ids leave gaps.
• Semantics B — BIOS renumbers contiguously; surviving physical CPUs are renumbered from 0.

The design draft's §3.2 / §3.3 diagrams show semantics B ("6 survivors: CPU0…CPU5", "4 survivors: CPU0…CPU3") and explicitly warn: "CPU ids are drawn as the contiguous numbering BIOS exposes in each scenario… the implementation must not bind the diagram's CPU ids to fixed physical positions." The /2 and /4 derivation is only valid under semantics A.

Worked example 1: PG2 (Die0/Cluster1 and Die1/Cluster3 faulted)

Survivors: Cluster0 (Die0, true CPU0/1) and Cluster2 (Die1, true CPU4/5).

	Semantics A (gaps)	Semantics B (renumbered)
phy_cpu_id	{0,1,4,5}	{0,1,2,3}
cluster = phy/2	{0,2}, size 2 ✅	{0,1}, size 2 ✅
die = phy/4	{0,1} ✅	{0} ❌
clusters_per_die	(1,1) ✅	—
Classification	kPg2 ✅	kUnknown ❌

Worked example 2: PG1 where the faulted cluster is not the trailing one

The draft's §2.1 states the fault can land on either die. Say Die0/Cluster1 is faulted, leaving Cluster0 (Die0), Cluster2 (Die1), Cluster3 (Die1) — true distribution is Die0:1 / Die1:2.

	Semantics A	Semantics B
phy_cpu_id	{0,1,4,5,6,7}	{0,1,2,3,4,5}
cluster	{0,2,3}, size 3 ✅	{0,1,2}, size 3 ✅
dies	{0,1} ✅	{0,1} ✅
clusters_per_die	(1,2) ✅ matches truth	(2,1) ❌ inverted
Classification	kPg1 ✅ die attribution correct	kPg1 ✅ but die attribution is wrong

Example 2 is the dangerous one: classification "succeeds", selection runs normally, and the result is silently wrong with no anomaly in any log. CPU4/CPU5 — truly on Die1, renumbered to phy 2/3 — are attributed to Die0.

Why this has not surfaced yet

1. Only FG has ever been measured. Both verified JSON entries (9599 FG, 9579) are fault-free topologies, and no PG machine has ever been dumped. FG has no fault, hence no renumbering, hence the derivation is necessarily correct — which is why production is fine today.

2. validate_cpu_topology() cannot catch it. It only checks fields >= 0, unique cpu_id, and unique (phy, ht). Both semantics pass.

3. The existing unit tests contradict each other on this very point:

   • test_aicpu_topology_fallback.cpp:304-320 ComputesPg2SharingPolicy builds the pool from make_physical_range(0, 1) + make_physical_range(4, 5), skipping phy 2/3 — i.e. it assumes semantics A.
   • test_aicpu_topology_fallback.cpp:351-363 MatchesPg2ComputeOnlyDiagram hand-writes {cpu_id=4, phy=2, cluster=2, die=1} — where cluster != phy/2 and die != phy/4, bypassing the derivation entirely.

   Neither test exercises the derivation itself.

4. Getting it wrong still runs. A misclassification or a wrong die only degrades placement affinity; it does not break functionality. The draft's §4.7 admission criterion "every supported PG topology completes runs stably" would still pass.

Blast radius

	Impact on today's CPU selection	Impact on die-affinity assignment
Example 1 (PG2 → kUnknown)	worse affinity, still runs	die info degrades — not fatal
Example 2 (PG1 die inverted)	worse affinity, still runs	AICore clusters land on the wrong die → anti-affinity, worse than not doing it

For today's code a wrong die is merely suboptimal. Die affinity consumes the die id as a hard input, so one bit wrong turns the feature negative. Hence this must be answered first.

---

What needs to be verified (please reply in a comment on this issue)

Run on a PG machine (one or two AICPU clusters faulted). If several are available, one PG1 and one PG2 would be ideal — and for PG1 especially, a machine whose faulted cluster is not the trailing one (the example-2 case).

# Build steps: tools/cann-examples/aicpu-device-query/README.md
export SIMPLER_DISPATCHER_SO=$REPO/build/lib/a5/dispatcher/libsimpler_aicpu_dispatcher.so
export SIMPLER_AICPU_QUERY_SO=$REPO/tools/cann-examples/aicpu-device-query/device/build/libaicpu_query.so

task-submit --device auto --device-num 1 \
    --run "$REPO/tools/cann-examples/aicpu-device-query/host/build/query_device_hal \$TASK_DEVICE --json"

Please paste the full JSON verbatim, plus answers to these five:

#	Question	How to decide
1	How many clusters are actually faulted on this machine, and on which die?	Confirm from the hardware/BIOS side — do not infer it from this tool's output
2	Are the os_schedulable_cpus[].phy_cpu_id values contiguous?	Gaps ⇒ semantics A; contiguous ⇒ semantics B or a trailing fault
3	What scenario_type does the JSON report?	Compare against the true fault count from #1
4	Does each CPU's die_id match the true die from #1?	The decisive one
5	Does surviving_clusters match the true surviving cluster ids?

Notes:

• --json emits only the OCCUPY-filtered os_schedulable_cpus, whereas classification runs over the full CPU_TOPO table. If possible, please additionally attach the unfiltered (cpu_id, phy_cpu_id, hyperthread_id, cpu_mask) list.
• If --json fails because the SoC is unsupported, fall back to the pretty-print output of query_device_hal <dev> plus the host-side aicpu_topology_probe: CPU_TOPO source=... soc=... logical=N LOG_INFO line.

How the answer branches the design

Verification result	Conclusion
Semantics A (gaps present, die_id matches truth)	The derivation is trustworthy; die affinity is viable on PG1/PG2
Semantics B (contiguous, die_id disagrees with truth)	The derivation is not trustworthy → either switch to reading cluster/die from DSTI, or fall back to plain round-robin on any PG topology

---

Location

• src/a5/platform/onboard/host/aicpu_topology_probe.cpp:58-64 — DsmiSingleCpu; the driver gives four fields, no cluster/die
• src/a5/platform/onboard/host/aicpu_topology_probe.cpp:233-234 — live CPU_TOPO path derives cluster_id / die_id
• src/a5/platform/onboard/host/aicpu_topology_probe.cpp:1158-1159 — the same derivation on the JSON fallback path
• src/a5/platform/onboard/host/aicpu_topology_probe.cpp:570-580 — classify_aicpu_scenario(), the FG/PG1/PG2 criteria
• tests/ut/cpp/a5/test_aicpu_topology_fallback.cpp:304-320 — ComputesPg2SharingPolicy (assumes semantics A)
• tests/ut/cpp/a5/test_aicpu_topology_fallback.cpp:351-363 — MatchesPg2ComputeOnlyDiagram (hand-written cluster/die, bypasses the derivation)
• src/a5/docs/hardware.md:200 — where the derivation is documented

Line numbers are against upstream/main @ 748c39fbe7825e8411f44e91b754a3b2449f633c.

Proposed Fix

Step 1 — get the verification result first. Nothing in the existing logic changes until then.

Step 2 — two items worth doing regardless of the answer (decoupled from die affinity, can ship as their own PR):

1. Add a cold-path diagnostic that prints the raw phy_cpu_id sequence and whether it is contiguous, so the next PG machine answers this question by simply running:

   LOG_INFO("aicpu_topology: phy_cpu_ids=[%s] distinct=%zu contiguous=%d",
            dump.c_str(), distinct_phy.size(), is_contiguous);

2. Resolve the contradiction between the two PG2 tests: either add a case that genuinely goes through probe_aicpu_topology's derivation, or state in a comment that MatchesPg2ComputeOnlyDiagram assumes cluster/die are supplied correctly upstream and does not cover the derivation. Today the two tests assume incompatible phy_cpu_id semantics for the same scenario.

Step 3 — when the die-affinity feature ships, gate it on a conservative predicate (leaves the existing classification untouched, so the blast radius is zero):

distinct_phy = deduplicated set of phy_cpu_id
  size == 8                      → FG, no fault, die trustworthy
  max(distinct_phy) + 1 > size   → gaps present ⇒ semantics A, die trustworthy
  otherwise (contiguous, < 8)    → indistinguishable ⇒ die not trustworthy, fall back to round-robin

This is fail-safe: it also conservatively rejects "semantics A + trailing-cluster fault" (where the die would in fact be correct), giving up a little benefit but never producing a wrong assignment. It can be relaxed once a PG dump lands here.

---

Follow-up questions

Questions 2, 3, … will be appended to this issue in the same shape (facts → problem → what to verify → how the answer branches the design). Known directions still to write up:

• Chain (b): whether the worker_id → physical_core_id mapping is dense and die-monotone (this decides whether die affinity can preserve tmr's barrier-free init path)
• Whether the AICPU die numbering and the AICore die numbering refer to the same physical die (if inverted, affinity becomes anti-affinity)
• The measured magnitude of the die-affinity benefit (if it cannot be measured, do not build it)

Priority

Medium (minor risk, should fix in next few releases)

[ChaoZheng109]

2026-09-13: rewrote the title and body in English to match the repo's issue-language convention. No substantive change — same facts, worked examples, verification request and proposed fix as the original Chinese version.

[yanghaoran29]

Measurement: OCCUPY / CPU_TOPO / cluster·die (one 9579 / one 9581 / one 9599)

1. halGetDeviceInfo(..., MODULE_TYPE_AICPU, INFO_TYPE_OCCUPY, &val)

Platform	val	Binary	Meaning
9579	0x3e	0b111110	Threads 1–5 usable
9581 / 9599	0x1f8	0b111111000	Threads 3–8 usable

2. dsmi_get_device_info(device_id, main_cmd, sub_cmd, buf, size) → AICPU logical CPU topology

• 9579: this interface is not supported (no CPU_TOPO obtained in this run).
• 9581 / 9599: supported; results on the two machines are identical.

Return contract:

• Function return value 0 = success
• *size is written with the number of bytes actually written
• buf holds a naturally aligned structure: total_nums + cpus[i]{cpu_mask, cpu_id, is_share, phy_cpu_id, hyperthread_id}
• total_nums = 9

cpu_id	phy_cpu_id	ht	is_share	cpu_mask
0	0	0	0	0x1
1	1	0	1	0x6
2	1	1	1	0x6
3	2	0	0	0x8
4	3	0	0	0x10
5	4	0	0	0x20
6	5	0	0	0x40
7	6	0	0	0x80
8	7	0	0	0x100

Meaning: 8 physical cores + one extra hyperthread on phy1 → 9 logical CPUs; only cpu1/2 are an SMT pair (is_share=1, masks cross-point to 0x6).

Combined with OCCUPY=0x1f8: threads 3–8 in the table are usable; 0–2 are system-reserved.

3. cluster_id / die_id

No interface was found that directly returns cluster_id / die_id for entries in this table. The runtime derives them from the physical id (see hardware.md / aicpu_topology_probe):

cluster_id = phy_cpu_id / 2
die_id     = phy_cpu_id / 4

On FG with phy=0..7:

phy	cluster	die
0,1	0	0
2,3	1	0
4,5	2	1
6,7	3	1

[doraemonmj]

Follow-up to the measurement above: the live CPU_TOPO path has never worked, and the fallback is silent

Independently re-ran the probe on two boxes — one 9579 and one 9581. The raw numbers reproduce @yanghaoran29's table exactly (same OCCUPY, same 9-row CPU_TOPO on 9581, same "not supported" on 9579), so I won't repeat them. What follows is what that run turned up around the data, which changes what Step 2 should contain.

TL;DR — three things, none of which depend on ever finding a PG machine:

1. kInfoCpuTopo = 59 is wrong; it should be 60. The HAL primary path has never succeeded on any box. Everything we have ever read came from the DSMI fallback.
2. The source comment explaining rc=65534 is wrong, and it actively misleads.
3. On 9579 the tool emits a complete, plausible-looking "scenario_type": "PG1" with a full die distribution that is 100% synthetic — and the only tell is one field and two LOG_WARNs. This is a live instance of exactly the failure mode this issue is about.

---

1. kInfoCpuTopo = 59 should be 60 — the primary path has never worked

aicpu_topology_probe.cpp:40:

constexpr int32_t kModuleSystem = 0;
constexpr int32_t kInfoCpuTopo = 59;   // ← EVENT_RESUME, not CPU_TOPO

In driver/ascend_hal_base.h, INFO_TYPE_CPU_TOPO == 60; 59 is INFO_TYPE_EVENT_RESUME. Confirmed by compiling the header and printing the enum (not by counting entries by hand) on both boxes.

Measured on the 9581 box, same binary, same device, only the constant varied:

call	rc	total
halGetDeviceInfoByBuff(SYSTEM, infoType=59)	65534	0
halGetDeviceInfoByBuff(SYSTEM, infoType=60)	0	9
dsmi_get_device_info(main=14, sub=0) — documented	65534	0
dsmi_get_device_info(main=14, sub=1) — documented	65534	0
dsmi_get_device_info(main=14, sub=2) — not in the enum	0	9

Patched 59 → 60, rebuilt query_device_hal, re-ran end to end: the halGetDeviceInfoByBuff(CPU_TOPO) LOG_WARN disappears, the primary path returns the topology directly, and the emitted JSON is byte-identical to what the DSMI fallback produced.

So the CPU_TOPO we have been reading on 9581/9599 comes entirely from the DSMI fallback at :134, and that fallback works by way of a sub-command the current headers do not define (see §3). The designed primary path has been dead the whole time. Worth fixing before any of this feeds a hard input like die id.

kModuleSystem = 0 is correct — see the sweep in §4, where SYSTEM+60 is the only combination that returns anything.

2. rc=65534 is DRV_ERROR_NOT_SUPPORT, and the comment above the struct says otherwise

aicpu_topology_probe.cpp:54-57:

// Natural-alignment layout (no pack pragma). Mirrors
// tools/cann-examples/query/query.cpp's struct; the HAL/DSMI driver
// expects this byte layout — packing breaks the size check inside the
// driver and yields rc=65534.

65534 == 0xfffe == DRV_ERROR_NOT_SUPPORT (driver/ascend_hal_error.h:144). It means the driver does not implement the query — nothing to do with struct packing or a size check.

The alignment advice itself may well be right, but attaching rc=65534 to it sends the next reader after buffer layout. It cost me a round of testing five different buffer sizes against a driver that was never going to answer. Suggest decoupling the two and decoding the error by name at the LOG_WARN sites (:125, :137).

3. The DSMI path that does work is undocumented

aicpu_topology_probe.cpp:42-49 says the constants were verified against dsmi_common_interface.h. kDsmiSocInfoMainCmd = 14 does check out. kDsmiSocInfoSubCmdCpuTopo = 2 does not:

/* DSMI sub command for DSMI_MAIN_CMD_SOC_INFO */
typedef enum {
    DSMI_SOC_INFO_SUB_CMD_DOMAIN_INFO = 0,
    DSMI_SOC_INFO_SUB_CMD_CUST_OP_ENHANCE = 1,
    DSMI_SOC_INFO_SUB_CMD_MAX,          /* == 2 */
} DSMI_SOC_INFO_SUB_CMD;

There is no SUB_CMD_CPU_TOPO; 2 is the end-of-enum sentinel. (Read from the 25.7.rc1.6 driver header on the 9579 box; the CANN 9.2.0 toolkit header carries the same enum. I could not read the driver headers on the 9581 box — permissions — so this is not confirmed across both driver versions.) And empirically it is inverted: the two documented sub-commands return NOT_SUPPORT while the sentinel value returns the topology.

Not asking anyone to change it — it works. But the comment claims header backing it does not have, so the next person to "clean this up" against the header will silently delete the only path that currently functions. Worth a comment saying it is empirical.

4. 9579: this is a hard negative, not "not obtained in this run"

To pin down whether 9579 lacks the query or we were asking wrong, I swept, as root, with the device held under the box's task queue:

• halGetDeviceInfoByBuff, module ∈ {0..16, 0x8000} × infoType ∈ {55..64} — 180 probes, zero rc=0
• dsmi_get_device_info, main=14 × sub ∈ {0..7} — all NOT_SUPPORT
• also with kInfoCpuTopo patched to 60, end to end through query_device_hal — still NOT_SUPPORT on both paths

Positive control: the identical sweep binary on the 9581 box hits module=0 (SYSTEM) infoType=60 → rc=0, total=9 on the first matching probe and dumps the full table. So the sweep detects a working combination when one exists.

Conclusion: 9579 does not implement CPU_TOPO at any module/infoType, and no constant fix reaches it. Also worth recording that the box lacking support is on the newer driver:

	9579 box	9581 box
chip	Ascend950PR_9579 (28 AICore)	Ascend950DT_9581 (32 AICore, V120)
host	x86_64 / Ubuntu 24.04	aarch64 / openEuler 24.03
driver	25.7.rc1.6	25.6.rc2.b023
firmware	9.0.0.105.229	7.0.9.100.202
CPU_TOPO	none	SYSTEM + 60 ✅
topology_source	json_fallback	driver

Two variables move at once (SoC and driver version), so this does not say which one decides it.

5. The silent fallback, demonstrated

This is the part I think matters most for Step 2. On the 9579 box, query_device_hal --json exits 0 and prints:

{
  "soc_name": "Ascend950PR_9579",
  "topology_source": "json_fallback",
  "scenario_type": "PG1",
  "surviving_clusters": [0, 1, 2],
  "os_schedulable_cpus": [
    {"cpu_id": 1, "phy_cpu_id": 1, "hyperthread_id": 0, "cluster_id": 0, "die_id": 0},
    {"cpu_id": 2, "phy_cpu_id": 2, "hyperthread_id": 0, "cluster_id": 1, "die_id": 0},
    ...
    {"cpu_id": 5, "phy_cpu_id": 5, "hyperthread_id": 0, "cluster_id": 2, "die_id": 1}
  ]
}

A PG1 classification, a surviving-cluster list, and a per-CPU die distribution — none of it measured. It is the hardcoded table keyed on soc_name + occupy. The only signals are the topology_source field and two LOG_WARNs that scroll past in a task-queue log.

To be fair to the fallback: its kVerifiedFallbackOccupy = 0x3e matches what the device actually reports, so the occupy mask it keys on is confirmed. What is unconfirmed is the cluster/die layout it asserts on top of that — which is precisely the quantity this issue is about.

Two consequences:

• This is a live example of the issue's own thesis. §"Why this has not surfaced yet" argues a wrong die would run normally and log nothing anomalous. Here is a box doing exactly that today, in the field, with a fabricated PG1.
• It is a hazard for this issue's own verification request. The request asks people to run query_device_hal --json and paste the result. Anyone doing that on a 9579-class box will paste a synthetic PG1 topology in good faith and it will read as a genuine PG dump. Suggest adding to the request: check topology_source first — if it is not driver, the run answers nothing.

So Step 2 item 1's diagnostic should be a loud warning on the fallback branch, not only a LOG_INFO of the phy_cpu_id sequence. The phy dump is useful, but it does not help if the numbers being dumped are invented.

Relatedly, for Step 3: the predicate should take topology_source == driver as a precondition. As written it is a pure function of distinct_phy, so on a fallback box it would classify a synthetic table — and whichever branch that lands in, the verdict is meaningless.

6. What the 9581 box answers, and what it does not

Against the five questions — noting up front that this box is FG, so it cannot settle Question 1:

#	Question	Answer on the 9581 box
1	clusters actually faulted	0 — 4 clusters present, no fault
2	phy_cpu_id contiguous?	Yes — {0..7}, distinct=8, max+1 == size, no gaps
3	reported scenario_type	FG — matches
4	does die_id match the true die?	no contradiction, but vacuous: with no fault there is no renumbering, so semantics A and B coincide. We did not obtain an independent hardware/BIOS die ground truth
5	surviving_clusters correct?	Yes — [0,1,2,3]

Checked on all 8 devices of that box, not just device 0 — all identical.

One genuinely new data point: this is the first real input to the Step 3 predicate. distinct_phy == 8 → FG → die trustworthy. The predicate's FG branch now has hardware behind it.

Question 1 itself is still open. Both boxes are FG. The example-2 case — PG1 with a non-trailing faulted cluster — remains the thing that has to be found, and neither of these boxes can stand in for it.

---

Suggested change to Step 2

Current Step 2 lists (1) a phy_cpu_id diagnostic and (2) resolving the two contradictory PG2 tests. Both still worth doing. Adding, all decoupled from die affinity and shippable in the same PR:

3. kInfoCpuTopo: 59 → 60.
4. Fix the rc=65534 comment at :54-57; decode driver errors by name at :125 / :137.
5. Note at :49 that kDsmiSocInfoSubCmdCpuTopo = 2 is empirical, not header-backed, so it does not get "corrected" away.
6. Make the fallback branch warn loudly, and gate the Step 3 predicate on topology_source == driver.

Happy to put these up as a PR if that is useful.

_{Verified against aicpu_topology_probe.cpp at upstream/main; constant line numbers (:40, :48-49, :54-57, :125, :137) checked against the current file.}