TransferBench v1.71.00 by AtlantaPepsi · Pull Request #359 · ROCm/TransferBench

AtlantaPepsi · 2026-09-23T15:27:20Z

Motivation

NIC executor check against max message size
fixing logical CU ID report
inclusion of gfx1250-strict target
Pingpong latency testing integration and presets

Technical Details

Test Plan

Validate HSA_DISABLE_GFX12_STRICT=0 ./TransferBench output of gfx1250-strict target
On gfx1250, gfx942 and gfx950 compare SHOW_ITERATIONS output CU ID against CU_MASK bits
Latency Testing
- p2p_latency preset: in/cross-domain, and AMD and Nvidia platform
- Concurrent transfers + pingpongs run, single/multistream
- validation of all memory types support

Test Result

p2p_latency example output

[Latency Related]
GPU_MEM_TYPE         =            0 : Using default GPU memory for flags (0=default, 1=fine-grained, 2=uncached, 3=managed)
NUM_GPU_DEVICES      =            8 : Using 8 GPUs
NUM_LAPS             =         1000 : Timing 1000 round trips per iteration
USE_REMOTE_READ      =            0 : Executors write to their partner's memory and poll their own

Pingpong round-trip latency per lap (us), each pair run by itself
[1000 laps] [default GPU memory flags] [remote write / local poll]
 PING\PONG     GPU 00     GPU 01     GPU 02     GPU 03     GPU 04     GPU 05     GPU 06     GPU 07
    GPU 00      0.550      1.973      4.093      4.504      4.155      4.501      4.092      4.504
    GPU 01      1.956      0.585      4.502      4.930      4.507      4.860      4.505      4.925
    GPU 02      4.079      4.501      0.548      1.993      4.095      4.503      4.067      4.503
    GPU 03      4.501      4.919      1.995      0.592      4.503      4.873      4.502      5.010
    GPU 04      4.111      4.501      4.050      4.501      0.578      1.973      4.111      4.501
    GPU 05      4.501      4.901      4.501      4.837      1.993      0.581      4.503      4.818
    GPU 06      4.060      4.501      4.034      4.500      4.088      4.501      0.562      1.954
    GPU 07      4.502      4.957      4.502      4.873      4.511      4.841      1.956      0.575

Concurrent transfer + pingpong hybrid run example output

./TransferBench cmdline 16M "6 16 (G0->G1->G1) (G1->G2->G2) (G2->G3->G3) (G3->G0->G0
) (N G0 G1 +1000 N G1 G0) (C0 G0 G1 +50 C0 G1 G0)"

Test 1:
-------------------┬--------------┬------------┬-------------------┬---------------------------
  Executor: GPU 00 │ 302.684 GB/s │   0.055 ms │    16777216 bytes │ 332.617 GB/s (sum)
-------------------┼--------------┼------------┼-------------------┼---------------------------
     Transfer 3    │ 332.617 GB/s │   0.050 ms │    16777216 bytes │ G3 -> G0:16 -> G0
     PingPong 4    │     1.046 us │   1.046 ms │         1000 laps │ N->G0->G1 <+> N->G1->G0
     PingPong 5    │     1.544 us │   0.077 ms │           50 laps │ C0->G0->G1 <+> C0->G1->G0
-------------------┼--------------┼------------┼-------------------┼---------------------------
  Executor: GPU 01 │ 331.512 GB/s │   0.051 ms │    16777216 bytes │ 355.359 GB/s (sum)
-------------------┼--------------┼------------┼-------------------┼---------------------------
     Transfer 0    │ 355.359 GB/s │   0.047 ms │    16777216 bytes │ G0 -> G1:16 -> G1
-------------------┼--------------┼------------┼-------------------┼---------------------------
  Executor: GPU 02 │ 345.778 GB/s │   0.049 ms │    16777216 bytes │ 350.870 GB/s (sum)
-------------------┼--------------┼------------┼-------------------┼---------------------------
     Transfer 1    │ 350.870 GB/s │   0.048 ms │    16777216 bytes │ G1 -> G2:16 -> G2
-------------------┼--------------┼------------┼-------------------┼---------------------------
  Executor: GPU 03 │ 358.855 GB/s │   0.047 ms │    16777216 bytes │ 364.532 GB/s (sum)
-------------------┼--------------┼------------┼-------------------┼---------------------------
     Transfer 2    │ 364.532 GB/s │   0.046 ms │    16777216 bytes │ G2 -> G3:16 -> G3
-------------------┼--------------┼------------┼-------------------┼---------------------------
   Aggregate (CPU) │  58.392 GB/s │   1.149 ms │    67108864 bytes │ Overhead 1.094 ms
-------------------┴--------------┴------------┴-------------------┴---------------------------

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate correctness, portability, parsing, and build-target issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 4 · 4

Open (8)

Restrict GetXccId CDNA5 instruction to CDNA5 targets · New
Include pong subindex when enabling GFX subindices · New
Emit advanced transfer fields before the pong token · New
Align TDM device guard with supported architectures · New
Avoid indexing stale or empty transfer results after failure · New
Validate NIC message size for the selected port · New
Fix pingpong timing scaling by subiterations · New
Reject multi-device pingpong memory halves · New

What changed in this PR

Adds pingpong latency testing, NIC message-size validation, logical CU reporting fixes, and gfx1250-strict support.

Changes:

Adds pingpong parsing, execution, timing, and latency presets.
Adds NIC max_msg_sz validation and reporting.
Updates GPU architecture and TDM handling.

File	Review summary
`src/header/TransferBench.hpp`	Critical and moderate issues in XCC handling, pingpong validation, NIC limits, subindex selection, dump parsing, and timing scaling.
`src/header/tdmCopy.h`	Critical architecture guard mismatch enables unsupported TDM targets.
`src/client/Utilities.hpp`	Reviewed result and topology utilities.
`src/client/Topology.hpp`	Reviewed NIC message-size reporting.
`src/client/Presets/Presets.hpp`	Reviewed latency preset registration.
`src/client/Presets/Latency.hpp`	Moderate issues with failed-run result handling and diagonal pair measurement.
`src/client/EnvVars.hpp`	Reviewed pingpong configuration variables.
`src/client/Client.cpp`	Reviewed pingpong transfer display updates.
`docs/install/build_from_source.rst`	Reviewed strict GPU target documentation.
`CMakeLists.txt`	Moderate issue: package build targets omit `gfx1250-strict`.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        exeInfo.totalSubExecs += t.numSubExecs;
+      } else {
+        exeInfo.totalPingpong ++;
+      }
      exeInfo.useSubIndices |= (t.exeSubIndex != -1 || (t.exeDevice.exeType == EXE_GPU_GFX && !cfg.gfx.prefXccTable.empty()));


+            auto singleMemOrNull = [](vector<MemDevice> const& mems) {
+              if (mems.empty()) return MemDevice{MEM_NULL, 0, 0};
+              return mems[0];
+            };


Copilot

Copilot review overview

🟡 Changes recommended

Unresolved correctness issues remain in ping-pong execution, validation, parsing, and target gating.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 · 4

Open (9)

Run unpinned pingpong parameters on only one XCC · New
Align TDM device guard with supported architectures
Emit advanced transfer fields before the pong token
Include pong subindex when enabling GFX subindices
Restrict GetXccId CDNA5 instruction to CDNA5 targets
Reject multi-device pingpong memory halves
Fix pingpong timing scaling by subiterations
Validate NIC message size for the selected port
Avoid indexing stale or empty transfer results after failure

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues remain in pingpong execution, validation, result handling, and architecture gating.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 7 · 5

Open (12)

Reject pingpong subindices below -1 · New
Prevent persistent pingpong dependency cycles from hanging · New
Run unpinned pingpong parameters on only one XCC
Align TDM device guard with supported architectures
Emit advanced transfer fields before the pong token
Include pong subindex when enabling GFX subindices
Restrict GetXccId CDNA5 instruction to CDNA5 targets
Allow default numSubExecs for pingpong transfers · New
Reject multi-device pingpong memory halves
Fix pingpong timing scaling by subiterations
Validate NIC message size for the selected port
Avoid indexing stale or empty transfer results after failure

+#else
+            useSubIndexCount[exe]++;
+            int numSubIndices = GetNumExecutorSubIndices(exe);
+            if (subIndex >= numSubIndices) {


+    dim3 const gridSize(xccDim, numPingpong, 1);
+    dim3 const blockSize(1);


      if (t.numSubExecs <= 0)
        errors.push_back({ERR_FATAL, "Transfer %d: # of subexecutors must be positive", i});
-      else
+      else if (isPingpong) {
+        if (t.numSubExecs != 1)
+          errors.push_back({ERR_WARN,


Copilot

Copilot review overview

🟡 Changes recommended

Unresolved correctness issues affect ping-pong execution, target guards, NIC validation, and package builds.

Review effort: Lite
Findings: 7 · 4

Open (11)

Guard peer access setup to the executor's local MPI rank · New
Prevent persistent pingpong dependency cycles from hanging
Reject pingpong subindices below -1
Run unpinned pingpong parameters on only one XCC
Align TDM device guard with supported architectures
Include pong subindex when enabling GFX subindices
Restrict GetXccId CDNA5 instruction to CDNA5 targets
Propagate RunTransfers failures from latency presets · New
Allow default numSubExecs for pingpong transfers
Reject multi-device pingpong memory halves
Validate NIC message size for the selected port

Resolved since last review (3)

Emit advanced transfer fields before the pong token
Fix pingpong timing scaling by subiterations
Avoid indexing stale or empty transfer results after failure

+          } else if (partnerMem.memIndex != exeDevice.exeIndex) {
+            if (System::Get().IsVerbose()) {
+              System::Get().Log("[INFO]   Enabling pingpong peer access: GPU %d -> GPU %d\n",
+                                exeDevice.exeIndex, partnerMem.memIndex);
+            }
+            ERR_CHECK(EnablePeerAccess(exeDevice.exeIndex, partnerMem.memIndex));


AtlantaPepsi added 8 commits September 22, 2026 09:39

ibv port maxMsgSize check

3d321ae

fixing logical CU ID report

98a43f5

gfx1250strict

bed30a5

pingpong latency test

85d92c1

adjusting output format

6f775df

relaxed atomics for pingpong kernel

93227a0

refactor GFX executor timing

6954cdd

cuda fix and cross pod fix

81865fa

Copilot AI lite review requested due to automatic review settings September 23, 2026 15:27

AtlantaPepsi requested review from a team as code owners September 23, 2026 15:27

Copilot started reviewing on behalf of AtlantaPepsi September 23, 2026 15:28 View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 23, 2026 15:36

Copilot started reviewing on behalf of AtlantaPepsi September 23, 2026 15:37 View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

Comment thread src/header/TransferBench.hpp

AtlantaPepsi changed the title ~~Candidate 1.71~~ TransferBench v1.71.00 Sep 23, 2026

Latency Presets

e5166b6

Copilot AI review requested due to automatic review settings September 23, 2026 21:06

AtlantaPepsi force-pushed the candidate-1.71 branch from e37e7de to 07e9f67 Compare September 23, 2026 21:06

Copilot started reviewing on behalf of AtlantaPepsi September 23, 2026 21:07 View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

AtlantaPepsi added 4 commits September 24, 2026 20:20

xcc subindices reporting + advanced mode fix

f75f9c1

fix pingpong timing wrt subiterations

352cd1d

addition of latency ring; skip failing latency tests

60c7d5c

TransferBench v1.71.00

91f99bd

Copilot AI review requested due to automatic review settings September 28, 2026 15:27

AtlantaPepsi force-pushed the candidate-1.71 branch from 07e9f67 to 91f99bd Compare September 28, 2026 15:27

Copilot started reviewing on behalf of AtlantaPepsi September 28, 2026 15:27 View session

Copilot AI reviewed Sep 28, 2026

View reviewed changes

Navigation Menu

TransferBench v1.71.00 - #359

TransferBench v1.71.00#359
AtlantaPepsi wants to merge 13 commits into
developfrom
candidate-1.71

AtlantaPepsi commented Sep 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		dim3 const gridSize(xccDim, numPingpong, 1);
		dim3 const blockSize(1);

Conversation

AtlantaPepsi commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

p2p_latency example output

Concurrent transfer + pingpong hybrid run example output

Submission Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AtlantaPepsi commented Sep 23, 2026 •

edited

Loading