Skip to content

fix(iluvatar): avoid unavailable CUDA driver target - #70

Open
gongchensu wants to merge 1 commit into
masterfrom
fix/iluvatar-cmake-driver-link
Open

fix(iluvatar): avoid unavailable CUDA driver target#70
gongchensu wants to merge 1 commit into
masterfrom
fix/iluvatar-cmake-driver-link

Conversation

@gongchensu

@gongchensu gongchensu commented Sep 3, 2026

Copy link
Copy Markdown

Summary

Fix the Iluvatar build by removing the unavailable CUDA::cuda_driver CMake
imported target from the InfiniCCL library and example link configurations.

CoreX provides the CUDA-compatible Runtime API through CUDA::cudart, while
the selected collective backend is linked separately through the configured
NCCL library. InfiniCCL and its examples do not call the CUDA Driver API
directly, so linking CUDA::cuda_driver is unnecessary and prevents otherwise
valid Iluvatar builds.

Changes

  • Iluvatar library build

    • Remove CUDA::cuda_driver from the Iluvatar infiniccl link libraries in src/CMakeLists.txt.
    • Retain CUDA::cudart; NCCL continues to be linked independently when WITH_NCCL=ON.
  • Example build

    • Remove CUDA::cuda_driver from Iluvatar example targets in examples/CMakeLists.txt.
    • Retain the CUDA-compatible runtime dependency required by device allocation and stream operations.
  • Scope

    • Keep NVIDIA and all other platform/backend link configurations unchanged.
    • Do not change any public API, collective algorithm, or runtime dispatch behavior.

Platform and Backend Affected

Platform

  • CPU
  • NVIDIA GPU
  • Iluvatar GPU
  • MetaX GPU
  • Moore Threads GPU
  • Cambricon MLU
  • HYGON DCU

Backend

  • OpenMPI
  • MPICH
  • NCCL
  • MCCL

Performance Impact

  • No performance impact
  • Performance improved
  • Performance regression possible

If applicable, provide benchmark results.

Known Issues & Future Work

Test Results

Test Involved Platform

  • CPU
  • NVIDIA GPU
  • Iluvatar GPU
  • MetaX GPU
  • Moore Threads GPU
  • Cambricon MLU
  • HYGON DCU

Test Involved Backend

  • OpenMPI
  • MPICH
  • NCCL
  • MCCL

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

  • PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
  • Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
  • Each commit message follows Conventional Commits.
  • Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
  • No stray merge commits from master — the branch is rebased cleanly on top of the current master.
  • No fixup! / squash! / wip commits remain.

Scope and Design

  • Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
  • No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
  • No unrelated formatting churn that would obscure the diff.
  • Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

  • The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
  • Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
  • No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
  • Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
  • All comments and error messages are in English (CONTRIBUTING.md §Code/General).
  • Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

  • Code follows the Google C++ Style Guide strictly.
  • clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
  • No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
  • Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
  • Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
  • Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
  • Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
  • Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

  • Code is PEP 8 compliant; ruff check passes cleanly on CI (see .github/workflows/ruff.yml).
  • ruff format --check passes cleanly — if not, run ruff format and commit the result.
  • Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
  • Framework-specific conventions (e.g. lowercase pytest.skip messages without terminal period) are honored where applicable (CONTRIBUTING.md §Python).
  • No blank line between the function signature and the body when there is no docstring or comment (CONTRIBUTING.md §Python).
  • A blank line is present before and after if, for, and similar control-flow statements (CONTRIBUTING.md §Python).
  • A blank line appears before each return, except when it directly follows a control-flow statement (CONTRIBUTING.md §Python).
  • Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
  • Type hints are added / kept consistent with the surrounding code.

Testing

  • All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

  • New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
  • Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

  • README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
  • Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

  • No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
  • Third-party code is license-compatible and attributed.
  • No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

@gongchensu gongchensu self-assigned this Sep 3, 2026
@gongchensu
gongchensu force-pushed the fix/iluvatar-cmake-driver-link branch from 0e8e201 to 67c9f4d Compare September 3, 2026 08:18
@gongchensu

Copy link
Copy Markdown
Author

root@gpu01:/home/zhuyue/codes/InfiniCore/submodules/InfiniCCL# python3 scripts/run_examples.py
--config /tmp/infiniccl-iluvatar-cluster.yaml
--launcher none
--examples ccl/all_reduce
--log-path /tmp/infiniccl-example-logs
--verbose
--
-g 2 -w 1 -p 1 -n 1048576

           InfiniCCL Distributed Verification                 

Target Configuration File: /tmp/infiniccl-iluvatar-cluster.yaml
Selected Launcher Engine : none
Logs Target Destination : /tmp/infiniccl-example-logs/run_20260903_164337
Verbose Console Output : ENABLED

🚀 Running via icclrun (with build): ccl/all_reduce
--- [VERBOSE OUTPUT START: ccl/all_reduce] ---
[*] Orchestrating iluvatar on localhost...
-- Iluvatar: CUDA compiler /usr/local/corex/bin/clang++, arch ivcore11, toolkit /usr/local/corex
-- InfiniCCL Config: Devices [cpu, iluvatar] | Backends [nccl]
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/zhuyue/codes/InfiniCore/submodules/InfiniCCL/build/iluvatar
[ 12%] Built target infiniccl
[ 16%] Building CXX object examples/CMakeFiles/ccl_all_reduce.dir/ccl/all_reduce.cc.o
[ 20%] Building CXX object examples/CMakeFiles/ccl_mpi_hybrid_all_reduce.dir/ccl_mpi_hybrid/all_reduce.cc.o
[ 32%] Building CXX object examples/CMakeFiles/mpi_all_to_all.dir/mpi/all_to_all.cc.o
[ 32%] Building CXX object examples/CMakeFiles/mpi_reduce.dir/mpi/reduce.cc.o
[ 32%] Building CXX object examples/CMakeFiles/mpi_all_gather.dir/mpi/all_gather.cc.o
[ 36%] Building CXX object examples/CMakeFiles/mpi_gather.dir/mpi/gather.cc.o
[ 44%] Building CXX object examples/CMakeFiles/mpi_reduce_scatter.dir/mpi/reduce_scatter.cc.o
[ 44%] Building CXX object examples/CMakeFiles/mpi_broadcast.dir/mpi/broadcast.cc.o
[ 48%] Building CXX object examples/CMakeFiles/mpi_all_reduce.dir/mpi/all_reduce.cc.o
[ 52%] Building CXX object examples/CMakeFiles/mpi_scatter.dir/mpi/scatter.cc.o
[ 56%] Building CXX object examples/CMakeFiles/mpi_send_recv.dir/mpi/send_recv.cc.o
[ 60%] Linking CXX executable mpi/reduce
[ 64%] Linking CXX executable mpi/all_reduce
[ 68%] Linking CXX executable mpi/reduce_scatter
[ 72%] Linking CXX executable ccl_mpi_hybrid/all_reduce
[ 76%] Linking CXX executable mpi/scatter
[ 80%] Linking CXX executable mpi/all_gather
[ 84%] Linking CXX executable mpi/gather
[ 84%] Built target ccl_mpi_hybrid_all_reduce
[ 84%] Built target mpi_reduce
[ 84%] Built target mpi_all_reduce
[ 84%] Built target mpi_reduce_scatter
[ 88%] Linking CXX executable mpi/all_to_all
[ 88%] Built target mpi_scatter
[ 88%] Built target mpi_all_gather
[ 88%] Built target mpi_gather
[ 88%] Built target mpi_all_to_all
[ 96%] Linking CXX executable mpi/send_recv
[ 96%] Linking CXX executable ccl/all_reduce
[100%] Linking CXX executable mpi/broadcast
[100%] Built target mpi_send_recv
[100%] Built target ccl_all_reduce
[100%] Built target mpi_broadcast
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/libinfiniccl.so
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/include/infiniccl
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/include/infiniccl/comm.h
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/include/infiniccl/data_type.h
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/include/infiniccl/return_status.h
-- Installing: /tmp/infiniccl-pr-validation/install/iluvatar/include/infiniccl/infiniccl.h
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/bin/icclrun
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/cmake/InfiniCCL/InfiniCCLTargets.cmake
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/cmake/InfiniCCL/InfiniCCLTargets-release.cmake
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/cmake/InfiniCCL/InfiniCCLConfig.cmake
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/cmake/InfiniCCL/InfiniCCLConfigVersion.cmake
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/infiniccl/icclrun_logic.py
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/infiniccl/backends
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/infiniccl/backends/mpi_base.py
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/infiniccl/backends/mpich.py
-- Up-to-date: /tmp/infiniccl-pr-validation/install/iluvatar/lib/infiniccl/backends/ompi.py
[Main Process] Host: gpu01 | Target GPUs: 2

=== AllReduce Results ===
Correct: YES
Expect: 3.00
Actual: 3.00

=== Single-Node Threaded AllReduce Results ===
Data size: 1048576 floats (4 MB)
Time: 0.284 ms
Throughput: 13.74 GB/s (Bus BW)
Alg Bandwidth: 13.74 GB/s
[Main Process] All worker threads joined. InfiniCCL finalized safely.
--- [VERBOSE OUTPUT END: ccl/all_reduce] ---
✓ PASSED (Log saved to ccl_all_reduce.log)

==================================================================
EXECUTION SUMMARY

Total Framework Targets : 1
Successfully Passed : 1
Failed / Crashed Targets : 0
Overall Success Rate : 100.00%

🎉 All targets executed successfully!
root@gpu01:/home/zhuyue/codes/InfiniCore/submodules/InfiniCCL#

@gongchensu
gongchensu requested a review from Ziminli September 3, 2026 08:44
Comment thread src/CMakeLists.txt
Comment on lines +79 to +80
# CoreX does not export `CUDA::cuda_driver`; this backend only uses the
# CUDA runtime API and NCCL.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个注释可以麻烦去掉一下,不太必要。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants