Assignment #4: CUDA integration (NVIDIA + Iluvatar CoreX) - #67
Open
tangjinchi23-cmd wants to merge 41 commits into
Open
Assignment #4: CUDA integration (NVIDIA + Iluvatar CoreX)#67tangjinchi23-cmd wants to merge 41 commits into
tangjinchi23-cmd wants to merge 41 commits into
Conversation
added 30 commits
July 24, 2026 15:28
Add CPU kernels for add, argmax, embedding, linear, rms_norm, rope, self_attention (causal + GQA + numerically stable softmax), and swiglu, plus supporting tensor/op-dispatch changes. Add docs/SELF_ATTENTION_ZH.md recording the self_attention implementation process.
Adds full Nvidia device support: runtime API, xmake/nvidia.lua build config, and CUDA kernels for all 9 ops (add, argmax, embedding, linear, rearrange stub, rms_norm, rope, self_attention, swiglu). Wires the Qwen2 model (src/llaisys/models/qwen2.cc, python bindings) through to run on --device nvidia, plus doc/reference notes from the Assignment wooway777#3/wooway777#4 work.
…mbing DeviceResource is now created per Runtime and reaches ops through context().runtime().resource(); nvidia::Resource owns a cublasHandle_t. linear's F32/F16/BF16 cases now call cublasSgemm/cublasSgemmEx with a bias broadcast-add kernel afterward, instead of the hand-written tiled-GEMM kernel (kept commented out for reference).
CI's Linux runner has no GPU, so ENABLE_NVIDIA_API isn't defined and the NVIDIA case in linear's switch compiles out entirely, leaving `resource` declared but unused under -Werror. Move the declaration inside the NVIDIA-only case block so it doesn't exist at all on CPU-only builds.
…as fallback Head dims that are multiples of 8 now go through a cudnn_frontend SDPA graph (causal + native GQA support) instead of the hand-written V1 kernel; V1 stays as the fallback for head dims that aren't. Resource gains a cudnnHandle_t alongside the existing cublasHandle_t, and xmake links cudnn/nvrtc for the device and ops nvidia targets. Update CLAUDE.md: the self_attention section now documents the cuDNN path and marks the earlier staged flash-attention plan as superseded.
…el comments Prep for debugging on remote GPU server: sync CLAUDE.md notes, drop stale argmax TODO block in favor of a comment explaining why the V1 kernel was kept, and add a matching note to rope's V1 kernel.
…DNN 9.24.0 SDPA regression on sm_120 and Iluvatar CoreX compat findings - self_attention_cuda.cu: close the #if CUDNN_MAJOR >= 8 guard right after the cudnn_frontend include instead of leaving it open to EOF. - CLAUDE.md: record the cuDNN 9.24.0 SDPA runtime regression on sm_120 (fixed by reverting to 9.20.0) and the Iluvatar CoreX toolchain findings relevant to the corex-cudnn-compat branch.
Adds LLAISYS_DEVICE_ILUVATAR device type and per-op src/ops/*/iluvatar/ directories (verbatim copies of nvidia/ sources, renamed to llaisys::ops::iluvatar / llaisys::device::iluvatar namespaces) plus xmake/iluvatar.lua wiring the clang++ -x ivcore toolchain used by the remote Iluvatar CoreX box. Dispatch layer (runtime_api.cpp, device_resource.cpp, op.cpp) and test scripts are not yet wired to the new device type; still bootstrap state pending build validation on the remote box.
set_languages("cxx17") only propagates to the cc/cxx tools, not the
custom cu tool (bare clang++ invoked with -x ivcore). Without an
explicit -std=c++17 cuflag, C++17 features (std::byte, if constexpr,
nested namespace defs) used by the copied nvidia/ sources failed to
compile on the remote box. Found via test/ops/add.py-style xmake
build of add_iluvatar.cu alone.
add_iluvatar.cu now compiles and archives cleanly through xmake on the remote box after the -std=c++17 cuflag fix. Records the xmake install/root gotchas hit along the way and sets the next step: widen back to the full op glob plus llaisys-device-iluvatar.
Checks in the two on-box test scripts used so far so they can be pulled directly on the remote Iluvatar box instead of pasted by hand: verify_add_narrow.sh (standalone clang++ compile + narrowed xmake build of just add_iluvatar.cu) and verify_full.sh (full 9-op + device-iluvatar xmake build, the next step to run).
All 9 ops plus llaisys-device-iluvatar now compile cleanly on the remote box with no source changes needed beyond the -std=c++17 cuflag fix. Confirms the CUDNN_MAJOR>=8 guard correctly falls back to V1 for self_attention/linear on this box's cuDNN 7.6.5. Source-level compilation is fully verified; remaining work is aggregate xmake deps + dispatch-layer wiring, not more toolchain debugging.
Adds ENABLE_ILUVATAR_API-guarded LLAISYS_DEVICE_ILUVATAR branches to
runtime_api.{hpp,cpp} and device_resource.{hpp,cpp}, and to all 9 ops'
op.cpp (mirroring the existing NVIDIA case 1:1 - every iluvatar/*.cuh
signature already matched its nvidia counterpart exactly). Hooks
llaisys-device-iluvatar/llaisys-ops-iluvatar into the llaisys-device/
llaisys-ops aggregate targets via has_config("iluvatar-gpu"). Adds an
"iluvatar" branch to test_utils.py's device mapping functions and an
"iluvatar" --device choice to all 10 test scripts.
Verified no regression by rebuilding locally with --nv-gpu=y (ENABLE_
ILUVATAR_API is undefined here so the new branches compile out) and
rerunning add/self_attention/linear --device nvidia - all still pass.
Not yet build- or run-tested on the actual Iluvatar box.
Full xmake build llaisys --iluvatar-gpu=y (not narrowed to one target), install, then test/ops/*.py --device iluvatar for all 8 ops plus test_runtime.py --device iluvatar. This exercises the dispatch layer wired in the previous commit for the first time.
xmake's built-in cuda.env rule unconditionally adds -lcudadevrt (and
-lcudart_static unless cudart/cudart_static is already linked) to any
target containing .cu files, regardless of the cuda.rdc setting.
Iluvatar's corex SDK only ships libcudart.so (no static variant) and
no libcudadevrt at all.
Fix: explicitly add_links("cudart") in xmake/iluvatar.lua so the real
shared library satisfies xmake's check and cudart_static never gets
auto-added. cudadevrt has no such escape hatch and is genuinely absent
on this platform, but also genuinely unneeded (cuda.rdc = false, no
relocatable device code linking) - verify_ops.sh now creates an empty
stub libcudadevrt.a via `ar rcs` before building, which satisfies the
linker without needing any real symbols.
xmake install only copies the .so into python/llaisys/libllaisys/; `import llaisys` also needs the editable pip install. Script now checks importability and runs pip install -e ./python if needed, instead of leaving that as a manual step to remember each time.
test/ops/linear.py --device iluvatar fails numerically on bf16 (f32/ f16 pass). The cublasSgemmEx status return value was never checked; asserting it will show whether the call is silently failing on Iluvatar's CUDA-10.2-era cuBLAS compat lib, which may predate CUDA_R_16BF support in cublasSgemmEx.
assert() compiles out under xmake's release build (-DNDEBUG), so it never actually ran. printf isn't compiled out, giving a real signal.
Confirmed cublasSgemmEx returns status 15 (CUBLAS_STATUS_NOT_SUPPORTED) for bf16 on Iluvatar's CUDA-10.2-era cuBLAS compat lib - it doesn't support CUDA_R_16BF in this call. Tagging the print with "bf16" makes that unambiguous in future runs.
7 of 8 ops plus test_runtime.py now pass --device iluvatar on the actual remote box. linear's bf16 failure is root-caused to Iluvatar's CUDA-10.2-era cuBLAS compat lib not supporting CUDA_R_16BF in cublasSgemmEx (confirmed via a status printf - assert() was silently compiled out under xmake's release -DNDEBUG build). Fix approach not yet decided.
Confirmed cublasSgemmEx returns CUBLAS_STATUS_NOT_SUPPORTED (15) for CUDA_R_16BF on Iluvatar's CUDA-10.2-era cuBLAS. Before deciding on a fix, this dumps what the actual headers on the box declare: whether cublasGemmEx (the more general function, distinct from cublasSgemmEx) exists and what computeType it expects, whether cublasComputeType_t exists at all (a newer enum vs. the older cudaDataType-based signature), and whether CUDA_R_16BF/bf16 appear anywhere else.
cublasSgemmEx returns CUBLAS_STATUS_NOT_SUPPORTED (15) for CUDA_R_16BF on Iluvatar's cuBLAS. Headers show cublasGemmEx is a distinct, more general function that might support bf16 via an explicit computeType. Fixed a type mismatch in the switch: cublasGemmEx's computeType param is declared as cudaDataType (not cublasComputeType_t) in this header, so it needs CUDA_R_32F, not CUBLAS_COMPUTE_32F - passing the wrong enum type there wouldn't compile.
cublasGemmEx (with the correct cudaDataType-typed CUDA_R_32F computeType, not cublasComputeType_t's CUBLAS_COMPUTE_32F) succeeds where cublasSgemmEx returned CUBLAS_STATUS_NOT_SUPPORTED for bf16. test/ops/linear.py --device iluvatar now passes for all shapes/ dtypes. All 8 ops pass on the actual Iluvatar box as of this commit.
Removed the temporary status asserts/printfs and the now-unused <cassert> include added while diagnosing the bf16 cublasSgemmEx failure. Matches the existing NVIDIA linear_cuda.cu style: status is computed but not checked.
Documents the standard cuBLAS trick to remove the separate add_bias_kernel launch: broadcast bias into out before the GEMM call, then use beta=1.0f so the matrix product accumulates onto the already-bias-filled out instead of a follow-up elementwise add. No functional change - just leaves the plan in comments across all three dtype branches for whoever implements it next.
qwen2.cc:251 read a GPU device pointer (max_idx->data()) directly from host code via reinterpret_cast, which is an illegal memory access on any real GPU device - it only "worked" on CPU because CPU device pointers and host pointers share the same address space there. This masked the bug since full end-to-end inference was never re-verified on NVIDIA after linear/self_attention switched to cuBLAS/cuDNN. Fixed by copying the single int64 back to host via memcpy_sync (mirrors the existing pattern in Tensor::debug(), tensor.cpp:149-164) before returning it on non-CPU devices. Found while building test/benchmark_infer.py, a new script that times prefill vs decode steps separately (via an optional step_context hook added to qwen2.py's generate(), a no-op by default) and can optionally wrap each step in an NVTX range for nsys. Verified test_infer.py --device nvidia --test now passes end-to-end (32-token match against HF) in release mode.
…ode bottleneck benchmark_infer.py showed decode running at 640ms/token (1.56 tok/s) on the real 1.5B model. A standalone micro-benchmark at the actual decode shape isolated self_attention at 20.36ms/call vs linear's 0.012-0.039ms/call (negligible) - 28 layers x ~20ms accounts for nearly all of the observed per-token latency. Confirms the existing no-graph-caching TODO is the dominant real cost, not linear (which was the other open question after the cuBLAS switch).
added 11 commits
August 5, 2026 18:50
Replaces the TODO-only draft with actual environment details, per-op and per-platform status (NVIDIA 8/8, Iluvatar 8/8), the linear bf16 cuBLAS-compat root cause and fix, the qwen2.cc NVIDIA segfault found/fixed, and the performance-analysis findings so far (self_attention's per-call cuDNN graph rebuild as the confirmed decode bottleneck). Meant to keep growing as the performance-optimization phase continues.
…d-KV paths both blocked, bucketing+padding-mask is the chosen next step
…ernel Per-call cudnn_frontend graph rebuild was the dominant decode-time cost (~20ms/call, ~570ms/token across 28 layers). A proper fix needed bucketing total_len plus padded/masked K/V, which was never worked out safely; on top of that, cuDNN has real cross-platform fragility (9.24.0 SDPA crashes on this machine's sm_120 GPU, and Iluvatar's cuDNN 7.6.5 predates the Graph API entirely). Drop cuDNN from self_attention on both NVIDIA and Iluvatar, unify on the V1 hand-written kernel, and stop linking cudnn/nvrtc. Verified: xmake build llaisys --nv-gpu=y builds clean with no cuDNN linked, all 8 ops' test/ops/*.py --device nvidia pass, and test_infer.py --device nvidia --test still matches HF token-for-token.
…path self_attention's cuDNN graph-cache exploration concluded cuDNN acceleration isn't a good fit here (per-call graph rebuild dominates decode time, and a proper fix needs bucketing+padding-mask that was never worked out safely, plus real cross-platform fragility). self_attention is back to the V1 hand-written kernel on both NVIDIA and Iluvatar, and neither platform links cudnn/nvrtc anymore.
…ify report Split KV direction across warps for decode when total_len > 256 (flash_attention_decode_splitkv_cuda.cu), fixing the SM-occupancy gap left by the single-warp-per-head decode kernel on long KV caches. Verified against the existing decode kernel (bit-for-bit within fp tolerance, F32/F16/BF16, total_len 1-4000) and benchmarked at both kernel level (up to 7.3x at total_len=16000) and end-to-end (~1.29x decode speedup on a real generation). Also folds in an existing pass of comment cleanup/dead-code removal across the CPU/NVIDIA/Iluvatar op implementations (no functional changes), and trims REPORT.md for the upcoming submission.
…verification Rebuilds against latest code, re-runs the 8 op tests, auto-locates the local model weights if present, and runs test_infer.py --test on the Iluvatar box.
…tar instance The previous remote Iluvatar instance was destroyed. This script covers the full path on a fresh instance of the same course image: install xmake, clone the repo over HTTPS, install the lightweight platform-agnostic Python deps (leaves the Iluvatar-patched torch build alone), build, and run the op + end-to-end inference tests, falling back to a HuggingFace download if no local model weights are found.
…ce recreation The original remote Iluvatar instance was destroyed; bootstrap_and_verify.sh was used to stand up a fresh instance of the same course image and re-verify the full stack: build, all 8 ops, and test_infer.py --test (model auto-downloaded from Hugging Face since the fresh instance had no local weights) all passed, confirming the earlier cuDNN-removal change didn't regress Iluvatar.
add_cugencodes("native") silently adds no -gencode flags if no GPU is
visible at build time, which could produce a binary with no A100
(sm_80) machine code if the build environment can't see the GPU.
Keep native detection for local dev but always also target sm_80.
…ary table Note the actual grading hardware (A100 / Iluvatar TG150-200) differs from this session's dev hardware, and record the sm_80 gencode fallback already applied for that gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
test/ops/*.py --device nvidia全部通过test/ops/*.py --device iluvatar全部通过(远程验证)test/test_infer.py --device nvidia --test与 HF transformers 逐 token 匹配test/test_infer.py --device iluvatar --test远程端到端验证通过