[CI] CUDA 13.3 pip devcontainer builds fail during CMake configuration

[csadorf]

Summary

The CUDA 13.3 pip devcontainer jobs fail during CMake configuration when the build reaches an import cuda.tile check.

Failing test/component: CUDA 13.3 pip devcontainer build

Example failures:

• Update Dask RF to use the new distributed algo #8466
  • amd64: https://github.com/NVIDIA/cuml/actions/runs/33825593157/job/100877469580
  • arm64: https://github.com/NVIDIA/cuml/actions/runs/33825593157/job/100877469643
• Use latest intersphinx inventories on main #8558
  • amd64: https://github.com/NVIDIA/cuml/actions/runs/33858605010/job/100977618161
  • arm64: https://github.com/NVIDIA/cuml/actions/runs/33858605010/job/100977618041

Test Details

• Test file: build/pip/cuda-13.3/release/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61

• Test name: devcontainer / {amd64, arm64}, 13.3, pip

• Error message:

  -- CUML_CPP: Build and statically link CUVS libraries: OFF
  -- CPM: Adding package cuvs@26.10.00 (main)
  
  CMake Error at build/pip/cuda-13.3/release/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61 (message):
    cuda.tile (cuTile Python) is required to build cuTile embedded kernels.
    Install cutile-python and cuda-tileiras (conda), or cuda-tile[tileiras]
    (pip).
  
    Interpreter: /home/coder/.local/share/venvs/rapids/bin/python3.14
  
    Import error: Traceback (most recent call last):
      File "<string>", line 1, in <module>
        import cuda.tile
    ModuleNotFoundError: No module named 'cuda.tile'
  

The same error occurs in all four example jobs before compilation starts. CMake reports CUML_CPP: Build and statically link CUVS libraries: OFF before adding cuvs@26.10.00 (main).

Related: #8520 tracks pip devcontainer builds fetching and building cuVS source instead of using the installed package.

Root Cause Analysis