RuntimeError: operator torchvision::nms does not exist

[vennand]

Hi,

Following the instructions to install topaz on Ubuntu 22.04, 24.04 and Centos 7.9, I've run into this error:

(topaz) cryosparcuser@hades:~$ topaz --version
Traceback (most recent call last):
  File "/usr/local/anaconda/anaconda3/envs/topaz/bin/topaz", line 9, in <module>
    sys.exit(main())
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/topaz/main.py", line 60, in main
    import topaz.commands.train
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/topaz/commands/train.py", line 8, in <module>
    from topaz.training import load_data,make_model,train_model
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/topaz/training.py", line 32, in <module>
    from topaz.utils.data.sampler import (RandomImageTransforms,
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/topaz/utils/data/sampler.py", line 14, in <module>
    from torchvision.transforms.functional import rotate as rotate2d
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/torchvision/__init__.py", line 6, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/torch/library.py", line 467, in inner
    handle = entry.abstract_impl.register(func_to_register, source)
  File "/usr/local/anaconda/anaconda3/envs/topaz/lib/python3.8/site-packages/torch/_library/abstract_impl.py", line 30, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist

I have run:

conda create -p /usr/local/anaconda/anaconda3/envs/topaz python=3.8
conda activate topaz
conda install topaz pytorch-cuda=11.8 -c tbepler -c pytorch -c nvidia

And conda list returns:

(topaz) cryosparcuser@hades:~$ conda list
# packages in environment at /usr/local/anaconda/anaconda3/envs/topaz:
#
# Name                     Version          Build               Channel
_libgcc_mutex              0.1              main
_openmp_mutex              5.1              1_gnu
aom                        3.12.1           h7934f7d_0
blas                       1.0              openblas
bottleneck                 1.3.7            py38ha9d4c09_0
brotli-python              1.0.9            py38h6a678d5_8
bzip2                      1.0.8            h5eee18b_6
c-ares                     1.34.5           hef5626c_0
ca-certificates            2025.11.4        h06a4308_0
cairo                      1.18.4           h44eff21_0
certifi                    2024.8.30        py38h06a4308_0
charset-normalizer         3.3.2            pyhd3eb1b0_0
cuda-cudart                11.8.89          0                   nvidia
cuda-cupti                 11.8.87          0                   nvidia
cuda-libraries             11.8.0           0                   nvidia
cuda-nvrtc                 11.8.89          0                   nvidia
cuda-nvtx                  11.8.86          0                   nvidia
cuda-runtime               11.8.0           0                   nvidia
cuda-version               12.9             3                   nvidia
dav1d                      1.2.1            h5eee18b_0
expat                      2.7.3            h3385a95_0
ffmpeg                     6.1.1            hc01ab50_4
filelock                   3.13.1           py38h06a4308_0
fontconfig                 2.15.0           h2c49b7f_0
freetype                   2.13.3           h4a9f257_0
fsspec                     2024.6.1         py38h06a4308_0
future                     0.18.3           py38h06a4308_0
gettext                    0.21.0           hedfda30_2
giflib                     5.2.2            h5eee18b_0
gmp                        6.3.0            h6a678d5_0
gmpy2                      2.1.2            py38heeb90bb_0
graphite2                  1.3.14           h295c915_1
h5py                       3.11.0           py38hbe37b52_0
harfbuzz                   10.2.0           hdfddeaa_1
hdf5                       1.12.1           h2b7332f_3
icu                        73.1             h6a678d5_0
idna                       3.7              py38h06a4308_0
jinja2                     3.1.4            py38h06a4308_0
joblib                     1.4.2            py38h06a4308_0
jpeg                       9f               h5ce9db8_0
lame                       3.100            h7b6447c_0
lcms2                      2.16             hb9589c4_0
ld_impl_linux-64           2.44             h153f514_2
leptonica                  1.82.0           h42c8aad_2
lerc                       4.0.0            h6a678d5_0
libabseil                  20240116.2       cxx17_h6a678d5_0
libarchive                 3.8.2            h3ec8f01_0
libcublas                  11.11.3.6        0                   nvidia
libcufft                   10.9.0.58        0                   nvidia
libcufile                  1.14.1.1         4                   nvidia
libcurand                  10.3.10.19       0                   nvidia
libcurl                    8.16.0           heebcbe5_0
libcusolver                11.4.1.48        0                   nvidia
libcusparse                11.7.5.86        0                   nvidia
libdeflate                 1.22             h5eee18b_0
libev                      4.33             h7f8727e_1
libffi                     3.4.4            h6a678d5_1
libgcc-ng                  11.2.0           h1234567_1
libgfortran-ng             11.2.0           h00389a5_1
libgfortran5               11.2.0           h1234567_1
libglib                    2.84.4           h77a78f3_0
libgomp                    11.2.0           h1234567_1
libiconv                   1.16             h5eee18b_3
libidn2                    2.3.8            hf80d704_0
libjpeg-turbo              2.0.0            h9bf148f_0          pytorch
libnghttp2                 1.57.0           h2d74bed_0
libnpp                     11.8.0.86        0                   nvidia
libnvjpeg                  11.9.0.86        0                   nvidia
libogg                     1.3.5            h27cfd23_1
libopenblas                0.3.30           h46f56fc_0
libopus                    1.3.1            h5eee18b_1
libpng                     1.6.50           h2ed474d_0
libprotobuf                4.25.3           he621ea3_0
libssh2                    1.11.1           h251f7ec_0
libstdcxx-ng               11.2.0           h1234567_1
libtheora                  1.2.0            h32ad74f_1
libtiff                    4.5.1            hffd6297_1
libunistring               1.3              hb25bd0a_0
libuuid                    1.41.5           h5eee18b_0
libvorbis                  1.3.7            h7b6447c_0
libvpx                     1.13.1           h6a678d5_0
libwebp                    1.3.2            h11a3e52_0
libwebp-base               1.3.2            h5eee18b_1
libxcb                     1.17.0           h9b100fa_0
libxml2                    2.13.9           h2c43086_0
libzlib                    1.3.1            hb25bd0a_0
lz4-c                      1.9.4            h6a678d5_1
markupsafe                 2.1.3            py38h5eee18b_0
mpc                        1.3.1            h5eee18b_0
mpfr                       4.2.1            h5eee18b_0
mpmath                     1.3.0            py38h06a4308_0
ncurses                    6.5              h7934f7d_0
networkx                   3.1              py38h06a4308_0
numexpr                    2.8.4            py38hd2a5715_1
numpy                      1.24.3           py38hf838250_0
numpy-base                 1.24.3           py38h1e6e340_0
openh264                   2.6.0            he621ea3_0
openjpeg                   2.5.2            he7f1fd0_0
openssl                    3.0.18           hd6dcaed_0
packaging                  24.1             py38h06a4308_0
pandas                     2.0.3            py38h1128e8f_0
pcre2                      10.46            hf426167_0
pillow                     10.4.0           py38h5eee18b_0
pip                        24.2             py38h06a4308_0
pixman                     0.46.4           h7934f7d_0
platformdirs               3.10.0           py38h06a4308_0
pooch                      1.7.0            py38h06a4308_0
pthread-stubs              0.3              h0ce48e5_1
pysocks                    1.7.1            py38h06a4308_0
python                     3.8.20           he870216_0
python-dateutil            2.9.0post0       py38h06a4308_2
python-tzdata              2025.2           pyhd3eb1b0_0
pytorch                    2.3.0            cpu_py38h1ce4368_1
pytorch-cuda               11.8             h7e8668a_6          pytorch
pytorch-mutex              1.0              cuda                pytorch
pytz                       2024.1           py38h06a4308_0
readline                   8.3              hc2a1206_0
requests                   2.32.3           py38h06a4308_0
scikit-learn               1.3.0            py38h1128e8f_1
scipy                      1.10.1           py38h32ae08f_1
setuptools                 75.1.0           py38h06a4308_0
six                        1.16.0           pyhd3eb1b0_1
sqlite                     3.50.2           hb25bd0a_1
sympy                      1.13.3           py38h06a4308_0
tesseract                  5.2.0            hb0d2e87_3
threadpoolctl              3.5.0            py38h2f386ee_0
tk                         8.6.15           h54e0aa7_0
topaz                      0.3.16           py_0                tbepler
torchvision                0.18.0           py38_cu118          pytorch
tqdm                       4.66.5           py38h2f386ee_0
typing_extensions          4.11.0           py38h06a4308_0
urllib3                    2.2.3            py38h06a4308_0
wheel                      0.44.0           py38h06a4308_0
xorg-libx11                1.8.12           h9b100fa_1
xorg-libxau                1.0.12           h9b100fa_0
xorg-libxdmcp              1.1.5            h9b100fa_0
xorg-libxext               1.3.6            h9b100fa_0
xorg-libxrender            0.9.12           h9b100fa_0
xorg-xorgproto             2024.1           h5eee18b_1
xz                         5.6.4            h5eee18b_1
zlib                       1.3.1            hb25bd0a_0
zstd                       1.5.7            h11fc155_0

I managed to fix the issue by running this:
pip install --upgrade torch torchvision

(solution from https://stackoverflow.com/questions/60247432/no-such-operator-torchvisionnms)

Any idea why I'm getting this error, and would this solution cause more issues in the future?

[DarnellGranberry]

That's odd, I'm not able to replicate that issue. It sounds like you tried this in multiple OS's; if that's the case, did it happen in all of them? It seems like this happens when pytorch and torchvision have mismatched binaries, but if you're installing topaz in a brand new environment then they should come matched on their own.

[vennand]

Yeah, it happened on all of them. All those systems (Ubuntu and Centos) have similar setup though (firewall rules, software, user permissions, etc.) so I don't know if any specific configuration is causing this.

I'll pull conda versions and channels on Monday, see if that could have an impact. My first thought right now is that I might have changed the default channels at some point.