RuntimeError: Expected tensor for argument #1 'input' to have the same type as tensor for argument #2 'weight'; but type torch.cuda.FloatTensor does not equal torch.cuda.HalfTensor (while checking arguments for cudnn_batch_norm)

[wzr0108]

Got this error when running torch2trt.py

(PersonaLive) wzr@DESKTOP-H41D2AV:~/PersonaLive$ uv run torch2trt.py 
Uninstalled 1 package in 0.66ms
Installed 1 package in 23ms
/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  _torch_pytree._register_pytree_node(
/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  _torch_pytree._register_pytree_node(
Warning: Accessing the `severity` property of G_LOGGER is deprecated and will be removed in v0.50.0. Use `module_severity` instead
['pred_video', 'latents', 'pose_cond_fea_out', 'motion_hidden_states_out', 'motion_out', 'latent_first']
开始导出 ONNX 模型到: ./pretrained_weights/onnx/unet/unet.onnx ...
/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/diffusers/models/embeddings.py:175: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.height != height or self.width != width:
/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py:100: UserWarning: `nn.functional.upsample` is deprecated. Use `nn.functional.interpolate` instead.
  up2 = F.upsample(low3, size=rescale_size, mode='bilinear')
Traceback (most recent call last):
  File "/home/wzr/PersonaLive/torch2trt.py", line 93, in <module>
    export_onnx(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/home/wzr/PersonaLive/src/modeling/onnx_export.py", line 59, in export_onnx
    torch.onnx.utils.export(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 522, in export
    _export(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1457, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1080, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 964, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 871, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 1504, in _get_trace_graph
    outs = ONNXTracedModule(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 138, in forward
    graph, _out = torch._C._create_graph_by_tracing(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 129, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1763, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/wzr/PersonaLive/src/modeling/framed_models.py", line 28, in forward
    new_motion_hidden_states = self.motion_encoder(motion)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1763, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/encoder.py", line 37, in forward
    latent = self.model(rearrange(x, "b c f h w -> (b f) c h w"))
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1763, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py", line 314, in forward
    hg = self._modules['m' + str(i)](previous)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1763, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py", line 105, in forward
    return self._forward(self.depth, x)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py", line 91, in _forward
    low2 = self._forward(level - 1, low1)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py", line 91, in _forward
    low2 = self._forward(level - 1, low1)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py", line 97, in _forward
    low3 = self._modules['b3_' + str(level)](low3)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1763, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/wzr/PersonaLive/src/models/motion_encoder/FAN_feature_extractor.py", line 37, in forward
    out1 = self.bn1(x)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1763, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/modules/batchnorm.py", line 193, in forward
    return F.batch_norm(
  File "/home/wzr/PersonaLive/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 2817, in batch_norm
    return torch.batch_norm(
RuntimeError: Expected tensor for argument #1 'input' to have the same type as tensor for argument #2 'weight'; but type torch.cuda.FloatTensor does not equal torch.cuda.HalfTensor (while checking arguments for cudnn_batch_norm)

[RASPIAUDIOadmin]

same here ! windows 4090

[h93910]

same +1

[huai-chang]

Got this error when running torch2trt.py

I have tested torch2trt.py on my end, but I was unable to reproduce the issue you reported.

For reference, my test environment is:

OS: Ubuntu
GPU: RTX 4090D
PyTorch: 2.1.0

Warning: Accessing the `severity` property of G_LOGGER is deprecated and will be removed in v0.50.0. Use `module_severity` instead
/home/lzy/anaconda3/envs/personalive/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
['pred_video', 'latents', 'pose_cond_fea_out', 'motion_hidden_states_out', 'motion_out', 'latent_first']
开始导出 ONNX 模型到: /home/lzy/PA/PersonaLive/pretrained_weights/onnx/unet/unet.onnx ...
/home/lzy/anaconda3/envs/personalive/lib/python3.10/site-packages/diffusers/models/embeddings.py:175: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.height != height or self.width != width:
/home/lzy/anaconda3/envs/personalive/lib/python3.10/site-packages/torch/nn/functional.py:3769: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
/home/lzy/PersonaLive/src/models/unet_3d_explicit_reference.py:475: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if any(s % default_overall_up_factor != 0 for s in sample.shape[-2:]):
/home/lzy/PersonaLive/src/models/transformer_3d.py:118: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if encoder_hidden_states.shape[0] != hidden_states.shape[0]:
/home/lzy/PersonaLive/src/models/motion_module.py:195: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert encoder_hidden_states.shape[1] == video_length, (video_length, encoder_hidden_states.shape)
/home/lzy/PersonaLive/src/models/resnet.py:112: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert hidden_states.shape[1] == self.channels
/home/lzy/PersonaLive/src/models/resnet.py:116: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert hidden_states.shape[1] == self.channels
/home/lzy/PersonaLive/src/models/resnet.py:53: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert hidden_states.shape[1] == self.channels
/home/lzy/PersonaLive/src/models/resnet.py:64: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if hidden_states.shape[0] >= 64:
/home/lzy/anaconda3/envs/personalive/lib/python3.10/site-packages/diffusers/models/upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert hidden_states.shape[1] == self.channels
/home/lzy/anaconda3/envs/personalive/lib/python3.10/site-packages/diffusers/models/upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if hidden_states.shape[0] >= 64:
Dynamic: d00   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: d01   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: d10   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: d11   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: d20   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: d21   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: m     | Base(1x): 64    | Range: 64 ~ 256 | Opt: 128
Dynamic: u10   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: u11   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: u12   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: u20   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: u21   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: u22   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: u30   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: u31   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: u32   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
finished
Optimizing Onnx Model...
Saving to /home/lzy/PA/PersonaLive/pretrained_weights/onnx/unet_opt/unet_opt.onnx...
Optimization done.
[V] Loaded Module: tensorrt | Version: 10.14.1.48.post1 | Path: ['/home/lzy/anaconda3/envs/personalive/lib/python3.10/site-packages/tensorrt']
[V] [MemUsageChange] Init CUDA: CPU -2, GPU +0, now: CPU 5866, GPU 4119 (MiB)
[V] ----------------------------------------------------------------
[V] Input filename:   /home/lzy/PA/PersonaLive/pretrained_weights/onnx/unet_opt/unet_opt.onnx
[V] ONNX IR version:  0.0.8
[V] Opset version:    17
[V] Producer name:    pytorch
[V] Producer version: 2.1.0
[V] Domain:           
[V] Model version:    0
[V] Doc string:       
[V] ----------------------------------------------------------------
[I] TF32 is disabled by default. Turn on TF32 for better performance with minor accuracy differences.
[V] Setting TensorRT Optimization Profiles
[V] Input tensor: sample (dtype=DataType.HALF, shape=(1, 4, 16, -1, -1)) | Setting input tensor shapes to: (min=(1, 4, 16, 64, 64), opt=(1, 4, 16, 64, 64), max=(1, 4, 16, 64, 64))
[V] Input tensor: encoder_hidden_states (dtype=DataType.HALF, shape=(1, 1, 768)) | Setting input tensor shapes to: (min=(1, 1, 768), opt=(1, 1, 768), max=(1, 1, 768))
[V] Input tensor: motion_hidden_states (dtype=DataType.HALF, shape=(1, 12, 32, 16)) | Setting input tensor shapes to: (min=(1, 12, 32, 16), opt=(1, 12, 32, 16), max=(1, 12, 32, 16))
[V] Input tensor: motion (dtype=DataType.HALF, shape=(1, 3, 4, 224, 224)) | Setting input tensor shapes to: (min=(1, 3, 4, 224, 224), opt=(1, 3, 4, 224, 224), max=(1, 3, 4, 224, 224))
[V] Input tensor: pose_cond_fea (dtype=DataType.HALF, shape=(1, 320, 12, -1, -1)) | Setting input tensor shapes to: (min=(1, 320, 12, 64, 64), opt=(1, 320, 12, 64, 64), max=(1, 320, 12, 64, 64))
[V] Input tensor: pose (dtype=DataType.HALF, shape=(1, 3, 4, -1, -1)) | Setting input tensor shapes to: (min=(1, 3, 4, 512, 512), opt=(1, 3, 4, 512, 512), max=(1, 3, 4, 512, 512))
[V] Input tensor: new_noise (dtype=DataType.HALF, shape=(1, 4, 4, -1, -1)) | Setting input tensor shapes to: (min=(1, 4, 4, 64, 64), opt=(1, 4, 4, 64, 64), max=(1, 4, 4, 64, 64))
[V] Input tensor: d00 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: d01 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: d10 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: d11 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: d20 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: d21 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: m (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 64, 1280), opt=(1, 128, 1280), max=(1, 256, 1280))
[V] Input tensor: u10 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: u11 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: u12 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: u20 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: u21 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: u22 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: u30 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: u31 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: u32 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))

[huai-chang]

same here ! windows 4090

We haven't tested PersonaLive on Windows. This discussion might be helpful: #8 & #11

[malikmayank]

ONNX export is using autocast while the model is already in fp16. Update the export call in torch2trt. py (auto_cast = False) should fix it.