On Custom Finetuning : RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same #bug

[sanjay7178]

Traceback (most recent call last): | 0/8 [00:00<?, ?it/s]
File "/home/vitap/Desktop/Hotshot-XL/fine_tune.py", line 994, in
main()
File "/home/vitap/Desktop/Hotshot-XL/fine_tune.py", line 976, in main
process_batch(batch)
File "/home/vitap/Desktop/Hotshot-XL/fine_tune.py", line 922, in process_batch
run_validation(step=global_step, node_index=accelerator.process_index // 8)
File "/home/vitap/Desktop/Hotshot-XL/fine_tune.py", line 718, in run_validation
video = pipe(prompt,
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/vitap/Desktop/Hotshot-XL/hotshot_xl/pipelines/hotshot_xl_pipeline.py", line 891, in call
video = self.decode_latents(latents)
File "/home/vitap/Desktop/Hotshot-XL/hotshot_xl/pipelines/hotshot_xl_pipeline.py", line 985, in decode_latents
video.append(self.vae.decode(
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/diffusers/models/autoencoder_kl.py", line 288, in decode
decoded = self._decode(z).sample
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/diffusers/models/autoencoder_kl.py", line 274, in _decode
z = self.post_quant_conv(z)
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/vitap/miniconda3/envs/hotshot/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same

[howardgriffin]

Same promble. How to solve it?

[jrohsc]

I fixed by changing the latents dtype to bfloat16 in the hotshot_xl_pipeline.py. Hope this helps everyone.

[sanjay7178]

I fixed by changing the latents dtype to bfloat16 in the hotshot_xl_pipeline.py. Hope this helps everyone.

thanks , I do the same but in my hardware doesnt works for me