RuntimeError: Input type (float) and bias type (c10::Half) should be the same

[Adoni]

When I finetune dream booth with flux and lora, there is an error when validating:

  File "/root/code/diffusers/src/diffusers/models/autoencoders/autoencoder_kl.py", line 292, in _decode
    dec = self.decoder(z)
  File "/root/miniconda3/envs/flux23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/flux23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/code/xcode/diffusers/src/diffusers/models/autoencoders/vae.py", line 291, in forward
    sample = self.conv_in(sample)
  File "/root/miniconda3/envs/flux23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/flux23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/envs/flux23/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/root/miniconda3/envs/flux23/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (float) and bias type (c10::Half) should be the same

Then I tried to finetune sd3 and it shows the same error.

Anyone know the solution?

The GPU is H100 and I use exactly the code in diffusers dream booth example: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux.md

[asomoza]

cc: @linoytsaban

[linoytsaban]

Hi @Adoni can you please share here exactly the config you used, and specifically which script you used- dreambooth or dreambooth lora?

[linoytsaban]

Hi @Adoni, I wasn't able to reproduce your bug - I ran the exact same config and it worked well. Could it be an issue with your env?

[Adoni]

Thanks. I will try to use a new env.

[Adoni]

New env doesn't help.

Did you use deepspeed to run this example?

[linoytsaban]

No I didnt, maybe that's the issue. Could you share what version of diffusers, transformers, accelerate & peft you are using?

[Adoni]

When I run with deepspeed, the error is solved.
When I run without deepspeed, it occurs again. Here is the version you need:

• diffusers: 0.31.0.dev0 (up to today)
• transformers: 4.45.2
• accelerate: 1.0.1
• peft: 0.13.2