Questions of pre-training LoRA with other modules simutaneously

[George0726]

Describe the bug

Sorry for asking questions unrelated to diffusers... I am trying to jountly train an ELLA model with a UNET LoRA (rank: 64). However, I confronted a weird situation, the ELLA model worked well while the LoRA crashed and made the output like random noise. Have you guys witnessed this phenomenon?

Reproduction

I have tried multiple ways, like FP16, FP32, Deepspeed, and checked my code many times. It confused me for a long time. I would appreciate if the community conld provide some suggestions or guidiances.

Logs

No response

System Info

• diffusers version: 0.28.0.dev0
• Platform: Linux-6.5.0-35-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version (GPU?): 2.2.0+cu121 (True)
• Huggingface_hub version: 0.21.3
• Transformers version: 4.30.2
• Accelerate version: 0.21.0
• xFormers version: 0.0.24
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[sayakpaul]

This is NOT a library issue and is better off here as a discussion.