Generated images are blurry after using train_dreambooth_lora_sdxl_advanced.py to train a lora model

[joey0922]

I used script train_dreambooth_lora_sdxl_advanced.py from offical examples of diffusers to train a dreambooth and ti lora model by my own dataset, but images generated by this lora model are always blurry. I just changed the parameter max_train_steps without anything else changed and whatever how many steps I trained, the generated images are blurry. I believe my dataset to train is good with high quality and resolution. Can anybody help me out of this?

[tolgacangoz]

Could you examine its blog post?

[joey0922]

yes, I read it and tried, but the result is still the same.

[asomoza]

Hi, if you want help you will need to provide more info, we can't just guess what's your problem is. We have tested this script a lot of times and it works, so it can be literally anything.

First test it with a known dataset, so you can test it with just the dog example and see if it generates good results. If it works, the problem is your dataset.

if not, you will need to provide more information so we can help you, ideally it should be:

• your environment, including the diffusers version
• the model you're using.
• your dataset.
• your train command.

Without these, it's impossible for us to know what the problem is.

[joey0922]

Thanks for your reply, I fixed it myself.

[tolgacangoz]

How? I wonder what the reason was. Is it possible for you to share your experience with us?

[joey0922]

I am not quite sure, I just raise the value of repeats, and modify a suitable value for max train steps then it worked. But I found though I set the value of repeats back to 1, it still worked. I have no idea why it happened. I didn't change anything else.

[joey0922]

@asomoza @tolgacangoz I want to ask another question. How to train multiple concepts at the same time by this script? What I know for now is that I can set multiple values to parameter "token_abstraction" according to specific concepts. For example, if I just want train a model to generate different car such as Benz and Audi, I can set a string "Benz,Audi" to token_abstraction, and I also have captioned images for train. According to the car in image, I used different identifier in the prompt of the image. That is to say a image of Benz car, I captioned it starting with "a photo of Benz car". Is it correct? What else I need to do for multi-concept training? It seems that the parameters "instance_prompt" and "validation_prompt" are not able to cover all identifiers. It is invalid to set more than one prompts to them?

[asomoza]

I think that script is not made for multiple concepts. The main target of dreambooth is to train a single concept really fast with a very small dataset, so the best traditional answer to your question is that you should probably train multiple loras and then merge them and see if they work together. Specially if you don't want to put too much effort in it.

My personal answer and my experience is that I never train with a TI (because I think it makes the loading more complicated and the benefits are not clear for me) and I don't train on special tokens.

So what I do is to just generate a dataset with normal text descriptions and let the model learn, but for this to work you will need a bigger dataset and better descriptions, because the model needs to learn the difference between the concepts of each image and this will also take longer to train than a single concept DB LoRA.

This is more similar to traditional LoRA training but in my experience it works better/faster if you have similar concepts, like for example some cars, or a person with different attributes like long hair, short hair, glasses, etc.

I think most of the popular loras are trained like this too, you can test this because the trigger words are not something like sks or some special token, they train with something normal like "pixel art", "papercut" or depending on what you're training you can even do it without a trigger word.

[joey0922]

Thanks for you valuable advice. And is there any experience or material you can share with me? I've thought of training normal lora models, but I don't have enough images for train. If I wanna train a lora model by a dataset with normal text descriptions, how many images in dataset do you think is good for this job?

[asomoza]

is there any experience or material you can share with me

Sadly there isn't that much material too read about this because even a difference in a couple of images will affect the training a lot, specially with dreambooth. So there isn't a standard "recipe" to use, finetuning and training LoRAs (with image generation), IMO is learned a lot more by experimenting than by reading.

If I wanna train a lora model by a dataset with normal text descriptions, how many images in dataset do you think is good for this job?

It depends a lot in what you want so as before there isn't a number I can't give you about this.

Training specific persons doesn't need that much quantity of images but they need a really good description and quality, one bad image will make the training useless the same if you miss one description or put a wrong word. The hard part here with the descriptions is to know which is considered the "person" you want to train and what isn't.

Also people normally use really high rank numbers and learning rates which makes the training faster and to "seem" better but in reality they overfit the model and mostly the person trained can do just a couple of fixed poses and you won't be able to merge different loras with it.

Styles are more forgiving in the sense of quality and descriptions but you will need a lot more images for the model to learn it, I'm taking hundreds or even thousand of them, depending on the style and if the model knows something similar from before. For styles you can just use a good VLLM to generate the descriptions.

We're going to do sometime soon a blog post about this, also personally I really want to do a guide but I need a dataset and this is really hard to do because I need a public domain (or truly open source one) and safe one, and most of the models are already trained with those.