Changing StableDiffusion default Text Encoder to a different one.

[danielemolino]

Hi, probably that's a simple question but I wasn't able to find any tutorial. I need to use the Stable Diffusion model with a different Text Encoder for some tests. I'm doing something like this but it isn't working.

from transformers import AutoModel, AutoTokenizer
from diffusers import StableDiffusionPipeline

text_encoder = AutoModel.from_pretrained("StanfordAIMI/RadBERT")
text_tokenizer = AutoTokenizer.from_pretrained("StanfordAIMI/RadBERT")

text_prompts = ["A Chest X-ray of a patient with pneumonia."]

pipeline = StableDiffusionPipeline.from_pretrained(f"runwayml/stable-diffusion-v1-5")
pipeline.text_encoder = text_encoder
pipeline.tokenizer = text_tokenizer
pipeline.safety_checker = None

images = pipeline(
prompt=text_prompts,
height=512,
width=512,
num_inference_steps=25,
num_images_per_prompt=1,
).images[0]

But I get an error, especially this one:

OverflowError: int too big to convert

Anybody willing to help?

[tolgacangoz]

Hi Daniele!
This seems to work:

from transformers import AutoModel, AutoTokenizer
from diffusers import StableDiffusionPipeline
#import torch

text_encoder = AutoModel.from_pretrained("StanfordAIMI/RadBERT")#, load_in_8bit=True)
text_tokenizer = AutoTokenizer.from_pretrained("StanfordAIMI/RadBERT")#, load_in_8bit=True)

text_prompts = ["A Chest X-ray of a patient with pneumonia."]

pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5",
													text_encoder=text_encoder, safety_checker=None,
													#torch_dtype=torch.float16, variant='fp16'
													).to("cuda")
pipeline.tokenizer = text_tokenizer
pipeline.tokenizer.model_max_length = 512

image = pipeline(prompt=text_prompts,
			     height=512,
			     width=512,
			     num_inference_steps=25,
			     num_images_per_prompt=1,
			     ).images[0]
image

[tolgacangoz]

The current model_max_length is 1000000000000000019884624838656.

[s9anus98a]

Hi Daniele! This seems to work:

from transformers import AutoModel, AutoTokenizer
from diffusers import StableDiffusionPipeline
#import torch

text_encoder = AutoModel.from_pretrained("StanfordAIMI/RadBERT")#, load_in_8bit=True)
text_tokenizer = AutoTokenizer.from_pretrained("StanfordAIMI/RadBERT")#, load_in_8bit=True)

text_prompts = ["A Chest X-ray of a patient with pneumonia."]

pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5",
													text_encoder=text_encoder, safety_checker=None,
													#torch_dtype=torch.float16, variant='fp16'
													).to("cuda")
pipeline.tokenizer = text_tokenizer
pipeline.tokenizer.model_max_length = 512

image = pipeline(prompt=text_prompts,
			     height=512,
			     width=512,
			     num_inference_steps=25,
			     num_images_per_prompt=1,
			     ).images[0]
image

How to change text encoder to T5 XXL for sd1.5?

[asomoza]

is not a transformers issue, if you use the model as intended it doesn't output any error:

text_encoder = AutoModel.from_pretrained(
    "StanfordAIMI/RadBERT",
    torch_dtype=torch.float16,
)
text_tokenizer = AutoTokenizer.from_pretrained(
    "StanfordAIMI/RadBERT",
    torch_dtype=torch.float16,
)

text = "A Chest X-ray of a patient with pneumonia."
encoded_input = text_tokenizer(text, return_tensors="pt")
output = text_encoder(**encoded_input)

There's seems to be a misunderstanding here, you can't just replace any model as a text encoder, the SD Pipeline was coded to work with the CLIP models which is a totally different architecture than bert (only text). Also SD was trained with those, if you save the image you'll see that it outputs really bad images because the embeddings are different.

The same answer for the the T5, you can't just replace it, you need to train the model with it for it to work, that's why you can see that just recently there are some models trained with T5. If it was this easy, everyone will be just working with the latest and the greatest.

But there's a project for using a LLM with SD 1.5 if you want to test it: https://github.com/TencentQQGYLab/ELLA

[s9anus98a]

is not a transformers issue, if you use the model as intended it doesn't output any error:

text_encoder = AutoModel.from_pretrained(
    "StanfordAIMI/RadBERT",
    torch_dtype=torch.float16,
)
text_tokenizer = AutoTokenizer.from_pretrained(
    "StanfordAIMI/RadBERT",
    torch_dtype=torch.float16,
)

text = "A Chest X-ray of a patient with pneumonia."
encoded_input = text_tokenizer(text, return_tensors="pt")
output = text_encoder(**encoded_input)

There's seems to be a misunderstanding here, you can't just replace any model as a text encoder, the SD Pipeline was coded to work with the CLIP models which is a totally different architecture than bert (only text). Also SD was trained with those, if you save the image you'll see that it outputs really bad images because the embeddings are different.

The same answer for the the T5, you can't just replace it, you need to train the model with it for it to work, that's why you can see that just recently there are some models trained with T5. If it was this easy, everyone will be just working with the latest and the greatest.

But there's a project for using a LLM with SD 1.5 if you want to test it: https://github.com/TencentQQGYLab/ELLA

Thank you! What i mean how to use ELLA in colab? i have tried and got an error please help. error:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x2048 and 768x320)

my full script to produce error & using ELLA:
https://colab.research.google.com/drive/1izYMstRblS65wGoVVSwDR4FuZA6_ISOT

[asomoza]

that's another discussion and that's not related to diffusers, nevertheless I have some kind of interest in it so I can help you when I have the time, but please open another discussion because this one is not about ELLA.

The short answer is that ELLA is not just about replacing the text_encoder, they have a trained unet that plugs in too, so you need to load the unet and patch the forward method for it to work.