Incompatible Safetensors Metadata and Weight Name Issues in SDXL ControlNet Setup

[Artemis1111]

The following code is part of my SDXL ControlNet execution code.

# controlnet model select
controlnet = ControlNetModel.from_pretrained(
    controlnet_model_directory,
    torch_dtype=torch.float16,
    weights_name=controlnet_model_name,
    use_safetensors=True,
)

# vae model select
vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix",
    torch_dtype=torch.float16
)

# unet select - important
unet = UNet2DConditionModel.from_pretrained(
    model, 
    weights_name="model.safetensors",
    low_cpu_mem_usage=False, 
    device_map=None,
    torch_dtype=torch.float16,
    use_safetensors=True,
)


# scheduler select
scheduler = DPMSolverMultistepScheduler.from_config(scheduler_config, use_karras_sigmas=True)
# scheduler = DPMSolverMultistepScheduler.from_config(scheduler_config, use_karras_sigmas=True, algorithm_type = "sde", solver_order = 2) # if you want 'DPM++ 2M SDE Karras', use this



pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    model,
    unet=unet,
    controlnet=controlnet,
    vae=vae,
    scheduler=scheduler,
    torch_dtype=torch.float16,
    use_safetensors=True,
).to('cuda')

pipe.enable_model_cpu_offload()
pipe.save_pretrained(model)

I have two questions:

While running this code, I noticed that the weights_name="model.safetensors" in the UNet is not being executed. After digging into the internal code, I found out that the weights name always has to be defined as something like what's set in diffusers>utils>constants.py (e.g., SAFETENSORS_WEIGHTS_NAME = "model.safetensors").

Similarly, in pipe, it requires weights files to be named as pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index, or flax_model.msgpack.

The problem I'm facing is that the required model weight names are different between these two parts, which causes issues. Could you help me find a solution for this?

I temporarily solved this issue by copying and pasting the same weights model (6.5GB) to make two files. However, a new problem has arisen:

Traceback (most recent call last):
  File "/my/path", line 155, in <module>
    pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py", line 876, in from_pretrained
    loaded_sub_model = load_sub_model(
  File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_loading_utils.py", line 703, in load_sub_model
    loaded_sub_model = load_method(cached_folder, **loading_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3804, in from_pretrained
    raise ValueError(
ValueError: Incompatible safetensors file. File metadata is not ['pt', 'tf', 'flax', 'mlx'] but None

I encounter this same issue even when trying with the original SDXL 1.0 base model, not just my finetuned model. It says there’s no file metadata. How can I resolve this issue?

The versions of each library are as follows:

transformers 4.45.2
torch 2.1.0+cu118
diffusers 0.30.3

[asomoza]

Hi, usually when you use UNet2DConditionModel.from_pretrained is with a diffusers unet model, which requires a config.json, so this should be always a directory and not a file.

You're saying that the weights_name doesn't get executed but this is not correct, the from_pretrained doesn't have a weights_name argument, so you're just passing a random argument.

If you're using from_pretrained to load a unet, it should be a directory that has a config.json file and the model weights filename should be either diffusion_pytorch_model.safetensors or diffusion_pytorch_model.fp16.safetensors

Also you're mixing more stuff, the error you're getting is with the pipeline, probably the model variable is not a directory either or it doesn't have a model_index.json

It's hard to tell the specific error because you have unknowns to us, if you want more help you'll need to provide a full reproducible code with a link to the weights and the file structure.

Are you sure you need from_pretrained though? it's really weird that someone saved the models weights in the diffusers format and then practically changed the whole file structure and the naming afterwards for some unknown reason.

If you still need to use something like this, you probably will need to load the models with vanilla pytorch.

As an example, you can load the unet without problems using this code:

import torch

from diffusers import AutoencoderKL, StableDiffusionXLPipeline, UNet2DConditionModel


vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

unet = UNet2DConditionModel.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    subfolder="unet",
    torch_dtype=torch.float16,
    variant="fp16",
)

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", unet=unet, vae=vae, variant="fp16"
)

If you want to load a custom unet, for example, the lightings ones, you can use something like this:

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

from diffusers import AutoencoderKL, StableDiffusionXLPipeline, UNet2DConditionModel


vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

config = UNet2DConditionModel.load_config("stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet")
unet = UNet2DConditionModel.from_config(config)
unet.load_state_dict(
    load_file(hf_hub_download("ByteDance/SDXL-Lightning", "sdxl_lightning_4step_unet.safetensors"), device="cuda")
)

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", unet=unet, vae=vae, variant="fp16"
)

[Artemis1111]

Thank you.

I confirmed that the from_pretrained function does not have a weights_name argument.

Additionally, regarding the second question, I did indeed have the model_index.json file.

However, what I realized while solving the issue is that I misunderstood the structure of safetensors.

There are two types of safetensors used in Stable Diffusion.

The first is the safetensors provided by Huggingface.

The second is a kind of safetensors that not only contains the safetensors provided by Huggingface, but also the config.json, various other files, and weights, packaged together like a zip file.

I did not know the difference and mistakenly thought the second type of safetensors was the first one. This led me to ask the rather silly question about the second issue.

I resolved the problem in the following way:

import torch
from diffusers import StableDiffusionXLPipeline

pipeline = StableDiffusionXLPipeline.from_single_file(
    "model_name.safetensors", 
    torch_dtype=torch.float16,
).to("cuda")

pipeline.push_to_hub("{huggingface_id}/{project_name}")