Outpainting I - Controlnet version

[asomoza]

Outpainting with controlnet

There are at least three methods that I know of to do the outpainting, each with different variations and steps, so I'll post a series of outpainting articles and try to cover all of them.

Outpainting with controlnet requires using a mask, so this method only works when you can paint a white mask around the area you want to expand. With this method it is not necessary to prepare the area before but it has the limit that the image can only be as big as your VRAM allows it.

For this case I'll use a wolf image that was provided by @Laidawang in this comment:

This time I won't do a comparison with the popular applications because the intention of this post is not to write about other webuis but to do an example on how to do this with diffusers. I did use them to have a baseline though.

The starting prompt is a wolf playing basketball and I'll use the Juggernaut V9 model.

There's a controlnet for SDXL trained for inpainting by destitech: https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl. It's an early alpha version but I think it works well for most cases.

controlnet = ControlNetModel.from_pretrained(
    "destitech/controlnet-inpaint-dreamer-sdxl", torch_dtype=torch.float16, variant="fp16"
)

This controlnet is really easy to use, you just need to paint white the parts you want to replace, so in this case what I'm going to do is paint white the transparent part of the image.

To paint white the alpha channel of an image I use this code:

response = requests.get("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/outpainting/313891870-adb6dc80-2e9e-420c-bac3-f93e6de8d06b.png?download=true")

control_image = Image.open(BytesIO(response.content))
new_controlnet_image = Image.new("RGBA", control_image.size, "WHITE")
new_controlnet_image.alpha_composite(control_image)

With this we get the same image but with a white background:

This image has a 720x720 px size, SDXL works better with 1024x1024 image, so the generations will be upscaled, you can use the smaller image but you'll probably get a lower quality image.

The conditional scale affects how much of the original image will be preserved, since this is an outpaint, it's safe to use higher values, for inpainting and complex images is better to use lower values around 0.5.

With the same seed I get these results:

0.2	0.5	0.7	0.9	1.0

I always prefer to allow the model to have a little freedom so it can adjust tiny details to make the image more coherent, so for this case I'll use 0.9

At this point I think we are at the level of other solutions, but let's say we want the wolf to look just like the original image, for that I want to give the model more context of the wolf and where I want it to be so I'll use an IP adapter for that.

There's a little trick that works for me and is that I use the generated image I want as a base and paint the mask of the wolf over it and then use this as an attention mask for the IP Adapter.

The process I use to generate the mask is like this:

base image	wolf mask	background

For me doing this with something like GIMP doesn't take me more than a minute to do.

I'm using the IP Plus Adapter with a scale of 0.4, and with these settings we now get these results:

My guess as to why this works is that since we’re drawing a shape of a wolf and giving it an input image of a wolf, the model tries to maintain the shape, position, and coherence. In short, we’re giving the model more context for it to generate the final image, and now it looks a lot more like the original.

For example, if we don’t use a mask for this ip adapter we get this:

Now we're at the point where the image looks good but I always want more, so lets improve it with the prompt.

I'll change the prompt to this:

"high quality photo of a wolf playing basketball, highly detailed, professional, dramatic ambient light, cinematic, dynamic background, focus"

Also I'll give it a little more freedom to the controlnet with control_guidance_end=0.9 so it can finish the details without restrictions.

Finally, if you’ve ever worked with compositing images or video before, you’ll know that it is a common practice to apply a filter to the whole composition to unite the final look. This is true for stable diffusion too, and it sometimes hides the seams better, if there are any. For this, I’ll do a final pass to the whole image with an image-to-image pipeline.

Diffusers allows to just change the pipeline while preserving the loaded models which is what we need here.

stable_img2img_pipeline = StableDiffusionXLImg2ImgPipeline(
    vae=pipeline.vae,
    text_encoder=pipeline.text_encoder,
    text_encoder_2=pipeline.text_encoder_2,
    tokenizer=pipeline.tokenizer,
    tokenizer_2=pipeline.tokenizer_2,
    unet=pipeline.unet,
    scheduler=pipeline.scheduler,
    image_encoder=pipeline.image_encoder,
    feature_extractor=pipeline.feature_extractor,
)

The VAE decoding is a lossy process, so everytime we encode or decode we're lossing details and lowering the quality of the image, to prevent this we need to stay in the latent space as much as possible.

Diffusers allows this if you pass output_type="latent" to the pipeline. We then feed the latents to the image to image pipeline but before that I want to also give it a more cinematic look, so I'll change the prompt again:

prompt = "cinematic film still of a wolf playing basketball, highly detailed, high budget hollywood movie, cinemascope, epic, gorgeous, film grain"

This should be just a quick pass, so I'll set the steps to 30 and the strength to 0.2.

These are the final results:

In my opinion these images are as good as or better than the outpainting done with other UIs and hopefully these will help people to better understand what you can do with diffusers.

As a bonus, there's a neat little trick. Before I used the IP Adapter with a mask to give more of the initial image to the generation. However, this can also be used without a mask. This has the effect of feeding the model more context and makes it better able to guess the rest of the image, for example with just the prompt "high quality":

source image	without ip adapter	with ip adapter

So if I do what I did before but with the exception of the ip adapter mask, this is the final result:

Don't ask me why is the Eiffel tower there ^^

This is the code:

import random
from io import BytesIO

import requests
import torch
from PIL import Image

from diffusers import (
    AutoPipelineForImage2Image,
    ControlNetModel,
    DPMSolverMultistepScheduler,
    StableDiffusionXLControlNetPipeline,
)
from diffusers.image_processor import IPAdapterMaskProcessor
from diffusers.utils import load_image, logging
from diffusers.utils.logging import set_verbosity


set_verbosity(logging.ERROR)  # to not show cross_attention_kwargs..by AttnProcessor2_0 warnings

controlnet = ControlNetModel.from_pretrained(
    "destitech/controlnet-inpaint-dreamer-sdxl", torch_dtype=torch.float16, variant="fp16"
)

pipeline = StableDiffusionXLControlNetPipeline.from_pretrained(
    "RunDiffusion/Juggernaut-XL-v9",
    torch_dtype=torch.float16,
    variant="fp16",
    controlnet=controlnet,
).to("cuda")

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
pipeline.scheduler.config.use_karras_sigmas = True

pipeline.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter-plus_sdxl_vit-h.safetensors",
    image_encoder_folder="models/image_encoder",
)
pipeline.set_ip_adapter_scale(0.4)

ip_wolf_image = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/outpainting/ip_wolf_source.png?download=true"
)

ip_mask = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/outpainting/wolf_position_mask.png?download=true"
)

processor = IPAdapterMaskProcessor()
ip_masks = processor.preprocess(ip_mask, height=1024, width=1024)

response = requests.get(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/outpainting/313891870-adb6dc80-2e9e-420c-bac3-f93e6de8d06b.png?download=true"
)
control_image = Image.open(BytesIO(response.content))
new_controlnet_image = Image.new("RGBA", control_image.size, "WHITE")
new_controlnet_image.alpha_composite(control_image)

prompt = "high quality photo of a wolf playing basketball, highly detailed, professional, dramatic ambient light, cinematic, dynamic background, focus"
negative_prompt = ""

seed = random.randint(0, 2**32 - 1)
generator = torch.Generator(device="cpu").manual_seed(seed)

latents = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=1024,
    width=1024,
    guidance_scale=6.5,
    num_inference_steps=25,
    generator=generator,
    image=new_controlnet_image,
    controlnet_conditioning_scale=0.9,
    control_guidance_end=0.9,
    ip_adapter_image=ip_wolf_image,
    cross_attention_kwargs={"ip_adapter_masks": ip_masks},
    output_type="latent",
).images[0]

pipeline_img2img = AutoPipelineForImage2Image.from_pipe(pipeline, controlnet=None)

prompt = "cinematic film still of a wolf playing basketball, highly detailed, high budget hollywood movie, cinemascope, epic, gorgeous, film grain"

image = pipeline_img2img(
    prompt=prompt,
    negative_prompt=negative_prompt,
    guidance_scale=3.0,
    num_inference_steps=30,
    generator=generator,
    image=latents,
    strength=0.2,
    ip_adapter_image=ip_wolf_image,
    cross_attention_kwargs={"ip_adapter_masks": ip_masks},
).images[0]

image.save("result.png")

[yiyixuxu]

awesome! thank you for sharing your knowledge! this is super educational and tremendously valuable to our community :)

[yiyixuxu]

oh thanks! will look into them!

[asomoza]

thanks, in the meantime I'll ask the model owner if he can upload the config.json and publish a fp16 version.

[yiyixuxu]

for from_pipe, you can use this, if you don't do controlnet=None, it will switch to controlnet img2img

stable_img2img_pipeline= AutoPipelineForImage2Image.from_pipe(pipeline, controlnet=None)

[yiyixuxu]

for the controlnet checkpoint, it is a diffusers checkpoint, just named differently?!

[asomoza]

oh ok, I'll fix that for now, so I was always doing the second pass without a controlnet, my mistake. I will test it with the controlnet one because my original intention was to use it in the second pass too.

The controlnet was trained with the diffusers script, the model owner says it here: https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl/discussions/2#65b20645b4225e3e423f50d3

[FurkanGozukara]

hello great guide. is this different than what we have in controlnet of automatic1111?

can we replicate this in controlnet of Mikubill in auto1111? anyone tested?

[asomoza]

if automatic1111 can load the controlnet (I only know that it works in comfyui) and you can use IP Adapters at the same time, I don't see why not. For the second pass you'll need to send it to img2img, but that isn't that necessary if you're happy with just the result from controlnet. Is just that you can't do it all in a single run.

[Laidawang]

@asomoza soooo cool, according to your pipeline, I think it is possible to achieve the same quality of fooocus or even better.
1.Why don't we try fooocus's process? That is to say, you use controlnet-inpaint-dreamer-sdxl + Juggernaut V9 in steps 0-15 and Juggernaut V9 in steps 15-30. You may need to modify the pipeline code, pass in two models and modify them in the intermediate steps.（Why do I think this? I think controlnet will affect the generation quality of sdxl model, so 0.9 may be too lagging）
2.We should use https://github.com/exx8/differential-diffusion to make the seams better, so that you don't need to use the second stage strength of 0.2 to blend the images.

[asomoza]

I did that too, it works but I wanted to keep as much of the original wolf as I can. If you take out the controlnet, you can do a lot more creative stuff but at some point is not just an outpaint but a whole different image like the anthropomorphic wolf I posted in the original issue.

Differential diffusion is one of the other methods I mentioned, I'll do a full guide on that one too, Also keep in mind that I'm just using juggernaut here, but in the other guides I'll switch to other checkpoints depending on the original image.

But those are good ideas, if you have more, you can share them and I'll try test them all.

[Laidawang]

https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py try to use this pipeline which accepts a mask so you can keep the original area unchanged.

[asomoza]

Yeah that is going to be the the third method I'm going to do, I mentioned that I know three methods to do the outpainting, these are:

1. With controlnet
2. With an inpainting model and using the inpainting pipeline
3. With differential diffusion

IMO all three of them are good.

[Laidawang]

We should fuse them, this will have the best effect.In my opinion, they can coexist.

[asomoza]

I don't remember if I tried that, but I think they won't work good together, but when I'm done with each one I'll try again doing a mix of them and test the results.

Also differential diffusers works with gradients and/or depthmaps, that means you can use a soft brush for the mask, the controlnet and the inpainting model won't know what to do with that.

[peki12345]

Nice work! I noticed a problem. The source image has been significantly changed (especially the color change). After all, ip_adapter can't perfectly retain the image information, which causes the image to be unnatural after source image being pasted. Fooocus does not have this problem.

[asomoza]

yeah, this could be considered a more creative outpainting, for keeping the source image the same as the original, is better to use the other methods, either with an inpainting model or with differential diffusion, I'll soon post those guides too.

If I remove the background of the source image I can blend the original on top of the inpainting but I wanted to keep it simple as most people won't remove the background.

Also I'm probably not good at using fooocus because the results I got are like this:

For those I had to do the mask a little inside of the wolf image, if I just do the outside, like in this example, I get this:

So I'm probably doing something wrong since a lot of people say that fooocus is really good at this, with comfy I get a lot better results but that's probably because I use it a lot too and know how it works.

[Laidawang]

@asomoza https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py this pipeline can use both sdxl inpaint and controlnet, and by insert the code here https://github.com/exx8/differential-diffusion/blob/main/SDXL/diff_pipe.py#L976 you can use differential diffusers too

[asomoza]

I know, I have my own code where I don't use pipelines, I can load all three and even more stuff at the same time without restrictions. What I meant is that it probably won't look good since there's going to be a race between them trying to inpaint. Let me do a quick test and show you.

Also, again, diff-diff uses soft inpainting, if you use diff-diff with a binary mask it won't do anything better than regular inpaint if not worse.

[asomoza]

This wasn't as easy as it sounds, but finally I got it working, here's the results with an inpainting pipeline, juggernaut inpainting and controlnet:

strength=1.0	strength=1.0

strength=0.99	strength=0.99

As I told you, with a strength of 1.0, the inpainting model is the one that has full control over the generation. However, if I lower it just 0.01, they start to conflict with each other and generate images that are surprisingly similar to what fooocus produces.

So maybe with this experiment we now know what is the inpainting patch that fooocus does but for me, they aren't good or usable.

[Laidawang]

I know why you got black, because you did not process the initial image. For a transparent image, the outpaint area is black as the initial latent. You can consider using a super large Gaussian blur or lamma to modify the outpaint area in advance as init encode image.

[Laidawang]

check the code here, https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py#L908, when you use strength =1 init_latent = noise , when your strength < 1,init_latent = lantent_image + noise

[asomoza]

I know that for the other methods I will need to do something with the outpainting area, this was just a quick test that I did for you.

I'll do the other methods alone first then and at the end I'll try to mix them as an experiment and see if there's an improvement, because I'll need to probably do a custom pipeline for that.

[joey0922]

That's wonderful!!!! But I encountered a little bug, I use a image of car with white background to generate new background, but i found it is not able to retain every detail of the car, especially the logo of car always changed and destoryed. why did it happen? How to solve it? Thank you!

[asomoza]

For that one I did a lot of the steps manually, but I'm doing a guide with something similar to your problem and I'll post the full code very soon.

If you want to try before that, the trick here is to generate the background before the outpainting.

Using your image, use the pre-processors so you can and use it with controlnet or t2i adapters to generate the background, it doesn't really matter the quality at this step, you just need something similar of what you want, we need this to fill the background with the right perspective:

Lineart	Depth

Now that you have the background and the foreground (your car) you can use them to generate the final image (with a good mask). The model likes to add detail to the car, so you'll need to be very specific with the prompt or use a controlnet to prevent it. You can play with the colors of the background before this or maybe add blur, it all depends on what you want.

If you can lose some of the original detail you can add a second pass with image-to-image, or just use another software to add a filter or LUT to blend the result better.

[joey0922]

There are two goals I think I want when I try to generate background for a specific car:

1. Keep the key features of the car, such as the logo, outward appearance or shape of the car which make it easily identified and so on. In one word, I want the car I generated is real so that we can recognize what car it is easily.
2. The car blends in the generated background well, especially the color, light, shadows and space matched perfectly, which makes the image look nature, like a real photo or painting. I think it must be seamless.

The method you mentioned above sounds good. I will take a shot later. Could you give me more details about fusing background and foreground(my car) please? I know the mask is important, but I am not good at it. May I ask you to teach me how to make a perfect mask, or is there any model you can recommend to fuse car and background naturally? Thanks again!

[joey0922]

@asomoza Is there any update about your third guide? I am looking forward to it.

[asomoza]

yeah, sorry I forgot to update this, I published it yesterday, here it is: https://huggingface.co/blog/OzzyGT/outpainting-inpaint-model. Hope it helps, I'll start another discussion about the 3 of them and outpainting.

[joey0922]

got it, really appreciate it!

[yiyixuxu]

here is part2 of the outpainting guide https://huggingface.co/blog/OzzyGT/outpainting-differential-diffusion

[asomoza]

I'm thinking to group all three of them in one post when I'm done with the next one, so each one as a blog post and post here a short intro with a link to each of them.

[AbhinavJangra29]

i've been trying to add this in my personal project but its eating up my vram like crazy. i am not able to figure out how can i streamline all the 3 pipelines into a single one. this is my current schema-

• pipeline_1 gives a raw outpaint using a control image , which is then used to make up the mask for inpaint
• pipeline 2 and 3 are the ip adapter inpaint pipeline and the i2i pipeline.

now the thing is the flow of the whole process is pipeline_1 to pipeline_3 and consecutive pipelines are just modification of the previous ones. if i modify the pipelines in place, i wont be able to use it on the next image(since image needs to pass from pipeline_1 , which is now modified into pipeline_3 due to the previous pass). if i make 3 seperate pipeline it just eats up my vram (25gb rtx3090). is there a way i can make a single reusable block ? @asomoza

[asomoza]

I don't fully understand all you're trying to do but you can use all the pipelines you want without the need of additional VRAM.

If I have the need of using multiple pipelines, I would do:

• Make all the pipelines return the latents only when possible, this makes everything faster and with less VRAM consumption.
• Always use from_pipe to transfer the modules, you just need to make the modules None when you don't need them or load them again when you need.
• Unload loras and the IP Adapters when not needed (image encoder too).

If you really need a better memory management I would load each model separately (unet, text encoders, vae), unload or load them to RAM and VRAM and then create the pipelines with them when needed, but the from_pipe should be enough.

You can share the model, but if you're using a normal model and an inpaint model, you can't do this, you will have to unload one or you will be using more VRAM, in this case the best solution will be to offload the one you're not using to the RAM.

[Bilal143260]

@asomoza thanks for this amazing guide. I have gone through all of your three blog posts and I have observed that all of the workflows struggle to generate human faces and hands. For example I take an image of cloth and try to outpaint a human inside it, it will mess up the facial structure and even anatomy. Any work around that?

[asomoza]

outpainting refers more to expanding an image, what you're suggesting is just plain inpainting.

Usually what you're suggesting its done in reverse, given a specific person you try to dress them with specific clothes that adapts to them.

What you want to do is more like what the use in this discussion wanted to do, so depending on the quality you can follow the same advice.

Also there's a lot more stuff we can use right now, these guides are old (in AI time) and I have to update them but haven't had the time yet.

Finally take into account that hands and faces are the hardest parts in humans that diffusion models struggle with, that's why it's better to do it the other way around, clothes are a lot more easier to adapt than human anatomy, even Flux produces blobs of humans.

[pietrobolcato]

Amazing work! Thank you so much 🙏 would it be possible to provide a similar guide to obtain the equivalent of Generative Fill from photoshop? it would be super valuable!

[pietrobolcato]

Hey there dear @asomoza, thanks so much again for the amazing work 🙏
I got the chance to play with it, and in general is awesome 🔥 often times though, the model would generate a new subject instead of filling the background, see example attached:

Any idea how this could be solved? or is it just a limitation of the model we need to accept? :)

[asomoza]

Hi, sorry for the late answer, the only way to solve this if it keeps generating a new subject is to use a prompt and give a description of what you want and don't want, but this makes it just a regular inpainting.

Also as a more advanced solution for this that I often use is to additionally use img2img and draw a raw shape or color of what I want, so for example in the case of the bear, I would paint white the area around the hands and it won't generate any object in them.

[pietrobolcato]

Hi, sorry for the late answer, the only way to solve this if it keeps generating a new subject is to use a prompt and give a description of what you want and don't want, but this makes it just a regular inpainting.

Also as a more advanced solution for this that I often use is to additionally use img2img and draw a raw shape or color of what I want, so for example in the case of the bear, I would paint white the area around the hands and it won't generate any object in them.

No worries man thanks so much again! 🙏 the second solution sounds really good! i don't want to steal much of your time - but if you have a second in the future, do you think it would be possible to put together a small example of what you mean in code? would be extremely helpful for this purpose

thanks a lot again 🫶

[asomoza]

sure, I'll try to do one when I have the time soon.

Also please don't hesitate to ask for other real use cases, the image fill one was really popular and I'm always looking for real use tasks to do guides with diffusers.

[pietrobolcato]

sure, I'll try to do one when I have the time soon.

Also please don't hesitate to ask for other real use cases, the image fill one was really popular and I'm always looking for real use tasks to do guides with diffusers.

Amazing thank you so much man you're a real hero! Very greatly appreciated ❤️