Outpainting with Diffusers adding extra artifacts

[levoz92]

Hi.

I've been struggling with this problem for sometime so I decided to revert to the original code in this blog post to see where I was going wrong. First, I ran the code as it is with no changes to produce the car and I got this:

And then, I only changed the filename and the prompts to see if I can do the same inpainting with humans. Here is what the code looks like now:

import random

import requests
import torch
from controlnet_aux import ZoeDetector
from PIL import Image, ImageOps

from diffusers import (
    AutoencoderKL,
    ControlNetModel,
    StableDiffusionXLControlNetPipeline,
    StableDiffusionXLInpaintPipeline,
)


def scale_and_paste(original_image):
    aspect_ratio = original_image.width / original_image.height

    if original_image.width > original_image.height:
        new_width = 1024
        new_height = round(new_width / aspect_ratio)
    else:
        new_height = 1024
        new_width = round(new_height * aspect_ratio)

    # make the subject a little smaller
    new_width = new_width - 20
    new_height = new_height - 20

    resized_original = original_image.resize((new_width, new_height), Image.LANCZOS)
    white_background = Image.new("RGBA", (1024, 1024), "white")
    x = (1024 - new_width) // 2
    y = (1024 - new_height) // 2
    white_background.paste(resized_original, (x, y), resized_original)

    return resized_original, white_background


# load the original image with alpha
original_image = Image.open('drive/MyDrive/woman-running.png').convert("RGBA")
resized_img, white_bg_image = scale_and_paste(original_image)

# load preprocessor and generate depth map
zoe = ZoeDetector.from_pretrained("lllyasviel/Annotators")
image_zoe = zoe(white_bg_image, detect_resolution=512, image_resolution=1024)

# load controlnets
controlnets = [
    ControlNetModel.from_pretrained(
        "destitech/controlnet-inpaint-dreamer-sdxl", torch_dtype=torch.float16, variant="fp16"
    ),
    ControlNetModel.from_pretrained("diffusers/controlnet-zoe-depth-sdxl-1.0", torch_dtype=torch.float16),
]

# vae in case it doesn't come with model
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16).to("cuda")

# initial pipeline for temp background
pipeline = StableDiffusionXLControlNetPipeline.from_pretrained(
    "SG161222/RealVisXL_V4.0", torch_dtype=torch.float16, variant="fp16", controlnet=controlnets, vae=vae
).to("cuda")


# function to generate
def generate_image(prompt, negative_prompt, inpaint_image, zoe_image, seed: int = None):
    if seed is None:
        seed = random.randint(0, 2**32 - 1)

    generator = torch.Generator(device="cpu").manual_seed(seed)

    image = pipeline(
        prompt,
        negative_prompt=negative_prompt,
        image=[inpaint_image, zoe_image],
        guidance_scale=6.5,
        num_inference_steps=25,
        generator=generator,
        controlnet_conditioning_scale=[0.5, 0.8],
        control_guidance_end=[0.9, 0.6],
    ).images[0]

    return image


# initial prompt
prompt = "a runner on the sidewalk"
negative_prompt = ""

temp_image = generate_image(prompt, negative_prompt, white_bg_image, image_zoe, 4138619029)

# paste original subject over temporal background
x = (1024 - resized_img.width) // 2
y = (1024 - resized_img.height) // 2
temp_image.paste(resized_img, (x, y), resized_img)

# create a mask for the final outpainting
mask = Image.new("L", temp_image.size)
mask.paste(resized_img.split()[3], (x, y))
mask = ImageOps.invert(mask)
final_mask = mask.point(lambda p: p > 128 and 255)

# clear old pipeline for VRAM savings
pipeline = None
torch.cuda.empty_cache()

# new pipeline with inpaiting model
pipeline = StableDiffusionXLInpaintPipeline.from_pretrained(
    "OzzyGT/RealVisXL_V4.0_inpainting",
    torch_dtype=torch.float16,
    variant="fp16",
    vae=vae,
).to("cuda")

# Use a blurred mask for better blend
mask_blurred = pipeline.mask_processor.blur(final_mask, blur_factor=20)


# function for final outpainting
def generate_outpaint(prompt, negative_prompt, image, mask, seed: int = None):
    if seed is None:
        seed = random.randint(0, 2**32 - 1)

    generator = torch.Generator(device="cpu").manual_seed(seed)

    image = pipeline(
        prompt,
        negative_prompt=negative_prompt,
        image=image,
        mask_image=mask,
        guidance_scale=10.0,
        strength=0.8,
        num_inference_steps=30,
        generator=generator,
    ).images[0]

    return image


# better prompt for final outpainting
prompt = "high quality photo of a runner on the sidewalk, shadows, highly detailed"
negative_prompt = ""

# generate the image
final_image = generate_outpaint(prompt, negative_prompt, temp_image, mask_blurred, 3352253467)

# paste original subject over final background
x = (1024 - resized_img.width) // 2
y = (1024 - resized_img.height) // 2
final_image.paste(resized_img, (x, y), resized_img)
final_image.save("result2.png")

And this is the output:

The input image also has a transparent background like the input image for the car example.

Where am I going wrong?

[asomoza]

Thanks for opening another discussion with this. I will look into this, I'm still deciding if I should try to test and fix this or just update and publish a new guide with the new techniques and also with an example like this one.

Probably what's happening here is that the subject is too small in comparison to what you're generating as a background, so probably the model doesn't know what to do, in your examples also the perspective and the shadows are wrong.

If you can provide an image (isolated subject) I can test it with it or else, I'll have to find one that might not be the same as your problem.

[levoz92]

[levoz92]

Any updates?

[asomoza]

After doing some tests I realized that this is really old, what seemed like a real good method before I think now it's really old and not that good.

Nevertheless, the problem is as I told you, the subject is too small, the model doesn't know how to generate the background without doing something weird to the subject like bigger shoes or the artifacts you mention.

The only way I found to mitigate this with that solution is to increase the zoe depth guidance scale and end:

...
        controlnet_conditioning_scale=[0.5, 1.0],
        control_guidance_end=[0.9, 0.8],
...

Even with this, some of the images in the first pass do get additional details, so it all depends on the luck with your seeds, after I found a good seed for the first one, the second pass gets easier:

I can still see some additional unwanted details but it's a lot better than your example.

I also found that the image is not that good to work with, the man has some black borders which prevents the generation to outpaint the correct background with the lighting and blending.

With the car and more large subjects, I did a small scale up/down to prevent the details from being leaked outwards but with a small subject with details this doesn't work that well.

Just remember, there's a reason why I post this as tutorial guides, these guides are not final products or something to deploy in a production environment, this is just for learning and to teach yourself as a first step for doing these kind of tasks.

Currently, there's a much better inpainting controlnet for SDXL and a really good one for SD3, also people are saying that flux is really amazing for image to image which can help to generate the background. The zoe depth model and controlnet are also outdated, and I found a really better way to generate coherent backgrounds that don't distort the original image that much, so it's easier to paste the original over it.

I will post a more modern guide on how to do this but I have a lot more stuff in my TODO list before it, so don't wait on it and I encourage you to try to experiment.