Diffusers Image Fill guide & space

[asomoza]

Hi, I made a guide about how to use diffusers for erasing parts of an image or fill it with something new. You can read it here.

I also created a space so you can play with it and test the results here.

You can post your results here of if you have any suggestion or comment about it, please let me know.

[peki12345]

Hi, I tried outpainting and found the results to be quite good. I want to further improve it by incorporating Differential Diffusion, as demonstrated in your blog: https://huggingface.co/blog/OzzyGT/outpainting-differential-diffusion. I'm aiming to make ControlNet and Differential Diffusion work together, but I haven't succeeded yet. Have you tried this?

[asomoza]

yes, they're not mutually exclusive so it can be done, so far I haven't encountered a single use case where differential diffusion can't be used.

I don't have a simple pipeline with just controlnet and diff-diff, but I plan to release an upscaler that has them soon.

[peki12345]

yes, they're not mutually exclusive so it can be done, so far I haven't encountered a single use case where differential diffusion can't be used.

I don't have a simple pipeline with just controlnet and diff-diff, but I plan to release an upscaler that has them soon.

Looking forward to it! I seem to have found my problem, I provided a black image of the outpaint area for the original_image of diff-diff, so the result I obtained is mainly in black. Actually, I feel that the ’gray wall‘ filled in your blog(https://huggingface.co/blog/OzzyGT/outpainting-differential-diffusion) before using XComposter2 are also due to the introduction of the same color information in your squared image. I tried to pre-fill some simple information in the outpaint area of original_image and get the correct result, but I don't know if my idea is correct or what information is better to pre-fill. Do you have any suggestions?

[asomoza]

yeah, I demonstrated the gray wall as that effect precisely, if you don't prefill the image, it will take into account whatever was before it to generate the new image, in this case since there's nothing, it will be black which is very bad.

I haven't played anymore with the outpainting after this, but the idea I have in mind is to use the current one as a prefill, so I essentially was going to use it instead of the prefill, color information and Ip Adapter, and after the first pass, do a second pass with diff-diff to remove the seams that you currently see.

Of course this will be a more "professional" grade solution but also it will be slower because what I wanted to do is to do a full second pass without the lighting model to get in all the tiny details that the current one doesn't do and to match perfectly the new outpainted area with the old one.

To do something in the middle, like a fast outpaint without the seams I have a couple of ideas but I need to test them, basically is to play with the steps (beginning) of diff-diff and to also to dynamically change the strength.