[SD-XL] Add inpainting by patrickvonplaten · Pull Request #4098 · huggingface/diffusers

patrickvonplaten · 2023-07-14T11:44:38Z

SD-XL inpainting

This PR solves: #4080 and is ready for a review.

Inpainting works well for both the vanilla case and the "Ensemble of Expert Denoisers case".

You can try the following to see for yourself:

Vanilla inpainting:

import torch
from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
)
pipe.to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A red cat sitting on a bench"
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=0.80).images[0]

Ensemble of Expert of denoisers

which should give slightly better quality:

from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
)
pipe.to("cuda")

refiner = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-0.9",
    text_encoder_2=pipe.text_encoder_2,
    vae=pipe.vae,
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)
refiner.to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A red cat sitting on a bench"
num_inference_steps = 75
high_noise_frac = 0.7

image = pipe(
    prompt=prompt,
    image=init_image,
    mask_image=mask_image,
    num_inference_steps=num_inference_steps,
    strength=0.80,
    denoising_start=high_noise_frac,
    output_type="latent",
).images
image = refiner(
    prompt=prompt,
    image=image,
    mask_image=mask_image,
    num_inference_steps=num_inference_steps,
    denoising_start=high_noise_frac,
).images[0]

HuggingFaceDocBuilderDev · 2023-07-14T11:53:12Z

The documentation is not available anymore as the PR was closed or merged.

…add_inpaint_sd_xl

gkorepanov · 2023-07-14T13:05:25Z

Hi, is there already a SD XL checkpoint with unet having 9 channels? Seems like no specific inpainting model was released for SD XL, but without it inpainting results are meaningless (I mean there is little to no semantic match between inpainted regions and present regions in generated images)

AmericanPresidentJimmyCarter · 2023-07-14T13:07:27Z

Is this the same method as InpaintLegacy in SD? There are now 3 inpainting methods for LDM, the "InpaintLegacy" method, the model using extra channels from RunwayML, and PSLD. I think we should maintain consistent naming.

AmericanPresidentJimmyCarter · 2023-07-14T13:10:02Z

And yes, I agree with @gkorepanov , the "InpaintLegacy" method is more or less useless.

patrickvonplaten · 2023-07-14T14:52:28Z

StableDiffusionInpaintPipelineLegacy is deprecated and will be removed. Everything you were able to do with StableDiffusionInpaintPipelineLegacy you can now do with StableDiffusionInpaintPipeline

In that sense there will only be one "true" StableDiffusionXLInpaintPipeline

adhikjoshi · 2023-07-14T14:55:20Z

As inpaiting checkpoint isn't there, does it affects quality in general?

patrickvonplaten · 2023-07-14T14:57:21Z

As inpaiting checkpoint isn't there, does it affects quality in general?

Works pretty well for me for now, I recommend making sure to pass strength=0.7 or strength=0.8.

I think the checkpoint will however have problems when you want to replace the masked area with something very different to what was there before.

nhnt11 · 2023-07-14T14:57:30Z

+from diffusers.utils import load_image
+
+pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
+    "stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, variant="fp16", use_safetensors=True


Is it really intended to use the refiner model for general img2img? I've been trying to understand this - I've also seen it here for example - but I think I am missing something. My understanding is that the refiner model is intended as a kind of de-noising and/or fidelity-increasing step and it isn't good at generating the kind of baseline content of the image. If that's correct, feels like it'd perform poorly for img2img with lower strength values.

You can use both! The refiner might be better suited for images that look already like the prompt which is the case here. We should maybe improve the docs after the official release.

Gotcha - thanks for clarifying!

patrickvonplaten · 2023-07-14T14:58:45Z

            generator,
            do_classifier_free_guidance,
        )
-        init_image = init_image.to(device=device, dtype=masked_image_latents.dtype)


This is never used actually and was a copy-paste bug I think

* Add more * more * up * Get ensemble of expert denoisers working * Fix code * add tests * up

patrickvonplaten added 2 commits July 14, 2023 13:31

Add more

00babb9

more

456256a

patrickvonplaten mentioned this pull request Jul 14, 2023

Add in-painting pipeline for Stable Diffusion XL #4080

Closed

patrickvonplaten added 2 commits July 14, 2023 12:51

up

b1830ab

Merge branch 'main' of https://github.com/huggingface/diffusers into …

61343f4

…add_inpaint_sd_xl

patrickvonplaten added 4 commits July 14, 2023 13:49

Get ensemble of expert denoisers working

45cc8bd

Fix code

5924554

add tests

945e99b

up

b377267

nhnt11 reviewed Jul 14, 2023

View reviewed changes

patrickvonplaten commented Jul 14, 2023

View reviewed changes

patrickvonplaten merged commit b024ebb into main Jul 14, 2023

darhsu mentioned this pull request Jul 21, 2023

Stable Diffusion XL Inpaint not working with refiner #4186

Closed

orpatashnik pushed a commit to orpatashnik/diffusers that referenced this pull request Aug 1, 2023

[SD-XL] Add inpainting (huggingface#4098)

faa020e

* Add more * more * up * Get ensemble of expert denoisers working * Fix code * add tests * up

orpatashnik pushed a commit to orpatashnik/diffusers that referenced this pull request Aug 1, 2023

[SD-XL] Add inpainting (huggingface#4098)

df40f97

* Add more * more * up * Get ensemble of expert denoisers working * Fix code * add tests * up

orpatashnik pushed a commit to orpatashnik/diffusers that referenced this pull request Aug 1, 2023

[SD-XL] Add inpainting (huggingface#4098)

83f1496

* Add more * more * up * Get ensemble of expert denoisers working * Fix code * add tests * up

kashif deleted the add_inpaint_sd_xl branch September 11, 2023 19:07

yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023

[SD-XL] Add inpainting (huggingface#4098)

262f34e

* Add more * more * up * Get ensemble of expert denoisers working * Fix code * add tests * up

AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024

[SD-XL] Add inpainting (huggingface#4098)

3fb04b9

* Add more * more * up * Get ensemble of expert denoisers working * Fix code * add tests * up

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SD-XL] Add inpainting#4098

[SD-XL] Add inpainting#4098
patrickvonplaten merged 8 commits intomainfrom
add_inpaint_sd_xl

patrickvonplaten commented Jul 14, 2023 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Jul 14, 2023 •

edited

Loading

Uh oh!

gkorepanov commented Jul 14, 2023

Uh oh!

AmericanPresidentJimmyCarter commented Jul 14, 2023

Uh oh!

AmericanPresidentJimmyCarter commented Jul 14, 2023

Uh oh!

patrickvonplaten commented Jul 14, 2023

Uh oh!

adhikjoshi commented Jul 14, 2023

Uh oh!

patrickvonplaten commented Jul 14, 2023

Uh oh!

nhnt11 Jul 14, 2023

Uh oh!

patrickvonplaten Jul 14, 2023

Uh oh!

nhnt11 Jul 14, 2023

Uh oh!

patrickvonplaten Jul 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

patrickvonplaten commented Jul 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SD-XL inpainting

Vanilla inpainting:

Ensemble of Expert of denoisers

Uh oh!

HuggingFaceDocBuilderDev commented Jul 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gkorepanov commented Jul 14, 2023

Uh oh!

AmericanPresidentJimmyCarter commented Jul 14, 2023

Uh oh!

AmericanPresidentJimmyCarter commented Jul 14, 2023

Uh oh!

patrickvonplaten commented Jul 14, 2023

Uh oh!

adhikjoshi commented Jul 14, 2023

Uh oh!

patrickvonplaten commented Jul 14, 2023

Uh oh!

nhnt11 Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

nhnt11 Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

patrickvonplaten commented Jul 14, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jul 14, 2023 •

edited

Loading