Describe the bug

The results obtained using DPM++ 2M SDE Karras contain artifacts that suggest there is some bug. This happens across different XL models, although is not so visible in the base SDXL. Using Automatic1111 with the same models and the same kind of scheduler results in high-quality results, and it seems impossible to achieve the same quality using the diffusers equivalent.

An image generated using Automatic1111 DPM++ 2M SDE Karras with the Juggernaut XL v6 model

An image generated using the equivalent scheduler from diffusers:

The artifacts are very visible in the last image.

The following didn't help:

• Loading the model in fp32
• use_karras_sigmas=False
• use_karras_sigmas=False and dpmsolver++
• euler_at_final=True and use_lu_lambdas=True

I see this happening clearly at least with the following models:

• frankjoshua/juggernautXL_version6Rundiffusion
• Lykon/dreamshaper-xl-1-0

Reproduction

I've created a github repo to help reproducing the problem. Also, a simpler code snippet is provided below:

from diffusers import AutoPipelineForText2Image
from diffusers.schedulers import DPMSolverMultistepScheduler
import torch

model_id = "frankjoshua/juggernautXL_version6Rundiffusion"
pipeline = AutoPipelineForText2Image.from_pretrained(
    model_id, 
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
).to("cuda")

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
        pipeline.scheduler.config, 
        use_karras_sigmas=True,
        sde_type="sde-dpmsolver++",
        euler_at_final=True,
        use_lu_lambdas=True
        )

prompt = "Adorable infant playing with a variety of colorful rattle toys."

results = pipeline(
    prompt=prompt, 
    guidance_scale=3,
    generator=torch.Generator(device="cuda").manual_seed(42),  
    num_inference_steps=25, 
    height=768, 
    width=1344)
display(results.images[0])

Logs

No response

System Info

• diffusers version: 0.24.0
• Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.31
• Python version: 3.10.13
• PyTorch version (GPU?): 2.1.1 (True)
• Huggingface_hub version: 0.20.1
• Transformers version: 4.36.2
• Accelerate version: 0.25.0
• xFormers version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@yiyixuxu @patrickvonplaten