The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

just a few comments, not a full review:

• there is some overlap with Fix qwen encoder hidden states mask #12655
• this code has the same issue mentioned in Fix qwen encoder hidden states mask #12655 (expecting boolean semantics in a FloatTensor - but float attention masks are interpreted differently)
• Could you clarify what the purpose of this PR is?
  If the purpose is to remove the txt_seq_lens parameters, and infer the sequence lengths from the attention mask: why is it still a parameter of the transformer model?
  If the purpose is towards passing sequence lengths to the attention dispatch (see Qwen Image: txt_seq_lens is redundant and not used #12344 (comment)), the sequence lengths for each batch sample must be inferred from the mask and passed to the transformer blocks, not only the max sequence length across all batch samples for RoPE

thanks @dxqb the idea was to remove txt_seq_lens all together and work with any mask pattern

#12655 provides some benchmarks on the speed, as well. Possible to provide them here too? @kashif

some benchmarks with various backends:

code:
benchmark_backends_qwen.py

@dxqb you can also send a PR to this branch or fix this PR directly

I'll send the small changes in your PR that I needed (like support for text sequence length per sample), but keep the PR overall separate - it's a separate feature

@kashif @dxqb
is this ready for final round of review?

Hi @yiyixuxu and @kashif,

Quick update: this PR passed our internal regression tests with batch_size=1. Heads up though — the EditPlus pipeline still doesn’t support batch_size > 1 yet.

Since this is a large PR with valuable fixes, I suggest we merge it now and handle batch>1 support separately. What are your thoughts?

Below is a minimal reproduction script for the EditPlus pipeline error when batch_size > 1:

import os
import torch
from diffusers import QwenImageEditPlusPipeline
from PIL import Image
torch.manual_seed(0)

pipeline = QwenImageEditPlusPipeline.from_pretrained("Qwen/Qwen-Image-Edit-2511")
print("pipeline loaded")
pipeline.to(torch.bfloat16)
pipeline.to('cuda')
pipeline.set_progress_bar_config(disable=None)

image0 = Image.open("input1.png").convert("RGB")
image1 = Image.open("input2.png").convert("RGB")
prompt = "merge picture 1 with picture 2"
inputs = {
    "image": [[image0, image1], [image0, image1]],
    "prompt": [prompt, prompt],
    "height": 1024,
    "width": 1024,
    "generator": torch.manual_seed(0),
    "true_cfg_scale": 4.0,
    "negative_prompt": " ",
    "num_inference_steps": 50,
    "guidance_scale": 1.0,
    "num_images_per_prompt": 1,
}
with torch.inference_mode():
    output = pipeline(**inputs)
    output_image = output.images[0]
    output_image.save("output_image_edit_plus.png")
    print("image saved at", os.path.abspath("output_image_edit_plus.png"))

Thanks a lot @naykun! WDYT about raising an error from the EditPlus pipeline when batch_size > 1? Would that work for you?

Thanks a lot @naykun! WDYT about raising an error from the EditPlus pipeline when batch_size > 1? Would that work for you?

Sounds good to me! 😊

Uff feels like a big burden off everyone's shoulders! Thanks a lot @cdutr @kashif for jamming. @cdutr please obtain your MVP certificate from here. Let us also know about your HF username, so that we can grant you some credits :)

@sayakpaul @yiyixuxu Have you tested Context Parallel? It seems that this PR has caused an error in Context Parallel.

[rank3]:     output = function_reference.forward(*args, **kwargs)
[rank3]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/dev/vipshop/diffusers/src/diffusers/models/transformers/transformer_qwenimage.py", line 959, in forward
[rank3]:     encoder_hidden_states, hidden_states = block(
[rank3]:                                            ^^^^^^
[rank3]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/dev/vipshop/diffusers/src/diffusers/models/transformers/transformer_qwenimage.py", line 696, in forward
[rank3]:     attn_output = self.attn(
[rank3]:                   ^^^^^^^^^^
[rank3]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/dev/vipshop/diffusers/src/diffusers/models/attention_processor.py", line 605, in forward
[rank3]:     return self.processor(
[rank3]:            ^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/dev/vipshop/diffusers/src/diffusers/models/transformers/transformer_qwenimage.py", line 535, in __call__
[rank3]:     txt_query = apply_rotary_emb_qwen(txt_query, txt_freqs, use_real=False)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/dev/vipshop/diffusers/src/diffusers/models/transformers/transformer_qwenimage.py", line 140, in apply_rotary_emb_qwen
[rank3]:     x_out = torch.view_as_real(x_rotated * freqs_cis).flatten(3)
[rank3]:                                ~~~~~~~~~~^~~~~~~~~~~
[rank3]: RuntimeError: The size of tensor a (109) must match the size of tensor b (27) at non-singleton dimension 1

I have checkout to b86bd99 (the last commit before this pr) and the CP is work.

Was it working before in the CP setup? Cc: @kashif

@sayakpaul @kashif The txt_seq_len is error if context parallel enabled, can we fix it.

Let's please not sound condescending here. I am politely asking if it was working as expected for Qwen before this PR. Could you confirm that?

If you have a fix, then we appreciate any PR otherwise, I kindly ask you to have patience. It will surely be fixed.

@sayakpaul Hi~

1. I'm very sorry. I didn't mean to offend. I think I was just describing the problem I found. I'm trying to solve this problem. If it works, I will submit a PR for the fix. I really like your works, my friends ~

2. I have checkout to b86bd99 (the last commit before this pr) and the CP is work.

@sayakpaul Change the CP plan from transformer-level to block-level seems can avoid sequence length mismatch in rope. I will submit a pr to fix this issue after I have conducted thorough tests.

        _cp_plan = {
            "transformer_blocks.0": {
                "hidden_states": ContextParallelInput(
                    split_dim=1, expected_dims=3, split_output=False
                ),
                "encoder_hidden_states": ContextParallelInput(
                    split_dim=1, expected_dims=3, split_output=False
                ),
            },
            "pos_embed": {
                0: ContextParallelInput(split_dim=0, expected_dims=2, split_output=True),
                1: ContextParallelInput(split_dim=0, expected_dims=2, split_output=True),
            },
            "proj_out": ContextParallelOutput(gather_dim=1, expected_dims=3),
        }

@sayakpaul I have make a draft pr at #12970, I'm sorry again for my straightforward comments. I really didn't mean to offend. I've just been paying more attention to diffusers' CP's ability. As you can see, I have also made a lot of efforts to fix the bugs related to CP. I truly hope that diffusers will keep getting better and better, because I am also a loyal user of it and have benefited a lot from its code.

some benchmarks with various backends:

code: benchmark_backends_qwen.py

Have any of you run into this issue: when running qwen-image-edit-2511 at 1024×1024 resolution on H20, the single-step inference speed dropped from 5.7s in version 0.36.0 to 7.8s? @kashif

import os
import torch
from PIL import Image
from diffusers import QwenImageEditPlusPipeline

pipeline = QwenImageEditPlusPipeline.from_pretrained("/data00/models/Qwen-Image-Edit-2511", torch_dtype=torch.bfloat16)
print("pipeline loaded")

pipeline.to('cuda')
pipeline.set_progress_bar_config(disable=None)
image1 = Image.open("/data00/image_examples/qwen-image-edit-input1.jpg")
image2 = Image.open("/data00/image_examples/qwen-image-edit-input2.jpg")
prompt = "The magician bear is on the left, the alchemist bear is on the right, facing each other in the central park square."
inputs = {
    "image": [image1, image2],
    "prompt": prompt,
    "generator": torch.Generator(device="cuda").manual_seed(0),
    "true_cfg_scale": 4.0,
    "negative_prompt": " ",
    "num_inference_steps": 40,
    "guidance_scale": 1.0,
    "num_images_per_prompt": 1,
    "height": 1024,
    "width": 1024,
}
with torch.inference_mode():
    output = pipeline(**inputs)
    output_image = output.images[0]
    output_image.save("output_image_edit_2511.png")
    print("image saved at", os.path.abspath("output_image_edit_2511.png"))

@GengarWu would you be able to test with the PR #12987 Thanks!