Skip to content

Support UniDiffuser model and pipeline#5487

Merged
JunnYu merged 35 commits intoPaddlePaddle:developfrom
nemonameless:add_unidiffuser
Apr 24, 2023
Merged

Support UniDiffuser model and pipeline#5487
JunnYu merged 35 commits intoPaddlePaddle:developfrom
nemonameless:add_unidiffuser

Conversation

@nemonameless
Copy link
Copy Markdown
Contributor

@nemonameless nemonameless commented Mar 31, 2023 •
edited
Loading

PR types

New features

PR changes

Models

Description

UniDiffuser (One Transformer Fits All Distributions in Multi-Modal Diffusion)

Inference

Task Paddle Memory PyTorch Memory
Text-to-image Generation 7.01 GB 8.65 GB
Image-to-text Generation 5.88 GB 7.62 GB
Unconditional Joint Generation 7.10 GB 8.66 GB
Unconditional Image Generation 7.02 GB 8.65 GB
Unconditional Text Generation 5.52 GB 7.49 GB
Image Variation 7.02 GB 8.40 GB
Text Variation 5.53 GB 7.49 GB
  • (1) Text-to-image Generation
CUDA_VISIBLE_DEVICES=0 python examples/inference/text_to_image_generation-unidiffuser.py

image

  • (2) Image-to-text Generation
CUDA_VISIBLE_DEVICES=0 python examples/inference/image_to_text_generation-unidiffuser.py

An image of an astronaut floating in space

  • (3) Unconditional Joint Generation
CUDA_VISIBLE_DEVICES=0 python examples/inference/unconditional_image_text_joint_generation-unidiffuser.py

image

A living room and dining area with a chandelier

  • (4) Unconditional Image Generation
CUDA_VISIBLE_DEVICES=0 python examples/inference/unconditional_image_generation-unidiffuser.py

image

  • (5) Unconditional Text Generation
CUDA_VISIBLE_DEVICES=0 python examples/inference/unconditional_text_generation-unidiffuser.py

A man standing in front of a bunch of a waterfall

  • (6) Image Variation
CUDA_VISIBLE_DEVICES=0 python examples/inference/image_variation-unidiffuser.py

image

  • (7) Text Variation
CUDA_VISIBLE_DEVICES=0 python examples/inference/text_variation-unidiffuser.py

An elephant swimming in the water in the ocean

@paddle-bot
Copy link
Copy Markdown

paddle-bot Bot commented Mar 31, 2023

Thanks for your contribution!

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 31, 2023 •
edited
Loading

Codecov Report

Merging #5487 (02307a2) into develop (feb7c05) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #5487   +/-   ##
========================================
  Coverage    61.70%   61.70%           
========================================
  Files          487      487           
  Lines        68287    68287           
========================================
  Hits         42135    42135           
  Misses       26152    26152

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@guoshengCS guoshengCS requested review from JunnYu and guoshengCS March 31, 2023 11:05
@nemonameless nemonameless changed the title UniDiffuser initial version [WIP] UniDiffuser initial version Apr 3, 2023
JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/models/caption_decoder.py
JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/models/uvit.py Outdated
JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/models/uvit.py
return pos_emb


class Attention(nn.Layer):
Copy link
Copy Markdown
Member

@JunnYu JunnYu Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这应该也可以复用，

PaddleNLP/ppdiffusers/ppdiffusers/models/cross_attention.py

Line 27 in 706e5d6

class CrossAttention(nn.Layer):
只是权重命名和shape发生变化了

JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/models/uvit.py
return x


class Block(nn.Layer):
Copy link
Copy Markdown
Member

@JunnYu JunnYu Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PaddleNLP/ppdiffusers/ppdiffusers/models/attention.py

Line 164 in 706e5d6

class BasicTransformerBlock(nn.Layer):
这个加个 DropPath 和 skip_linear 估计就能用了

JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/models/uvit.py Outdated
JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/models/uvit.py
sample_text: paddle.Tensor


class UViTModel(ModelMixin, ConfigMixin):
Copy link
Copy Markdown
Member

@JunnYu JunnYu Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里应该跟

PaddleNLP/ppdiffusers/ppdiffusers/models/transformer_2d.py

Line 42 in 706e5d6

class Transformer2DModel(ModelMixin, ConfigMixin):
这个很像，不过这里面没有区分，in_blocks，mid_block，out_bloks，但是里面的模型结构都是 BasicTransformerBlock

JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/examples/inference/unidiffuser.py Outdated
JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/pipelines/unidiffuser/pipeline_unidiffuser_i.py Outdated
JunnYu
JunnYu reviewed Apr 7, 2023
View reviewed changes
Comment thread ppdiffusers/ppdiffusers/pipelines/unidiffuser/pipeline_unidiffuser_i2t.py Outdated
@nemonameless nemonameless mentioned this pull request Apr 7, 2023
Merged
8 tasks
@JunnYu
Copy link
Copy Markdown
Member

JunnYu commented Apr 12, 2023

thu-ml/unidiffuser/text_encoder 和 thu-ml/unidiffuser/image_encoder 权重应该多传了，text里面多了vision的权重，vision里面多了text的权重，下载的时候可能多下载好多

@JunnYu
Copy link
Copy Markdown
Member

JunnYu commented Apr 12, 2023

  • unidiffuser-t.txt
    A WORRIED MACHINE SOUTH LAKE IN THE RAIN
  • unidiffuser-i2t.txt
    An image of an astronaut flying in space
  • unidiffuser-joint.txt
    A man embraces with rubble in the direction strike by Israeli air strikes during the Gaza Strip
  • unidiffuser-t2i2t.txt
    Owns Owns Own Own Own Own Own Own Own Own Own Own Own Own Own Own Own Own Own Own Own Owns Own Own Owns Own Owns Own Owns Own Owns Owns Owns Owns Owns Owns Owns Owns Owns Owns Owns Owns Owns Owns Owns
  • unidiffuser-i2t2i.png
    image
  • unidiffuser-t2i.png
    image
  • unidiffuser-i.png
    image
  • unidiffuser-joint.png
    image

JunnYu
JunnYu reviewed Apr 12, 2023
View reviewed changes
Comment thread ppdiffusers/examples/inference/unidiffuser.py Outdated
JunnYu
JunnYu reviewed Apr 12, 2023
View reviewed changes
Comment thread ppdiffusers/requirements.txt Outdated
@nemonameless nemonameless changed the title [WIP] UniDiffuser initial version Support UniDiffuser model and pipeline Apr 17, 2023
@JunnYu
Copy link
Copy Markdown
Member

JunnYu commented Apr 17, 2023 •
edited
Loading

（1）image_to_text_generation-unidiffuser.py
astronaut in space in space spaceship spaceship space background
（2）image_variation-unidiffuser.py
image
（3） text_to_image_generation-unidiffuser.py
image
（4） text_variation-unidiffuser.py
An Elephant Underwater and Caring for Its World
（5）unconditional_image_generation-unidiffuser.py
image
（6）unconditional_image_text_joint_generation-unidiffuser.py
Used BMW X6 M Competition 30i
（7）unconditional_text_generation-unidiffuser.py
Outdoor Lighting For Unique Carpeted Garden Garden Design Ideas Carpeted Lights

JunnYu
JunnYu approved these changes Apr 21, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@JunnYu JunnYu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JunnYu JunnYu merged commit e70e9d8 into PaddlePaddle:develop Apr 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JunnYu JunnYu JunnYu approved these changes

@guoshengCS guoshengCS Awaiting requested review from guoshengCS

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nemonameless @JunnYu