Skip to content

fix extra batch mode P Transparency#15664

Merged
AUTOMATIC1111 merged 1 commit intodevfrom
fix-extra-batch-mode-P-Transparency
Jun 8, 2024
Merged

fix extra batch mode P Transparency#15664
AUTOMATIC1111 merged 1 commit intodevfrom
fix-extra-batch-mode-P-Transparency

Conversation

@w-e-w
Copy link
Copy Markdown
Collaborator

@w-e-w w-e-w commented Apr 30, 2024 •
edited
Loading

Description

fix

when enable_pnginfo using extra "batch" the existing_pnginfo will also be restored

note: issue is only in Batch process not single
I believe gradio image component already does a conversion once as a result this issue does not occur when using Single image

when converting a none RGB mode image to RGB some existing_pnginfo keys will also have to be converted to make it work
this is the case with a P mode image with Transparency key

the issue is we currently extract the existing_pnginfo "before" we convert the image into RGB mode

as suche the existing_pnginfo we restore later will have the old unconverted existing_pnginfo from P mode

in the case of P -> to RGB if Transparency : 0 key exist, it will have to be converted in to a 3 value tuple (0, 0, 0)
this is why some users are seeing error

*** Arguments: ('task(io7z85oegmsjbmy)', 1.0, <PIL.Image.Image image mode=RGBA size=119x152 at 0x1EAA924E8F0>, [<tempfile._TemporaryFileWrapper object at 0x000001EAA924E350>], '', '', True, True, 0.0, 2, 0.0, 512, 512, True, 'R-ESRGAN 4x+ Anime6B', 'None', 0, False, 1, False, 1, 0, False, 0.5, 0.2, False, 0.9
, 0.15, 0.5, False, False, 384, 768, 4096, 409600, 'Maximize area', 0.1, False, ['Horizontal'], False, ['Deepbooru']) {}
    Traceback (most recent call last):
      File "B:\GitHub\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "B:\GitHub\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "B:\GitHub\stable-diffusion-webui\modules\postprocessing.py", line 131, in run_postprocessing_webui
        return run_postprocessing(*args, **kwargs)
      File "B:\GitHub\stable-diffusion-webui\modules\postprocessing.py", line 96, in run_postprocessing
        fullfn, _ = images.save_image(pp.image, path=outpath, basename=basename, extension=opts.samples_format, info=infotext, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=existing_pnginfo, forced_filename=forced_filename, suffix=suffix)
      File "B:\GitHub\stable-diffusion-webui\modules\images.py", line 728, in save_image
        _atomically_save_image(image, fullfn_without_extension, extension)
      File "B:\GitHub\stable-diffusion-webui\modules\images.py", line 712, in _atomically_save_image
        save_image_with_geninfo(image_to_save, info, temp_file_path, extension, existing_pnginfo=params.pnginfo, pnginfo_section_name=pnginfo_section_name)
      File "B:\GitHub\stable-diffusion-webui\modules\images.py", line 584, in save_image_with_geninfo
        image.save(filename, format=image_format, quality=opts.jpeg_quality, pnginfo=pnginfo_data)
      File "B:\GitHub\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 2432, in save
        save_handler(self, fp, filename)
      File "B:\GitHub\stable-diffusion-webui\venv\lib\site-packages\PIL\PngImagePlugin.py", line 1364, in _save
        red, green, blue = transparency   #  <---- the unconverted transparency key from P mode is just 0, the not the expected (0, 0, 0)
    TypeError: cannot unpack non-iterable int object

the easiest solution I found is to convert the image to the to our desired mode before parseing the info with read_info_from_image()

I have tested with the 4 formats of images generated by webui (png jpg webp avif) the info (the parts we wish to preserve) read after conversion seems to be intact

test images
test images.zip
test from #15421 #6534
2 p mode image with and without transparency key

before the PR the image with transparency key should faile to save using extra batch
should be fixed after the PR

note
I'm not an expert on image formats it's it is possible this modification can potentially messed up other things

Checklist:

@w-e-w
fix extra batch mode P Transparency
9d39380 
red, green, blue = transparency TypeError: cannot unpack non-iterable int object
@w-e-w w-e-w requested a review from AUTOMATIC1111 as a code owner April 30, 2024 10:34
@w-e-w w-e-w mentioned this pull request Apr 30, 2024
Open
1 task
Rabcor
Rabcor approved these changes Apr 30, 2024
View reviewed changes
Copy link
Copy Markdown

@Rabcor Rabcor left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked for my sample files in #6534 (comment)

Not seeing any detrimental effects at all. It does change some behavior between batch and single image processing though.

Namely that batches now try to preserve alpha while single image processing does not, this is in my opinion an improvement though and maybe it'd be worth it to figure out a way to get transparency working on single image processing as well.

Original:
tx_de_lutestrings_01
Single Image Upscale:
00017
Batch Upscale:
tx_de_lutestrings_01

@w-e-w
Copy link
Copy Markdown
Collaborator Author

w-e-w commented Apr 30, 2024

about the single image
I know the cause and I have a way to fix it, but this method may brakes some extensions
if you're interested you can see https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/more-extra-transparency-fix-again

basically the cause of the losing of transparency for P mode images is due the the gradio convertion the input image into RGBA
so when it gets to our script is already a RGBA image, the
when a P image gets converted into RGBA the transparency key in removed as it is encoded into Alpha channel
up to this point is fine

next depending on the process module you use such as upscale
upscale is for RGB only so the output of upscale is RGB not RGBA
what has happened the transparency information is lost

it workd for Batch because we are the ones doing the converting from P to RGB, and since we saved the tansparency key befor converting and restore it after it's finished
transparency information is preserved

in this branch https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/more-extra-transparency-fix-again what I did is disable Gradios auto converting of image to RGBA, by disalbeing preprocess, this way it retruns the raw image in base64 string format, doing so loud us more control on the converting process and allowing the preservation of the transparency key

the issue is as far as I'm aware disalbeing preprocess applies for every component for that submit,
if there is currently any other extensions or modules that requires preprocessing (like if that moduls has a second image input)
they will also receive base64 as opposed to the image they're expecting, stuff wll break

so if there's a way of preserving the transparency key after it's after it has been converted to RGBA then things would work

another method is to set

stable-diffusion-webui/modules/ui_postprocessing.py

Line 15 in 1c0a0c4

extras_image = gr.Image(label="Source", source="upload", interactive=True, type="pil", elem_id="extras_image", image_mode="RGBA")

from RGBA bask to RGB
the change to RGB was done in

yeah it's kind of ironic making RGBA possible for some modules actually breaks transparency for other image mode

@Rabcor
Copy link
Copy Markdown

Rabcor commented May 1, 2024

yeah it's kind of ironic making RGBA possible for some modules actually breaks transparency for other image mode

Actually, here's an extra sample file to test this.
655dc9d16b55c103ad53e26f

And here's the old sample file I am still using here as well:
tx_de_lutestrings_01

Unmodified 1.9.3 baseline

With this, we have established that this commit: #15334

Is only enabling transparency for the reactor extension, not upscaling.

Details

Upscale
00045
00046
Reactor
00047

#15664 applied to 1.9.3

Identical to baseline.

Details

Upscale
00045
00046
Reactor
00047

#15334 undone for 1.9.3

Interestingly undoing the commit made it preserve transparency on one sample file but not the other when upscaling.

Details

Upscale
0004900046

Reactor
00050

#15664 applied and #15334 undone for 1.9.3

Identical to above.

Details

Upscale
0004900046

Reactor
00050

master...more-extra-transparency-fix-again changes applied to 1.9.3

One of the sample files still isn't transparent. Reactor successfully preserves transparency but fails to apply any changes to the face.

Details

Upscale
0004900046

Reactor
00057

@Rabcor Rabcor mentioned this pull request May 1, 2024
Merged
4 tasks
@w-e-w
Copy link
Copy Markdown
Collaborator Author

w-e-w commented May 1, 2024 •
edited
Loading

no we have not established that

what that PR dose it's enabling possibility of "Proper" RGBA support for single image input for ANY module as long as the module "itself" support transparency

upscaller depending on the upscaler method you use, and most dose not support transparency (Alpha Channel)

the reason you were able to get desirable transparency results is mostly a combination of luck

your image is a P mode image
https://pillow.readthedocs.io/en/stable/handbook/concepts.html#modes

P mode: 8-bit pixels, mapped to any other mode using a color palette

which uses a different method of encoding transparency then RGBA
as far as I'm weird image is essentially a grayscale image with a color palette information encoded alongside
including the transparency of certain color palettes

the lucky part is if the color palettes transparency is save and re-apply back to the image you would get the transparency back
as long as the processing module you're using does not shift the color

you can test it with different upscaling methods the more a particular message shifts the color the more likely that the transparency will be ruined

some of the test image you sent, image is basically composed of non don't transparent color parts
and transparent background

img1 img2 img3
p with transparency key 326719922-b986075a-ef14-4e11-bc7b-0caddd42a36e p no transparency key

test of img1 with different upscaling methods

Lanczos R-ESRGAN 4x+ Anime6B R-ESRGAN 4x+ 16x-ESRGAN
Lanczos R-ESRGAN 4x+ Anime6B R-ESRGAN 4x+ 16x-ESRGAN
transparency preserved with artifacts image is complete gone leaving only transparent background because of color shift you got blocks of black transparency is completely gone

if you are "lucky" and the color of the output image has the same color pallette as the input, it we then restore the info back transparency gets restored
if you're unlucky then you might have results ranging from getting artifacts to completely ruined

the issue is not with that PR it's issue with lots of modules does not support transparency

in fact after taht PR it has better transparency support
as long as all the modules you use in the processing pipeline support transparency

the reason that PR accidentally break transparency support for certain images you are using
is because

the issue with gradio preprocess auto converting image into RGBA,
during during conversion the the partial transparency metadata you have in your P mode image
just converted to the more widely accepted and fully featured Alpha Channel
this conversion is done before we have a chance to preserve the origin transparency metadata

and when later down the pipeling after going through upstairs that does not support transparency (Alpha) is lost
and so when saving the image the transparency is lost

so the way I found to "fix" that is to take converting image into our own hands and not handled gradio
but the orthodox method of doing so requires disableing preporcessing for the entire call
the issues if we do so it has a potential of breaking extensions

what I would consider doing would be to patch gradio so that we have more contorl
but I'm hesitant to do this now because I'm not sure when will we be upgrading to gradio4.x,
because if we do so stuff like this sort of pathces will probably break

but the thing is I'm not entirely sure if doing all this is worth it as transparency is at best support finicky
depending on your image contents and the upscaling method you're using

  • the intent of this PR fix extra batch mode P Transparency #15664
    is not providing transparency support, it is to fix P mode image with transparency information throwing TypeError: cannot unpack non-iterable int object and fail to save the image

personally I think it might be a good idea to just remove the transparency altogether because it could potentially ruin the image
but I figure it might be a good idea to still preserve it on off chance that if it works for certain images

@Rabcor
Copy link
Copy Markdown

Rabcor commented May 1, 2024 •
edited
Loading

Yeah, including transparency is never going to be harmful, it's pretty hard to make an opaque image transparent, but the reverse is extremely easy, so it's never a bad thing.

But yeah, this PR solves the error it set out to solve perfectly.

@AUTOMATIC1111 AUTOMATIC1111 merged commit a1130c2 into dev Jun 8, 2024
@AUTOMATIC1111 AUTOMATIC1111 deleted the fix-extra-batch-mode-P-Transparency branch June 8, 2024 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AUTOMATIC1111 AUTOMATIC1111 AUTOMATIC1111 approved these changes

+1 more reviewer

@Rabcor Rabcor Rabcor approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@w-e-w @Rabcor @AUTOMATIC1111