Skip to content

Better error message when comparing images#7140

Closed
Yay295 wants to merge 3 commits intopython-pillow:mainfrom
Yay295:assert_images_equal_diff
Closed

Better error message when comparing images#7140
Yay295 wants to merge 3 commits intopython-pillow:mainfrom
Yay295:assert_images_equal_diff

Conversation

@Yay295
Copy link
Contributor

@Yay295 Yay295 commented May 7, 2023

When comparing images with assert_image_equal(), assert their bytes instead of asserting False. This gives a better error message:

import pytest

def test_assert_false():
    assert False, "message"

def test_assert_compare():
    assert b"abc" == b"abb", "message"
================ FAILURES =================
____________ test_assert_false ____________

    def test_assert_false():
>       assert False, "message"
E       AssertionError: message
E       assert False

Tests/test_assert.py:4: AssertionError
___________ test_assert_compare ___________

    def test_assert_compare():
>       assert b"abc" == b"abb", "message"
E       AssertionError: message
E       assert b'abc' == b'abb'
E         At index 2 diff: b'c' != b'b'
E         Use -v to get more diff

Tests/test_assert.py:7: AssertionError

However, if the verbosity is too high, it will show a diff of the full image bytes. We can avoid this by temporarily changing the verbosity.

@Yay295 Yay295 mentioned this pull request May 7, 2023
Closed
radarhere
radarhere reviewed May 7, 2023
View reviewed changes
Tests/helper.py
from PIL import Image, ImageMath, features

# overridden in conftest.py
pytestconfig = {"option": {"verbose": 0}}
Copy link
Member

@radarhere radarhere May 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was previously discussed in #6653 (comment)
Screenshot 2023-05-08 at 7 30 46 am

@radarhere
Copy link
Member

radarhere commented May 8, 2023

Before

self = <Tests.test_image.TestImage object at 0x1081b8fa0>

    def test_p_from_rgb_rgba(self):
        for mode, color in [
            ("RGB", "#DDEEFF"),
            ("RGB", (221, 238, 255)),
            ("RGBA", (221, 238, 255, 255)),
        ]:
            im = Image.new("P", (100, 100), color)
            expected = Image.new(mode, (100, 100), color).rotate(45)
>           assert_image_equal(im.convert(mode), expected)

Tests/test_image.py:681: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = <PIL.Image.Image image mode=RGB size=100x100 at 0x10818C0D0>, b = <PIL.Image.Image image mode=RGB size=100x100 at 0x10818C430>, msg = None

    def assert_image_equal(a, b, msg=None):
        assert a.mode == b.mode, msg or f"got mode {repr(a.mode)}, expected {repr(b.mode)}"
        assert a.size == b.size, msg or f"got size {repr(a.size)}, expected {repr(b.size)}"
        if a.tobytes() != b.tobytes():
            if HAS_UPLOADER:
                try:
                    url = test_image_results.upload(a, b)
                    logger.error(f"Url for test images: {url}")
                except Exception:
                    pass
    
>           assert False, msg or "got different content"
E           AssertionError: got different content
E           assert False

Tests/helper.py:98: AssertionError
=============================================================================================================== short test summary info ================================================================================================================
FAILED Tests/test_image.py::TestImage::test_p_from_rgb_rgba - AssertionError: got different content

After

self = <Tests.test_image.TestImage object at 0x103793c70>

    def test_p_from_rgb_rgba(self):
        for mode, color in [
            ("RGB", "#DDEEFF"),
            ("RGB", (221, 238, 255)),
            ("RGBA", (221, 238, 255, 255)),
        ]:
            im = Image.new("P", (100, 100), color)
            expected = Image.new(mode, (100, 100), color).rotate(45)
>           assert_image_equal(im.convert(mode), expected)

Tests/test_image.py:681: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = <PIL.Image.Image image mode=RGB size=100x100 at 0x103997400>, b = <PIL.Image.Image image mode=RGB size=100x100 at 0x103997A00>, msg = None

    def assert_image_equal(a, b, msg=None):
        assert a.mode == b.mode, msg or f"got mode {repr(a.mode)}, expected {repr(b.mode)}"
        assert a.size == b.size, msg or f"got size {repr(a.size)}, expected {repr(b.size)}"
    
        original_verbosity = pytestconfig.option.verbose
        try:
            # set pytest's verbosity to 0 so that it doesn't show the full bytes diff
            pytestconfig.option.verbose = 0
>           assert a.tobytes() == b.tobytes(), msg or "got different content"
E           AssertionError: got different content
E           assert b'\xdd\xee\xf...f\xdd\xee\xff' == b'\x00\x00\x0...0\x00\x00\x00'
E             At index 0 diff: b'\xdd' != b'\x00'
E             Use -v to get more diff

Tests/helper.py:98: AssertionError
=============================================================================================================== short test summary info ================================================================================================================
FAILED Tests/test_image.py::TestImage::test_p_from_rgb_rgba - AssertionError: got different content

This might just be personal preference, but "got different content" grabs my attention more in the before version.

@Yay295
Copy link
Contributor Author

Yay295 commented May 8, 2023

It's still there. This change just also tells you what the different content is.

@radarhere
Copy link
Member

radarhere commented May 11, 2023

Yes, but I think in 99% of cases, the difference in bytes is not going to be meaningful to me as a human - it is highly likely that I'd want to visually compare the images. I'm saying that I think placing 'got different content' more prominently is more helpful.

@hugovk
Copy link
Member

hugovk commented May 12, 2023

Agreed, we don't need to print out bytes to the screen.

If someone does want to see them, they temporarily can add prints to the test.

@radarhere radarhere closed this May 20, 2023
@Yay295 Yay295 deleted the assert_images_equal_diff branch June 25, 2024 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@radarhere radarhere radarhere left review comments

Assignees

No one assigned

Labels

Testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Yay295 @radarhere @hugovk