Skip to content

Merge test data + make tests directory which contain tests#1807

Merged
j9ac9k merged 15 commits intopyqtgraph:masterfrom
j9ac9k:merge-test-data
Jun 5, 2021
Merged

Merge test data + make tests directory which contain tests#1807
j9ac9k merged 15 commits intopyqtgraph:masterfrom
j9ac9k:merge-test-data

Conversation

@j9ac9k
Copy link
Copy Markdown
Member

@j9ac9k j9ac9k commented May 28, 2021

This PR aims to get pyqtgraph in-line with standard python package directory structures, by creating a tests directory on the root level, and putting all tests in there, including the images from test-data repository. The best example of this is in the "Good Integration Practices" on the pytest documentation page.

Instead of just copying the images, I generated new images with Qt6, which the test suite passed with out of the box for the most part, even after resetting the pxCount to 0 for Qt5 and Qt6. On windows, some of the ROI tests had a few pixels off, but setting pxCount to 2 there seemed to address the issue (which is much smaller than the 10 it was before for Qt5).

With setup.py we can ensure that test data is not bundled with the pip package, and we can get more reliable use of conftest.py files (which we make minimal use of presently).

While doing the huge shuffle of the test suite, a number of issues presented themselves (including a call to QApplication.instance().deleteLater()). As I addressed them; I realized that the test suite no longer emits warnings; so I addressed #1025. Turns out pytest runs into an unraisable exception with how we call our examples, which this was addressed.

I took the opportunity to fix some of the issues the static code checker had flagged (non-secure temporary files), and I modified debug.printExc so it doesn't just print an exception (while suppressing it), but it actually raises a Runtime Warning. This is to prevent the case of #1804 where the exception was swallowed by this mechanism, despite there being a test to catch that very thing.

Unfortunately, on my local machine, I'm triggering segfaults with test_ImageItem.py, test_NonUniformImage.py and test_InfiniteLine.py. Until these segfaults are resolved, I'm leaving this PR in draft form.

Fixes #1025
Fixes #1064

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 28, 2021

For the segfaults, initially I was suspicious of qWaitForWindowExposed vs. qWaitForWindowActive but after changing those, while the segfaults appeared to happen less regularly; they still do occur.

Furthermore, they do not happen when running any single test by itself.

tests/graphicsItems/test_ArrowItem.py .                                                                                                                                                                  [ 74%]
tests/graphicsItems/test_AxisItem.py ......                                                                                                                                                              [ 78%]
tests/graphicsItems/test_ErrorBarItem.py .                                                                                                                                                               [ 79%]
tests/graphicsItems/test_GraphicsItem.py ..                                                                                                                                                              [ 80%]
tests/graphicsItems/test_ImageItem.py sszsh: segmentation fault  pytest -x -k "not examples"

~/Developer/pyqtgraph merge-test-data* 10s
pyqtgraph-pyside2_515-py38 ❯ pytest -k "test_ImageItem"
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.8.6, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/ogi/Developer/pyqtgraph, configfile: pytest.ini
plugins: cov-2.11.1, xdist-2.2.0, forked-1.3.0
collected 247 items / 239 deselected / 8 selected

tests/graphicsItems/test_ImageItem.py ss...                                                                                                                                                              [ 62%]
tests/graphicsItems/test_ImageItemFormat.py ...                                                                                                                                                          [100%]

================================================================================= 6 passed, 2 skipped, 239 deselected in 2.88s =================================================================================

~/Developer/pyqtgraph merge-test-data*
pyqtgraph-pyside2_515-py38 ❯

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 28, 2021

found a bunch more tests, migrated them over as well, ...segfaults occur in

  • test_ImageItem.py
  • test_ViewBox.py
  • test_InfiniteLine.py
  • test_NonUniformImage.py

this command runs without any errors! 😆

pytest tests -k "not test_ImageItem and not test_InfiniteLine and not test_NonUniformImage and not test_ViewBox"

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 28, 2021

now that the CI is looking at the tests directory, the segfault shows up there too, so at least we have that going for us...

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 28, 2021

I can no longer reproduce a segfault locally, but two segfaults in the CI is enough to make me suspicious.

@j9ac9k j9ac9k marked this pull request as ready for review May 28, 2021 07:52
@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 28, 2021

Last two runs have shown no segfaults; will rerun a few more times to build some confidence in the results.

@j9ac9k j9ac9k force-pushed the merge-test-data branch 4 times, most recently from b8e625a to 1a04650 Compare May 29, 2021 05:59
@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 29, 2021

Tests are migrated over, static code quality checker is fine, "silenced" exceptions have been addressed. I think the next step is to regenerate images with Qt5 (or Qt6), and tighten the tolerance for assertImageApproved for Qt5 and Qt6.

@NilsNemitz
Copy link
Copy Markdown
Contributor

NilsNemitz commented May 29, 2021

Dear @j9ac9k , thanks for the massive overhaul.
With the revised tests, you might want to check if
#1025 and #1064 can be closed!

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 29, 2021

Good news regarding the pixel tolerance, it does not appear to be needed when the images are generated using Qt6 bindings. I've only done the PlotCurveItem images, I'm going to do the other images shortly (I'm setup to do one at a time, so it's going to take a bit!).

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 29, 2021

ok, I made CI angry; ...will investigate more later. With the new images, tox passed on macOS across the board with the new images

EDIT: I can replicate locally; must be something I just did....will investigate more later tonight

@j9ac9k j9ac9k force-pushed the merge-test-data branch from f5598e0 to de791fa Compare May 30, 2021 03:02
@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

passes on macOS, segafaults on linux, crashes on windows, "works on my machine!"

EDIT: looks like it didn't like my changes to image_testing.py thank you sublime merge for letting me easily restore the change to that one file in the one commit (and leaving the rest of the commit untouched).

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

image

So this is the hold up on windows, there appears to be 1 blue and 1 yellow pixel that is the difference between what was shown, and what was expected. Will tinker with scaling settings....

@pijyoi
Copy link
Copy Markdown
Contributor

pijyoi commented May 30, 2021

If you want to refactor into the function getImageFromWidget(), then imageToArray() needs to be called with copy=True, or manually make a copy of the resultant ndarray.

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

If you want to refactor into the function getImageFromWidget(), then imageToArray() needs to be called with copy=True, or manually make a copy of the resultant ndarray.

I'll keep that in mind, but I reverted that change (due to the segfaults on ubuntu).... I may look to refactor it out, as I think it can provide useful functionality on its own; but right now I'm debating how to pass the assertImageAppoved tests, without having to provide a fudge-factor for Windows systems, which at a glance looks like what I will have to do.

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

Oh the test_getArrayRegion tests, only half of the tests within there fail, specifically the following two fail

        (pg.ROI([1, 1], [27, 28], pen='y'), 'baseroi'),
        (pg.RectROI([1, 1], [27, 28], pen='y'), 'rectroi'),

The other ones pass

        (pg.EllipseROI([1, 1], [27, 28], pen='y'), 'ellipseroi'),
        (pr, 'polylineroi'),

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

Just noticed the following comment in the code:

# on windows, one edge of one ROI handle is shifted slightly; letting this slide with pxCount=10

This was applied to one of the tests failing; but not to all the ones failing...

@j9ac9k j9ac9k force-pushed the merge-test-data branch 2 times, most recently from 8cae3fe to 550a512 Compare May 30, 2021 05:17
@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

I'm doing some last minute cleanup on this PR, I'm going to let it sit for a day, and merge, so we can get back to #1796 and ensure the image tests pass that way.

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 30, 2021

and now I screwed up my git history here 🤦🏻

Ok, now I think this PR is ready.

@j9ac9k j9ac9k force-pushed the merge-test-data branch from 019d189 to cc30ec9 Compare May 30, 2021 06:43
@j9ac9k j9ac9k force-pushed the merge-test-data branch 2 times, most recently from dff4a89 to 62cc14c Compare May 31, 2021 15:25
@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented May 31, 2021

@pijyoi the static code analyzer doesn't like the change to functions.py

    if depth in (8, 24, 32):
        dtype = np.uint8
        nchan = depth // 8
    elif depth in (16, 64):
        dtype = np.uint16
        nchan = depth // 16
    shape = qimg.height(), qimg.width()
    if nchan != 1:
        shape = shape + (nchan,)
    data = np.frombuffer(img_ptr, dtype=dtype).reshape(shape)
    return data

Specifically, it is complaining about nchan and dtype being potentially uninitialized. Should I provide an else with a runtime error or something along those lines?

@pijyoi
Copy link
Copy Markdown
Contributor

pijyoi commented May 31, 2021

Sure, please fix it with a ValueError, unsupported image type.

@j9ac9k j9ac9k force-pushed the merge-test-data branch 2 times, most recently from 3aceb60 to d69b9fa Compare May 31, 2021 17:05
@pijyoi
Copy link
Copy Markdown
Contributor

pijyoi commented Jun 1, 2021

nonuniform_image/colormap-3x3.png looks very different. Probably due to some older PR breaking things w/o the test suite catching it.

master image from separate github repo
colormap-3x3

this PR
colormap-3x3

UPDATE: It seems that the commit 510626c that added the test was already generating the bottom image!
The offending line:
https://github.com/pyqtgraph/pyqtgraph/blob/master/pyqtgraph/graphicsItems/tests/test_NonUniformImage.py#L96

cmap = ColorMap(pos=[0.0, 1.0], color=[(0.0, 0.0, 0.0, 1.0), (1.0, 1.0, 1.0, 1.0)])

should perhaps be

cmap = ColorMap(pos=[0.0, 1.0], color=[(0, 0, 0), (255, 255, 255)])

I am not familiar with ColorMap usage, so maybe @NilsNemitz could help here.

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented Jun 1, 2021 via email

@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented Jun 1, 2021

@pijyoi you are spot on:

image

I'll see if I can reuse the image that's still on test-data, if not, can easily regenerate.

EDIT:

Sure enough, looks like I'll have to regenerate:

image

j9ac9k added 14 commits May 31, 2021 21:05
To reduce complexity, and make it easier to add more images and tests,
the images in the `test-data` repository should be merged with the main
repository.  Furthermore, we can remove a lot of the subprocess work in
the image_testing.py file, as we no longer need to have it interact with
git.

The images are not the same.  Images were regenerated with Qt6, and now
have proper big and little endian handling thanks to @pijyoi

Second commit is a slightly modified variant of
2e135ab authored by @pijyoi
it is to convert qimages to RGBA8888 for testing.  Image
files were regenerated images for the big/little handling

Fixed issue with bogus test from test_NonUniformImage and generated a
new image
the test_signalproxy.py had a fixture for the QApplication instance,
only problem is at the end of each use of the fixture, it would mark
the application instance for deletion, which is most definitely not what
we want
Remove if __name__ == "__main__" bits, replace some == None to is None
checks

Cleanup variety of static code checker issue

Removed missing imports, restructured some logic to make a little
cleaner, fixed a function in test_dockarea, unfortunately broke the test
so now I tell pytest to expect the Exception.
While this fix prevents an assertion error, the assertion error was
being suppressed; and was only noticeable via pytest -s where the error
was printed to console.  Future work should be done to minimize the use
of bare exceptions so these tests do not fail silently
This file contains two tests, one of which has been skipped forever,
and the second (test_pg_exit) has been a flakey test that does not test
in general, testing a use-case we likely do not see any more.  So
therefore I am removing this test from the library.
This issue was flagged as a security issue in the static code analyzer
On Windows, pyreadline is emitting a deprecation warning we can ignore.
Furthermore, test_svg was using NamedTemporaryFile in a manner that was
causing permission denied errors, so this commit switches to the use of
pytest friendly temporary files
The way we called subprocess.Popen, it looks like we didn't close all
the pipes; this PR addresses the warning that pytest generates when
running the examples.

In addition, we toggle the pytest setting to error on any warning
Several bugs have snuck through due to being wrapped with printExc,
which would prevent the exception from raising, but printing the trace
to the console.  In pytest, this output is not captured at all, and is
invisible unless the -s parameter is added.

This PR changes the print statement to a runtime warning, which pytest
will capture.
@j9ac9k j9ac9k force-pushed the merge-test-data branch from d69b9fa to f01f3d4 Compare June 1, 2021 04:07
@j9ac9k
Copy link
Copy Markdown
Member Author

j9ac9k commented Jun 1, 2021

Updated the PR , unless anyone spots anything funky (including my screwing up of my git history, which I've only done a dozen or so times on this branch), I think this should be ready to merge (of course pending the result of the static code checker).

@j9ac9k j9ac9k merged commit 5c54577 into pyqtgraph:master Jun 5, 2021
@j9ac9k j9ac9k deleted the merge-test-data branch June 5, 2021 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PyQt5 test images missing Tests do not fail on warnings

3 participants