Skip to content

Improve image comparison decorator#17267

Merged
timhoffm merged 4 commits intomatplotlib:masterfrom
QuLogic:testing-lock
May 5, 2020
Merged

Improve image comparison decorator#17267
timhoffm merged 4 commits intomatplotlib:masterfrom
QuLogic:testing-lock

Conversation

@QuLogic
Copy link
Member

@QuLogic QuLogic commented Apr 30, 2020

PR Summary

Firstly, re-write the parameter indirection in image_comparison to use signature modifications, like check_figures_equal. Secondly add a lock file for any comparisons that are externally parametrized.

We've been seeing test_loglog_nonpos fail a lot on macOS, and this should help with that case. Since there's now a lock, revert some of my previous workarounds to avoid parallel writes.

mpl_image_comparison_parameters is no longer used, but I'm not sure if we consider matplotlib.testing.conftest public or whether I can just remove it. It's really only useful when applied via image_comparison.

PR Checklist

  • Has Pytest style unit tests
  • Code is Flake 8 compliant
  • [N/A] New features are documented, with examples if plot related
  • [N/A] Documentation is sphinx and numpydoc compliant
  • [N/A] Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
  • [N/A] Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

@QuLogic QuLogic added this to the v3.3.0 milestone Apr 30, 2020
marker.args[0] not in {'extension'}
for marker in request.node.iter_markers('parametrize'))

if baseline_images is not None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just do if baseline_images is None: baseline_images = request.getfixturevalue(...)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That causes a read-before-use error on the variable, though maybe it'd work if marked non-local? It didn't need to modify the external variable so I didn't try that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sure, no worries.

QuLogic added 4 commits May 4, 2020 15:32
It saves a couple lines to add-if-needed, instead of remove-if-unneeded.
Namely, use signature manipulation, instead of an indirection via
fixtures, to add `extension` and `request`. Also, use the closure of
`baseline_images`, instead of passing it indirectly by marker.
This should prevent tests (if run with xdist) from overwriting each
other's files in the middle of checking them.
These should no longer be needed with the lock, and are hopefully a bit
clearer again.

This reverts commit 95aa968,
"Make test_imagegrid_cbar_mode_edge less flaky."

This reverts commit 2e20058,
"Make test_stem less flaky."
@timhoffm timhoffm merged commit a975e48 into matplotlib:master May 5, 2020
@QuLogic QuLogic deleted the testing-lock branch May 5, 2020 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants