I recently spent a day (yes really) looking into the figure test builds for sunpy, specifically surrounding how we pin the version of freetype to minimize differences with the reference images (for sunpy they are hashes of the figures so have no tolerance unlike the currently astropy test setup). This is documented in sunpy/sunpy#4271.

What I learned is that matplotlib actually does a very good job of managing its own freetype versions under normal (pip) operations. When you install matplotlib from wheels a version of freetype is included, which is determined by matplotlib. More interestingly though is that is is also true when compiling matplotlib from a git checkout, and optionally true when compiling matplotlib from a release tarball (controlled by an environment variable).

Therefore, what I think is a good compromise between reliability and simplicity is to rely on these versions of freetype as provided by matplotlib, and not use docker etc on the CI, but to have tox install a known version of matplotlib, which will have an enclosed known version of freetype. To support this approach in sunpy we also check the version of freetype at runtime, and then stop the test run before the figure comparison fails.

The upshot of this is that, as long as you run the figure tests with tox, you should always get the right version of matplotlib and freetype to be able to accurately reproduce the reference figures.

@larrybradley @astrofrog as visualization maintainers.