I'm putting this up for discussion at the coordination meeting this week

After several people mentioned tox as a potential way to simplify our infrastructure, I decided to look into it to see if it could serve our purposes. Just for background, the way tox works is that you can specify any number of configurations for different virtual environments containing different sets of dependencies. When you run one of the configurations, it will set up the environment (if not already existing), install the developer version of astropy into it, then run any custom commands.

Compare this to how we run tests - we install astropy to a temporary directory and then run pytest in that directory - and how we run docs: we build astropy then patch the sphinx build to point to the built version of astropy. What we are trying to achieve is similar to what tox does, except we have developed our own (in places complex and hacky) infrastructure.

With this PR, you can do e.g.:

tox -e py37-build-docs

and this will set up a virtual environment with the build requirements then run Sphinx, and it does not use the custom sphinx docs runner in astropy-helpers. Likewise, you can do e.g.:

tox -e py36-test

and it will run the tests on Python 3.6 after building astropy.

This opens up several possibilities:

• If we were to use this, we could possibly consider getting rid in the long term of our custom test runner infrastructure and custom sphinx build infrastructure, meaning less code to maintain.

• For continuous integration purposes, we could define different combinations of dependencies in tox.ini and then simply refer to those environments in the various CI configurations, rather than having to e.g. repeat dependencies in each CI config. The CI config would be a lot simpler, and as a bonus it would be a lot easier to replicate those environments locally with tox

In terms of downsides/issues, I've found that for ccache to work well I had to do:

tox -e py36-test --develop

and I'm not sure why it misses the cache otherwise, so something to investigate but not really a roadblock.

Finally, one thing to consider is how to deal with the fact that people will have muscle memory for e.g. python setup.py build_docs. For now I'm not suggesting we actually get rid of this, though we could imagine making it so that the setup commands become aliases for tox, or recommend the tox command to run. In the long run, we do have to recognize that the Python community is moving away from setup.py commands, so tox is a way to solve that problem.