-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Use py.test for our testing #256
Description
I would like to switch our testing over to use py.test. It has a number of benefits over unittest and I've had good experiences with it in the past.
Advantages:
- The reporting of test results is (IMO) cleaner than
unittest, with a better summary, colour-coded output, and more detailed reporting of failures. py.testrecognisesunitteststyle tests, so all of our current tests work - for proof, we have a "tox_pytest.ini" config that runs the tests under tox using py.test.- There is a powerful "fixtures" system to support cleaner setup/teardown code, which supports per-test, per-class, per-module and global fixtures.
- There is a built-in monkeypatching facility that handles most of what mock does for us.
- There are built-in fixtures for using a temporary directory for output and capturing stdout/stderr.
- Tests typically need less boilerplate than with
unittest.
Disadvantages
- Another tool to learn for contributors (although personally, I've found that "copy and modify an existing test" is a pretty good way of learning as much as you need).
- There's a lot of "magic" involved in the internals, which can be confusing.
- Either we end up with multiple styles of test or there's a lot of work in rewriting existing tests.
I am offering to do all of the conversion work myself, initially by creating a "parallel" copy of the test suite and porting some or all of the tests to use py.test directly, so that we have some examples of how they would look. I'd deliberately use some of the features of py.test (for example, using the temp directory support to get test output automatically cleaned up, rather than just putting them in the current directory) so people get a feel for how it would work. I'm not sure yet whether it's advisable to do a wholesale rewrite of all of the tests (for consistency, but there are obviously risks that I get something wrong) or just change a few (see above regarding lack of consistency in the test suite).
I'd also see if I can add a section to the documentation explaining how the tests work, and how to add new tests.
I don't see this as being an urgent activity, but I'd find it useful personally for some changes I want to look at, so I'd expect to start work on this when I get back from holiday (early Sep).
Does anyone have any comments? Objections? Can I give any more information to help people understand the implications of the change?