This patch still does 2 things, I'm afraid. I'm happy with the change to the output check, but less sure about the tests->test change (tests is pretty standard when using py.test AIUI, and is what pip uses as well). Why not fix setup.py to correctly use tests instead of renaming?

(I'm not going to block this on the renaming, though - if another core committer is happy with it going in, that's fine with me).

I can split it out easily enough. Sorry about misunderstanding splitting out the testing fixes into two separate pulls. The current setup.py is hijacking setuptools's builtin test command to execute py.test but whatever arguments are passed to setup.py pass through to py.test which is assuming they are a file or directory.

I can change python setup.py test to use the built-in (no-op for this project) and instead use python setup.py tests

In this case the test is both the setup.py command and the directory in which to find tests.

The fix for setup.py is in its own PR now: #638

Merged this. Thanks for your persistence with this. Improving the tests is really useful, and sadly doesn't get much developer focus, so your help is much appreciated.