When we moved our tests out of our packages, we accidentally stopped running pylint on them. Unfortunately, this isn't an entirely trivial fix. pylint does not support running tests on all files in a directory that's not part of a package. See pylint-dev/pylint#352.

At least a few ways to fix this are:

1. Give pylint the path to each individual file in tests directories. This is tricky because tests may contain arbitrary subdirectories.
2. Make our tests directories modules by adding __init__.py. My understanding is this is not normally done and I am not sure if there are any downsides to do this.
3. Switch to flake8 which happily will run tests on all files in subdirectories.

I'm personally most excited by the last option although it is almost certainly the most work.