Currently any raised DeprecationWarning makes the test suite fail due to the pytest settings in setup.cfg.

Opening this issue to see whether we want to keep this setup in the future. In my opinion it is too strict, and prone to failures due to reasons we can't control.

For instance, consider the following use case from #12041 . After installing numpy 1.8.2 and scipy 0.13.3 (minimal versions supported by scikit-learn), the importing scipy.sparse raises a deprecation warning,

python -W error -c "import scipy.sparse"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/rth/.miniconda3/envs/test-32/lib/python2.7/site-packages/scipy/__init__.py", line 77, in <module>
    from numpy import oldnumeric
  File "/home/rth/.miniconda3/envs/test-32/lib/python2.7/site-packages/numpy/oldnumeric/__init__.py", line 11, in <module>
    warnings.warn(_msg, ModuleDeprecationWarning)
numpy.ModuleDeprecationWarning: The oldnumeric module will be dropped in Numpy 1.9

Because scipy.sparse is indirectly imported in sklearn.__init__, this means that, technically, the test suite will always fails for these numpy & scipy versions. The only reason why it passed so far was due to a pytest limitation/bug (pytest-dev/pytest#3251). The workaround that we can use, now that that issue is fixed in pytest, is fairly awkward #12042.

This is just an example, but it illustrates that once 0.20 is released, with the current setting, running the test suite in some undetermined Python environment is likely to produce failures that are not representative.

Removing the setup.cfg altogether as it's done in Appveyor is also not ideal, as then e.g. the --doctest-modules option is also removed.

Possibly keeping this configuration for the CI and then removing it from the final .tar.gz before the release could work?
Another solution could be to fail only on DeprecationWarnings raised within scikit-learn code (if that's technically possible).

cc @jnothman @amueller