Har, the test fails without other fixes... But can circumvent by using a custom warning I think.

Ah no, the error was just that I put a new test into this commit, which should have been two commits later. Should be good now.

Did this make a difference for the numpy tests in practice? Note that I don't see extra warnings or test failures running the tests with Python 3.4.

Well, it did show a load of wrong tests of warnings, and exposes when new warnings are created and should surface in the tests. Basically, without this (or the followup in some similar manner just using catch warnings), testing warning only works half the time.

I guess I'm hoping for something simpler than fixing up all our tests, plus nagging new contributors to use the new functions. For instance, the warnings.py module from Python 3.5 is only 435 lines long. Admittedly, there is also a C component. So i'd be happy to change imports to from numpy.testing import warnings or from numpy.compat import warnings if we could fix things there.

Do Python 2.4 and 3.5 actually fix the problem?

I don't love it either, but getting deprecations and future warnings right in the tests is bordering impossible as well. Maybe @rgommers has an opinion/idea? We previously added another attempt to fix this stuff once even, but I don't think it helped much in the end, it is too difficult to get right (the catch and clear contex).

Python 3.4 (dunno which version) and 3.5 solve the problem in that:

with warnings.catch_warnings():
    warnings.filtewarnings("ignore", ...)

should not mess things up as long as it inside that context. This adds a few extra features (mostly nesting when the inner context uses record=True).

Does this still need a __repr__ method?

I guess I'm hoping for something simpler than fixing up all our tests, plus nagging new contributors to use the new functions.

I don't love it either, but getting deprecations and future warnings right in the tests is bordering impossible as well. Maybe @rgommers has an opinion/idea? We previously added another attempt to fix this stuff once even, but I don't think it helped much in the end, it is too difficult to get right (the catch and clear contex).

That previous attempt was np.testing.class clear_and_catch_warnings I guess - can't remember ever using that.

How about renaming suppress_warnings to catch_warnings? This has a few practical advantages:

• less existing test code to change
• less likely that new contributors will not use it (if it's already imported in a file, then hard not to use it)
• if we ever drop Python 2.7 support, then this context manager may become unnecessary and we can more easily stop using it (just change one import)

Hmm, don't mind renaming, but right now the syntax is a bit different from the typical catch_warnings, since I don't use warnings.simplefiler/filterwarnings anymore directly. The name is a bit off though in any case maybe.

Does it need a repr method? It seems catch warnings has one... The clear and catch warnings is probably used only in those specific tests that were added when it was added.... Frankly, removed all occurances in the later commits.

Does it need a repr method? It seems catch warnings has one

That's what I noticed indeed, hence I asked. It's not that important.

Hmm, don't mind renaming, but right now the syntax is a bit different from the typical catch_warnings, since I don't use warnings.simplefiler/filterwarnings anymore directly. The name is a bit off though in any case maybe.

Yes, I didn't think too hard about the exact API. Maybe they're too different to make it work. The alternative is just be consistent and review properly.

How about renaming suppress_warnings to catch_warnings? This has a few practical advantages:

I like that idea if it can be made to work.

You would have to at the very least inject new functions into the warnings module (i.e replace simplefilter/filterwarnings to actually do what the current filter/record methods do) to keep the API close enough to be more or less compatible. I am not sure that is a good/sustainable design choice, though warnings always works by replacing the show warning function, etc.

I'd like more explanation of just why this is needed. I expect you are tired of doing that at this point, but bear with me. For instance, if no warnings are seen in our tests, are we good? Do we worry about this in the two standard ways of running test: the test scripts for CI and runtests.py? If we didn't need to support Python 2.7 would we be good? Or does this help expose tests that are not working correctly? So on and so on.

I guess you can tell I'm hesitant about this and need convincing that the complexity tradeoff is worth it. And other folks are welcome to weigh in if they feel I'm missing something.

Well, let me start a bit, will try to explain a bit more later. There is one bug in python testing, which should be gone in new pythons. This causes warnings to only be shown once in some cases, and that breaks testinging for them later:

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    function_that_gives_a_warning()

with warnings.catch_warnings(record=True):
    warnings.simplefilter("always")
    function_that_gives_a_warning()  # may not give a warning (depends on warning origin details).

Now you can fix this, by making the first instance there with record=True and "always" instead of "ignore". However, that simple method fails if you want it on an outer level:

with warnings.catch_warnings(record=True):
    warnings.simplefilter(DeprecationWarning, "always")
    with warnings.catch_warnings(record=True):
        warnings.simplefilter(UserWarning, "always")
        function_to_test()  # gives Deprecation and UserWarning

Won't work (the inner one will catch the DeprecationWarning), you could not nest them, but then you still should do the bookkeeping that you don't actually care about the DeprecationWarning, but are testing for the UserWarning.

I would have to look through to figure out how much I actually use the "extra" features like better nesting support in the tests. New python versions resolving the bug is nice, but may be a bit tricky too, since you would only be able to test the existence of warnings reliably on newer python versions. Maybe that would be an option, I am not sure.

Another possible approach: Maybe nose could be interested in including such a chunk of code making it future compat not to warnings but to nose or so?

☔ The latest upstream changes (presumably #7760) made this pull request unmergeable. Please resolve the merge conflicts.

OK, thought about it some more. If we can use a compat warning context, and that warning context on older pythons injects itself into the warnings module (replacing filterwarnings and simplefilter seems fine really; might not support everything), most things should probably work. The only advantage this would still have is any extra features (decorator) and the specificity.

With specificity I mean that if you have two warnings inside the catch warning context with recording, and you only check the first one, the second one is simply lost, while this context will have it show up (with correct nesting). Also it gives a specific log for each warning type.

Of course most of the functionality, it would still need, module based "ignore" filters might not work (not sure, likely can be, but it is a bit trickier IIRC), but the public API would be to try to match catch_warnings.

Hmmm, but injecting would mean that you can't do from warnings import simplefilter, though at least some of those things could be caught by a test, like I did that test to see that nobody uses simplefilter("ignore", ...).

Well, we may need to go with this as is, but I'm glad you are thinking hard about it ;)

The problem with trying to get this into nose is that nose is abandonware, and the nose maintainers recommend that everyone switch to something else like nose2 (which is a complete rewrite and I don't know anyone using it), py.test (which is what everyone seems to use in practice), or plain unittest (which has gotten better but last I checked it still requires annoying boilerplate).

It looks like py.test's warning stuff might suffer from some of the same issues and they might appreciate a patch. But that is less useful to numpy unless we switch to py.test :-). And I suppose we probably should do that at some point, but there's no need to make that a blocker for this patch...

True, might be worth to get their feedback anyway/steal ideas. On first sight, their stuff does not seem to take into account these things. It seems a bit nifty in some sense, but I don't think they do this.

☔ The latest upstream changes (presumably #7268) made this pull request unmergeable. Please resolve the merge conflicts.

The problem with trying to get this into nose is that nose is abandonware, and the nose maintainers recommend that everyone switch to something else like nose2 (which is a complete rewrite and I don't know anyone using it)

That might be because no one knows it's abandonware - I certainly didn't.

Rebased and committed in #7985, so closing this. Thanks Sebastian.