Can you elaborate? Why are the non-alphanumeric characters an issue? They can be escaped, just like you did, right? But maybe we do want to keep the function around just for this purpose. I don't have a strong opinion.

LGTM :)

Can you elaborate? Why are the non-alphanumeric characters an issue? They can be escaped, just like you did, right? But maybe we do want to keep the function around just for this purpose. I don't have a strong opinion.

Some of our expected messages in tests have characters like "(" that got interpreted by the regex engine and caused tests to fail.

well but you could wrap that with a re.escape. The question is whether we want to do that in the tests or have this function that does it for us.

I think it's simpler / cleaner to keep that assert_raise_message function as it's going to be a very recurrent pattern.

+1 for merge on my side.

This is a nice PR with negative LOC :)

feel free to merge. this solution is fine with me.

Thanks @jfraj!

unfortunately this messes with the error messages, and if you nest assert_raise_message each level will add additional escapes to the message, which makes it impossible to check the message. See https://travis-ci.org/scikit-learn/scikit-learn/jobs/59146101

I had not thought of that issue. I think we should restore the previous assert_raise_message implementation then.

Not that the behavior expected in #4559 is not what the current assert_raises_regex from Python 3's unittest module (which is exposed by nose) does: if the type of the exception is wrong, the exception is not caught:

>>> from nose.tools import assert_raises_regex
>>> def f(): raise ValueError()
>>> assert_raises_regex(TypeError, 'expected message', f)
Traceback (most recent call last):
  File "<ipython-input-8-e4ebd5ea6314>", line 1, in <module>
    assert_raises_regex(TypeError, 'expected message', f)
  File "/volatile/ogrisel/opt/lib/python3.5/unittest/case.py", line 1238, in assertRaisesRegex
    return context.handle('assertRaisesRegex', callable_obj, args, kwargs)
  File "/volatile/ogrisel/opt/lib/python3.5/unittest/case.py", line 162, in handle
    callable_obj(*args, **kwargs)
  File "<ipython-input-5-ccbc7be77110>", line 1, in f
    def f(): raise ValueError()
ValueError

I think we should follow the same behavior in our utilities for consistency.

So we could restore the previous assert_raise_message with the following fix for #4559. I can prepare a PR if it LGTY.

def assert_raise_message(exception, message, function, *args, **kwargs):
    """Helper function to test error messages in exceptions"""

    try:
        function(*args, **kwargs)
        raise AssertionError("Should have raised %r" % exception(message))
    except exception as e:
        error_message = str(e)
        assert_in(message, error_message)
    except:
        raise AssertionError("Didn't raise %s" % exception.__name__)

@ogrisel so the behavior of assert_raises_regex changed from 2 to 3?
That is a bit odd. @jfraj I guess we will just revert, as I agree we should behave as the unittest module.

I felt the behavior was odd and was confirmed by the old assert_raise_regex behavior, but maybe that was a bad idea.

Sorry about the confusion.

Ok I see. I guess there must be some deep reason for this change of behavior in 3.

Actually it did not change in Python 3, this was an inconsistency introduced in our backport. Under Python 2, assert_raises_regex does not exist but it existed with the deprecated name assert_raises_regexp with the same behavior as under Python 3:

>>> from nose.tools import assert_raises_regexp
>>> def f(): raise ValueError()
>>> assert_raises_regexp(TypeError, 'expected message', f)
Traceback (most recent call last):
  File "<ipython-input-14-74b379a93b2a>", line 1, in <module>
    assert_raises_regexp(TypeError, 'expected message', f)
  File "/usr/lib/python2.7/unittest/case.py", line 993, in assertRaisesRegexp
    callable_obj(*args, **kwargs)
  File "<ipython-input-13-ccbc7be77110>", line 1, in f
    def f(): raise ValueError()
ValueError

So the problem was definitely in our backport which was inconsistent with the behavior of the standard library wrapped by nose.