fixes #5023.
Thanks @lvrzhn

So the problem here is that deprecate can deprecate classes or functions, but the scorers are actually instances of classes, that is they are objects.

The way I propose to solve this, is to deprecate the __call__ method of these objects.
The way to do this is

message = "Function returns negative %s so that greater=better. This function is deprecated in version 0.18 and will be removed in version 0.20. Use neg_mean_squared_error instead"

mean_squared_error_scorer.__call__ = deprecated(message % "MSE")(mean_squared_error_scorer.__call__)

above that you have to define a mean_squared_error_scorer as before.

(I also defined the message as an extra variable so it is in the source code only once.)

hm... I just realized that this object can then no longer be pickled. So this solution doesn't actually work.

Maybe @ogrisel @agramfort @GaelVaroquaux or @jnothman have ideas how to solve this?

My best idea at the moment would be to add an additional "deprecation" parameter that takes a message, and the scorer classes store this in an attribute, and when they are called, they check if the attribute is set, and if so, raise a deprecation warning.

something like adding in _PredictScorer.__call__

if self.deprecation is not None:
    raise DeprecationWarning(self.deprecation)

and similarly in the other scorers.

The scorers are not part of the public API, so should not be deprecated like that (apart from the verbosity of the code where a loop should be used instead). The deprecation message should be triggered in get_scorer instead, which is public, and should be in use every time a scorer is needed (if not, we need to fix it!)

So get_scorer needs a list of scorer names for which it raises a warning suggesting that the name with the neg_ prefix should be used instead.

IMO, SCORERS should also be removed from sklearn.metrics.__all__, and perhaps sklearn.metrics.scorer.__all__ should be defined just for clarity of what is in the public API.

Hmm... sklearn/linear_model/logistic.py uses SCORERS and should use get_scorer. model_evaluation.rst also references SCORERS and should not.

When I previously needed deprecation of scorers (but not in the merged version of the code), I also introduced a list_scorers function that meant a list of scorers (ignoring those that are deprecated) was available in a machine-readable form (not just in an error message from get_scorer or the keys of SCORERS).

It's a bit unclear currently whether SCORERS are pubic API.
It's mentioned in a note here:
http://scikit-learn.org/dev/modules/model_evaluation.html#common-cases-predefined-values

Maybe that was not a good idea and it should be private and the deprecation should be in get_scorer. Do you have a snippet of your list_scorers? Do you define a global list of deprecated scorers in the scorers module?

Sorry, I've lost that code. I think I may have defined a SCORER_DEPRECATION
dict mapping name to deprecation warning text

On 16 December 2015 at 02:49, Andreas Mueller notifications@github.com
wrote:

Closed #6028 #6028.

—
Reply to this email directly or view it on GitHub
#6028 (comment).

@amueller, was your closing this PR accidental? Not that we heard more from @lvrzhn, but you can't exactly blame them for going quiet after their first PR was mysteriously closed a day after submission...

@lvrzhn could you kindly update us if you are still interested/available to work on this PR?

Yes, I am still interested in collaborating on this!

First could you rebase please?

@lvrzhn This is an important issue, would you mind if I cherry picked your commit (you will be the author) and continued on your work in a separate PR?

Hi raghavr, sorry for the delay. I wouldn't mind, please go ahead. Thanks a lot.

replaced by #7261.