"Brier score" seems to be the accepted name so I'm inclined to keep it that way.

IIRC, in "Transforming Classifier Scores into Accurate Multiclass Probability Estimates" (KDD 2002), they study several methods and conclude that one-vs-rest in the most practical solution.

"Brier score" seems to be the accepted name so I'm inclined to keep it
that way.

hum. I am sure it will pass the consistency brigade :)

IIRC, in "Transforming Classifier Scores into Accurate Multiclass
Probability Estimates" (KDD 2002), they study several methods and conclude
that one-vs-rest in the most practical solution.

ok so IsotonicCalibrator should use OneVsRestClassifier internally and
the proba are estimated for each pair and normalized to sum to 1
right?

I guess IsotanicCalibrator could take the 2d output of decision_function as an input and produce multiclass probabilities. Please read the reference I gave, I don't remember it well.

About the Brier score being not a sklearn consistent score (higher == worst in this case): I don't really know what would be best as the sklearn naming convention is indeed conflicting with this official name.

http://en.wikipedia.org/wiki/Brier_score

I would be +0 for keeping the brier_score name and emphasizing the fact that higher values means less confident estimates in the docstring and the narrative doc.

I guess IsotanicCalibrator could take the 2d output of decision_function
as an input and produce multiclass probabilities. Please read the reference
I gave, I don't remember it well.

indeed in there experimental results they use OvR

I think it's a good idea to fit an IR to each decision score.

About the Brier score being not a sklearn consistent score (higher ==
worst in this case): I don't really no what would be best as the sklearn
naming convention is indeed conflicting with this official name.

http://en.wikipedia.org/wiki/Brier_score

I would be +0 for keeping the brier_score name and emphasizing the fact
that higher values means less confident estimates in the docstring and the
narrative doc.

can we call it brier ? or brier_error ? as it's a mean squared error
on the proba.

One solution would be to introduce decorators to label some functions as scores and others as losses. Of course, this is out of the scope of this PR... (This idea would also be useful to define whether a metric accepts predicted labels or predicted scores, c.f. the AUC issue)

Another solution is to introduce a function negative_brier_score, which is just the brier score multiplied by -1. But I think it's important to use the most commonly used names so -1 for this solution.

So adding a note to the documentation as @ogrisel suggested seems like a good temporary solution to me.

Some quick comments ...

• +1 for the name brier_score
• I would consider passing a cross validation class to IsotonicCalibration.__init__().
  Using one fold (oo_bag data) in the case of small number of samples problems could be insufficient.
• I would also add log_likelihood_loss for probability estimation evaluation.
  See "The problem with the Brier score" by Stephen Jewson
• I would warn in the doc agains using Isotonic Calibration with less than 1000 calibration samples since it tends
  overfit (Platt's calibration is advisable in that case). I should be able to find a paper on the subject if anyone
  is interested
• +1 for default number of bins = sqrt(samples) for calibration_plot

hi paolo,

thanks for this valuable feedback. I won't work on this in the next few
days to if you want to improve on my PR please do so. I'll merge your
commits into my PR.

I would warn in the doc agains using Isotonic Calibration with less than 1000 calibration samples since it tends overfit (Platt's calibration is advisable in that case). I should be able to find a paper on the subject if anyone is interested

+1

Using one fold (oo_bag data) in the case of small number of samples problems could be insufficient.

If you use more than one fold, how do you combine the results?

rebased on master + address some comments

I needed a coding break... ;)

If you feel like playing with it please do...

You should provide a default value for the estimator constructor param maybe, a baseline that is fast such as MultinomialNB that is both fast, has few hyperparameters and benefit from calibration:

======================================================================
ERROR: sklearn.tests.test_common.test_all_estimators
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/scikit-learn/scikit-learn/sklearn/tests/test_common.py", line 68, in test_all_estimators
    estimator = Estimator()
TypeError: __init__() takes exactly 2 arguments (1 given)

I've rebased and cleanup the example. I'am open to discussion regarding API and multiclass handling. I don't know how to take care of the OOB to estimate the probas.

@mblondel any chance you can provide feedback on this? I'll get back to it tomorrow morning.

Changes Unknown when pulling bd727f3 on agramfort:isotonic_calibration into * on scikit-learn:master*.

@ogrisel @mblondel have a look :)

Alright, I squashed it now into 5 generic commits (calibration module, brier score, tests, examples, narrative doc). I've also added @mblondel to the list of authors in calibration.py

Coverage increased (+0.02%) to 95.06% when pulling 1a2a9ae on agramfort:isotonic_calibration into f20ff86 on scikit-learn:master.

I pushed a couple of commits (cosmit + coverage improvement). The coverage could be slightly improved. Currently we have 95% coverage.

Alright, I squashed it now into 5 generic commits (calibration module, brier score, tests, examples, narrative doc). I've also added @mblondel to the list of authors in calibration.py

Great, thank you very much!

The coverage could be slightly improved. Currently we have 95% coverage.

Indeed there is a couple of exceptions that should be covered by additional assert_raises or assert_raise_message checks:

https://coveralls.io/builds/1947924/source?filename=sklearn%2Fcalibration.py

It should be easy to raise the coverage close to 99% on that file.

Coverage increased (+0.03%) to 95.07% when pulling 838b06e on agramfort:isotonic_calibration into f20ff86 on scikit-learn:master.

I've added some assert_raises, coverage of calibration.py should now be effectively 100%

Thanks. I think this is ok for merge. @agramfort @mblondel any more comments?

+1 for merge on my side.

Just to confirm: do we want really want sigmoid_calibration to be a public function?

Other than that +1 as well!

+1 for a private sigmoid_calibration as well. I will do the change and merge.

Oops I had already merged, I will cherry pick this.

Done! 🍻 Thank you very much everyone!

Thanks for merging! 🍻

Congrats!

When was this effort started again? Two? Three years ago? :)