Calibration Metrics and Multi-class Calibration #21785
Replies: 2 comments 2 replies
-
|
@MischaPanch Please feel free to add anything that I may have missed. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for opening this discussion. We had recently some work going on regarding the calibration module and we are looking forward to improving it. A discussion related to the way to go forward with metrics is ongoing on there: #21718. Implementing metrics should not be too controversial. We should only make sure that our documentation exposes well the characteristics and usage of such. The discussion on the issue should probably drive this improvement. Regarding the integration of a new calibration method, it will be more tenuous:
However, this is only to be sure that the implementation will be relevant in the long term. I will probably bring this topic at the monthly developer topic together with the discussion of the RFC. |
Beta Was this translation helpful? Give feedback.
-
|
@MischaPanch, @glemaitre I have updated the post above with papers and equations for the calibration metrics. Here are, in no particular order, some arguments for the inclusion of these metrics:
I will hopefully do the same for the calibration algorithms. |
Beta Was this translation helpful? Give feedback.
-
|
As already mentioned, #21718 tries to give some background. The problem I see is that some calibration metrics, even when often used, lack a theoretical foundation. Probabilistic classification models trained with log loss (or Brier score) are often not bad (or even well) calibrated. Here, assessing calibration amounts to a good old analysis of residuals. My personal wish is to improve on the options to assess calibration in scikit-learn. Let‘s see. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
My colleagues from appliedAI have developed kyle a library for probability calibration that uses the Calibration Framework under the hood for some algorithms.
We feel that this is not the right way to proceed and would like to start a discussion to find a way to integrate these features into scikit-learn.
The existing calibration module in scikit-learn only handles binary classification with two methods, namely isotonic regression and platt scaling, and doesn't have an implementation of the expected calibration error (ECE). We are aware of the existence of a pull request, #11096, to implement calibration metrics but it wasn't given much attention in a while.
As a first step we would like to implement several calibration metrics such as ECE, cwECE, MCE, ACE as well as others with different binning strategies. After that we would implement temperature scaling as the simplest and most well known parametric multi-class calibration method.
EDIT: 14/12/2021 - Added links to papers for some of the metrics
- Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International Conference on Machine Learning (pp. 1321-1330). PMLR.
- Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International Conference on Machine Learning (pp. 1321-1330). PMLR.
Beta Was this translation helpful? Give feedback.
All reactions