Skip to content

Add PAV algorithm for calibration_curve/reliability diagrams #23132

@lorentzenchr

Description

@lorentzenchr

Describe the workflow you want to enable

import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")

Describe your proposed solution

Add the strategy PAV as [1] and [2] to calibration_curve, there called CORP. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

Describe alternatives you've considered, if relevant

No response

Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    Discussion

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions