The current docstring for calibration.calibration_curve states that the return values are of shape (n_bins,) The code for calibration_curve however, will remove bins that are empty (which is not mentioned in the current docstring), thus the number of bins in the return value may be smaller than n_bins. This commit fixes the docstring to document the (existing) behavior of removing empty bins.

What does this implement/fix? Explain your changes.

When a bin is empty, calibration_curve will remove it from the list of bins, resulting in return values with less than n_bins:

>>> from sklearn.calibration import calibration_curve
>>> y_true = [True, True, False]
>>> y_prob = [0.9, 0.8, 0.2]
>>> prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=3)
>>> prob_true.shape
(2,)

The existing documentation for this function makes no mention of removing empty bins, however, and states that the return values with be of shape (n_bins,):

    prob_true : array, shape (n_bins,)
        The true probability in each bin (fraction of positives).
 
    prob_pred : array, shape (n_bins,)
        The mean predicted probability in each bin.

Obviously it's best of the documentation matches the behavior of the code, and changing the behavior of the code would risk breaking existing uses of the code. This check in thus updates the documentation to include the behavior of dropping empty bins.

Any other comments?

I encountered this in with real data when trying to generate confidence intervals for a calibration curve using cross validation. The model used weakly correlating features and thus high predicted probabilities were rare and not present in some folds. This lead to differing numbers of bins in each fold causing errors downstream, and the undocumented behavior slowed down the debugging process.

It some cases (such as mine) it would be better to support the behavior of the original documentation (always returning the requested number of bins, even if some are empty) by returning NaNs for empty bins which the end user can then handle appropriately. This could be safely added with a parameter such as drop_empty which could default to True for compatibility. I would be happy to create the pull request if there is interest, but I am assuming there is not enough interest to justify maintaining and testing another control path in the code.