What do you think about raising an error if a metric_func is passed that accepts y_score instead of y_pred?

Right now, this just passes and returns useless values.

Though trying something like this

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import top_k_accuracy_score, metric_at_thresholds

X, y = make_classification(n_samples=100, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
clf = LogisticRegression()
clf.fit(X_train, y_train)
y_score = clf.predict_proba(X_test)[:, 1]
metric_values, thresholds = metric_at_thresholds(y_test, y_score, top_k_accuracy_score)

returns metric_values, thresholds and raises

UndefinedMetricWarning: 'k' (2) greater than or equal to 'n_classes' (2) will result in a perfect score and is therefore meaningless.

(n_thresholds times).

It can still be confusing. I think we can help users if we raise a ValueError early on.

I don't think we are consistent enough that y_pred always means thresholded values, e.g., d2_log_loss_score has y_pred, which takes "Predicted probabilities".

I think the docstring updates you suggested should be enough for now but if we get issues we can look into changing ?

Do we want to (later) have a section in the user guide where we show metric_at_thresholds in action?

I'm working through the workflow, and this is what I currently understand:

• The thresholds we calculate via y_score = clf.predict_proba(X) are dependent on the data (X) we use to get y_score, so they're limited to what exists in that specific dataset
• if new data (X_test) produces different y_score, its best threshold might not have been evaluated at all before and we cannot be sure the best threshold for X is the best threshold for X_test, too
• on the other hand, if we tried to evaluate a metric on the best threshold using the same data (X) that we used to find the threshold, we have data leakage

(Can you verify this is correct, @lucyleeow?)

I'd be happy to document the workflow so users can use this new tool safely.

Do we want to (later) have a section in the user guide where we show metric_at_thresholds in action?

The second step will be to create a Display object to easily plot curves (metric vs threshold) from the results of this function.

if new data (X_test) produces different y_score, its best threshold might not have been evaluated at all before and we cannot be sure the best threshold for X is the best threshold for X_test, too
on the other hand, if we tried to evaluate a metric on the best threshold using the same data (X) that we used to find the threshold, we have data leakage

Not sure what you mean by that. The purpose of this function is not to find the optimal threshold (for that we have the TunedThresholdClassifier). It's to plot a metric against all threshold values to visualize how the metric depends on the threshold. The curve may indeed be a bit different depending on the input data but that's not an issue.

Oh I see, I misunderstood how this function would be used. Thanks for settling this, @jeremiedbb.

Maybe we can be a bit more explicit about it being for visualisation purposes then in the docstring and also refer to TunedThresholdClassifier from there for threshold tuning, @lucyleeow?

Made some amendments, see comment for details.

Could we use a more intuitive description? The other functions in this file phase it like this:

(Though I'm not sure what "confidence values" means.)
Would that also be correct here?

This does seem to be the phrasing used for most (all?) y_score definitions in _ranking.py.

Searching for "confidence values", I see in SVC we say: "confidence values of :term:decision_function". I can also see some other cases where we say:

probability estimates of the positive class, confidence values, or binary decisions values.

So we seem to be implying "confidence values" is decision function output? Though here, we specifically list output of decision functions in these _ranking.py metrics.

I don't know if "confidence values" is a technical term? I also wonder if it may cause confusion with "confidence intervals" in statistics. I would consider re-phrasing of the y_score parameter descriptions to be honest but @jeremiedbb would know much more than me though, WDYT?

It also seems to me that there's redundancy between "confidence values" and "non-thresholded measure of decision", but I'm not sure. Looking at the blame, all these "confidence values" occurrences come from 11 years ago.

For this function I'd rather be coherent with the docstring of confusion_matrix_at_thresholds:
"Estimated probabilities or output of a decision function."

(We could still amend both to be a bit more precise: "Estimated probabilities of the positive class or output of a decision function.")

"Estimated probabilities of the positive class or output of a decision function."

That's pretty nice.

(And maybe we can remove "confidence values" from the param description on the other functions in a later PR.)

Looking at the blame, all these "confidence values" occurrences come from 11 years ago.

Agreed and it seems we've copy pasted all the parameter descriptions for y_score.

Done, and I agree about removing "confidence values" - maybe open a PR and see what others have to say? @StefanieSenger are you interested?

I've moved this section further down to improve the flow of the docs - now all the TunedThresholdClassifierCV stuff is sequentially together.

This does mean that I do not have a metric per threshold plot to refer to.

I thought about adding a plot to bottom of the example (with the distribution of decision threshold plot) https://scikit-learn.org/dev/auto_examples/model_selection/plot_tuned_decision_threshold.html but since we set: cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42) - there are 50 different estimators and 50 lines to plot. I could have done some interpolation (since y_score would be different for each estimator) and summary statistic (quantile or mean/std bands) but I was not confident of the statistical validity of this.

I imagine once we add the display class we will want to use it in existing example (probably https://scikit-learn.org/dev/auto_examples/model_selection/plot_cost_sensitive_learning.html) and possibly add another example? We could expand this section then as well.

cc @AnneBeyer who may be interested in adding the display class following this PR