Thanks! Can you also change the entry in roc_curve, from the issue that sounds like it was a bit ambiguous, too.

I changed the y_true description for roc_curve to be:

True binary labels (either {0, 1} or {-1, 1}). If labels are not binary, pos_label should be explicitly given.

roc_curve is a little strange because it allows labels other than {-1, 1} or {0, 1}, but the user has to specify a positive label. But I think the accepted values are (hopefully) clear in the above description.

I'm not sure I understand If labels are not binary, pos_label should be explicitly given. (I haven't checked the code).

Yeah, it's kind of weird how there are two modes that roc_curve operates in, depending on if pos_label is given. Without specifying pos_label, {0, 1} or {-1, 1} are the only values that can be in y_true. But if pos_label is specified, then any two values can be in y_true. For example, from the roc_curve doc (http://scikit-learn.org/dev/modules/generated/sklearn.metrics.roc_curve.html)

import numpy as np
from sklearn import metrics
y = np.array([1, 1, 2, 2])
scores = np.array([0.1, 0.4, 0.35, 0.8])
fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)

Is totally fine because pos_label=2 is specified. If that was left out, then the error

ValueError: Data is not binary and pos_label is not specified

would be raised.

Here is where the values in y_true get explicitly checked if pos_label isn't given

https://github.com/scikit-learn/scikit-learn/blob/baa20488/sklearn/metrics/ranking.py#L311

I would clarify "labels are not binary" as "labels are not {-1, 1} or {0, 1}". {2, 5} are also binary labels

Thanks for the comment @amueller. I changed the y_true description in roc_curve to

y_true : array, shape = [n_samples]
    True binary labels. If labels are not either {-1, 1} or {0, 1},
    then pos_label should be explicitly given.

LGTM