LGTM pending comments.

there is no test.

set the PR to MRG when ready

@agramfort, where is the proper place for the tests? Is it test_grid_search.py? Seems like the only place in sklearn/tests where roc_auc_score is mentioned.

test_ranking.py

Ok, will do.

Is there something else I should do within this issue?

(Appveyor failure unrelated)

Please add, or at least try adding, this metric to METRICS_WITH_POS_LABEL in test_common.py

I'd like to clarify the semantics. Let's say we're using this measure with a binary problem with labels [1, 2] and 1 is the positive class. AFAIK, the predict_proba and decision_function of our classifiers will output a 1d array representing a greater-is-better score for the higher-valued class, i.e. 2. So roc_auc_score(y_true, y_score, pos_label=1) is correctly implemented wrt scikit-learn classifiers if it treats 1 as the positive class in y_true, but such that a lower value of y_score means an instance is more likely classified as positive... Is that what's going on here? (I think not.)

@tswr are you still interested in working on this?
Maybe we should also add a labels parameter?
Hm roc_curve already has a pos_label parameter that assumes you give it the "right" scores so that higher is positive, but here we should take care of that ourselves.
I could swear somewhere in metrics is already the logic for treating the binary special case but I couldn't find it (or is it still in a PR?). We should also do the same for average_precision.

tagging as "needs contributor" as there was no reply from @tswr

Hello I am quite new to scikit-learn and would like to contribute.

@aradhyamathur I think this is not a good issue for a beginner, it might be tricky to get right.

@amueller ohk I understand thanks for your insight :)

This work is continued in #9567. Could you please have a look? Thanks.

Closing given #2723 (comment)