Interesting pr!

So the handling of ties is implementation specific (depends on the initial ordering of the target score and the tie handling of the np.argsort which is not even stable by default).

I would rather implement the pessimistic tie handling as described in #2580 (comment), at least as an option.

Also it would be great to add a sample_group=None to at least ndcg_score and mean_ndcg_score (and maybe others) to make it easy for users to pass evaluate a predictions made on a multi-query Learning to Rank test set.

@ogrisel Handling of ties and edge cases is on my todo list.

I have some scoring functions working (NDCG@k and DCG@k) with make_scorer and willing to make a pull request if this one is discontinued.

what happened to this? Also @davidgasquez I guess go ahead?

Hey @amueller! Not 100% sure if I should make a PR with the current implementation. As you can see, NDCG@K requires LabelBinarizer and I don't know if that's something acceptable for Scikit Learn! Looking forward to hearing your thoughts on this!

Using a LabelBinarizer is fine, but I'm not sure if you mean to be handling
multilabel ground truth as well.

On 11 October 2016 at 22:24, David Gasquez notifications@github.com wrote:

Hey @amueller https://github.com/amueller! Not 100% sure if I should
make a PR with the current implementation
https://github.com/davidgasquez/kaggle-airbnb/blob/master/kairbnb/metrics.py.
As you can see, NDCG@K requires LabelBinarizer and I don't know if
that's something acceptable for Scikit Learn! Looking forward to hearing
your thoughts on this!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2805 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6_g9R8LLsG6iCyYSnUk3AErUi_x-ks5qy3HVgaJpZM4BeGxi
.

Using a LabelBinarizer is fine, but I'm not sure if you mean to be handling
multilabel ground truth as well.

Thanks! I'll read the contributing guidelines and try to send a PR in the next few days!

Started working on it with #7739. Any feedback is appreciated! 😄

Are we interested on implementing kendall_tau_score, pairwise_accuracy_score and spearman_rho_score metrics (I am +1).

Indeed, I would be happy to implement these metrics.

Summary: dcg_score and ndcg_score have been added in #7739, and kenall's tau or spearman's rho could be very easily used by scipy's implementations:

from scipy.stats import kendalltau, spearmanr
from sklearn.metrics import make_scorer

kenall_tau_score = make_scorer(kendalltau)
spearman_rho_score = make_scorer(kendalltau)

This leaves us with pairwise_accuracy_score.

I'm content with that and would therefore close. Any objections?

Remark: We could add this in an example, for instance.