[MRG] Add DCG and NDCG by davidgasquez · Pull Request #7739 · scikit-learn/scikit-learn

davidgasquez · 2016-10-24T15:55:27Z

Reference Issue

DCG and NDCG implementation. Issue #2805

What does this implement/fix? Explain your changes.

Add two scores, dcg_score and ndcg_score to compute Discounted Cumulative Gain (DCG) or the Normalized one (NDCG) at rank K.

Any other comments?

As a first time collaborator, let me know if I should add or change something!

TomDLT · 2016-10-24T16:36:02Z

sklearn/metrics/ranking.py

+    return np.sum(gain / discounts)
+
+
+def ndcg_score(ground_truth, predictions, k=5):


you probably should use y_true and y_pred, or did I miss something?

TomDLT · 2016-10-24T16:37:00Z

sklearn/metrics/ranking.py

+    """
+    lb = LabelBinarizer()
+    lb.fit(range(len(predictions) + 1))
+    T = lb.transform(ground_truth)


We generally avoid single-letter variable name

TomDLT · 2016-10-24T16:39:39Z

Could you add a reference link for these metrics?
Also, we will need some unit testing, that you can add in sklearn/metrics/tests/test_ranking.py.
And you can add the metrics in scikit-learn/sklearn/metrics/__init__.py
Ideally, you could write a small text on the documentation, with the maths formula and explaining when these metrics should be used.

TomDLT · 2016-10-24T16:40:32Z

sklearn/metrics/ranking.py

+    >>> ground_truth = [1, 0, 2]
+    >>> predictions = [[0.15, 0.55, 0.2], [0.7, 0.2, 0.1], [0.06, 0.04, 0.9]]
+    >>> score = ndcg_score(ground_truth, predictions, k=2)
+    1.0


If you store the result in score, there is no printing, which is why this doctest is failing

…into dcg_metrics

davidgasquez · 2016-10-27T11:50:28Z

Hey there @TomDLT! Thanks a lot for this useful tips for someone like me! I've tried to fix and improve the implementation a bit. I still have to:

Add reference links to the metrics
Include unit testing
Add the metrics in scikit-learn/sklearn/metrics/__init__.py

Again, I'd appreciate a lot if you want to let me know anything that I could improve. Thanks again! 😄

…into dcg_metrics

davidgasquez · 2016-12-14T20:33:02Z

Hey there @TomDLT! Sorry for the delay with this one. Would love to hear how it feels now that it has his own test and they are complete. 😄

Also, I'd love to get some input on which might be the next steps for me. Thanks for your help!

TomDLT

Thanks for the update, it looks much better!
You have to fix your unit test, it's failing.

Can you also add a small text on the documentation (doc/modules/model_evaluation.rst, in the ranking section), with the maths formula and explaining when these metrics should be used.

TomDLT · 2016-12-15T10:00:21Z

sklearn/metrics/ranking.py

+    ----------
+    .. [1] `Wikipedia entry for the Discounted Cumulative Gain
+           <https://en.wikipedia.org/wiki/Discounted_cumulative_gain>`_
+    .. [2] `Gist about Ranking Metrics`


I am not sure we want to keep this reference.

TomDLT · 2016-12-15T10:26:06Z

sklearn/metrics/ranking.py

+    y_score, y_true = check_X_y(y_score, y_true)
+
+    lb = LabelBinarizer()
+    lb.fit(range(max(max(y_true) + 1, len(y_true))))


why do you need this?
can you add a comment?

This is to avoid skipping some test labels when the input is [0, 1, 3] in the test set but in the training step we were using [0, 1, 2, 3]

…into dcg_metrics

davidgasquez · 2017-01-31T19:23:03Z

Just pushed another series of small commits @TomDLT!

Can you also add a small text on the documentation (doc/modules/model_evaluation.rst, in the ranking section), with the maths formula and explaining when these metrics should be used.

Not sure if I'm the best one to do that as I've only used it once and briefly!

…into dcg_metrics

davidgasquez · 2017-03-24T11:19:10Z

Any updates on this?

TomDLT · 2017-03-24T12:38:44Z

If your pull-request is ready to be merged, change the prefix from [WIP] to [MRG], and wait for some reviews. I'll try to take a look shortly.

…into dcg_metrics

agramfort · 2017-06-08T11:20:01Z

thanks @davidgasquez

I'll send a PR now to fix cosmits + update what's new

* Add DCG and NDCG ranking functions

jnothman · 2017-10-16T03:10:53Z

This appears to have been merged without review..? Issues at #9921 (comment), #9930, #9929, #9931

jnothman · 2017-10-16T04:34:43Z

I think we should consider reverting this for 0.19.1. It implements a ?non-standard definition of NDCG without sufficient documentation, and does so in a way that makes it unusable.

* Add DCG and NDCG ranking functions

Add DCG and NDCG ranking functions

8933452

davidgasquez mentioned this pull request Oct 24, 2016

[WIP] Ranking metrics #2805

Closed

TomDLT reviewed Oct 24, 2016

View reviewed changes

davidgasquez added 7 commits October 25, 2016 16:40

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

81c8199

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

c940ae7

…into dcg_metrics

Fix parameters names

9ded773

Add array checks

e51d2a0

Add check for value ranges in array

535559a

Simplify loop

d078bac

Fix doctest

f84d22a

davidgasquez added 9 commits November 5, 2016 20:01

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

62fa44f

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

d239c2f

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

3165911

…into dcg_metrics

Add metric references

cb2791a

Add metrics in init file

fa1b2b2

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

fddfa37

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

91b9910

…into dcg_metrics

Fix y_score shape in documentation

5333543

Add unit test for ndcg

9708cbe

TomDLT reviewed Dec 15, 2016

View reviewed changes

davidgasquez added 5 commits January 25, 2017 13:51

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

b448253

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

d847b9f

…into dcg_metrics

Fix test

f671a5b

Clean references

0215f12

Add small comment

632d4c0

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

a934123

…into dcg_metrics

davidgasquez changed the title ~~[WIP] Add DCG and NDCG~~ [MRG] Add DCG and NDCG Mar 24, 2017

davidgasquez added 4 commits March 31, 2017 20:15

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

7b5ab20

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

30e89f7

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

527ba37

…into dcg_metrics

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

85b317f

…into dcg_metrics

agramfort merged commit 419f21b into scikit-learn:master Jun 8, 2017

agramfort mentioned this pull request Jun 8, 2017

[MRG+1] DCG + NDCG continuation #9052

Merged

Sundrique pushed a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

92bef6c

* Add DCG and NDCG ranking functions

dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

43f0b0c

* Add DCG and NDCG ranking functions

dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

c6c14d2

* Add DCG and NDCG ranking functions

NelleV pushed a commit to NelleV/scikit-learn that referenced this pull request Aug 11, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

827f9a9

* Add DCG and NDCG ranking functions

paulha pushed a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

3614ddb

* Add DCG and NDCG ranking functions

AishwaryaRK pushed a commit to AishwaryaRK/scikit-learn that referenced this pull request Aug 29, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

8f2f0e6

* Add DCG and NDCG ranking functions

jnothman mentioned this pull request Oct 16, 2017

metrics.ndcg_score is busted #9921

Closed

maskani-moh pushed a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

9a38c6d

* Add DCG and NDCG ranking functions

jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017

[MRG] Add DCG and NDCG (scikit-learn#7739)

1183b17

* Add DCG and NDCG ranking functions

lorentzenchr mentioned this pull request Sep 24, 2021

Example with ranking metrices #21138

Open

		return np.sum(gain / discounts)


		def ndcg_score(ground_truth, predictions, k=5):

Uh oh!

Conversation

davidgasquez commented Oct 24, 2016

Reference Issue

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

TomDLT Oct 24, 2016

Choose a reason for hiding this comment

Uh oh!

TomDLT Oct 24, 2016

Choose a reason for hiding this comment

Uh oh!

TomDLT commented Oct 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomDLT Oct 24, 2016

Choose a reason for hiding this comment

Uh oh!

davidgasquez commented Oct 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidgasquez commented Dec 14, 2016

Uh oh!

TomDLT left a comment

Choose a reason for hiding this comment

Uh oh!

TomDLT Dec 15, 2016

Choose a reason for hiding this comment

Uh oh!

TomDLT Dec 15, 2016

Choose a reason for hiding this comment

Uh oh!

davidgasquez Jan 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidgasquez commented Jan 31, 2017

Uh oh!

davidgasquez commented Mar 24, 2017

Uh oh!

TomDLT commented Mar 24, 2017

Uh oh!

agramfort commented Jun 8, 2017

Uh oh!

jnothman commented Oct 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnothman commented Oct 16, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TomDLT commented Oct 24, 2016 •

edited

Loading

davidgasquez commented Oct 27, 2016 •

edited

Loading

davidgasquez Jan 31, 2017 •

edited

Loading

jnothman commented Oct 16, 2017 •

edited

Loading