Skip to content

[MRG+1] Extension of v_measure_score metric to include beta parameter#13607

Merged
thomasjpfan merged 24 commits intoscikit-learn:masterfrom
scouvreur:v_measure_score_beta_param
Apr 16, 2019
Merged

[MRG+1] Extension of v_measure_score metric to include beta parameter#13607
thomasjpfan merged 24 commits intoscikit-learn:masterfrom
scouvreur:v_measure_score_beta_param

Conversation

@scouvreur
Copy link
Copy Markdown
Contributor

@scouvreur scouvreur commented Apr 10, 2019

Reference Issues/PRs

This PR is an enhancement proposed in #13595 to the v_measure_score metric.

As mentioned there by @ivsanro1 :

Currently, there is no way to set weight to Homogeneity and Completeness when calculating the V-Measure score.

As the paper states (A. Rosenberg, ‎2007), so as the F1-Score is to Precision and Recall, V-Measure score is the harmonic mean of Homogeneity and Completeness. However, we have the function fbeta_score to calculate the Fb-Score, but (as far as I know) we do not have a way of calculating a weighted V-Measure score.

What does this implement/fix? Explain your changes.

This PR adds an additional parameter beta to the v_measure_score metric in sklearn/metrics/cluster/supervised.py to allow for different weight to be attributed to homogeneity or completeness as proposed in "V-Measure: A conditional entropy-based external cluster evaluation measure" (A. Rosenthal et al.).

If beta = 1.0 (default), the function returns unchanged the homogeneity_completeness_v_measure. Else, it returns:

v_measure_score = ((1 + beta) * homogeneity * completeness
                   / ((beta * homogeneity) + completeness))

as explained p. 412 of the paper.

Tasks

  • Add beta parameter to:
    • homogeneity_completeness_v_measure
    • v_measure_score
  • Add test coverage for when beta is passed to:
    • homogeneity_completeness_v_measure
    • v_measure_score
  • Document changes in function docstring
  • Document changes in v0.21 what's new

…ed formula in case beta != 1.0 is passed [ci skip]
Copy link
Copy Markdown
Member

@jnothman jnothman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test

@scouvreur
Copy link
Copy Markdown
Contributor Author

Do you think this test coverage is sufficient ? Should I add a test for when beta is passed directly to v_measure_score ?

It seems the scikit-learn.scikit-learn build is failing due to an error in Debian Jessie in #13590

@scouvreur scouvreur changed the title [WIP] Extension of v_measure_score metric to include beta parameter [MRG] Extension of v_measure_score metric to include beta parameter Apr 12, 2019
Copy link
Copy Markdown
Member

@jnothman jnothman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise LGTM

Co-Authored-By: scouvreur <stephane.couvreur@feedzai.com>
@scouvreur scouvreur changed the title [MRG] Extension of v_measure_score metric to include beta parameter [MRG+1] Extension of v_measure_score metric to include beta parameter Apr 15, 2019
@scouvreur
Copy link
Copy Markdown
Contributor Author

Thanks for your help and feedback @jnothman @thomasjpfan !

@thomasjpfan thomasjpfan merged commit a353b7f into scikit-learn:master Apr 16, 2019
@thomasjpfan
Copy link
Copy Markdown
Member

Thank you! @scouvreur

jeremiedbb pushed a commit to jeremiedbb/scikit-learn that referenced this pull request Apr 25, 2019
xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019
xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019
xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019
koenvandevelde pushed a commit to koenvandevelde/scikit-learn that referenced this pull request Jul 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants