Description

label_ranking_average_precision_score offers a sample_weighting argument to allow nonuniform contribution of individual samples to the reported metric. Separately, individual samples whose labels are the same for all classes (all true or all false) are treated as a special case (precision == 1, line 732). However, this special case bypasses the application of sample_weight (line 740). So, in the case where there is both non-default sample_weighting and samples with, for instance, zero labels, the reported metric is wrong.

Steps/Code to Reproduce

See example in this colab

import numpy as np
import sklearn.metrics

# Per sample APs are 0.5, 0.75, and 1.0 (default for zero labels).
truth = np.array([[1, 0, 0, 0], [1, 0, 0, 1], [0, 0, 0, 0]], dtype=np.bool)
scores = np.array([[0.3, 0.4, 0.2, 0.1], [0.1, 0.2, 0.3, 0.4], [0.4, 0.3, 0.2, 0.1]])
print(sklearn.metrics.label_ranking_average_precision_score(
    truth, scores, sample_weight=[1.0, 1.0, 0.0]))

Expected Results

Average of AP of first and second samples = 0.625

Actual Results

Sum of AP of all three samples, divided by sum of weighting vector = 2.25/2 = 1.125

Versions

System:
python: 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]
executable: /usr/bin/python3
machine: Linux-4.14.79+-x86_64-with-Ubuntu-18.04-bionic

BLAS:
macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
lib_dirs: /usr/local/lib
cblas_libs: mkl_rt, pthread

Python deps:
pip: 19.0.3
setuptools: 40.8.0
sklearn: 0.20.3
numpy: 1.14.6
scipy: 1.1.0
Cython: 0.29.6
pandas: 0.22.0