I'm not sure about all the terminology and codes returned.

Can you have a look at #1643 ?

I would use something like _get_label_type and separate the datatype identification and the check.
Your check_multilabel_type doesn't follow the same behaviour as check_arrays which is confusing.

I don't mind removing the size checks, and I'm happy to change check_ to get_. However, I've realised now that this is in fact just an extension on is_multilabel. Anything that returns is_multilabel True will return non-False here. So I say we rename check_multilabel_type -> is_multilabel -> _is_multilabel. The name then being clearer, I will remove the shape checks...

And after looking at #1643, I'll adopt 'sequences' and 'indicator' instead of 'matrix' and 'sos'.

Please check this again @arjoly, and @amueller given your earlier implementation of something similar.

Also, @arjoly, can we not as a rule assume that a label indicator matrix
consists of 0s and 1s or Falses and Trues? Do we need to worry about pos_label?

I am thinking the same: it's a source of confusion and makes code more
complex.

Also, @arjoly, can we not as a rule assume that a label indicator matrix consists of 0s and 1s or Falses and Trues? Do we need to worry about pos_label?
I am thinking the same: it's a source of confusion and makes code more complex.

Yes, this can be remove.

Could you add tests in sklearn/utils/tests/test_multiclass.py?
We do not really use docstring for testing.

Since you want to add the support for list of list of string, I think that you should extend the invariance test for multilabel data format in test_metrics.py.

I did support binary and multiclass, just not in the case of multiple. Your comments are fair. I think get_classification_type (or type_of_targets) is a great idea for general utility.

I think the returned strings should be:

• 'multilabel-indicator'
• 'multilabel-sequences'
• 'multiclass'
• 'binary'? I'm not sure we want to check for this as it involves looking through all the labels.
• 'regression'? but if we're going to look through all the labels, we might as well check for non-discrete labels?

Could you add tests in sklearn/utils/tests/test_multiclass.py?
We do not really use docstring for testing.

I thought this might be okay in utils, but can change it.

Since you want to add the support for list of list of string, I think that you should extend the invariance test for multilabel data format in test_metrics.py.

List of list of string was already supported, effectively (it passed is_multilabel and you map labels to integer indices in your _tp_tn_fp_fn). I'm not sure it's necessary to support, but if we support list of string for multiclass, then why not?

maximally:

def type_of_target(y):
    if is_sequence_of_sequences(y):
        return 'multilabel-sequences'
    elif is_label_indicator_matrix(y):
        return 'multilabel-indicator'
    y = np.asarray(y)
    if issubclass(y.dtype.type, np.float) and np.any(y != y.astype(int)):
        return 'regression'  # or 'continuous'? 'real'? delete this case?
    if len(np.unique(y)) == 2:
        return 'binary'
    else:
        return 'multiclass'

And maybe something like this would be useful to help metrics pass invariance tests (this is untested):

def _check_clf_targets(y_true, y_pred, encode=False, ravel=True):
    y_true, y_pred = check_arrays(y_true, y_pred, allow_lists=True)
    type_true = type_of_target(y_true)
    type_pred = type_of_target(y_pred)
    labels = unique_labels(y_true, y_pred)

    if type_true.startswith('multilabel'):
        if not type_pred.startswith('multilabel'):
            raise ValueError("Can't handle mix of multilabel and multiclass targets")

        if type_true != type_pred:
            enc = LabelBinarizer()
            enc.fit([labels.tolist()])
            y_true = enc.transform(y_true)
            y_pred = enc.transform(y_pred)
            type_true = type_pred = 'multilabel-indicator'

    elif type_pred.startswith('multilabel'):
        raise ValueError("Can't handle mix of multilabel and multiclass targets")

    elif 'regression' in (type_true, type_pred):
        raise ValueError("Can't handle continuous targets")

    else:
        if 'multiclass' in (type_true, type_pred):
            # 'binary' can be removed
            type_true = type_pred = 'multiclass'

        y_true, y_pred = _check_1d_array(y_true, y_pred, ravel=ravel)

    if encode and type_true != 'multilabel-indicator':
        enc = LabelEncoder()
        enc.fit([labels.tolist()] if type_true.startswith('multilabel')
                else labels.tolist())
        y_true = enc.transform(y_true)
        y_pred = enc.transform(y_pred)

    return labels, type_true, y_true, y_pred

I like your type_of_target and _check_clf_targets functions.

Will it simplify anything to support the binary case or regression case?

Well, precision_recall_fscore_support and LabelBinarizer both identify the binary case in contrast with multilabel. So it avoids the condition if not multilabel and len(labels) == 2, and hence the bug introduced if if len(labels) == 2 is used alone.

Regression? (Or "continuous"... or "real-valued".) I can't think of anything, except to produce a ValueError for categorical metrics if the output of decision_function or a regressor is provided.

okay, have another look.

You can add your name in the authors of sklearn/utils/multiclass.py.

Rebased on master.

In terms of performance, I guess the slow checks in type_of_target are np.unique and y.astype(int) == y.astype. And by ignoring the continuous case, we can avoid the latter... Not really sure if there's any nice way around this, though.

PRF benchmark (10 classes; multiclass len(y)=1e6, multlabel list of lists 1e3; multilabel label indicator 1e6) averaged over 50 trials (on a not-very-fast laptop, script @ https://gist.github.com/jnothman/5734967):

• master: 0.854, 0.573, 0.000979
• multilabel_type: 1.12, 0.566, 0.00123
• prf_rewrite: 0.658, 0.721, 0.00059
• d33634d^ (before multilabel support): 0.947, N/A, N/A

These results make me wonder how long binarizing takes for multiclass data and whether it's worth doing in general, and the answer seems to be "almost certainly":

In [6]: %timeit sklearn.preprocessing.LabelBinarizer().fit_transform(y)
1 loops, best of 3: 421 ms per loop
In [7]: y.shape
Out[7]: (1000000,)

@arjoly might be interested...

Thanks for the benchmark!

I take back "almost certainly": of course, LabelBinarizer needs to transform two arrays of that size, so it adds up to a bit more; and it blows out as we increase the number of labels:

• n_classes ; direct ; binarized
• 10 0.564 2.79
• 100 0.76 26.7

That's multiclass 1e6 samples.

What about sequences of sequences 1e3 samples?

• 10 0.694 0.0288
• 100 0.849 0.293
• 1000 2.65 2.85

Well, that's a nice reduction. Perhaps we should always binarize sequences of sequences in PRF.

I should note that much of that binarizing time is taken up by finding the unique labels in fit. Note that unique_labels is being rewritten (#1985), and yet if the scorer could receive a labels parameter, this could be avoided...

Thanks @jnothman for the rebase and the benchmarks. I think we are good to merge. 👍 on my side.

Well, what've we got to lose? Merged as fd3bd2d.

@arjoly: let the rebasing begin!

thanks a lot @jnothman !!! :-)