Moving the discussion with @amueller from pydata/patsy#77 (comment).

Proposing to:

• add an optional 'labels' argument to log_loss
• add an argument to make_scorer to enable passing labels argument to the custom loss function

Related to this - perhaps also allow make_scorer's need_threshold = True to be used for non-binary classification problems. Not sure why it's limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC doesn't have a cheap predict_proba, but has a cheap decision_function, so it would allow it to be used with rank-based metris). From make_scorer perspective, it just seems like a completely arbitrary restriction because it shouldn't care about what the custom loss function does with the decision_function output.