We should consider changing the metrics API. Rather than having individual metrics be functions, we should consider creating a GroupedData object which would accept y_true, y_pred and group_membership as constructor arguments (and optionally sample_weights). Individual metrics would then become methods on that (still returning a GroupMetricResult object). Users would specify their own metrics via an apply_function(my_metric_function) method (where my_metric_function took arguments of y_true, y_pred and sample_weights).