I found two issues with make_scorer's needs_threshold option.

First, it doesn't work if the base estimator is a regressor. Using a regressor is a perfectly valid use case, e.g., if you want do use a regressor but still want to optimize your hyper-parameters against AUC (pointwise ranking with two relevance levels). It does work with Ridge but this is because decision_function and predict are aliases of each others. See also the discussion in issue #1404.

Second, _ThresholdScorer checks that the number of unique values in y_true is 2 but this need not be the case. For example, I can use a regressor and optimize my hyper-parameters against NDCG with more than 2 relevance levels.