We can be a little more careful here. If a user passes a scorer that is a callable without sample_weight, this would break their code. What about:

And a small test to make sure passing a callable scorer with the signature:

def my_scorer(estimator, X, y):
	return 1

does not error.

That's a good point. The only issue is that both permutation_importance and _calculate_permutation_scores use this call, so the suggested correction would have to be implemented in both functions.

An alternative solution is to create a new protected function that do this logic, for example:

        def _scorer_weights_robust(scorer, estimator, X, y, sample_weight):
            if sample_weight is not None:
                return scorer(estimator, X, y, sample_weight)
            return scorer(estimator, X, y)

then we can use it like this:

feature_score = _scorer_weights_robust(scorer, estimator, X_permuted, y, sample_weight)

What do you think?

Okay lets create this function to be called in both spots.

done