I'm wondering if we should raise a warning

The more I think about it, the more I think the answer is no. To me it makes sense to rely on pytest for anything more than just ignoring the test.

And I mean, pytest is definitely a standard now. It's not like we're asking developers to use an esoteric testing framework that nobody uses.

FWIW this is very similar to what I proposed in #16507.

I changed my mind about not raising the warning. We raise the warning for skipped tests in check_estimator so I think we should also do that for xfail_checks (or do for neither). Skipped and xfailed tests are basically the same thing with minor variations.

Since parametrize_with_checks uses check_estimator to generate the tests, this PR will filter out the xfail test from pytest as well.

Thanks for catching that! Should we use, _yield_all_checks directly in parametrize_with_checks, even if it means a couple of repeated lines?

I think the fact that parametrize_with_checks relies on check_estimator is the source of a number of constraints we have been experiencing when modifying anything in this mechanism..

Good catch, thanks, should be fixed.

I think the fact that parametrize_with_checks relies on check_estimator is the source of a number of constraints we have been experiencing when modifying anything in this mechanism..

Totally agree.

I'm also quite puzzled by the need to pass name everywhere... that parameter is never used in the checks themselves. well it's used in some tests but we could just check types instead of names, or retrieve the name from the instances.

Opened #17222 which is the same except that checks in the tag will be skipped with a SkipTestWarning.