RFC Expose xfail_checks with a more flexible API #29951
Description
xref: #29818 (comment)
Right now we have the tags._xfail_checks which seems private since it has the leading underscore.
We're refactoring tests and making them more modular and much nicer to deal with, but still there are going to be cases where an estimator developer might want to skip a few tests, and not a whole category.
So the proposal here is to rename _xfail_checks to xfail_checks (with a deprecation cycle of one release?), and also add the ability for the developers to set the whether the tests should fail, warn, or be skipped/xfailed.
There's also the question of granularity: do we want to set the warn/xfail/warn to be set on the estimator level, or for each test?
Some alternatives could be:
Option 1
class MyEstimator(BaseEstimator):
def __sklearn_tag__(self):
tags = super().__sklearn_tag__()
tags.xfail_checks = {
"check_estimators_dtypes": ("some-error", "warn"/"skip"/"raise"),
}
return tagsOption 2
class MyEstimator(BaseEstimator):
def __sklearn_tag__(self):
tags = super().__sklearn_tag__()
tags.xfailed_checks = "warn"/"skip"/"raise"
tags.xfail_checks = {
"check_estimators_dtypes": "some-error",
}
return tagsOption 3
class MyEstimator(BaseEstimator):
def __sklearn_tag__(self):
tags = super().__sklearn_tag__()
tags.xfailed_checks = {
"check_estimators_dtypes": "warn"/"skip"/"raise",
}
tags.xfail_checks = {
"check_estimators_dtypes": "some-error",
}
return tagscc @scikit-learn/core-devs since it's public /developer API RFC