OK, test failure is normal (it's because we print in a test, which is going to be removed of course).

@rth @jnothman @thomasjpfan WDYT, are you OK with this idea of marking strict checks with a decorator? I'm not a fan of the use of the global var, interested in your input

Thanks

Thanks for giving this a go @NicolasHug !

If we want to avoid global variable, I guess the other solution is to mark the checks to skip in a decorator in some way (e.g. set an attribute),

def strict_check(check):
    setattr(check, 'strict', True)
    # do we actually need to use the wrapper in this case?
    # the signature is preserved as far as I can tell
    return check

then pass strict_mode to _generate_class_checks and _generate_instance_checks, and filter/raise warnings there. Currently I feel that's a bit strange that an individual check is run or not based on a global variable even when run from outside of check_estimator.

Otherwise yes, why not for decorators.

Although the other use case this doesn't address is to run the test but skip /modify some of it's parts when strict=False. That would require passing strict variable to the check.

If we do that, calling some function skip_if_strict(strict) inside is only marginally longer than decorators. Though it would mean skipped tests still show in pytest reports (e.g. with parametrized_checks) so maybe decorators as a separate mechanism are still useful.

Although the other use case this doesn't address is to run the test but skip /modify some of it's parts when strict=False

Yeah if we want to allow this I think we need to pass a strict parameter to all checks. I'll try to submit another PR with that workflow enabled

(Thanks for the feedback!)