@ogrisel @jnothman All the builds fail with this error message at this line -

AssertionError: 0.0067716825379038909 not less than 0.001

I'm not sure if this has anything to do with the dataset per se.

@glouppe @jmschrei is it safe to reduce the tolerance of this test to 0.01 to pass the test?

Apparently only the regressors with "mae" fail here. Classifiers and other regression scores pass. It's probably the case that mae is more sensitive to changes in scale of the weights. I think it's ok to increase the threshold. I did some experiments in an interactive shell and the first three features (the informative onces) are always more important that the others.

Maybe you can replace this check by the following:

# The forest estimator can detect that only the first 3 features of the dataset
# are informative:
expected_mask = np.zeros(20, dtype=bool)
expected_mask[:3] = True
assert_array_equal(est.feature_importances_ > 0.1, expected_mask)

You call also add a check to rescale X instead of the sample_weight array and check that the features importances are exactly equal to the reference model.

Ah okay. In addition to your proposed test, is it okay if I check the criterion and iff it's mae, change the tolerance to 0.01?

yes let's do that.

Why not simply

assert np.all(importances[:3] > 0.1)

To make the test reports easier to read, please put dtype before X and y. Or even put X, y as global variables at the beginning of the module.

Sorry I don't follow you. I thought since there were check_* functions, the parameters of the functions show up in pytest and it's easier to debug? Which is why I included them as parameters rather than looping them.

It's just that the arrays are long and the end of the parameters are truncated. Putting dtype before X would render the report more informative. The actual content of X and y is not very informative.

Okay thanks