I think the failures are due to the use of the california dataset, this is a message from one of the failures:

Downloading Cal. housing from https://ndownloader.figshare.com/files/5976036 to C:\Users\VssAdministrator\scikit_learn_data
Downloading Cal. housing from https://ndownloader.figshare.com/files/5976036 to C:\Users\VssAdministrator\scikit_learn_data
WARNING: Failed to generate report: No data to report.

Link to line: https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=15400&view=logs&j=d32b16b6-cb9d-571b-e765-de83708fb8dd&t=b93f76c1-c2c9-579e-c2ec-c4f438af1261&l=34

But other tests use the california dataset as well so I don't understand the cause of the failure...

Hi @lucyleeow the failing test seems to be related to pytest-dev/pytest#6925, as only CI with pytest 5.4.1 are failing.
@thomasjpfan, I think you are quite familiar with dependency issues... do you have any suggestion? Thanks a lot!

It might be because i didn't add @pytest.mark.network. @thomasjpfan mentioned this in a different PR. I will rework this to use diabetes dataset and if we aren't happy about the score in test_oob_score_regressors I can amend just that test to use california.

While reviewing your PRs, this dataset looks have pretty poor results on test sets compared to the training set, which means the default parameters are overfitting:

from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_validate

X, y = load_diabetes(return_X_y=True)
results = cross_validate(RandomForestRegressor(random_state=0), X, y,return_train_score=True)

print("train score", results['train_score'].mean())
print("test score", results['test_score'].mean())
# train score 0.9210138461843774
# test score 0.4230661480472566

Oh wow that's a big difference. What kind of tests should I be careful on? I might not be good at assessing this e.g., check_diabetes_criterion doesn't split the data. I think it checks if the score is comparable for max_features=None and max_features=6 and the criteria "mse", "mae" and "friedman_mse". Will the overfitting make it unsuitable for this?

Edit: should I tune parameters and use tuned parameters for all the tests?

I would try to to use min_samples_leaf=10 or more to limit overfitting.

Hum maybe diabetes is too hard of a dataset to expect good generalization accuracy from the RandomForestRegressor model. Maybe you should try with an easier dataset for which is possible to get higher test scores.

Or maybe this is good enough for such tests. It's still significantly better than random.

Or feel free to use make_regression to generate an easy regression problem for the tests as discussed in #16918.

I tried using make_regression dataset but for test_oob_score_regressors,
assert test_score > est.oob_score_ doesn't pass.

Tried make_regression(n_samples=500, n_features=10) and a range of values for n_informative and noise.

Is this a problem with the way the dataset is generated? Since test_score should always (?) be higher than est.oob_score_, as oob_score_ uses unseen data.

from sklearn.datasets import make_regression
from sklearn.ensemble import RandomForestRegressor

X_reg, y_reg = make_regression(n_samples=500, n_features=10, n_informative=100,
                               random_state=1)

est = RandomForestRegressor(oob_score=True, random_state=0,
                            n_estimators=50, bootstrap=True)
n_samples = X_reg.shape[0]
est.fit(X_reg[:n_samples // 2, :], y_reg[:n_samples // 2])
test_score = est.score(X_reg[n_samples // 2:, :], y_reg[n_samples // 2:])

print(test_score)
print(est.oob_score_)

Gives:

0.7351186379920649
0.7425675493740419

Regardles, happy to keep as diabetes as well.

I think that one expects the OOB score to be really close to the score that you will obtain on the test set. So for this test, I would make the diff between the OOB and the test and check that it is smaller than 0.05 for instance.

I am really surprised for the diabetes results indeed.

I think that one expects the OOB score to be really close to the score that you will obtain on the test set.

I read the code wrong and thought we were fitting and testing on the same data, which is why I thought oob_score would always be worse. It makes sense. Will fix, thanks @glemaitre

Thanks @glemaitre. I amended all test to use the generated regression dataset.

Thanks @lucyleeow