✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 5afd7d0. Link to the linter CI: here}

I think the main reasons of the failures come from a too small dataset and not enough candidates. Here's a slightly modified version:

def test_halving_random_search_cv_results():
    X, y = make_classification(n_samples=150, n_features=4, random_state=42)

    params = [
        {"kernel": ["rbf"], "C": expon(scale=10), "gamma": expon(scale=0.1)},
        {"kernel": ["poly"], "degree": [2, 3]},
    ]
    param_keys = ("param_C", "param_degree", "param_gamma", "param_kernel")
    score_keys = (
        "mean_test_score",
        "mean_train_score",
        "rank_test_score",
        "split0_test_score",
        "split1_test_score",
        "split2_test_score",
        "split0_train_score",
        "split1_train_score",
        "split2_train_score",
        "std_test_score",
        "std_train_score",
        "mean_fit_time",
        "std_fit_time",
        "mean_score_time",
        "std_score_time",
    )
    extra_keys = ("n_resources", "iter")

    search = HalvingRandomSearchCV(
        SVC(),
        cv=3,
        param_distributions=params,
        return_train_score=True,
        random_state=0,
    )
    search.fit(X, y)
    n_candidates = sum(search.n_candidates_)

    cv_results = search.cv_results_
    # Check results structure
    check_cv_results_keys(
        cv_results, param_keys, score_keys, n_candidates, extra_keys
    )
    check_cv_results_array_types(search, param_keys, score_keys)

    assert all(
        (
            cv_results["param_C"].mask[i]
            and cv_results["param_gamma"].mask[i]
            and not cv_results["param_degree"].mask[i]
        )
        for i in range(n_candidates)
        if cv_results["param_kernel"][i] == "poly"
    )
    assert all(
        (
            not cv_results["param_C"].mask[i]
            and not cv_results["param_gamma"].mask[i]
            and cv_results["param_degree"].mask[i]
        )
        for i in range(n_candidates)
        if cv_results["param_kernel"][i] == "rbf"
    )

• more samples. 2 reasons:
  • we're testing something doing cross validation so the dataset will be splitted and we need to have the 2 classes in both train and test sets as often as possible.
  • Successing halving uses n_samples as a resource and relies on it to evaluate the number of candidates for the first round. So it means that it'll start with less samples which comes back to the previous point. Also, too small can lead to only select a single type of kernel for the first round for instance.
• n_candidates is not a paremeter so we can't easily know in advance what it'll be. Better to just use the n_candidates_ attribute
• set random_state to ensure reproducible results. I tested with 100 different seeds and the test always passed.
• I'd rather set iter and n_resources as extra_keys and modify check_cv_results_keys instead.

  def check_cv_results_keys(cv_results, param_keys, score_keys, n_cand, extra_keys=()):
      # Test the search.cv_results_ contains all the required results
      all_keys = param_keys + score_keys + extra_keys
      assert_array_equal(
          sorted(cv_results.keys()), sorted(all_keys + ("params",))
      )
      assert all(cv_results[key].shape == (n_cand,) for key in all_keys)

Thanks for reviewing, @jeremiedbb, for your help and the explanations. I have made the changes according to your suggestions and kind of understood your reasoning.

I still haven't understood the asserts for the masked arrays though (all the candidates appeared as masked_array for both kernels, when I checked), and I will talk with @adrinjalali about it tomorrow.

After reviewing this together with @adrinjalali, I have also modified the two not fully functioning tests I had idicated in the issue (#26885). Please have a look. :)

@glemaitre

And you would need to solve the conflict in the changelog.

I've finished the last few things. Thanks everyone for your support. :)