Skip to content

fix: fetch all trials in BruteForceSampler for HyperbandPruner#6107

Merged
not522 merged 4 commits intooptuna:masterfrom
himkt:bruto-force-trials
May 31, 2025
Merged

fix: fetch all trials in BruteForceSampler for HyperbandPruner#6107
not522 merged 4 commits intooptuna:masterfrom
himkt:bruto-force-trials

Conversation

@himkt
Copy link
Copy Markdown
Member

@himkt himkt commented May 31, 2025 •
edited
Loading

Motivation

HyperbandPruner has internally multiple sub studies due to its algorithm nation. Therefore study.get_trials is not guaranteed to fetch all trials of the root study.

Description of the changes

Before

Since trials is not comprehensive, duplicated combinations of parameters are selected as trials.

> uv run -m pytest tests/pruners_tests/test_hyperband.py -k test_hyperband_pruner_and_bruto_force_sampler -s
================================================================ test session starts =================================================================
platform darwin -- Python 3.12.0, pytest-8.3.5, pluggy-1.6.0
rootdir: /Users/himkt/work/github.com/himkt/optuna
configfile: pyproject.toml
collected 19 items / 18 deselected / 1 selected                                                                                                      

tests/pruners_tests/test_hyperband.py [I 2025-05-31 11:40:51,560] A new study created in memory with name: no-name-5edbcb58-cee2-4580-9051-ba1e58d49e85
[I 2025-05-31 11:40:51,568] Trial 0 finished with value: 1.0 and parameters: {'x': 1, 'y': 0}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:40:51,569] Trial 1 finished with value: 3.0 and parameters: {'x': 1, 'y': 2}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:40:51,569] Trial 2 finished with value: 3.0 and parameters: {'x': 2, 'y': 1}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:40:51,570] Trial 3 finished with value: 3.0 and parameters: {'x': 1, 'y': 2}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:40:51,570] Trial 4 pruned. 
[I 2025-05-31 11:40:51,571] Trial 5 finished with value: 1.0 and parameters: {'x': 0, 'y': 1}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:40:51,571] Trial 6 finished with value: 1.0 and parameters: {'x': 0, 'y': 1}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:40:51,572] Trial 7 pruned. 
[I 2025-05-31 11:40:51,572] Trial 8 pruned. 
[I 2025-05-31 11:40:51,572] Trial 9 finished with value: 0.0 and parameters: {'x': 0, 'y': 0}. Best is trial 9 with value: 0.0.
[I 2025-05-31 11:40:51,573] Trial 10 pruned. 
[I 2025-05-31 11:40:51,574] Trial 11 finished with value: 1.0 and parameters: {'x': 1, 'y': 0}. Best is trial 9 with value: 0.0.
[I 2025-05-31 11:40:51,575] Trial 12 pruned. 
[I 2025-05-31 11:40:51,575] Trial 13 finished with value: 2.0 and parameters: {'x': 0, 'y': 2}. Best is trial 9 with value: 0.0.
[I 2025-05-31 11:40:51,576] Trial 14 pruned. 
[I 2025-05-31 11:40:51,576] Trial 15 finished with value: 0.0 and parameters: {'x': 0, 'y': 0}. Best is trial 9 with value: 0.0.
F

====================================================================== FAILURES ======================================================================
___________________________________________________ test_hyperband_pruner_and_bruto_force_sampler ____________________________________________________

    def test_hyperband_pruner_and_bruto_force_sampler() -> None:
        def objective(trial):
            x = trial.suggest_int("x", 0, 2)
            y = trial.suggest_int("y", 0, 2)
            for i in range(10):
                trial.report(step=i, value=i*(x+y)/10)
                if trial.should_prune():
                    raise optuna.TrialPruned
            return x + y
    
        sampler = optuna.samplers.BruteForceSampler()
        pruner = optuna.pruners.HyperbandPruner()
        study = optuna.create_study(sampler=sampler, pruner=pruner)
        study.optimize(objective)
    
        trials = study.trials
>       assert len(trials) == 9
E       AssertionError: assert 16 == 9
E        +  where 16 = len([FrozenTrial(number=0, state=1, values=[1.0], datetime_start=datetime.datetime(2025, 5, 31, 11, 40, 51, 560270), datet...gh=2, log=False, low=0, step=1), 'y': IntDistribution(high=2, log=False, low=0, step=1)}, trial_id=5, value=None), ...])

tests/pruners_tests/test_hyperband.py:261: AssertionError
============================================================== short test summary info ===============================================================
FAILED tests/pruners_tests/test_hyperband.py::test_hyperband_pruner_and_bruto_force_sampler - AssertionError: assert 16 == 9
========================================================== 1 failed, 18 deselected in 0.10s ==========================================================

After

Now, the number of trials is the same as the total number of combinations.

> uv run -m pytest tests/pruners_tests/test_hyperband.py -k test_hyperband_pruner_and_bruto_force_sampler -s
=============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.12.0, pytest-8.3.5, pluggy-1.6.0
rootdir: /Users/himkt/work/github.com/himkt/optuna
configfile: pyproject.toml
collected 19 items / 18 deselected / 1 selected

tests/pruners_tests/test_hyperband.py [I 2025-05-31 11:42:01,042] A new study created in memory with name: no-name-d3cc5197-7e5d-451c-bd67-51db6824ccfb
[I 2025-05-31 11:42:01,058] Trial 0 finished with value: 1.0 and parameters: {'x': 0, 'y': 1}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:42:01,058] Trial 1 finished with value: 2.0 and parameters: {'x': 0, 'y': 2}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:42:01,059] Trial 2 pruned.
[I 2025-05-31 11:42:01,059] Trial 3 finished with value: 3.0 and parameters: {'x': 2, 'y': 1}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:42:01,060] Trial 4 finished with value: 1.0 and parameters: {'x': 1, 'y': 0}. Best is trial 0 with value: 1.0.
[I 2025-05-31 11:42:01,060] Trial 5 pruned.
[I 2025-05-31 11:42:01,060] Trial 6 pruned.
[I 2025-05-31 11:42:01,061] Trial 7 finished with value: 0.0 and parameters: {'x': 0, 'y': 0}. Best is trial 7 with value: 0.0.
[I 2025-05-31 11:42:01,061] Trial 8 pruned.
.

======================================================================== 1 passed, 18 deselected in 0.04s =========================================================================

@himkt himkt self-assigned this May 31, 2025
@himkt himkt force-pushed the bruto-force-trials branch from 69056e1 to 8c69550 Compare May 31, 2025 02:40
@toshihikoyanase toshihikoyanase added the sprint-20250531 PR from the online sprint event May 31, 2025. label May 31, 2025
@Alnusjaponica Alnusjaponica self-requested a review May 31, 2025 03:54
@not522 not522 changed the title fix: fetch all trials in BrutoForceSampler for HyperbandPruner fix: fetch all trials in BruteForceSampler for HyperbandPruner May 31, 2025
Alnusjaponica
Alnusjaponica approved these changes May 31, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Alnusjaponica Alnusjaponica left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for your contribution.

@Alnusjaponica Alnusjaponica requested a review from not522 May 31, 2025 03:58
@Alnusjaponica
Copy link
Copy Markdown
Contributor

Alnusjaponica commented May 31, 2025

@not522 Could you take another look?

not522
not522 reviewed May 31, 2025
View reviewed changes
@himkt @not522
Update tests/pruners_tests/test_hyperband.py
498ea10 
Co-authored-by: Naoto Mizuno <naotomizuno@preferred.jp>
not522
not522 approved these changes May 31, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@not522 not522 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@not522 not522 added the bug Issue/PR about behavior that is broken. Not for typos/examples/CI/test but for Optuna itself. label May 31, 2025
@not522 not522 added this to the v4.4.0 milestone May 31, 2025
@not522 not522 enabled auto-merge May 31, 2025 04:16
@not522 not522 merged commit 3128bee into optuna:master May 31, 2025
14 checks passed
@himkt himkt deleted the bruto-force-trials branch May 31, 2025 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@not522 not522 not522 approved these changes

@Alnusjaponica Alnusjaponica Alnusjaponica approved these changes

Assignees

@himkt himkt

Labels

bug Issue/PR about behavior that is broken. Not for typos/examples/CI/test but for Optuna itself. sprint-20250531 PR from the online sprint event May 31, 2025.

Projects

None yet

Milestone

v4.4.0

Development

Successfully merging this pull request may close these issues.

4 participants

@himkt @Alnusjaponica @not522 @toshihikoyanase