Update the FAQ on reproducible optimization results to remove note on HyperbandPruner#5075
Update the FAQ on reproducible optimization results to remove note on HyperbandPruner#5075eukaryo merged 1 commit intooptuna:masterfrom
Conversation
remove note on reproducibility for HyperbandPruner
Codecov Report
@@ Coverage Diff @@
## master #5075 +/- ##
=======================================
Coverage 89.40% 89.40%
=======================================
Files 205 205
Lines 15116 15116
=======================================
+ Hits 13514 13515 +1
+ Misses 1602 1601 -1 see 1 file with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
@eukaryo @nabenabe0928 Could you review this PR? |
There was a problem hiding this comment.
I tried the following code with different setups for study_name, n_jobs, and seed.
import optuna
study_name = ...
n_jobs = ...
seed = ...
def objective(trial):
x = trial.suggest_float("x", -5.0, 5.0)
for i in range(100):
trial.report(x + (100 - i) * (6 - x), i + 1)
return x
study = optuna.create_study(
sampler=optuna.samplers.TPESampler(seed=seed),
pruner=optuna.pruners.HyperbandPruner(min_resource=1, max_resource=101, reduction_factor=3),
study_name=study_name,
)
study.optimize(objective, n_trials=30, n_jobs=n_jobs)So basically, the result is reproducible only if n_jobs=1 and seed is not None.
Probably, we could mention this as HyperbandPruner by itself cannot guarantee reproducible optimizations.
Thanks for having a look! |
nabenabe0928
left a comment
There was a problem hiding this comment.
So basically, the result is reproducible only if
n_jobs=1andseed is not None. Probably, we could mention this asHyperbandPrunerby itself cannot guarantee reproducible optimizations.Thanks for having a look! Just above in the FAQ it says to set a seed in the sampler for reproducibility and below it says that reproducibility is only feasible when studies are run sequentially. I think the latter point corresponds to setting
n_jobs=1.
You are right! I approved the change:)
eukaryo
left a comment
There was a problem hiding this comment.
I read the conversation, LGTM.
|
Actually, we still need to |
Motivation
The FAQ contains a note on HyperbandPruner advising setting the python hash seed to ensure reproducible results.
It's in the paragraph after the code block.
It seems like this this is no longer applicable since #4131 and a similar note on in the API docs for HyperbandPruner was removed at that time
Description of the changes
Remove paragraph