HyperbandPruner might make optimisation irreproducible #3893
Description
Expected behavior
I'm unfamiliar with HyperbandPruner, so I might misunderstand the behaviour. When we run the following code in Steps to reproduce section, we expect the optimisation results including each trial value and trial state are the same because the pruner does not have any random seed-related argument. However, I could not get the same result sometimes.
One example of output
[I 2022-08-15 22:40:29,622] Trial 0 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.417022004702574}. Best is trial 0 with value: 0.5789473684210527.
[I 2022-08-15 22:40:29,726] Trial 1 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.7203244934421581}. Best is trial 0 with value: 0.5789473684210527.
[I 2022-08-15 22:40:29,857] Trial 2 finished with value: 0.7105263157894737 and parameters: {'alpha': 0.00011437481734488664}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:29,964] Trial 3 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.30233257263183977}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:29,994] Trial 4 pruned.
[I 2022-08-15 22:40:30,078] Trial 5 pruned.
[I 2022-08-15 22:40:30,107] Trial 6 pruned.
[I 2022-08-15 22:40:30,137] Trial 7 pruned.
[I 2022-08-15 22:40:30,166] Trial 8 pruned.
[I 2022-08-15 22:40:30,250] Trial 9 pruned.
[I 2022-08-15 22:40:30,353] Trial 10 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.4191945144032948}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:30,455] Trial 11 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.6852195003967595}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:30,560] Trial 12 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.9549862165766468}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:30,664] Trial 13 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.6502581893618595}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:30,769] Trial 14 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.20445224973151743}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:30,874] Trial 15 finished with value: 0.7105263157894737 and parameters: {'alpha': 0.013475387948539988}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:30,978] Trial 16 finished with value: 0.6842105263157895 and parameters: {'alpha': 0.02081449313817708}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:31,081] Trial 17 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.8781174363909454}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:31,183] Trial 18 finished with value: 0.7105263157894737 and parameters: {'alpha': 0.027387593197926163}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:31,188] Trial 19 pruned.
Another example
[I 2022-08-15 22:40:58,608] A new study created in memory with name: no-name-e9a15ed3-ae83-4ff8-ab70-36b7b2a23b01
[I 2022-08-15 22:40:58,712] Trial 0 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.417022004702574}. Best is trial 0 with value: 0.5789473684210527.
[I 2022-08-15 22:40:58,814] Trial 1 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.7203244934421581}. Best is trial 0 with value: 0.5789473684210527.
[I 2022-08-15 22:40:58,916] Trial 2 finished with value: 0.7105263157894737 and parameters: {'alpha': 0.00011437481734488664}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:58,944] Trial 3 pruned.
[I 2022-08-15 22:40:59,046] Trial 4 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.14675589081711304}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:59,131] Trial 5 pruned.
[I 2022-08-15 22:40:59,232] Trial 6 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.1862602113776709}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:59,235] Trial 7 pruned.
[I 2022-08-15 22:40:59,237] Trial 8 pruned.
[I 2022-08-15 22:40:59,339] Trial 9 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.538816734003357}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:59,350] Trial 10 pruned.
[I 2022-08-15 22:40:59,451] Trial 11 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.6852195003967595}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:59,462] Trial 12 pruned.
[I 2022-08-15 22:40:59,564] Trial 13 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.8781174363909454}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:59,665] Trial 14 finished with value: 0.7105263157894737 and parameters: {'alpha': 0.027387593197926163}. Best is trial 2 with value: 0.7105263157894737.
[I 2022-08-15 22:40:59,694] Trial 15 pruned.
[I 2022-08-15 22:40:59,723] Trial 16 pruned.
[I 2022-08-15 22:40:59,753] Trial 17 pruned.
[I 2022-08-15 22:40:59,839] Trial 18 pruned.
[I 2022-08-15 22:40:59,941] Trial 19 finished with value: 0.5789473684210527 and parameters: {'alpha': 0.5586898284457517}. Best is trial 2 with value: 0.7105263157894737.
Environment
- Optuna version:3.0.0.dev # 75b0189
- Python version:3.9.12
- OS:macOS-10.16-x86_64-i386-64bit
- (Optional) Other libraries and their versions:
- scikit-learn: 1.1.2
Error messages, stack traces, or logs
NASteps to reproduce
- Run the following code multiple times with
export PYTHONHASHSEED=7; python FOLLOWING_SCRIPT.py.
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
import optuna
X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=1)
classes = np.unique(y)
n_train_iter = 100
def objective(trial):
alpha = trial.suggest_float("alpha", 0.0, 1.0)
clf = SGDClassifier(alpha=alpha, random_state=1)
for step in range(n_train_iter):
clf.partial_fit(X_train, y_train, classes=classes)
intermediate_value = clf.score(X_valid, y_valid)
trial.report(intermediate_value, step)
if trial.should_prune():
raise optuna.TrialPruned()
return clf.score(X_valid, y_valid)
study = optuna.create_study(
direction="maximize",
pruner=optuna.pruners.HyperbandPruner(
min_resource=1, max_resource=n_train_iter, reduction_factor=3
),
sampler=optuna.samplers.TPESampler(seed=1)
)
study.optimize(objective, n_trials=20)The example code is mainly based on https://optuna.readthedocs.io/en/stable/reference/generated/optuna.pruners.HyperbandPruner.html.
Additional context (optional)
No response